Accessing Elasticsearch API from Python Script

Elasticsearch provides easy to use API and it can be access from kibana, postman, browser and curl.  You can read here how to access elasticsearch API from these options.

In this post we will look at very simple example of accessing elasticsearch API from python. Here is simple example along with results

import requests
import json

uri='http://localhost:9200/_cat/indices'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("this is : ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing document before creating : ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps({
    "query": {
            "user" : "NewUser",
            "post_date" : "2009-11-15T14:12:12",
            "message" : "trying out Elasticsearch from python"
            }
        })
response = requests.put(uri,headers=headers1,data=query)
print("*************************************")
print("Document is created ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing newly created document: ", response.text)

Here is output


$ python3 second.py 
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.7) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
this is :  yellow open ecommpediatest               R6WypBc2RtCS_ITA40_rFw 1 1    0 0    283b    283b
yellow open test-index                   OeILxjmFRhODhztu4GgE_w 1 1    1 0   4.5kb   4.5kb
yellow open twitter                      BUmjGbLpTNCbnDIPxht9vA 1 1   13 4  24.8kb  24.8kb
yellow open bookindex                    RKq8oJKvRb2HQHxuPZdEbw 1 1    1 0   5.4kb   5.4kb
green  open .kibana_task_manager_1       pc_LXegKQWu9690vT1Z-pA 1 0    2 1  16.9kb  16.9kb
green  open kibana_sample_data_ecommerce FkD1obNSSK6mfLDsy9ILPQ 1 0 4675 0   4.9mb   4.9mb
yellow open facebook                     4Wzax6UhThm5rXi03PiG7w 1 1    2 0     7kb     7kb
green  open .apm-agent-configuration     sxoXMmsoS4mRPyjcaPByzw 1 0    0 0    283b    283b
green  open .kibana_1                    pK7pO-SCSBORonZLDi8Vew 1 0   60 2 945.9kb 945.9kb
yellow open twitter1                     3iMzcYoJR3qY9JiM2sY8_g 1 1    1 0   4.5kb   4.5kb

*************************************
Accessing document before creating :  {"_index":"twitter","_type":"_doc","_id":"7","found":false}
*************************************
Document is created  {"_index":"twitter","_type":"_doc","_id":"7","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":28,"_primary_term":2}
*************************************
Accessing newly created document:  {"_index":"twitter","_type":"_doc","_id":"7","_version":1,"_seq_no":28,"_primary_term":2,"found":true,"_source":{"query": {"user": "NewUser", "post_date": "2009-11-15T14:12:12", "message": "trying out Elasticsearch from python"}}}

We will have a look at complex queries from python at later stages.

Command ‘pip’ not found on Ubuntu 18.04

While trying to install django I got following error


$ pip install Django==3.0.1

Command 'pip' not found, but can be installed with:

sudo apt install python-pip

Now before you install pip using above command check your python version

If you have python 2.X, use following command

$sudo apt update
$sudo apt install python-pip

If you have python 3.X use following command

$sudo apt update
$sudo apt install python3-pip

 

Windows 10 | ‘pip’ is not recognized as an internal or external command

After installing python on windows 10, I wanted to install few more packages such as numpy, scipy, pandas, however I got following error


C:\>pip install pandas
'pip' is not recognized as an internal or external command,
operable program or batch file.

pip script is present in following folder


C:\Users\USERNAME\AppData\Local\Programs\Python\Python36\Scripts
OR
C:\Python\Python36\Scripts

To make pip work, we need to add this to path variable.


C:\>path %path%;C:\Users\USERNAME\AppData\Local\Programs\Python\Python36\Scripts

outcome after adding pip to path


C:\>pip install pandas
Collecting pandas
Downloading https://files.pythonhosted.org/packages/d0/4e/9db3468e504ac9aeadb37eb32bcf0a74d063d24ad1471104bd8a7ba20c97/pandas-0.24.2-cp36-cp36m-win_amd64.whl (8.8MB)
100% |████████████████████████████████| 8.8MB 36kB/s
Collecting python-dateutil>=2.5.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
100% |████████████████████████████████| 235kB 67kB/s
Collecting pytz>=2011k (from pandas)
Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)
100% |████████████████████████████████| 512kB 64kB/s
Collecting numpy>=1.12.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/2e/11/f006363050b24fb19a235e5efd219e7ac549398d531110d80b8f2ba3a909/numpy-1.16.3-cp36-cp36m-win_amd64.whl (11.9MB)
100% |████████████████████████████████| 11.9MB 39kB/s
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas)
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
Successfully installed numpy-1.16.3 pandas-0.24.2 python-dateutil-2.8.0 pytz-2019.1 six-1.12.0
You are using pip version 9.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Windows 10 | ‘python’ is not recognized as an internal or external command

After installing python on Windows 10, I got following error.

C:\>python
'python' is not recognized as an internal or external command,
operable program or batch file.

To my surprise py works fine but python does not work.

C:\>py
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> 

To make python works, we need to set up path. For Windows 10, python will be at following folder

C:\Users\USERNAME\AppData\Local\Programs\Python\Python36
OR
C:\Python\Python36

Use following command to set up path


C:\>path %path%;C:\Users\USERNAME\AppData\Local\Programs\Python\Python36

Once this is done, you can check the path variable using below command


C:\>echo %path%

Now let us try using python

C:\>python
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

How to find all files in a directory with specific extension using Python

Whenever we need to check if file with specific extension exists in a directory or get list of files with specific extension from a directory, it can be done using python. Here is simple program for doing the same.

There are three ways to do this using python

  1. os.walk
  2. os.listdir
  3. glob.glob

All three methods are demonstrated in below program

Output of the program

*** Program Started ***
List of Files using os.walk: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using listdir: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using glob : ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
*** Program Completed ***

 

How to count number of files in directory using Python

Counting number of files using python in specific folder can be done using multiple ways.  Some of these are mentioned below

Here is the output

*** Program Started ***
Number of Files using os.walk : 7
Number of Files using listdir method#1 : 8
Number of Files using listdir method#2 : 7
Number of Files using listdir method#3 : 7
Number of Files using glob : 7
*** Program Completed ***

As you can see Number of Files using listdir method#1 are different, this is because, its counting a folder as a file.

 

How to write a list to a file and read a list from file using Python

Many a times we need to store list in a file for later usage, in such cases, its better to store list in a file and read this file into list whenever we need to use. Python has very easy methods for achieving this.

 

Output of the program

*** Program Started ***
lines : ['one', 'two', 'three', 'four', '']
type of lines : <class 'list'>
line : one
line : two
line : three
line : four
line :
*** Program Completed ***

file generated is as below

Here is another version of same program

output and file generated is same except file opening closing styles are different.

How to Check if a File Exists using Python

Recently I was working on file generation using python, before generating any file, I had to check if file exists or not, to check if file exists or not, I had to find a way to do in in program. As usual, this task is very easy using python.

Checking if file exists or not can be done in multiple ways using python, here is one using “os” module

os.path module has functions such as isfile, isdir and exists which helps us check if file or directory exists or not.

Here is the output

$ python3.6 file_exists_01.py 
file_exists : True
file_exists : False
dir_exists : False
dir_exists : True
dir_exists : False
exists : True
exists : True

If you use isfile() on directory, outcome will be False, you need to use isfile or isdir as per requirement. Alternatively, you can use exists function as well, this returns True if input file or directory path is valid. Please refer to os.path documentation for further details.

Here is another way to check if file exists or not using pathlib module.

Output is

$ python3.6 file_exists_02.py 
var : False
var : True
var : True
var : False
var : True
var : True

Both the modules have similar features, you can choose whichever is convenient to you.

How to crop image using Python

Image cropping is very easy using python library Pillow. Following is simple program to crop image in required parts.

Most important part of cropping is to know pixel size of your image, you can crop required portion using .crop method of Image module of Pillow.

Sample program

Please note that crop takes tuple  having four values as input. These four values represent left, upper, right, lower point .

Original image

Left Half of the image

Right Half of the image

Top Right corner of the image

Here is the output of the program, please note the pixel size of each cropped image.

$ python3.6 06_crop_image_01.py 
*** Program Started ***
im.size (1920, 1318)
im.size (960, 1318)
im.size (960, 1318)
im.size (960, 659)
*** Program Ended ***

How to compress images using Python

Python Library Pillow can be used very effectively to compress images. While doing some research on this I found that .JPG files can be compress very effectively however this does not work well with .PNG files. Here is sample program to reduce file size of an image.

Output of the program while using .JPG file as input and output

$ python3.6 05_compress_image_01.py 
*** Program Started ***
Input file size : (5456, 3632)
Input file name : 05_compress_image_01_input.jpg
Input Image Size : 1611664
Output file size : (5456, 3632)
Output file name : 05_compress_image_01_output.jpg
Output Image Size : 443479
*** Program Ended ***

Output of the program while using .PNG files as input and output

$ python3.6 05_compress_image_01.py 
*** Program Started ***
Input file size : (1920, 1282)
Input file name : 05_compress_image_01_input.png
Input Image Size : 3683320
Output file size : (1920, 1282)
Output file name : 05_compress_image_01_output.png
Output Image Size : 3619363
*** Program Ended ***

As you can see, while using .PNG files for input and output, there is hardly any change in file size however when you used .JPG files, output files is of 27% of original file size. Your percentage reduction might be different based on the file that you have used.

Please note input and out file dimensions, number of pixels stays the same.