Getting Started with django with ubuntu

Tag line for django says “The web framework for perfectionists with deadlines.” and most would agree. django is rediculously fast to implement, its fully loaded with lots of utilitis, its secure and at the same time its highly scalable. some of the most popular sites like instagram, pinterest are built with django.

Let us get our hands dirty by getting started with django instead of talking about django features.

Which django version should I use ?

Here are the details about djnago release. you can choose latest version, however, I will stick with LTS version

Currently I am using Django 2.2 for most of my projects.

Which Python version should I use ?

This is the most common question and following table with clarify your query

Django version Python versions
  1.11 2.7, 3.4, 3.5, 3.6
  2.0 3.4, 3.5, 3.6, 3.7
  2.1,2.2 3.5, 3.6, 3.7

In this tutorial we are going to use python version 3.6 (3.6.4 to be very specifuc) and django version 1.11.

Install django version 1.11

$ sudo python3.6 -m pip install django==1.11
Collecting django==1.11
Downloading https://files.pythonhosted.org/packages/47/a6/078ebcbd49b19e22fd560a2348cfc5cec9e5dcfe3c4fad8e64c9865135bb/Django-1.11-py2.py3-none-any.whl (6.9MB)
100% |████████████████████████████████| 6.9MB 339kB/s 
Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from django==1.11) (2017.3)
Installing collected packages: django
Successfully installed django-1.11

You can check the version as below

$ python3.6 -m django --version
1.11

Creating django project

$ django-admin startproject mysite

Now CD into directory “mysite” and run following command

$python3.6 manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

July 18, 2018 - 17:43:11
Django version 1.11, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[18/Jul/2018 17:43:31] "GET / HTTP/1.1" 200 1716
Not Found: /favicon.ico

Now you can access django website at location http://127.0.0.1:8000/

here is the output

Creating application

Now that your environment – a “project” – is set up, you’re set to start doing work.

$ python3.6 manage.py startapp polls

after creating polls app, please add following code to polls/views.py


from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world. You're at the polls index.")

Now create a file urls.py and add following code


from django.conf.urls import url

from . import views

urlpatterns = [
    url(r'^$', views.index, name='index'),
]

Now add following code in mysite.urls.py


from django.conf.urls import include, url
from django.contrib import admin

urlpatterns = [
    url(r'^polls/', include('polls.urls')),
    url(r'^admin/', admin.site.urls),
]

Here is the output

Hope this is helpful…

Calling one python script from another

Ideally should should import requited python file using import into another and call required function from other programs but there could be some instances where you would need to trigger one python script from another.

This  is fairly simple.

Lets say I have following two program located in same folder one.py and one_sub.py. Let us try calling one_sub.py from one.py is fairly simple

one.py

import os

os.system('python3.6 ' + 'one_sub.py')

This code will trigger one_sub.py.

Passing parameters to calling file

This needs minor changes. Whatever you need to pass, just mention that value or variable after a space.
one.py

import os

os.system('python3.6 ' + 'one_sub.py 11 ')

one_sub.py

import sys
print("---This is inside script 2")

input_value = int(sys.argv[1])
print(" 0 ", sys.argv[0], )
print(" 1 ", sys.argv[1], type(sys.argv[0]))

Please note, 0th parameter is always the script name, you can pass multiple parameters.

Also, received files are always string, you need to change to required datatype using datatype conversion operators e.g. int(sys.argv[1])

Trigger file located at different directory

import os

file_path = '/home/user/code/quant/source/library/'
os.system('python3.6 ' + file_path + 'one_sub.py 11 22')

Please note the training '/' in file_path

Accessing Elasticsearch API from Python Script

Elasticsearch provides easy to use API and it can be access from kibana, postman, browser and curl.  You can read here how to access elasticsearch API from these options.

In this post we will look at very simple example of accessing elasticsearch API from python. Here is simple example along with results

import requests
import json

uri='http://localhost:9200/_cat/indices'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("this is : ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing document before creating : ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps({
    "query": {
            "user" : "NewUser",
            "post_date" : "2009-11-15T14:12:12",
            "message" : "trying out Elasticsearch from python"
            }
        })
response = requests.put(uri,headers=headers1,data=query)
print("*************************************")
print("Document is created ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing newly created document: ", response.text)

Here is output


$ python3 second.py 
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.7) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
this is :  yellow open ecommpediatest               R6WypBc2RtCS_ITA40_rFw 1 1    0 0    283b    283b
yellow open test-index                   OeILxjmFRhODhztu4GgE_w 1 1    1 0   4.5kb   4.5kb
yellow open twitter                      BUmjGbLpTNCbnDIPxht9vA 1 1   13 4  24.8kb  24.8kb
yellow open bookindex                    RKq8oJKvRb2HQHxuPZdEbw 1 1    1 0   5.4kb   5.4kb
green  open .kibana_task_manager_1       pc_LXegKQWu9690vT1Z-pA 1 0    2 1  16.9kb  16.9kb
green  open kibana_sample_data_ecommerce FkD1obNSSK6mfLDsy9ILPQ 1 0 4675 0   4.9mb   4.9mb
yellow open facebook                     4Wzax6UhThm5rXi03PiG7w 1 1    2 0     7kb     7kb
green  open .apm-agent-configuration     sxoXMmsoS4mRPyjcaPByzw 1 0    0 0    283b    283b
green  open .kibana_1                    pK7pO-SCSBORonZLDi8Vew 1 0   60 2 945.9kb 945.9kb
yellow open twitter1                     3iMzcYoJR3qY9JiM2sY8_g 1 1    1 0   4.5kb   4.5kb

*************************************
Accessing document before creating :  {"_index":"twitter","_type":"_doc","_id":"7","found":false}
*************************************
Document is created  {"_index":"twitter","_type":"_doc","_id":"7","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":28,"_primary_term":2}
*************************************
Accessing newly created document:  {"_index":"twitter","_type":"_doc","_id":"7","_version":1,"_seq_no":28,"_primary_term":2,"found":true,"_source":{"query": {"user": "NewUser", "post_date": "2009-11-15T14:12:12", "message": "trying out Elasticsearch from python"}}}

We will have a look at complex queries from python at later stages.

Command ‘pip’ not found on Ubuntu 18.04

While trying to install django I got following error


$ pip install Django==3.0.1

Command 'pip' not found, but can be installed with:

sudo apt install python-pip

Now before you install pip using above command check your python version

If you have python 2.X, use following command

$sudo apt update
$sudo apt install python-pip

If you have python 3.X use following command

$sudo apt update
$sudo apt install python3-pip

 

Windows 10 | ‘pip’ is not recognized as an internal or external command

After installing python on windows 10, I wanted to install few more packages such as numpy, scipy, pandas, however I got following error


C:\>pip install pandas
'pip' is not recognized as an internal or external command,
operable program or batch file.

pip script is present in following folder


C:\Users\USERNAME\AppData\Local\Programs\Python\Python36\Scripts
OR
C:\Python\Python36\Scripts

To make pip work, we need to add this to path variable.


C:\>path %path%;C:\Users\USERNAME\AppData\Local\Programs\Python\Python36\Scripts

outcome after adding pip to path


C:\>pip install pandas
Collecting pandas
Downloading https://files.pythonhosted.org/packages/d0/4e/9db3468e504ac9aeadb37eb32bcf0a74d063d24ad1471104bd8a7ba20c97/pandas-0.24.2-cp36-cp36m-win_amd64.whl (8.8MB)
100% |████████████████████████████████| 8.8MB 36kB/s
Collecting python-dateutil>=2.5.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
100% |████████████████████████████████| 235kB 67kB/s
Collecting pytz>=2011k (from pandas)
Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)
100% |████████████████████████████████| 512kB 64kB/s
Collecting numpy>=1.12.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/2e/11/f006363050b24fb19a235e5efd219e7ac549398d531110d80b8f2ba3a909/numpy-1.16.3-cp36-cp36m-win_amd64.whl (11.9MB)
100% |████████████████████████████████| 11.9MB 39kB/s
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas)
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
Successfully installed numpy-1.16.3 pandas-0.24.2 python-dateutil-2.8.0 pytz-2019.1 six-1.12.0
You are using pip version 9.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Windows 10 | ‘python’ is not recognized as an internal or external command

After installing python on Windows 10, I got following error.

C:\>python
'python' is not recognized as an internal or external command,
operable program or batch file.

To my surprise py works fine but python does not work.

C:\>py
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> 

To make python works, we need to set up path. For Windows 10, python will be at following folder

C:\Users\USERNAME\AppData\Local\Programs\Python\Python36
OR
C:\Python\Python36

Use following command to set up path


C:\>path %path%;C:\Users\USERNAME\AppData\Local\Programs\Python\Python36

Once this is done, you can check the path variable using below command


C:\>echo %path%

Now let us try using python

C:\>python
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

How to find all files in a directory with specific extension using Python

Whenever we need to check if file with specific extension exists in a directory or get list of files with specific extension from a directory, it can be done using python. Here is simple program for doing the same.

There are three ways to do this using python

  1. os.walk
  2. os.listdir
  3. glob.glob

All three methods are demonstrated in below program

################################################################################################
# name: file_with_specific_extension_in_directory.py
# desc:
# date: 2019-03-30
# Author: conquistadorjd
################################################################################################
import os
import glob
print("*** Program Started ***")
dir_name = 'C:/sample/'
matches = []
for root, dirnames, filenames in os.walk(dir_name):
for filename in filenames:
# if filename.endswith(('.txt', '.MOV', '.avi', '.mpg')): ## Checking for multiple extenstions
if filename.endswith('.txt'):
matches.append(filename)
print("List of Files using os.walk:", matches)
matches = []
filenames = os.listdir(dir_name)
for filename in filenames:
# if filename.endswith(('.txt', '.MOV', '.avi', '.mpg')): ## Checking for multiple extenstions
if filename.endswith('.txt'):
matches.append(filename)
print("List of Files using listdir:", matches)
# filenames = glob.glob(dir_name+'*.txt') ## This will return file name with complete path name as well
matches = [os.path.basename(x) for x in glob.glob(dir_name+'*.txt')] ## This will return only file name
print("List of Files using glob :", matches)
print("*** Program Completed ***")

Output of the program

*** Program Started ***
List of Files using os.walk: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using listdir: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using glob : ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
*** Program Completed ***

 

How to count number of files in directory using Python

Counting number of files using python in specific folder can be done using multiple ways.  Some of these are mentioned below

################################################################################################
# name: file_count_in_directory.py
# desc:
# date: 2019-03-30
# Author: conquistadorjd
################################################################################################
import os
import glob
print("*** Program Started ***")
dir_name = 'C:/sample/'
file_count = sum([len(files) for r, d, files in os.walk(dir_name)])
print('Number of Files using os.walk :', file_count)
file_count = os.listdir(dir_name)
print("Number of Files using listdir method#1 :", len(file_count))
file_count = len([name for name in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, name))])
print("Number of Files using listdir method#2 :", file_count)
file_count = [name for name in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, name))]
print("Number of Files using listdir method#3 :", len(file_count))
file_count = glob.glob(dir_name+'*.*')
print("Number of Files using glob :", len(file_count))
print("*** Program Completed ***")

Here is the output

*** Program Started ***
Number of Files using os.walk : 7
Number of Files using listdir method#1 : 8
Number of Files using listdir method#2 : 7
Number of Files using listdir method#3 : 7
Number of Files using glob : 7
*** Program Completed ***

As you can see Number of Files using listdir method#1 are different, this is because, its counting a folder as a file.

 

How to write a list to a file and read a list from file using Python

Many a times we need to store list in a file for later usage, in such cases, its better to store list in a file and read this file into list whenever we need to use. Python has very easy methods for achieving this.

 

################################################################################################
# name: list_to_file_01.py
# desc:
# date: 2019-03-30
# Author: conquistadorjd
################################################################################################
print('*** Program Started ***')
score = ["one","two","three", "four"]
with open("list_to_file.txt", "w") as f_write:
for s in score:
f_write.write(str(s) +",")
with open("list_to_file.txt", "r") as f_read:
lines = f_read.read().split(',')
print('lines : ', lines )
print('type of lines : ', type(lines) )
for line in lines:
print("line : ", line)
print('*** Program Completed ***')

view raw
list_to_file_01.py
hosted with ❤ by GitHub

Output of the program

*** Program Started ***
lines : ['one', 'two', 'three', 'four', '']
type of lines : <class 'list'>
line : one
line : two
line : three
line : four
line :
*** Program Completed ***

file generated is as below

one,two,three,four,

view raw
list_to_file.txt
hosted with ❤ by GitHub

Here is another version of same program

################################################################################################
# name: list_to_file_02.py
# desc:
# date: 2019-03-30
# Author: conquistadorjd
################################################################################################
print('*** Program Started ***')
score = ["one","two","three", "four"]
f_write = open("list_to_file.txt", "w")
for s in score:
f_write.write(str(s) +",")
# Expkicite close is needed in this case
f_write.close()
f_read = open("list_to_file.txt", "r")
lines = f_read.read().split(',')
print('lines : ', lines )
print('type of lines : ', type(lines) )
for line in lines:
print("line : ", line)
## This close can be ommited considering program termination closes all open files.
f_read.close()
print('*** Program Completed ***')

view raw
list_to_file_02.py
hosted with ❤ by GitHub

output and file generated is same except file opening closing styles are different.

How to Check if a File Exists using Python

Recently I was working on file generation using python, before generating any file, I had to check if file exists or not, to check if file exists or not, I had to find a way to do in in program. As usual, this task is very easy using python.

Checking if file exists or not can be done in multiple ways using python, here is one using “os” module

os.path module has functions such as isfile, isdir and exists which helps us check if file or directory exists or not.

################################################################################################
# name: file_exists_01.py
# desc: Check if file exists
# date: 2019-02-13
# Author: conquistadorjd
################################################################################################
import os
input_file_path = '/home/conquistador/code/github/python-01-utilities/file/'
input_file_name = 'file_exists_01.py'
file_exists = os.path.isfile(input_file_path+input_file_name)
print('file_exists :', file_exists)
input_file_name = 'file_exists_99.py'
file_exists = os.path.isfile(input_file_path+input_file_name)
print('file_exists :', file_exists)
input_file_name = 'file_exists_01.py'
dir_exists = os.path.isdir(input_file_path+input_file_name)
print('dir_exists :', dir_exists)
dir_exists = os.path.isdir(input_file_path)
print('dir_exists :', dir_exists)
input_file_path = '/home/conquistador/code/github/python-01-utilities/fil e/'
dir_exists = os.path.isdir(input_file_path)
print('dir_exists :', dir_exists)
input_file_path = '/home/conquistador/code/github/python-01-utilities/file/'
exists = os.path.exists(input_file_path)
print('exists :', exists)
exists = os.path.exists(input_file_path+input_file_name)
print('exists :', exists)

view raw
file_exists_01.py
hosted with ❤ by GitHub

Here is the output

$ python3.6 file_exists_01.py 
file_exists : True
file_exists : False
dir_exists : False
dir_exists : True
dir_exists : False
exists : True
exists : True

If you use isfile() on directory, outcome will be False, you need to use isfile or isdir as per requirement. Alternatively, you can use exists function as well, this returns True if input file or directory path is valid. Please refer to os.path documentation for further details.

Here is another way to check if file exists or not using pathlib module.

################################################################################################
# name: file_exists_02.py
# desc: Check if file exists
# date: 2019-02-13
# Author: conquistadorjd
################################################################################################
from pathlib import Path
input_file_path = '/home/conquistador/code/github/python-01-utilities/file/'
input_file_name = 'file_exists_01.py'
var = Path(input_file_path)
print('var : ', var.is_file())
var = Path(input_file_path+input_file_name)
print('var : ', var.is_file())
var = Path(input_file_path)
print('var : ', var.is_dir())
var = Path(input_file_path+input_file_name)
print('var : ', var.is_dir())
var = Path(input_file_path)
print('var : ', var.exists())
var = Path(input_file_path+input_file_name)
print('var : ', var.exists())

view raw
file_exists_02.py
hosted with ❤ by GitHub

Output is

$ python3.6 file_exists_02.py 
var : False
var : True
var : True
var : False
var : True
var : True

Both the modules have similar features, you can choose whichever is convenient to you.