Python Archives - Technology Trekking

15/05/201923/05/2019

Windows 10 | ‘pip’ is not recognized as an internal or external command

After installing python on windows 10, I wanted to install few more packages such as numpy, scipy, pandas, however I got following error


C:\>pip install pandas
'pip' is not recognized as an internal or external command,
operable program or batch file.

pip script is present in following folder


C:\Users\USERNAME\AppData\Local\Programs\Python\Python36\Scripts
OR
C:\Python\Python36\Scripts

To make pip work, we need to add this to path variable.


C:\>path %path%;C:\Users\USERNAME\AppData\Local\Programs\Python\Python36\Scripts

outcome after adding pip to path


C:\>pip install pandas
Collecting pandas
Downloading https://files.pythonhosted.org/packages/d0/4e/9db3468e504ac9aeadb37eb32bcf0a74d063d24ad1471104bd8a7ba20c97/pandas-0.24.2-cp36-cp36m-win_amd64.whl (8.8MB)
100% |████████████████████████████████| 8.8MB 36kB/s
Collecting python-dateutil>=2.5.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
100% |████████████████████████████████| 235kB 67kB/s
Collecting pytz>=2011k (from pandas)
Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)
100% |████████████████████████████████| 512kB 64kB/s
Collecting numpy>=1.12.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/2e/11/f006363050b24fb19a235e5efd219e7ac549398d531110d80b8f2ba3a909/numpy-1.16.3-cp36-cp36m-win_amd64.whl (11.9MB)
100% |████████████████████████████████| 11.9MB 39kB/s
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas)
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
Successfully installed numpy-1.16.3 pandas-0.24.2 python-dateutil-2.8.0 pytz-2019.1 six-1.12.0
You are using pip version 9.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

15/05/2019

Windows 10 | ‘python’ is not recognized as an internal or external command

After installing python on Windows 10, I got following error.

C:\>python
'python' is not recognized as an internal or external command,
operable program or batch file.

To my surprise py works fine but python does not work.

C:\>py
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

To make python works, we need to set up path. For Windows 10, python will be at following folder

C:\Users\USERNAME\AppData\Local\Programs\Python\Python36
OR
C:\Python\Python36

Use following command to set up path


C:\>path %path%;C:\Users\USERNAME\AppData\Local\Programs\Python\Python36

Once this is done, you can check the path variable using below command


C:\>echo %path%

Now let us try using python

C:\>python
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

30/03/201930/03/2019

How to find all files in a directory with specific extension using Python

Whenever we need to check if file with specific extension exists in a directory or get list of files with specific extension from a directory, it can be done using python. Here is simple program for doing the same.

There are three ways to do this using python

os.walk
os.listdir
glob.glob

All three methods are demonstrated in below program

################################################################################################

# name: file_with_specific_extension_in_directory.py

# desc: 

# date: 2019-03-30

# Author: conquistadorjd

################################################################################################

import os

import glob

print("*** Program Started ***")

dir_name = 'C:/sample/'

matches = []

for root, dirnames, filenames in os.walk(dir_name):

	for filename in filenames:

		# if filename.endswith(('.txt', '.MOV', '.avi', '.mpg')):  ## Checking for multiple extenstions

		if filename.endswith('.txt'):

			matches.append(filename)

print("List of Files using os.walk:", matches)

matches = []

filenames = os.listdir(dir_name)

for filename in filenames:

	# if filename.endswith(('.txt', '.MOV', '.avi', '.mpg')):     ## Checking for multiple extenstions

	if filename.endswith('.txt'):

		matches.append(filename)

print("List of Files using listdir:", matches)

# filenames =  glob.glob(dir_name+'*.txt')		## This will return file name with complete path name as well

matches = [os.path.basename(x) for x in glob.glob(dir_name+'*.txt')] ## This will return only file name

print("List of Files using glob   :", matches)

print("*** Program Completed ***")

view raw
file_with_specific_extension_in_directory.py
hosted with ❤ by GitHub

Output of the program

*** Program Started ***
List of Files using os.walk: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using listdir: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using glob : ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
*** Program Completed ***

30/03/2019

How to count number of files in directory using Python

Counting number of files using python in specific folder can be done using multiple ways. Some of these are mentioned below

################################################################################################

# name: file_count_in_directory.py

# desc: 

# date: 2019-03-30

# Author: conquistadorjd

################################################################################################

import os

import glob

print("*** Program Started ***")

dir_name = 'C:/sample/'

file_count = sum([len(files) for r, d, files in os.walk(dir_name)])

print('Number of Files using os.walk          :', file_count)

file_count = os.listdir(dir_name)

print("Number of Files using listdir method#1 :", len(file_count))

file_count =  len([name for name in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, name))])

print("Number of Files using listdir method#2 :", file_count)

file_count =  [name for name in os.listdir(dir_name) if os.path.isfile(os.path.join(dir_name, name))]

print("Number of Files using listdir method#3 :", len(file_count))

file_count =  glob.glob(dir_name+'*.*')

print("Number of Files using glob             :", len(file_count))

print("*** Program Completed ***")

view raw
file_count_in_directory.py
hosted with ❤ by GitHub

Here is the output

*** Program Started ***
Number of Files using os.walk : 7
Number of Files using listdir method#1 : 8
Number of Files using listdir method#2 : 7
Number of Files using listdir method#3 : 7
Number of Files using glob : 7
*** Program Completed ***

As you can see Number of Files using listdir method#1 are different, this is because, its counting a folder as a file.

30/03/2019

How to write a list to a file and read a list from file using Python

Many a times we need to store list in a file for later usage, in such cases, its better to store list in a file and read this file into list whenever we need to use. Python has very easy methods for achieving this.

################################################################################################

# name: list_to_file_01.py

# desc: 

# date: 2019-03-30

# Author: conquistadorjd

################################################################################################

print('*** Program Started ***')

score = ["one","two","three", "four"]

with open("list_to_file.txt", "w") as f_write:

    for s in score:

        f_write.write(str(s) +",")

with open("list_to_file.txt", "r") as f_read:

	lines = f_read.read().split(',')

	print('lines : ', lines )

	print('type of lines : ', type(lines) )		

	for line in lines:

		print("line : ", line)

print('*** Program Completed ***')

view raw
list_to_file_01.py
hosted with ❤ by GitHub

Output of the program

*** Program Started ***
lines : ['one', 'two', 'three', 'four', '']
type of lines : <class 'list'>
line : one
line : two
line : three
line : four
line :
*** Program Completed ***

file generated is as below

one,two,three,four,

view raw
list_to_file.txt
hosted with ❤ by GitHub

Here is another version of same program

################################################################################################

# name: list_to_file_02.py

# desc: 

# date: 2019-03-30

# Author: conquistadorjd

################################################################################################

print('*** Program Started ***')

score = ["one","two","three", "four"]

f_write = open("list_to_file.txt", "w")

for s in score:

    f_write.write(str(s) +",")

# Expkicite close is needed in this case

f_write.close()

f_read = open("list_to_file.txt", "r")

lines = f_read.read().split(',')

print('lines : ', lines )

print('type of lines : ', type(lines) )		

for line in lines:

	print("line : ", line)

## This close can be ommited considering program termination closes all open files.

f_read.close()

print('*** Program Completed ***')

view raw
list_to_file_02.py
hosted with ❤ by GitHub

output and file generated is same except file opening closing styles are different.

13/02/2019

How to Check if a File Exists using Python

Recently I was working on file generation using python, before generating any file, I had to check if file exists or not, to check if file exists or not, I had to find a way to do in in program. As usual, this task is very easy using python.

Checking if file exists or not can be done in multiple ways using python, here is one using “os” module

os.path module has functions such as isfile, isdir and exists which helps us check if file or directory exists or not.

################################################################################################

# name: file_exists_01.py

# desc: Check if file exists

# date: 2019-02-13

# Author: conquistadorjd

################################################################################################

import os

input_file_path = '/home/conquistador/code/github/python-01-utilities/file/'

input_file_name = 'file_exists_01.py'

file_exists = os.path.isfile(input_file_path+input_file_name) 

print('file_exists :', file_exists)

input_file_name = 'file_exists_99.py'

file_exists = os.path.isfile(input_file_path+input_file_name) 

print('file_exists :', file_exists)

input_file_name = 'file_exists_01.py'

dir_exists = os.path.isdir(input_file_path+input_file_name) 

print('dir_exists :', dir_exists)

dir_exists = os.path.isdir(input_file_path) 

print('dir_exists :', dir_exists)

input_file_path = '/home/conquistador/code/github/python-01-utilities/fil e/'

dir_exists = os.path.isdir(input_file_path) 

print('dir_exists :', dir_exists)

input_file_path = '/home/conquistador/code/github/python-01-utilities/file/'

exists = os.path.exists(input_file_path) 

print('exists :', exists)

exists = os.path.exists(input_file_path+input_file_name) 

print('exists :', exists)

view raw
file_exists_01.py
hosted with ❤ by GitHub

Here is the output

$ python3.6 file_exists_01.py 
file_exists : True
file_exists : False
dir_exists : False
dir_exists : True
dir_exists : False
exists : True
exists : True

If you use isfile() on directory, outcome will be False, you need to use isfile or isdir as per requirement. Alternatively, you can use exists function as well, this returns True if input file or directory path is valid. Please refer to os.path documentation for further details.

Here is another way to check if file exists or not using pathlib module.

################################################################################################

# name: file_exists_02.py

# desc: Check if file exists

# date: 2019-02-13

# Author: conquistadorjd

################################################################################################

from pathlib import Path

input_file_path = '/home/conquistador/code/github/python-01-utilities/file/'

input_file_name = 'file_exists_01.py'

var = Path(input_file_path)

print('var : ', var.is_file())

var = Path(input_file_path+input_file_name)

print('var : ', var.is_file())

var = Path(input_file_path)

print('var : ', var.is_dir())

var = Path(input_file_path+input_file_name)

print('var : ', var.is_dir())

var = Path(input_file_path)

print('var : ', var.exists())

var = Path(input_file_path+input_file_name)

print('var : ', var.exists())

view raw
file_exists_02.py
hosted with ❤ by GitHub

Output is

$ python3.6 file_exists_02.py 
var : False
var : True
var : True
var : False
var : True
var : True

Both the modules have similar features, you can choose whichever is convenient to you.

11/02/201911/02/2019

How to read image using Pillow, Python and get image attributes

Pillow is the friendly PIL fork. PIL is the Python Imaging Library. This is the first article in series of image processing articles using python.

In this article we will see how to read file using pillow and get basic attributes.

Here is simplest program to read image file using pillow and get basic attributes

################################################################################################

# name: 01_read_image.py

# desc: 

# date: 2019-02-10

# Author: conquistadorjd

################################################################################################

from PIL import Image

print('*** Program Started ***')

# image_font_path = 'imagepath_input = '/home/conquistador/code/github/python-01-utilities/image/fonts/'

image_path_input = '/home/conquistador/code/github/python-01-utilities/image/input/'

# image_path_output = '/home/conquistador/code/github/python-01-utilities/image/output/'

image_name_input = '01_read_image.jpg'

im = Image.open(image_path_input + image_name_input)

print('im object: ',  im)

print('format   : ' , im.format)

print('size     : ' , im.size) 

print('mode     : ' , im.mode)

print('filename : ' , im.filename) 

print('width    : ' , im.width) 

print('height   : ' , im.height) 

print('info     : ' , im.info)

# This will display image on screen

# im.show()

print('*** Program Ended ***')

view raw
01_read_image.py
hosted with ❤ by GitHub

Output of this program

*** Program Started ***
im object: <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1920x1285 at 0x7F7FB79F6518>
format : JPEG
size : (1920, 1285)
mode : RGB
filename : /home/conquistador/code/github/python-01-utilities/image/input/01_read_image.jpg
width : 1920
height : 1285
info : {'jfif': 257, 'jfif_version': (1, 1), 'jfif_unit': 0, 'jfif_density': (1, 1), 'progressive': 1, 'progression': 1}
*** Program Ended ***

11/02/2019

How to check file size in Python

Many times while doing file processing python, we need to know file size in bytes/KBs/MBs. You can get file size using multiple methods, following are two simple methods to get file size using os module.

I have run this file for two inputs,one is image and another is empty file.

################################################################################################

# name: file_size.py

# desc: Getting gile size using python

# date: 2019-02-10

# Author: conquistadorjd

################################################################################################

import io, os

print('*** Program Started ***')

image_path_input = '/home/conquistador/code/github/python-01-utilities/file/input/'

image_name_input =  'file_size-01.jpg' #'file_size-02'

if os.stat(image_path_input + image_name_input).st_size ==0 :

	print('Input file is empty')

else :

	print('Input file is not empty')

print('File size (in Bytes) : ',os.stat(image_path_input + image_name_input).st_size)

print('File size (in Bytes) : ',os.path.getsize (image_path_input+image_name_input))

image_name_input =  'file_size-02'

if os.stat(image_path_input + image_name_input).st_size ==0 :

	print('Input file is empty')

else :

	print('Input file is not empty')

print('File size (in Bytes) : ',os.stat(image_path_input + image_name_input).st_size)

print('File size (in Bytes) : ',os.path.getsize (image_path_input+image_name_input))

print('*** Program Ended ***')

view raw
file_size.py
hosted with ❤ by GitHub

Here is the output of the program

$ python3.6 file_size.py 
*** Program Started ***
Input file is not empty
File size (in Bytes) : 147162
File size (in Bytes) : 147162
Input file is empty
File size (in Bytes) : 0
File size (in Bytes) : 0
*** Program Ended ***

Actual file size

I have added a check to see if file size is empty, you might need to check file size before doing any processing on files.

02/02/2019

Generating data for Linear Regression using NumPy

We have already seen how to generate random numbers in previous article, here we will have a look at how to generate data in specific format for linear regression.

To test data for linear regression, we will need a data which has somewhat linear relationship and one set of random data. Please find below code to generate data having some linear relation and random data using Python and NumPy. I have provided graphs which will help you understand the data created by using these programs.

Data with Linear Trend for Linear Regression

################################################################################################

#	name:	numpy_data_linear.py

#	desc:	Genarate test data having linear relationship

#	date:	2019-02-02

#	Author:	conquistadorjd

################################################################################################

import numpy as np

from matplotlib import pyplot as plt

print('*** Program Started ***')

n = 50

x = np.arange(–n/2,n/2,1,dtype=np.float64)

m = np.random.uniform(0.3,0.5,(n,))

b = np.random.uniform(5,10,(n,))

y = x*m +b 

print('x',x, type(x[0]))

print('y',y, type(y[0]))

plt.scatter(x,y,s=None, marker='o',color='g',edgecolors='g',alpha=0.9,label="Linear Relation")

plt.grid(color='black', linestyle='–', linewidth=0.5,markevery=int)

plt.legend(loc=2)

plt.axis('scaled')

plt.show()

plt.savefig('numpy_data_linear.jpeg')

print('*** Program ended ***')

view raw
numpy_data_linear.py
hosted with ❤ by GitHub

Data without any Trend for Linear Regression

################################################################################################

#	name:	numpy_data_random_01.py

#	desc:	Genarate test data having linear relationship

#	date:	2019-02-02

#	Author:	conquistadorjd

################################################################################################

import numpy as np

from matplotlib import pyplot as plt

print('*** Program Started ***')

n = 50

x = np.arange(–n/2,n/2,1,dtype=np.float64)

y = a=np.random.uniform(–15,15,(n,))

print('x',x)

print('y',y)

plt.scatter(x,y,s=None, marker='o',color='g',edgecolors='g',alpha=0.9,label="Random numbers")

plt.grid(color='black', linestyle='–', linewidth=0.5,markevery=int)

plt.legend(loc=2)

plt.axis('scaled')

plt.show()

plt.savefig('numpy_data_random_01.jpeg')

print('*** Program ended ***')

view raw
numpy_data_random_01.py
hosted with ❤ by GitHub

You can use this as an input data while training your model.

06/12/201811/12/2018

Python NumPy Tutorial : Getting started with NumPy

NumPy is BSD licensed fundamental package for scientific computing with Python. Most important feature is a powerful N-dimensional array object and sophisticated (broadcasting) functions. It also has useful linear algebra, Fourier transform, and random number capabilities

NumPy can also be used as an efficient multi-dimensional container of generic data. One of the most important feature is Arbitrary data-types.

Importing numpy and creating, accessing and modifying array

>>> import numpy as np
>>> a=np.array([1,2,3,4,5,6])
>>> a
array([1, 2, 3, 4, 5, 6])
>>> type(a)
<class 'numpy.ndarray'>
>>> a[1]
2
>>> a[1]=9
>>> a
array([1, 9, 3, 4, 5, 6])
>>> b = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
>>> b
array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])

Numpy Properties

>>> a.shape
(6,)
>>> b.shape
(3, 4)
>>> a.size
6
>>> b.size
12
>>> a.data
<memory at 0x7faf92bb0dc8>
>>> b.data
<memory at 0x7faf92bbba68>
>>> a.dtype
dtype('int64')
>>> b.dtype
dtype('int64')

Mathematical operations on numpy numpyarray

>>> a
array([1, 2, 3, 4, 5, 6])
>>> b
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
>>> c
array([ 3, 6, 9, 12, 15, 18])
>>> a+c
array([ 4, 8, 12, 16, 20, 24])
>>> a-c
array([ -2, -4, -6, -8, -10, -12])
>>> a*c
array([ 3, 12, 27, 48, 75, 108])
>>> a/c
array([0.33333333, 0.33333333, 0.33333333, 0.33333333, 0.33333333,
0.33333333])
>>> np.sqrt(a)
array([1. , 1.41421356, 1.73205081, 2. , 2.23606798,
2.44948974])
>>> np.sum(b,axis=0)
array([15, 18, 21, 24])
>>> np.sum(b,axis=1)
array([10, 26, 42])
>>> b
array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])
>>> b.T
array([[ 1,  5,  9],
       [ 2,  6, 10],
       [ 3,  7, 11],
       [ 4,  8, 12]])

Working on the arrays

>>> b
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
>>> b.reshape(4,3)
array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10, 11, 12]])
>>> b
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
>>> b.resize(4,3)
>>> b
array([[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9],
[10, 11, 12]])
>>> b[0:2,1:2] #[row range,column range]
array([[2],
       [5]])