How to find all files in a directory with specific extension using Python

Whenever we need to check if file with specific extension exists in a directory or get list of files with specific extension from a directory, it can be done using python. Here is simple program for doing the same.

There are three ways to do this using python

  1. os.walk
  2. os.listdir
  3. glob.glob

All three methods are demonstrated in below program

Output of the program

*** Program Started ***
List of Files using os.walk: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using listdir: ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
List of Files using glob : ['file3.txt', 'list_to_file_in_directory.txt', 'test.txt']
*** Program Completed ***

 

How to count number of files in directory using Python

Counting number of files using python in specific folder can be done using multiple ways.  Some of these are mentioned below

Here is the output

*** Program Started ***
Number of Files using os.walk : 7
Number of Files using listdir method#1 : 8
Number of Files using listdir method#2 : 7
Number of Files using listdir method#3 : 7
Number of Files using glob : 7
*** Program Completed ***

As you can see Number of Files using listdir method#1 are different, this is because, its counting a folder as a file.

 

How to write a list to a file and read a list from file using Python

Many a times we need to store list in a file for later usage, in such cases, its better to store list in a file and read this file into list whenever we need to use. Python has very easy methods for achieving this.

 

Output of the program

*** Program Started ***
lines : ['one', 'two', 'three', 'four', '']
type of lines : <class 'list'>
line : one
line : two
line : three
line : four
line :
*** Program Completed ***

file generated is as below

Here is another version of same program

output and file generated is same except file opening closing styles are different.

Getting started with Time Series Forecasting with Prophet

Facebook prophet is simplest way to get started with time series analysis with python. Please refer to this post to know how to install prophet on Ubuntu.

We will have a look at official example with one complete program.

Here is the output on terminal

$ python3.6 01_fbprophet_getting_started.py 
*** Program Started ***
ds y
0 2007-12-10 9.590761
1 2007-12-11 8.519590
2 2007-12-12 8.183677
3 2007-12-13 8.072467
4 2007-12-14 7.893572
INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Initial log joint probability = -19.4685
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
99 7975.37 0.00149529 224.247 1 1 128 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
186 7992.27 5.72063e-05 157.088 5.678e-07 0.001 261 LS failed, Hessian reset 
199 7993.26 0.000312701 314.644 0.1004 1 277 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
299 7997.05 0.0015387 170.701 1 1 408 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
321 7998.61 0.00020668 308.573 1.22e-06 0.001 478 LS failed, Hessian reset 
369 8000.52 2.98767e-05 97.9518 2.746e-07 0.001 566 LS failed, Hessian reset 
399 8000.98 0.000153501 134.602 0.7945 0.7945 601 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
457 8001.99 0.000276407 292.083 2.159e-06 0.001 718 LS failed, Hessian reset 
499 8002.58 0.000699641 197.602 1 1 770 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
550 8003.07 5.79234e-05 181.032 3.403e-07 0.001 874 LS failed, Hessian reset 
599 8003.43 0.000218596 78.2273 0.7213 0.7213 928 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
695 8004.08 3.66526e-05 116.76 2.994e-07 0.001 1095 LS failed, Hessian reset 
699 8004.11 0.000537041 111.615 1 1 1099 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
788 8004.7 3.21305e-06 76.4964 4.987e-08 0.001 1259 LS failed, Hessian reset 
797 8004.7 6.1457e-07 61.1166 0.6741 0.6741 1270 
Optimization terminated normally: 
Convergence detected: relative gradient magnitude is below tolerance
type of m <class 'fbprophet.forecaster.Prophet'>
type of future <class 'pandas.core.frame.DataFrame'>
type of forecast <class 'pandas.core.frame.DataFrame'>
*** Program Completed ***

 

 

Let us try to run the same by using column name as date and value instead of ds and y. When I tried this, I got following error.

 

$ python3.6 01_fbprophet_getting_started.py 
*** Program Started ***
time value
0 2007-12-10 9.590761
1 2007-12-11 8.519590
2 2007-12-12 8.183677
3 2007-12-13 8.072467
4 2007-12-14 7.893572
Traceback (most recent call last):
File "01_fbprophet_getting_started.py", line 19, in <module>
m.fit(df)
File "/usr/local/lib/python3.6/site-packages/fbprophet/forecaster.py", line 1016, in fit
"Dataframe must have columns 'ds' and 'y' with the dates and "
ValueError: Dataframe must have columns 'ds' and 'y' with the dates and values respectively.

Daily Seasonality

You might have observed following message

INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.

To get rid of this message, add daily_seasonality=True prophet object. It will look like below.

m = Prophet(daily_seasonality=True)

 

How to Install facebook prophet library Ubuntu

Prophet is forecasting librabry developed by facebook, it has been open sourced by facebook. Following is the simple command to install it.

 

$ sudo python3.6 -m pip install fbprophet
Collecting fbprophet
Downloading https://files.pythonhosted.org/packages/9b/a1/fef4ce00acbc28e75c0d33f60c9777527c4295656903b00ac4c9525cef7f/fbprophet-0.4.post2.tar.gz (45kB)
100% |████████████████████████████████| 51kB 354kB/s 
Collecting Cython>=0.22 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/e1/fd/711507fa396064bf716493861d6955af45369d2c470548e34af20b79d4d4/Cython-0.29.6-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)
100% |████████████████████████████████| 2.1MB 316kB/s 
Collecting pystan>=2.14 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/17/77/dd86797a7e7fccca117233c6d50cc171e0c2b2f5a0cd2a8d9753ee09b7be/pystan-2.18.1.0-cp36-cp36m-manylinux1_x86_64.whl (50.0MB)
100% |████████████████████████████████| 50.0MB 312kB/s 
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.6/site-packages (from fbprophet) (1.14.0)
Requirement already satisfied: pandas>=0.20.1 in /usr/local/lib/python3.6/site-packages (from fbprophet) (0.22.0)
Requirement already satisfied: matplotlib>=2.0.0 in /usr/local/lib/python3.6/site-packages (from fbprophet) (2.1.2)
Collecting lunardate>=0.1.5 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/4e/7e/377a3cbba646ec0cf79433ef858881d809a3b87eb887b0901cb83c66a758/lunardate-0.2.0-py3-none-any.whl
Collecting convertdate>=2.1.2 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/74/83/d0fa07078f4d4ae473a89d7d521aafc66d82641ea0af0ef04a47052e8f17/convertdate-2.1.3-py2.py3-none-any.whl
Collecting holidays>=0.9.5 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/16/09/c882bee98acfa310933b654697405260ec7657c78430a14e785ef0f1314b/holidays-0.9.10.tar.gz (73kB)
100% |████████████████████████████████| 81kB 370kB/s 
Collecting setuptools-git>=1.2 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/05/97/dd99fa9c0d9627a7b3c103a00f1566d8193aca8d473884ed258cca82b06f/setuptools_git-1.2-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/site-packages (from pandas>=0.20.1->fbprophet) (2.6.1)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/site-packages (from pandas>=0.20.1->fbprophet) (2017.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/site-packages (from matplotlib>=2.0.0->fbprophet) (0.10.0)
Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/site-packages (from matplotlib>=2.0.0->fbprophet) (1.11.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib>=2.0.0->fbprophet) (2.2.0)
Collecting ephem<3.8,>=3.7.5.3 (from convertdate>=2.1.2->fbprophet)
Downloading https://files.pythonhosted.org/packages/c3/2c/9e1a815add6c222a0d4bf7c644e095471a934a39bc90c201f9550a8f7f14/ephem-3.7.6.0.tar.gz (739kB)
100% |████████████████████████████████| 747kB 246kB/s 
Installing collected packages: Cython, pystan, lunardate, ephem, convertdate, holidays, setuptools-git, fbprophet
Running setup.py install for ephem ... done
Running setup.py install for holidays ... done
Running setup.py install for fbprophet ... done
Successfully installed Cython-0.29.6 convertdate-2.1.3 ephem-3.7.6.0 fbprophet-0.4.post2 holidays-0.9.10 lunardate-0.2.0 pystan-2.18.1.0 setuptools-git-1.2
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 

It takes some time considering number of dependencies. You can verify installation by login to python3.6 console, importing module and checking it.

$ python3.6
Python 3.6.4 (default, Jan 13 2018, 12:02:51) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fbprophet
>>> fbprophet.__version__
'0.4'
>>>

Please note that while using facebook prophet, correct way to import library is as below

from fbprophet import Prophet