Getting started with Time Series Forecasting with Prophet

Facebook prophet is simplest way to get started with time series analysis with python. Please refer to this post to know how to install prophet on Ubuntu.

We will have a look at official example with one complete program.

Here is the output on terminal

$ python3.6 01_fbprophet_getting_started.py 
*** Program Started ***
ds y
0 2007-12-10 9.590761
1 2007-12-11 8.519590
2 2007-12-12 8.183677
3 2007-12-13 8.072467
4 2007-12-14 7.893572
INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Initial log joint probability = -19.4685
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
99 7975.37 0.00149529 224.247 1 1 128 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
186 7992.27 5.72063e-05 157.088 5.678e-07 0.001 261 LS failed, Hessian reset 
199 7993.26 0.000312701 314.644 0.1004 1 277 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
299 7997.05 0.0015387 170.701 1 1 408 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
321 7998.61 0.00020668 308.573 1.22e-06 0.001 478 LS failed, Hessian reset 
369 8000.52 2.98767e-05 97.9518 2.746e-07 0.001 566 LS failed, Hessian reset 
399 8000.98 0.000153501 134.602 0.7945 0.7945 601 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
457 8001.99 0.000276407 292.083 2.159e-06 0.001 718 LS failed, Hessian reset 
499 8002.58 0.000699641 197.602 1 1 770 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
550 8003.07 5.79234e-05 181.032 3.403e-07 0.001 874 LS failed, Hessian reset 
599 8003.43 0.000218596 78.2273 0.7213 0.7213 928 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
695 8004.08 3.66526e-05 116.76 2.994e-07 0.001 1095 LS failed, Hessian reset 
699 8004.11 0.000537041 111.615 1 1 1099 
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes 
788 8004.7 3.21305e-06 76.4964 4.987e-08 0.001 1259 LS failed, Hessian reset 
797 8004.7 6.1457e-07 61.1166 0.6741 0.6741 1270 
Optimization terminated normally: 
Convergence detected: relative gradient magnitude is below tolerance
type of m <class 'fbprophet.forecaster.Prophet'>
type of future <class 'pandas.core.frame.DataFrame'>
type of forecast <class 'pandas.core.frame.DataFrame'>
*** Program Completed ***

 

 

Let us try to run the same by using column name as date and value instead of ds and y. When I tried this, I got following error.

 

$ python3.6 01_fbprophet_getting_started.py 
*** Program Started ***
time value
0 2007-12-10 9.590761
1 2007-12-11 8.519590
2 2007-12-12 8.183677
3 2007-12-13 8.072467
4 2007-12-14 7.893572
Traceback (most recent call last):
File "01_fbprophet_getting_started.py", line 19, in <module>
m.fit(df)
File "/usr/local/lib/python3.6/site-packages/fbprophet/forecaster.py", line 1016, in fit
"Dataframe must have columns 'ds' and 'y' with the dates and "
ValueError: Dataframe must have columns 'ds' and 'y' with the dates and values respectively.

Daily Seasonality

You might have observed following message

INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.

To get rid of this message, add daily_seasonality=True prophet object. It will look like below.

m = Prophet(daily_seasonality=True)

 

How to Install facebook prophet library Ubuntu

Prophet is forecasting librabry developed by facebook, it has been open sourced by facebook. Following is the simple command to install it.

 

$ sudo python3.6 -m pip install fbprophet
Collecting fbprophet
Downloading https://files.pythonhosted.org/packages/9b/a1/fef4ce00acbc28e75c0d33f60c9777527c4295656903b00ac4c9525cef7f/fbprophet-0.4.post2.tar.gz (45kB)
100% |████████████████████████████████| 51kB 354kB/s 
Collecting Cython>=0.22 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/e1/fd/711507fa396064bf716493861d6955af45369d2c470548e34af20b79d4d4/Cython-0.29.6-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)
100% |████████████████████████████████| 2.1MB 316kB/s 
Collecting pystan>=2.14 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/17/77/dd86797a7e7fccca117233c6d50cc171e0c2b2f5a0cd2a8d9753ee09b7be/pystan-2.18.1.0-cp36-cp36m-manylinux1_x86_64.whl (50.0MB)
100% |████████████████████████████████| 50.0MB 312kB/s 
Requirement already satisfied: numpy>=1.10.0 in /usr/local/lib/python3.6/site-packages (from fbprophet) (1.14.0)
Requirement already satisfied: pandas>=0.20.1 in /usr/local/lib/python3.6/site-packages (from fbprophet) (0.22.0)
Requirement already satisfied: matplotlib>=2.0.0 in /usr/local/lib/python3.6/site-packages (from fbprophet) (2.1.2)
Collecting lunardate>=0.1.5 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/4e/7e/377a3cbba646ec0cf79433ef858881d809a3b87eb887b0901cb83c66a758/lunardate-0.2.0-py3-none-any.whl
Collecting convertdate>=2.1.2 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/74/83/d0fa07078f4d4ae473a89d7d521aafc66d82641ea0af0ef04a47052e8f17/convertdate-2.1.3-py2.py3-none-any.whl
Collecting holidays>=0.9.5 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/16/09/c882bee98acfa310933b654697405260ec7657c78430a14e785ef0f1314b/holidays-0.9.10.tar.gz (73kB)
100% |████████████████████████████████| 81kB 370kB/s 
Collecting setuptools-git>=1.2 (from fbprophet)
Downloading https://files.pythonhosted.org/packages/05/97/dd99fa9c0d9627a7b3c103a00f1566d8193aca8d473884ed258cca82b06f/setuptools_git-1.2-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/site-packages (from pandas>=0.20.1->fbprophet) (2.6.1)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/site-packages (from pandas>=0.20.1->fbprophet) (2017.3)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/site-packages (from matplotlib>=2.0.0->fbprophet) (0.10.0)
Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/site-packages (from matplotlib>=2.0.0->fbprophet) (1.11.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib>=2.0.0->fbprophet) (2.2.0)
Collecting ephem<3.8,>=3.7.5.3 (from convertdate>=2.1.2->fbprophet)
Downloading https://files.pythonhosted.org/packages/c3/2c/9e1a815add6c222a0d4bf7c644e095471a934a39bc90c201f9550a8f7f14/ephem-3.7.6.0.tar.gz (739kB)
100% |████████████████████████████████| 747kB 246kB/s 
Installing collected packages: Cython, pystan, lunardate, ephem, convertdate, holidays, setuptools-git, fbprophet
Running setup.py install for ephem ... done
Running setup.py install for holidays ... done
Running setup.py install for fbprophet ... done
Successfully installed Cython-0.29.6 convertdate-2.1.3 ephem-3.7.6.0 fbprophet-0.4.post2 holidays-0.9.10 lunardate-0.2.0 pystan-2.18.1.0 setuptools-git-1.2
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 

It takes some time considering number of dependencies. You can verify installation by login to python3.6 console, importing module and checking it.

$ python3.6
Python 3.6.4 (default, Jan 13 2018, 12:02:51) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fbprophet
>>> fbprophet.__version__
'0.4'
>>>

Please note that while using facebook prophet, correct way to import library is as below

from fbprophet import Prophet

How to Handle Broken Images

If your website is displaying many images and some of their are from different sources, eventually you will run into a situation when some of the images no longer exists. They are either move to different path or got deleted altogether.

This is what you will start seeing something like below in your website.

This is very annoying to users and it needs to be fixed on topmost priority.

Here are some of the ways to fix broken images.

Remove img tag itself.

This is the simplest and most effective way of doing it.

<img src="Error.src" onerror="this.style.display='none'"/>

Another way of doing same thing

<img src="Error.src" onerror="this.parentNode.removeChild(this);">

This will remove img tag itself and user will never know if any image was to be displayed here, Only issue being, if you have referred image in your content, user might get confused.

Display Backup Image

Implementing this is also very easy but for various reasons I dont link this option.

<img src="Error.src" onerror="this.src='http://example.com/existent-image.jpg'"/>

This option remove awkwardness looking broken image picture, however it will be replace it will backup image. Humanly it’s not possible to set up back up image for each image so mostly, there will be one backup image for all images. This reduces the awkwardness of broken image but it replaces it with backup image. This highlights the broken image issue and if image is referred in content, it does not help user with anything.

How to Create Custom Query for WordPress

WordPress has a default query which fetches posts and display it in descending manner. If you are working on any WordPress customisation, you would come across a requirement to fetch WordPress posts in customised manner such as Fetching last 10 posts, Fetching post having specific category or tag.

Fortunately, we dont have to write MySQL level queries to get required data (and that what makes WordPress special), we can call a WP_Query function by passing required fields as input, we can get required data. This logic has two parts.

  1. Query : The query will select WordPress posts from the MySQL database based on our instructions.
  2. Loop : It will output the post information on the page. Each instance of output array will have attributes such as post thumbnail, title, date, and author etc.

Here is simple structure to help you understand custom WP_Query

// The Query
$the_query = new WP_Query( $args );

// The Loop
if ( $the_query->have_posts() ) {
    echo '<ul>';
    while ( $the_query->have_posts() ) {
        $the_query->the_post();
        echo '<li>' . get_the_title() . '</li>';
    }
    echo '</ul>';
    /* Restore original Post Data */
    wp_reset_postdata();
} else {
    // no posts found
}

 

You can customized the query as per your requirement.

<?php

// Creating array
$args = array( 
'posts_per_page' => 15, 
'orderby' => 'date', 
'order' ='ASC'
);
// Initiate the custom query
$custom_query = new WP_Query( $args );

?>

By Default, it will search for posts, if you want to search for specific custom post type, you can simply add a parameter in array

<?php

// Creating array
$args = array( 
'post_type' => 'vehicle',
'posts_per_page' => 15, 
'orderby' => 'date', 
'order' ='ASC'
);
// Initiate the custom query
$custom_query = new WP_Query( $args );

?>

How to Check if a File Exists using Python

Recently I was working on file generation using python, before generating any file, I had to check if file exists or not, to check if file exists or not, I had to find a way to do in in program. As usual, this task is very easy using python.

Checking if file exists or not can be done in multiple ways using python, here is one using “os” module

os.path module has functions such as isfile, isdir and exists which helps us check if file or directory exists or not.

Here is the output

$ python3.6 file_exists_01.py 
file_exists : True
file_exists : False
dir_exists : False
dir_exists : True
dir_exists : False
exists : True
exists : True

If you use isfile() on directory, outcome will be False, you need to use isfile or isdir as per requirement. Alternatively, you can use exists function as well, this returns True if input file or directory path is valid. Please refer to os.path documentation for further details.

Here is another way to check if file exists or not using pathlib module.

Output is

$ python3.6 file_exists_02.py 
var : False
var : True
var : True
var : False
var : True
var : True

Both the modules have similar features, you can choose whichever is convenient to you.

How to crop image using Python

Image cropping is very easy using python library Pillow. Following is simple program to crop image in required parts.

Most important part of cropping is to know pixel size of your image, you can crop required portion using .crop method of Image module of Pillow.

Sample program

Please note that crop takes tuple  having four values as input. These four values represent left, upper, right, lower point .

Original image

Left Half of the image

Right Half of the image

Top Right corner of the image

Here is the output of the program, please note the pixel size of each cropped image.

$ python3.6 06_crop_image_01.py 
*** Program Started ***
im.size (1920, 1318)
im.size (960, 1318)
im.size (960, 1318)
im.size (960, 659)
*** Program Ended ***

How to compress images using Python

Python Library Pillow can be used very effectively to compress images. While doing some research on this I found that .JPG files can be compress very effectively however this does not work well with .PNG files. Here is sample program to reduce file size of an image.

Output of the program while using .JPG file as input and output

$ python3.6 05_compress_image_01.py 
*** Program Started ***
Input file size : (5456, 3632)
Input file name : 05_compress_image_01_input.jpg
Input Image Size : 1611664
Output file size : (5456, 3632)
Output file name : 05_compress_image_01_output.jpg
Output Image Size : 443479
*** Program Ended ***

Output of the program while using .PNG files as input and output

$ python3.6 05_compress_image_01.py 
*** Program Started ***
Input file size : (1920, 1282)
Input file name : 05_compress_image_01_input.png
Input Image Size : 3683320
Output file size : (1920, 1282)
Output file name : 05_compress_image_01_output.png
Output Image Size : 3619363
*** Program Ended ***

As you can see, while using .PNG files for input and output, there is hardly any change in file size however when you used .JPG files, output files is of 27% of original file size. Your percentage reduction might be different based on the file that you have used.

Please note input and out file dimensions, number of pixels stays the same.

How to resize image using Python

Python library Pillow can be used to resize images. Please note resize does not mean compressing image. Yes, reducing pixels can lead to reduction in file size in terms of KBs however it will not be significant. If you need to compress image, please check How to compress image using python.

While resizing the image file, you need to maintain the aspect ration, else image might get distorted, we will see an example of the same.

Here is sample program to resize.

here is the output

$ python3.6 04_resize_image_01.py 
*** Program Started ***
im.size (1920, 1318)
im.size (960, 659)
*** Program Ended ***

Here we have maintained the aspect ratio so the files will look similar
Input

Output

Now lets look at resize by not maintaining the aspect ratio

Output of the program

$ python3.6 04_resize_image_02.py 
*** Program Started ***
im.size (1920, 1318)
im.size (500, 480)
*** Program Ended ***

You can see the issue with changing of aspect ratio.

Input

Output

Size comparison 

How to add text on image using Python

As a part of image processing, we sometimes need to write text on the image file. Pillow is a python library which can be used to add text on images using python. Using Pillow, which us a fork of PIL, is very easy for these kind of image processing activities.

Let us have a look at adding simple text on image file.

Please note that text location is determined by parameter position, its tuple with two parameters and it represents x and y axis position. In below example I have used position = (50, 50).

Now if you are wondering what these values represent, these are pixel numbers with (0,0) being top left corner.  If you image size is (640, 480) and you want to put any character exactly at center, you need to provide position as (320,240). Hope this clarifies the parameter

Here is the output

As you can see the text added does not look good sine we have not added any formatting to text.  Now let us look at formatted text  on image. Here is program with additional parameters for font

Here we have added two additional parameters, font and color, you can plan your desired font into font directory and can use it for adding text to image file.

Note: I tried using this for non english text e.g. devnagari text but it did not work.

Here is the output.

How to create blank image using Pillow, Python

Sometimes we need a blank image file to be created during execution of program, during processing required text or anything else can be added. Following is the sample program to create blank file using PIL, python.

You can decide file size and color as per your requirement.

Here is the output image