Troubleshooting django web application deployed using NGINX, uWSGI errors

For any type of error, first check if UWSGI is running fine ?

myusername@ubuntu-512:~$ sudo service uwsgi status
● uwsgi.service - uWSGI Emperor
   Loaded: loaded (/etc/systemd/system/uwsgi.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-09-09 08:39:27 IST; 9min ago
  Process: 1451 ExecStartPre=/bin/bash -c mkdir -p /run/uwsgi; chown myusername:www-data /run/uwsgi (code=exited, status=0/SUCCESS)
 Main PID: 1530 (uwsgi)
   Status: "The Emperor is governing 3 vassals"
    Tasks: 7
   Memory: 142.7M
      CPU: 1.874s
   CGroup: /system.slice/uwsgi.service
           ├─1530 /usr/local/bin/uwsgi --emperor /home/myusername/uwsgi/sites
           ├─1564 /usr/local/bin/uwsgi --ini website1.ini
           ├─1565 /usr/local/bin/uwsgi --ini website2.ini
           ├─1625 /usr/local/bin/uwsgi --ini website3.ini
           ├─2022 /usr/local/bin/uwsgi --ini website3.ini
           ├─2033 /usr/local/bin/uwsgi --ini website2.ini
           └─2035 /usr/local/bin/uwsgi --ini website1.ini

Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: Input to details english quote We-accept-the-love-we-think-we-deserve.
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: context: success
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: context: post_type quote
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: [pid: 2033|app: 0|req: 7/7] 114.119.146.195 () {40 vars in 768 bytes} [Wed Sep  9 03:14:29 2020] GET
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: [pid: 2035|app: 0|req: 1/1] 93.158.161.64 () {38 vars in 515 bytes} [Wed Sep  9 03:14:29 2020] GET /
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: announcing my loyalty to the Emperor...
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: Wed Sep  9 08:44:29 2020 - [emperor] vassal website1.ini is now loyal
Sep 09 08:46:03 ubuntu-512 uwsgi[1530]: [pid: 2035|app: 0|req: 2/2] 114.119.145.68 () {40 vars in 748 bytes} [Wed Sep  9 03:16:03 2020] GET 
Sep 09 08:49:09 ubuntu-512 uwsgi[1530]: --- no python application found, check your startup logs for errors ---
Sep 09 08:49:09 ubuntu-512 uwsgi[1530]: [pid: 2022|app: -1|req: -1/5] 49.35.75.185 () {46 vars in 692 bytes} [Wed Sep  9 03:19:09 2020] GET

You might see any error here. In above log its showing error as --- no python application found, check your startup logs for errors ---. In this case, we need to check the wsgi.py file settings. Your wsgi file should look something like below. Please not the lines where I have added comment #added this

import os
import sys #added this

from django.core.wsgi import get_wsgi_application

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) #added this
sys.path.append(BASE_DIR) #added this

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'yourapp.settings')

application = get_wsgi_application()

 

Another way to trouble shoot is to check if .sock is created at following location

/run/uwsgi

If nothing gives any clue, try to see if you can run application manually using following command

sudo uwsgi --http 0.0.0.0:8400 --home /var/www/mywebsite1/venvft --chdir /var/www/mywebsite1/mysite07 --wsgi-file /var/www/mywebsite1/mysite07/wsgi.py

You might get different error here. e.g. some library is not installed. I got an error that model requests not found. I installed it in virtualenv as below

$virtualenv venvft
created virtual environment CPython3.6.4.final.0-64 in 453ms
creator CPython3Posix(dest=/var/www/yourapp/venvft, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/user/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$source venvft/bin/activate
$pip install django 
$pip install psycopg2
$pip install requests
deactivate

Another error I got is as below

*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72920 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
  File "/var/www/website3/mysite07/wsgi.py", line 19, in 
    application = get_wsgi_application()
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 994, in _gcd_import
  File "", line 971, in _find_and_load
  File "", line 955, in _find_and_load_unlocked
  File "", line 665, in _load_unlocked
  File "", line 678, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/auth/apps.py", line 8, in 
    from .checks import check_models_permissions, check_user_model
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/auth/checks.py", line 8, in 
    from .management import _get_builtin_permissions
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/auth/management/__init__.py", line 9, in 
    from django.contrib.contenttypes.management import create_contenttypes
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/contenttypes/management/__init__.py", line 2, in 
    from django.db import (
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/db/migrations/__init__.py", line 1, in 
    from .migration import Migration, swappable_dependency  # NOQA
ModuleNotFoundError: No module named 'django.db.migrations.migration'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 2699, cores: 1)

Here if you see in details, error is with the migrations. In above screen error is ModuleNotFoundError: No module named 'django.db.migrations.migration' common solution is to reinstall django. Here fortunately we are using virtualenv so I can remove and reinstall it without impacting other applications

$rm -rf venvft
$virtualenv venvft
created virtual environment CPython3.6.4.final.0-64 in 453ms
creator CPython3Posix(dest=/var/www/yourapp/venvft, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/user/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$source venvft/bin/activate
$pip install django 
$pip install psycopg2
pip install requests
deactivate

once its done I rerun the following

sudo uwsgi --http 0.0.0.0:8400 --home /var/www/mywebsite1/venvft --chdir /var/www/mywebsite1/mysite07 --wsgi-file /var/www/mywebsite1/mysite07/wsgi.py

Here again I got below error

sFatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f7b5b79d700 (most recent call first):
Aborted (core dumped)

however when I tried to access my website, it was working properly. so for now, I decided to ignore the error

django QuerySet examples

If you want to fetch whole table or if you want to fetch n number of rows (lets try 30 in current case)

context = MyModel.objects.all()
context = MyModel.objects.all()[:30]
## You can iterate over the output as well 
for e in MyModel.objects.all():
      print(e.headline)

if you want to fetch data in specific order. – sign indicates descending

context = MyModel.objects.all().order_by('pub_date')
context = MyModel.objects.all().order_by('-pub_date')

Filtering

(name__startswith='Beatles')
#like clause
filter(name__contains='Smith')
Product.objects.filter(name__search='Shiny')

exluding

 context = MyModel.objects.exclude(pub_date__gt=datetime.date(2005, 1, 3)).exclude(headline='Hello')

if you want to fetch distinct values. Please note you can combineorder_by and distinct

context = MyModel.objects.all().distinct('blog')
context = MyModel.objects.all().order_by('pub_date').distinct('blog')

 

.values()

Returns a QuerySet that returns dictionaries, rather than model instances, when used as an iterable.

context = MyModel.objects.all().distinct('blog').values()

 

Django admin CSS or static file not working [Solved]

You might have came across a situation where your application works perfectly fine on local machine but when deployed to production, everything works except admin section, technically it functions as expected but it does not have CSS. Why this happens ?

If you want to recreate this problem, just following in settings.py from True to False.

DEBUG = False
This happens because you application is not able to load css files for admin  functionality. To address this problem, please follow below mentioned steps.

Step#1 collectstatic

First you need to create a folder called static in your project root directory, then run below command
$python3.6 manage.py collectstatic.
This will move admin css files under static folder. You can see new folder created under static folder called admin. This folder will have admin view related css, js , fonts, img.
You might wonder where were these files before ? these were server from site-packages folder of Django. You dont need to dig details here, but if you want you can.

Step#2 Change server config.

When you are running, runserver, you are running development server and it does a lot of things for you, including serving of static files but in production, Django expects that the static files will be served by your serer.

This is not difficult at all, just have below changes done to your nginx server config.

server {
    server_name domainname.com www.domainname.com;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/run/uwsgi/domainname.sock;
    }

    location /static {                        # add this
       alias /var/www/domainname/static;      # add this
    }                                         # add this

}

if you are hosting django application using uwsgi and NGINX, this will work for you. I am not sure about gunicorn server. You can check post deploying django with uwsgi and nginx for deployment of your application to server.

First make sure you application is working on production server and then work on to make admin successfully.  Deploying Django to server has its own challenges, don’t make it more complicated by adding this change too. Attack problems one at a time.

Please let me know if you faced any issue while getting this working.

How to Install correct django version

django has multiple version which are supported. There are few differences which can cause confusion while using it. If you want to install specific version of django, follow below steps.

Step#1 Check what is the current version.

$ python3.6 -m django --version
2.0

OR

$ python3.6
Python 3.6.4 (default, Jan 13 2018, 12:02:51) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.__version__
'1.11'
>>> exit()

If current version is expected version, the you don’t need any of below steps.

Step#2 Uninstall current version

Please note that here you don’t have to put any version while uninstalling. whatever be the current version that will be uninstalled.

$ sudo python3.6 -m pip uninstall django
[sudo] password for user: 
Uninstalling Django-1.11:
  Would remove:
    /usr/local/bin/django-admin
    /usr/local/bin/django-admin.py
    /usr/local/lib/python3.6/site-packages/Django-1.11.dist-info/*
    /usr/local/lib/python3.6/site-packages/django/*
Proceed (y/n)? y
  Successfully uninstalled Django-1.11

Step#3 Install expected version

If you want to install latest version use following command

$ sudo python3.6 -m pip install django
Collecting django
  Downloading https://files.pythonhosted.org/packages/ab/15/cfde97943f0db45e4f999c60b696fbb4df59e82bbccc686770f4e44c9094/Django-2.0.7-py3-none-any.whl (7.1MB)
    100% |████████████████████████████████| 7.1MB 103kB/s 
Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from django) (2017.3)
Installing collected packages: django
Successfully installed django-2.0.7

OR use below if you want to install any specific version

$ sudo python3.6 -m pip install django==2.0
Collecting django==2.0
  Downloading https://files.pythonhosted.org/packages/44/98/35b935a98a17e9a188efc2d53fc51ae0c8bf498a77bc224f9321ae5d111c/Django-2.0-py3-none-any.whl (7.1MB)
    100% |████████████████████████████████| 7.1MB 133kB/s 
Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from django==2.0) (2017.3)
Installing collected packages: django
Successfully installed django-2.0

and you are all set. Please let me if you get any issue while installing expected version

Getting started with django with python3 and ubuntu

Tag line for django says “The web framework for perfectionists with deadlines.” and most would agree. django is rediculously fast to implement, its fully loaded with lots of utilitis, its secure and at the same time its highly scalable. some of the most popular sites like instagram, pinterest are built with django.

Let us get our hands dirty by getting started with django instead of talking about django features.

Which django version should I use ?

Here are the details about djnago release. you can choose latest version, however, I will stick with LTS version

Which Python version should I use ?

This is the most common question and following table with clarify your query

Django version Python versions
  1.11 2.7, 3.4, 3.5, 3.6
  2.0 3.4, 3.5, 3.6, 3.7
  2.1,2.2 3.5, 3.6, 3.7

In this tutorial we are going to use python version 3.6 (3.6.4 to be very specifuc) and latest django version 3.0.1

Installation

If you want to install specific version use following

$ pip3 install Django==3.0.1

Alternative way to install Django is as below

$ sudo python3 -m pip install django
Collecting django  
  Downloading Django-X.X.X-py3-none-any.whl (7.1MB)
    100% |████████████████████████████████| 7.1MB 186kB/s 
Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from django)  
Installing collected packages: django  
Successfully installed django-X.X.X  

Checking the installed version

Simple way to check is using below command

$python3 -m django --version
3.0.1

another way

~$ python3
Python 3.6.4 (default, Jan 13 2018, 12:02:51)  
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.  
>>> import django
>>> print django.get_version()
  File "<stdin>", line 1
    print django.get_version()
               ^
SyntaxError: invalid syntax  
>>> print(django.get_version())
X.X.X  

Creating django project

Project is created using following command

$django-admin startproject mysite

This will create folder mysite having required files. Now let us start the application by running server.

$ python3 manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

December 28, 2019 - 07:55:49
Django version 3.0.1, using settings 'mysite01.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.  

Now your website is up and running

By default server status at port 8000. You can change port using following command

$python3.6 manage.py runserver 8080

Creating an app

The term application describes a Python package that provides some set of features. Applications may be reused in various projects. The term project describes a Django web application. The project Python package is defined primarily by a settings module, but it usually contains other things.Applications include some combination of models, views, templates, template tags, static files, URLs, middleware, etc. They’re generally wired into projects with the INSTALLED_APPS setting and optionally with other mechanisms such as URLconfs, the MIDDLEWARE setting, or template inheritance.

$ python3 manage.py startapp polls

This will create a sub-directory as below

├── mysite01
│   ├── db.sqlite3
│   ├── manage.py
│   ├── mysite01
│   │   ├── asgi.py
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-36.pyc
│   │   │   ├── settings.cpython-36.pyc
│   │   │   ├── urls.cpython-36.pyc
│   │   │   └── wsgi.cpython-36.pyc
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   └── polls
│       ├── admin.py
│       ├── apps.py
│       ├── __init__.py
│       ├── migrations
│       │   └── __init__.py
│       ├── models.py
│       ├── tests.py
│       └── views.py

Now to have our very basic application working, let us create urls.py file under folder polls and add below code

from django.urls import path
from . import views

app_name = 'polls'
urlpatterns = [
    # ex: /polls/
    path('', views.index, name='index'),
    path('first/', views.first, name='first'),
]

Please note “/’ after “first” url name.

We have updated url structure now let us code the view to be displayed if these urls are accessed. Update following code in polls/views.py

from django.shortcuts import render
from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world. You're at the polls index.")

def first(request):
    context = "Test Data"
    return render(request, 'polls/first.html', {'context': context})

As you see, we are displaying very basic message. This will give you basic idea about hor urls and views work. Now before you run the server again we need to inform our project about the application ‘polls’ by updating settings.py file. Update following in settings.py


INSTALLED_APPS = [
    'polls.apps.PollsConfig',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Also update main project urls.py to include polls/urls.py. any url starting with domain-name+polls will be served from ‘polls/urls.py’


from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('polls/',  include('polls.urls')),
    path('admin/', admin.site.urls),
]

Now let us run the server again

$ python3 manage.py runserver

Now you can access this website at below path
http://127.0.0.1:8000/polls/

http://127.0.0.1:8000/polls/first/