AWS VPC Architecture Diagram and notes

VPC

VPC stands for virtual private cloud. VPC is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your AWS resources, such as Amazon EC2 instances, into your VPC.

Following are the key components of VPC

  1. Virtual private cloud (VPC) — A virtual network dedicated to your AWS account.
  2. Internet gateway — A gateway that you attach to your VPC to enable communication between resources in your VPC and the internet.
  3. Route table — A set of rules, called routes, that are used to determine where network traffic is directed. You can explicitly associate a subnet with a particular route table. Otherwise, the subnet is implicitly associated with the main route table.
  4. Network ACL – It is a list of rules to determine whether traffic is allowed in or out of any subnet associated with the network ACL
  5. Subnet — A subnet is a logical partition within a VPC. It is a range of IP addresses in your VPC. You can launch AWS resources into a specified subnet. Use a public subnet for resources that must be connected to the internet, and a private subnet for resources that won’t be connected to the internet .
  6. VPC endpoint — Enables you to privately connect your VPC to supported AWS services and VPC endpoint services powered by PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.
  7. CIDR block —Classless Inter-Domain Routing. An internet protocol address allocation and route aggregation methodology. For more information, see Classless Inter-Domain Routing in Wikipedia.

When new account is created, following components are already created for you.

  1. Default VPC
  2. Internet Gateway
  3. Main route table
  4. Main network ACL : Associated with all subnets
  5. Subnets in each of the availability zones
  6. Security Group

When you create new VPC, it does not create subnet. In each region, AWS automatically provides a default VPC with default subnets, a main route table, a default security group, and a default NACL.

  • Subnet
    • NACL
  • Instance
    • Security Groups

If you delete VPC following is deleted

  • IGW
  • Subnets
  • Route tables

CIDR

There are different ways to represent a range of IP addresses. The shortest way is by CIDR notation, sometimes called slash notation. For example, the CIDR 172.16.0.0/16 includes all addresses from 172.16.0.0 to 172.16.255.255—a total of 65,536 addresses!

Valid IPv4 prefix lengths range from /0 to /32. Although you can specify any valid IP range for your VPC CIDR, it’s best to use one in the RFC 1918 range to avoid conflicts with public Internet addresses.

  • 10.0.0.0–10.255.255.255 (10.0.0.0/8)
  • 172.16.0.0–172.31.255.255 (172.16.0.0/12)
  • 192.168.0.0–192.168.255.255 (192.168.0.0/16)

192.988.0.0/24 – 24 indicates the first 6 numbers of network. When you manually creates Subnet IP range must fall within VPC IP range

  • VPC 11.0.0.0/16
  • sub 11.0.1.0/24
  • /32 is a network mask of 255.255.255.255
  • /24 is a network mask of 255.255.255.0

Point to be noted:

192.168.0.254/32 = ip address of 192.168.0.254

192.168.0.1/24 = range of ip’s from 192.168.0.1 to 192.168.0.255

Subnets

A subnet is a logical partition within a VPC that holds your EC2 instances. A subnet lets you isolate instances from each other, control how traffic flows to and from your instances, and lets you organize them by function. For example, you can create one subnet for public web servers that need to be accessible from the Internet and create another subnet for database servers that only the web instances can access.

Each subnet has its own CIDR block that must be a subset of the VPC CIDR that it resides in. For example, if your VPC has a CIDR of 172.16.0.0/16, one of your subnets may have a CIDR of 172.16.100.0/24. This range covers 172.16.100.0–172.16.100.255, which yields a total of 256 addresses

A subnet can exist within only one availability zone

VPC and Subnet

aws ec2 create-subnet-vpc-id [VPC resource ID] --cidr-block 172.16.100.0/24 --availability-zone us-east-1a

Each subnet must be associated with a route table, which specifies the allowed routes for outbound traffic leaving the subnet. Every subnet that you create is automatically associated with the main route table for the VPC. You can change the association, and you can change the contents of the main route table.

To connect to internet following is needed

  1. connection to internet (IGW)
  2. route (Route table)
  3. public ip (IP)

Internet Gateways

An Internet gateway gives instances the ability to receive a public IP address, connect to the Internet, and receive requests from the Internet.

When you create a VPC, it does not have an Internet gateway associated with it. You must create an Internet gateway and associate it with a VPC manually. You can associate only one Internet gateway with a VPC. But you may create multiple Internet gateways and associate each one with a different VPC.

Route Table

A route table contains a set of rules, called routes, that are used to determine where network traffic from your subnet or gateway is directed. Main route table The route table that automatically comes with your VPC. It controls the routing for all subnets that are not explicitly associated with any other route table.

Your VPC has an implicit router, and you use route tables to control where network traffic is directed. Each subnet in your VPC must be associated with a route table, which controls the routing for the subnet (subnet route table). You can explicitly associate a subnet with a particular route table. Otherwise, the subnet is implicitly associated with the main route table. A subnet can only be associated with one route table at a time, but you can associate multiple subnets with the same subnet route table.

IP routing is destination-based, meaning that routing decisions are based only on the destination IP address , not the source. To enable Internet access for your instances, you must create a default route pointing to the Internet gateway

Destination Target
172.31.0.0/16 Local
0.0.0.0/0 igw-0e538022a0fddc318
  1. A local route that allows instances in different subnets to communicate with each other.
  2. enable your subnet to access the internet through an internet gateway

Any subnet that is associated with a route table containing a default route pointing to an Internet gateway, meaning if a subnet’s traffic is routed to an internet gateway, it is called a public subnet. Contrast this with a private subnet that does not have a default route.

Security Groups

  • A security group acts as a virtual firewall for your instance to control inbound and outbound traffic.
  • Security groups act at the instance level, not the subnet level. Therefore, each instance in a subnet in your VPC can be assigned to a different set of security groups.

Distributed firewalls and firewalls are stateful

Basic Rules:

  • You can specify allow rules, but not deny rules.
  • You can specify separate rules for inbound and outbound traffic.
  • Security group rules enable you to filter traffic based on protocols and port numbers.
  • Security groups are stateful — if you send a request from your instance, the response traffic for that request is allowed to flow in regardless of inbound security group rules. Responses to allowed inbound traffic are allowed to flow out, regardless of outbound rules.
  • You can add and remove rules at any time. Your changes are automatically applied to the instances that are associated with the security group.
  • When you associate multiple security groups with an instance, the rules from each security group are effectively aggregated to create one set of rules.

Inbound Rules

When you create a security group, it doesn’t contain any inbound rules. Security groups use a default-deny approach, also called whitelisting, which denies all traffic that is not explicitly allowed by a rule.

Inbound rules specify what traffic is allowed into the attached ENI. An inbound rule consists of three required elements:

  • Source
  • Protocol
  • Port range
Source Protocol Port Range
198.51.100.10/32 TCP 22
0.0.0.0/0 TCP 443
  • SSH access only from IP 198.51.100.10
  • HTTPS access from internet. The prefix 0.0.0.0/0 covers all valid IP addresses

Outbound Rules

the outbound rules of a security group will be less restrictive than the inbound rules. When you create a security group, AWS automatically creates the outbound rule listed

  • Destination
  • Protocol
  • Port range
Destination Protocol Port Range
0.0.0.0/0 All All

Network Access Control Lists

Like a security group, a network access control list (NACL) functions as a firewall in that it contains inbound and outbound rules to allow traffic based on a source or destination CIDR, protocol, and port. Also, each VPC has a default NACL that can’t be deleted. But the similarities end there.

A NACL differs from a security group in many respects. Instead of being attached to an ENI, a NACL is attached to a subnet. The NACL associated with a subnet controls what traffic may enter and exit that subnet. This means that NACLs can’t be used to control traffic between instances in the same subnet. If you want to do that, you have to use security groups.

  • NACL rule order matters! Create a new NACL
  • NACL rules are processed in ascending order of the rule number. Any traffic not matching this rule would be processed by next rule.

Inbound rules determine what traffic is allowed to ingress the subnet. Each rule contains the following elements:

  1. Rule number
  2. Protocol
  3. Port range
  4. Source
  5. Action
Rule Number Protocol Port Range Source Action
100 All All 0.0.0.0/0 Allow

Outbound Rules As you might expect, the outbound NACL rules follow an almost identical format as the inbound rules. Each rule contains the following elements:

  1. Rule number
  2. Protocol
  3. Port range
  4. Destination
  5. Action
Rule Number Protocol Port Range Destination Action
100 All All 0.0.0.0/0 Allow

Using Network Access Control Lists and Security Groups Together

You may want to use a NACL in addition to a security group so that you aren’t dependent upon users to specify the correct security group when they launch an instance. Because a NACL is applied to the subnet, the rules of the NACL apply to all traffic ingressing and egressing the subnet, regardless of how the security groups are configured.

When you make a change to a NACL or security group rule, that change takes effect immediately

Network Address Translation Devices

Although network address translation occurs at the Internet gateway, there are two other resources that can also perform NAT.

  • NAT gateway
  • NAT instance

AWS calls these NAT devices. The purpose of a NAT device is to allow an instance to access the Internet while preventing hosts on the Internet from reaching the instance directly. This is useful when an instance needs to go out to the Internet to fetch updates or to upload data but does not need to service requests from clients.

When you use a NAT device, the instance needing Internet access does not have a public IP address allocated to it. Incidentally, this makes it impossible for hosts on the Internet to reach it directly. Instead, only the NAT device is configured with a public IP. Additionally, the NAT device has an interface in a public subnet

Configuring Route Tables to Use NAT Devices Instances that use the NAT device must send Internet-bound traffic to it, while the NAT device must send Internet-bound traffic to an Internet gateway. Hence, the NAT device and the instances that use it must use different default routes. Furthermore, they must also use different route tables and hence must reside in separate subnets.

Route Table for private instance using NAT gateway

Destination Target
10.0.0.0/16 local
0.0.0.0/0 NAT Device

Route table for public instance

Destination Target
10.0.0.0/16 local
0.0.0.0/0 igw-0e538022a0fddc318

NAT Device

NAT Gateway

A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances.

  • It automatically scales to accommodate your bandwidth requirements. You set it and forget it.
  • You create a public NAT gateway in a public subnet and must associate an elastic IP address with the NAT gateway at creation.
  • you can use a public NAT gateway to connect to other VPCs or your on-premises network. In this case, you route traffic from the NAT gateway through a transit gateway or a virtual private gateway.

When you create a NAT gateway, you must assign it an EIP. A NAT gateway can reside in only one subnet, which must be a public subnet for it to access the Internet. AWS selects a private IP address from the subnet and assigns it to the NAT gateway. For redundancy, you may create additional NAT gateways in different availability zones.

NAT Instance

A NAT instance is a normal EC2 instance that uses a preconfigured Linux-based AMI. You have to perform the same steps to launch it as you would any other instance. It functions like a NAT gateway in many respects, but there are some key differences.

Unlike a NAT gateway, a NAT instance doesn’t automatically scale to accommodate increased bandwidth requirements. Therefore, it’s important that you select an appropriately robust instance type. If you choose an instance type that’s too small, you must manually upgrade to a larger instance type.

Attribute NAT gateway NAT instance
Availability Highly available. NAT gateways in each Availability Zone are implemented with redundancy. Create a NAT gateway in each Availability Zone to ensure zone-independent architecture. Use a script to manage failover between instances.
Bandwidth Scale up to 45 Gbps. Depends on the bandwidth of the instance type.
Maintenance Managed by AWS. You do not need to perform any maintenance. Managed by you, for example, by installing software updates or operating system patches on the instance.
Performance Software is optimized for handling NAT traffic. A generic AMI that’s configured to perform NAT.
Cost Charged depending on the number of NAT gateways you use, duration of usage, and amount of data that you send through the NAT gateways. Charged depending on the number of NAT instances that you use, duration of usage, and instance type and size.
Type and size Uniform offering; you don’t need to decide on the type or size. Choose a suitable instance type and size, according to your predicted workload.
Public IP addresses Choose the Elastic IP address to associate with a public NAT gateway at creation. Use an Elastic IP address or a public IP address with a NAT instance. You can change the public IP address at any time by associating a new Elastic IP address with the instance.
Private IP addresses Automatically selected from the subnet’s IP address range when you create the gateway. Assign a specific private IP address from the subnet’s IP address range when you launch the instance.
Security groups You cannot associate security groups with NAT gateways. You can associate them with the resources behind the NAT gateway to control inbound and outbound traffic. Associate with your NAT instance and the resources behind your NAT instance to control inbound and outbound traffic.
Network ACLs Use a network ACL to control the traffic to and from the subnet in which your NAT gateway resides. Use a network ACL to control the traffic to and from the subnet in which your NAT instance resides.
Flow logs Use flow logs to capture the traffic. Use flow logs to capture the traffic.
Port forwarding Not supported. Manually customize the configuration to support port forwarding.
Bastion servers Not supported. Use as a bastion server.
Traffic metrics View CloudWatch metrics for the NAT gateway. View CloudWatch metrics for the instance.
Timeout behavior When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet). When a connection times out, a NAT instance sends a FIN packet to resources behind the NAT instance to close the connection.
IP fragmentation Supports forwarding of IP fragmented packets for the UDP protocol. Does not support fragmentation for the TCP and ICMP protocols. Fragmented packets for these protocols will get dropped. Supports reassembly of IP fragmented packets for the UDP, TCP, and ICMP protocols.

NAT getway allows only response to come back and does not allow internet to reach private submnet

Elastic IP Addresses

An elastic IP address (EIP) is a type of public IP address that AWS allocates to your account when you request it. Once AWS allocates an EIP to your account, you have exclusive use of that address until you manually release it. Outside of AWS, there’s no noticeable difference between an EIP and an automatically assigned public IP.

When you initially allocate an EIP, it is not bound to any instance. Instead, you must associate it with an ENI. You can move an EIP around to different ENIs, although you can associate it with only one ENI at a time. Once you associate an EIP with an ENI, it will remain associated for the life of the ENI or until you disassociate it.

Ingress traffic is composed of all the data communications and network traffic originating from external networks and destined for a node in the host network.

Egress traffic is the reverse of ingress traffic. Egress is all traffic is directed towards an external network and originated from inside the host network.

If you want your instances to be accessible from the Internet, you must provision an Internet gateway, create a default route, and assign public IP addresses. Those are the basics. If you choose to use a NAT gateway or instance or a VPC peering connection, you’ll have to modify multiple route tables.

NAT instance enables hosts in a private subnet within your VPC, outbound access to the internet. a NAT instance allows instances within your VPC to go out to the internet. Bastion host allows inbound access to known IP addresses and authenticated users. Bastion host allows access to internal private services to approved sources.

Mostly the bastian host is used for incoming access NAT instance is for providing outgoing access to the instance

VPC Flow logs

It captures the ip logs. Can be stored on S3 and send to CloudWatch logs.

  • CloudTrail : API Calls
  • Flow Logs : IP Traffic

All encompassing VPC design

Here are sample design presented in our of the AWS presentation

VPC Batteries include

Hope this is helpful !

Sample NGINX config files

Many times NGINX conf reference comes handy. Here I am copying couple of NGINX conf setups for future reference.

Single WordPress

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        root /var/www/html;
        index index.html index.htm index.nginx-debian.html;

        #server_name _;
        server_name example.com www.example.com;

        location / {
                #try_files $uri $uri/ =404;
                #try_files $uri/ index.php$args;
	        try_files $uri /index.php$is_args$args;
        }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;

                # With php7.0-cgi alone:
                #fastcgi_pass 127.0.0.1:9000;
                # With php7.0-fpm:
                fastcgi_pass unix:/run/php/php7.0-fpm.sock;
        }

	location /wp-admin/ {
		index index.php;
		try_files $uri $uri/ /index.php$args;
	}
}

WordPress multi site

Please not the additional details at top and before closing bracket of server block. For installation of multiple please check post How to Install WordPress Multisite

map $http_host $blogid {
    default 0;
    include /var/www/yourwebsite/wp-content/uploads/nginx-helper/map.conf;
}

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        root /var/www/html;
        index index.html index.htm index.nginx-debian.html;

        #server_name _;
        server_name example.com www.example.com;

        location / {
                #try_files $uri $uri/ =404;
                #try_files $uri/ index.php$args;
	        try_files $uri /index.php$is_args$args;
        }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;

                # With php7.0-cgi alone:
                #fastcgi_pass 127.0.0.1:9000;
                # With php7.0-fpm:
                fastcgi_pass unix:/run/php/php7.0-fpm.sock;
        }

	location /wp-admin/ {
		index index.php;
		try_files $uri $uri/ /index.php$args;
	}
	
	location ~ ^/files/(.*)$ {
	  try_files /wp-content/blogs.dir/$blogid/$uri /wp-includes/ms-files.php?file=$1 ;
	  access_log off; log_not_found off; expires max;
	}

	location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
		expires 24h;
		log_not_found off;
	}

	location ^~ /blogs.dir {
		internal;
		alias /var/www/fintrekking/wp-content/blogs.dir ;
		access_log off; log_not_found off;      expires max;
	}


	if (!-e $request_filename) {
		rewrite /wp-admin$ $scheme://$host$uri/ permanent;
		rewrite ^/[_0-9a-zA-Z-]+(/wp-.*) $1 last;
		rewrite ^/[_0-9a-zA-Z-]+(/.*\.php)$ $1 last;
	}
	
}

How to check and Change timezone in Ubuntu

When you create Ubuntu server, by default timezone is of the same regions where host is located or predefined by the cloud service provider. If you need to check the timezone use following command.

$ timedatectl
               Local time: Sun 2021-06-27 11:09:28 CEST
           Universal time: Sun 2021-06-27 09:09:28 UTC 
                 RTC time: Sun 2021-06-27 09:09:29     
                Time zone: Europe/Paris (CEST, +0200)  
System clock synchronized: yes                         
              NTP service: active                      
          RTC in local TZ: no 

You can check the timezone using following command as well

$ cat /etc/timezone
Europe/Paris

Now let us check how to change the timezone. First you need long name of timezone and you can fetch it using following command

$ timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Algiers
Africa/Bissau
Africa/Cairo
Africa/Casablanca
.
.
Asia/Kolkata
.
.
Pacific/Tongatapu
Pacific/Wake
Pacific/Wallis
UTC
lines 298-348/348 (END

To change timezone, you need to run below command

$sudo timedatectl set-timezone your_time_zone

In current scenario, I need to change it to IST (Indian Standard Time). Here is the the output of current timezone after changing timezone

$ sudo timedatectl set-timezone Asia/Kolkata
$ timedatectl
               Local time: Sun 2021-06-27 14:46:44 IST
           Universal time: Sun 2021-06-27 09:16:44 UTC
                 RTC time: Sun 2021-06-27 09:16:45    
                Time zone: Asia/Kolkata (IST, +0530)  
System clock synchronized: yes                        
              NTP service: active                     
          RTC in local TZ: no 

Installing NGINX on Ubuntu

First you need to update the Ubuntu and then simply run sudo apt install as below

sudo apt update

When NGINX is installed it will create a folder called as /www inside /var/ directory in Ubuntu. Here is how folders look like before installation

/var$ ll
total 52
drwxr-xr-x 13 root root 4096 Apr 30 23:26 ./
drwxr-xr-x 19 root root 4096 Jun 15 14:39 ../
drwxr-xr-x 2 root root 4096 Apr 15 2020 backups/
drwxr-xr-x 12 root root 4096 Jun 15 14:40 cache/
drwxrwxrwt 2 root root 4096 Apr 30 23:25 crash/
drwxr-xr-x 38 root root 4096 Jun 15 14:39 lib/
drwxrwsr-x 2 root staff 4096 Apr 15 2020 local/
lrwxrwxrwx 1 root root 9 Apr 30 23:15 lock -> /run/lock/
drwxrwxr-x 9 root syslog 4096 Jun 15 14:39 log/
drwxrwsr-x 2 root mail 4096 Apr 30 23:15 mail/
drwxr-xr-x 2 root root 4096 Apr 30 23:15 opt/
lrwxrwxrwx 1 root root 4 Apr 30 23:15 run -> /run/
drwxr-xr-x 6 root root 4096 Apr 30 23:36 snap/
drwxr-xr-x 4 root root 4096 Apr 30 23:17 spool/
drwxrwxrwt 6 root root 4096 Jun 15 14:39 tmp/

Please note there is no /www directory as we have not installed NGINX yet. Please note that, if you have any other server such as apache in install, /www will already be there.
Now let us install NGINX.

sudo apt install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6
  libxpm4 nginx-common nginx-core
Suggested packages:
  libgd-tools fcgiwrap nginx-doc ssl-cert
The following NEW packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6
  libxpm4 nginx nginx-common nginx-core
0 upgraded, 17 newly installed, 0 to remove and 60 not upgraded.
Need to get 2431 kB of archives.
After this operation, 7891 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 fonts-dejavu-core all 2.37-1 [1041 kB]
Get:2 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 fontconfig-config all 2.13.1-2ubuntu3 [28.8 kB]
Get:3 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libfontconfig1 amd64 2.13.1-2ubuntu3 [114 kB]
Get:4 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal-updates/main amd64 libjpeg-turbo8 amd64 2.0.3-0ubuntu1.20.04.1 [117 kB]
Get:5 http://us-west-1.ec2.archive.ubuntu.com/ubuntu focal/main amd64 libjpeg8 amd64 8c-2ubuntu8 [2194 B]
......
Setting up libnginx-mod-stream (1.18.0-0ubuntu1.2) ...
Setting up libtiff5:amd64 (4.1.0+git191117-2ubuntu0.20.04.1) ...
Setting up libfontconfig1:amd64 (2.13.1-2ubuntu3) ...
Setting up libgd3:amd64 (2.2.5-5.2ubuntu2) ...
Setting up libnginx-mod-http-image-filter (1.18.0-0ubuntu1.2) ...
Setting up nginx-core (1.18.0-0ubuntu1.2) ...
Setting up nginx (1.18.0-0ubuntu1.2) ...
Processing triggers for ufw (0.36-6) ...
Processing triggers for systemd (245.4-4ubuntu3.6) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...

After installation /www will be created. Here is how it looks like after installation

var$ ll
total 56
drwxr-xr-x 14 root root 4096 Jun 15 14:42 ./
drwxr-xr-x 19 root root 4096 Jun 15 14:39 ../
drwxr-xr-x 2 root root 4096 Apr 15 2020 backups/
drwxr-xr-x 12 root root 4096 Jun 15 14:40 cache/
drwxrwxrwt 2 root root 4096 Apr 30 23:25 crash/
drwxr-xr-x 39 root root 4096 Jun 15 14:42 lib/
drwxrwsr-x 2 root staff 4096 Apr 15 2020 local/
lrwxrwxrwx 1 root root 9 Apr 30 23:15 lock -> /run/lock/
drwxrwxr-x 10 root syslog 4096 Jun 15 14:42 log/
drwxrwsr-x 2 root mail 4096 Apr 30 23:15 mail/
drwxr-xr-x 2 root root 4096 Apr 30 23:15 opt/
lrwxrwxrwx 1 root root 4 Apr 30 23:15 run -> /run/
drwxr-xr-x 6 root root 4096 Apr 30 23:36 snap/
drwxr-xr-x 4 root root 4096 Apr 30 23:17 spool/
drwxrwxrwt 6 root root 4096 Jun 15 14:39 tmp/
drwxr-xr-x 3 root root 4096 Jun 15 14:42 www/

NGINX registers itself as a service with ufw after installation.

$ sudo ufw app list
Available applications:
Nginx Full
Nginx HTTP
Nginx HTTPS
OpenSSH

Let us allow traffic using NGINX HTTP by using following command

$sudo ufw allow 'Nginx HTTP'
Rules updated
Rules updated (v6)

Finally just to ensure NGINX is running file, you can check the status using following command

$ sudo service nginx status
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-06-15 14:42:10 UTC; 2min 35s ago
Docs: man:nginx(8)
Main PID: 2030 (nginx)
Tasks: 2 (limit: 1160)
Memory: 5.3M
CGroup: /system.slice/nginx.service
├─2030 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
└─2031 nginx: worker process

Jun 15 14:42:10 ip-172-31-7-66 systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 15 14:42:10 ip-172-31-7-66 systemd[1]: Started A high performance web server and a reverse proxy server.

other commands.

$sudo service nginx status
$sudo service nginx start
$sudo service nginx stop
$sudo service nginx restart
$sudo service nginx reload
$sudo service nginx disable
$sudo service nginx enable

Now you can access your server and it will display below page

Hope this is helpful. Please let me know if you have any query or getting any unexpected error.

Getting Started with django with ubuntu

Tag line for django says “The web framework for perfectionists with deadlines.” and most would agree. django is rediculously fast to implement, its fully loaded with lots of utilitis, its secure and at the same time its highly scalable. some of the most popular sites like instagram, pinterest are built with django.

Let us get our hands dirty by getting started with django instead of talking about django features.

Which django version should I use ?

Here are the details about djnago release. you can choose latest version, however, I will stick with LTS version

Currently I am using Django 2.2 for most of my projects.

Which Python version should I use ?

This is the most common question and following table with clarify your query

Django version Python versions
  1.11 2.7, 3.4, 3.5, 3.6
  2.0 3.4, 3.5, 3.6, 3.7
  2.1,2.2 3.5, 3.6, 3.7

In this tutorial we are going to use python version 3.6 (3.6.4 to be very specifuc) and django version 1.11.

Install django version 1.11

$ sudo python3.6 -m pip install django==1.11
Collecting django==1.11
Downloading https://files.pythonhosted.org/packages/47/a6/078ebcbd49b19e22fd560a2348cfc5cec9e5dcfe3c4fad8e64c9865135bb/Django-1.11-py2.py3-none-any.whl (6.9MB)
100% |████████████████████████████████| 6.9MB 339kB/s 
Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from django==1.11) (2017.3)
Installing collected packages: django
Successfully installed django-1.11

You can check the version as below

$ python3.6 -m django --version
1.11

Creating django project

$ django-admin startproject mysite

Now CD into directory “mysite” and run following command

$python3.6 manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

July 18, 2018 - 17:43:11
Django version 1.11, using settings 'mysite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[18/Jul/2018 17:43:31] "GET / HTTP/1.1" 200 1716
Not Found: /favicon.ico

Now you can access django website at location http://127.0.0.1:8000/

here is the output

Creating application

Now that your environment – a “project” – is set up, you’re set to start doing work.

$ python3.6 manage.py startapp polls

after creating polls app, please add following code to polls/views.py


from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello, world. You're at the polls index.")

Now create a file urls.py and add following code


from django.conf.urls import url

from . import views

urlpatterns = [
    url(r'^$', views.index, name='index'),
]

Now add following code in mysite.urls.py


from django.conf.urls import include, url
from django.contrib import admin

urlpatterns = [
    url(r'^polls/', include('polls.urls')),
    url(r'^admin/', admin.site.urls),
]

Here is the output

Hope this is helpful…

On Cloud9

I have long reticule cloud as mere hype. I have primarily worked in IT service Industry where applications are huge, they run on huge mainframe or UNIX server which are hosted in organizations premises. Since I started my career with healthcare provider, I know how much important data security is for any enterprise.

Also, upgrading server OS version to latest version itself used to be big task, usually companies delayed upgrade for the fear of breaking something unknown. Main goal used to be ” Don’t fix if its not broken”. his lead to OS upgrade to the very last moment when vendor decides to stop supporting older version. Nonetheless there used to be many applications which continued to run on unsupported OS version, and in some cases, product itself is out of support.

In such a cases I always used to wonder under which business case, organizations would like to migrate to cloud and loose control over infra.

 

Cloud does of easy scalability and infra support but my understanding was that, this is concern for mostly smaller organizations or start ups. For big corporations, spending money on scaling infra and acquiring experts to support is not a challenge.

However I do recognize now that many companies, not just new age companies like Netflix etc are using cloud, even the older generation Telecom companies are looking to use cloud for some of its product. It could be private could or hybrid cloud but still, there is some adoption.

obvious leaders in cloud as of now is AWS, followed by Azure by Microsoft. Google cloud platform (GCP) is distant third, however if you are interested in big data and ML services along with cloud, GCP is the go to option.

Currently, I am working in Telecom domain, however I think, I need to move from Telecom to cross domain area like Cloud, Machine Learning , Data Science, NLP. Considering this, I have decided to get certified  in GCP Cloud Architect.

For next couple of months, I will be focusing more on cloud technologies, starting with Virtualization to Cloudification. These articles will be my notes for future reference and can be supporting/guiding articles for fellow learners and professionals.

Using JSON dataype for database queries – PostgreSQL

I was torn between using relational database and document database like mongodb, elastic search etc for certain requirement. In fact I started learning elasticsearch but getting some of the tasks done with elasticseach is very painful compared to how easy its done in relational database. This was impacting my timeline and I was spending way too much time doing troubleshooting.

This is when I came across json datatype in PostgreSQL. This is combinations of easiness of relational database and document query capabilities of elastic seach. Of course this would be bit slower than elasticseach (I have not benchmarked the performance, buts its guess) but it is OK as I am in prototype phase, I just need to validate my business case, once it flies, I will make a switch be relevant document database.

Meanwhile, lets us see how to work with JSON datatype of PostgreSQL

select clause

Because -> operator returns a JSON object, you can chain it with the operator ->> to retrieve a it in text format.

SELECT id, productid,  productdetails -> 'productBaseInfoV1' ->> 'productUrl' AS customer
FROM public.ecomm_productdetails;
SELECT id, productid,  productdetails -> 'productBaseInfoV1' -> 'productUrl' AS customer
FROM public.ecomm_productdetails;

where clause

Please note you always need to use final element with ->> since this needs to be matched with text field.

SELECT id, productid,  productdetails
FROM public.ecomm_productdetails
WHERE productdetails -> 'productBaseInfoV1' ->> 'title' = 'Apple iPhone 6 (Grey, 128 GB)';

 

SELECT id, productid,  productdetails
FROM public.ecomm_productdetails
WHERE productdetails -> 'productBaseInfoV1' ->> 'title' like 'OPPO%';

Troubleshooting django web application deployed using NGINX, uWSGI errors

For any type of error, first check if UWSGI is running fine ?

myusername@ubuntu-512:~$ sudo service uwsgi status
● uwsgi.service - uWSGI Emperor
   Loaded: loaded (/etc/systemd/system/uwsgi.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-09-09 08:39:27 IST; 9min ago
  Process: 1451 ExecStartPre=/bin/bash -c mkdir -p /run/uwsgi; chown myusername:www-data /run/uwsgi (code=exited, status=0/SUCCESS)
 Main PID: 1530 (uwsgi)
   Status: "The Emperor is governing 3 vassals"
    Tasks: 7
   Memory: 142.7M
      CPU: 1.874s
   CGroup: /system.slice/uwsgi.service
           ├─1530 /usr/local/bin/uwsgi --emperor /home/myusername/uwsgi/sites
           ├─1564 /usr/local/bin/uwsgi --ini website1.ini
           ├─1565 /usr/local/bin/uwsgi --ini website2.ini
           ├─1625 /usr/local/bin/uwsgi --ini website3.ini
           ├─2022 /usr/local/bin/uwsgi --ini website3.ini
           ├─2033 /usr/local/bin/uwsgi --ini website2.ini
           └─2035 /usr/local/bin/uwsgi --ini website1.ini

Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: Input to details english quote We-accept-the-love-we-think-we-deserve.
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: context: success
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: context: post_type quote
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: [pid: 2033|app: 0|req: 7/7] 114.119.146.195 () {40 vars in 768 bytes} [Wed Sep  9 03:14:29 2020] GET
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: [pid: 2035|app: 0|req: 1/1] 93.158.161.64 () {38 vars in 515 bytes} [Wed Sep  9 03:14:29 2020] GET /
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: announcing my loyalty to the Emperor...
Sep 09 08:44:29 ubuntu-512 uwsgi[1530]: Wed Sep  9 08:44:29 2020 - [emperor] vassal website1.ini is now loyal
Sep 09 08:46:03 ubuntu-512 uwsgi[1530]: [pid: 2035|app: 0|req: 2/2] 114.119.145.68 () {40 vars in 748 bytes} [Wed Sep  9 03:16:03 2020] GET 
Sep 09 08:49:09 ubuntu-512 uwsgi[1530]: --- no python application found, check your startup logs for errors ---
Sep 09 08:49:09 ubuntu-512 uwsgi[1530]: [pid: 2022|app: -1|req: -1/5] 49.35.75.185 () {46 vars in 692 bytes} [Wed Sep  9 03:19:09 2020] GET

You might see any error here. In above log its showing error as --- no python application found, check your startup logs for errors ---. In this case, we need to check the wsgi.py file settings. Your wsgi file should look something like below. Please not the lines where I have added comment #added this

import os
import sys #added this

from django.core.wsgi import get_wsgi_application

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) #added this
sys.path.append(BASE_DIR) #added this

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'yourapp.settings')

application = get_wsgi_application()

 

Another way to trouble shoot is to check if .sock is created at following location

/run/uwsgi

If nothing gives any clue, try to see if you can run application manually using following command

sudo uwsgi --http 0.0.0.0:8400 --home /var/www/mywebsite1/venvft --chdir /var/www/mywebsite1/mysite07 --wsgi-file /var/www/mywebsite1/mysite07/wsgi.py

You might get different error here. e.g. some library is not installed. I got an error that model requests not found. I installed it in virtualenv as below

$virtualenv venvft
created virtual environment CPython3.6.4.final.0-64 in 453ms
creator CPython3Posix(dest=/var/www/yourapp/venvft, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/user/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$source venvft/bin/activate
$pip install django 
$pip install psycopg2
$pip install requests
deactivate

Another error I got is as below

*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72920 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
  File "/var/www/website3/mysite07/wsgi.py", line 19, in 
    application = get_wsgi_application()
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 994, in _gcd_import
  File "", line 971, in _find_and_load
  File "", line 955, in _find_and_load_unlocked
  File "", line 665, in _load_unlocked
  File "", line 678, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/auth/apps.py", line 8, in 
    from .checks import check_models_permissions, check_user_model
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/auth/checks.py", line 8, in 
    from .management import _get_builtin_permissions
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/auth/management/__init__.py", line 9, in 
    from django.contrib.contenttypes.management import create_contenttypes
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/contrib/contenttypes/management/__init__.py", line 2, in 
    from django.db import (
  File "/var/www/website3/venvft/lib/python3.6/site-packages/django/db/migrations/__init__.py", line 1, in 
    from .migration import Migration, swappable_dependency  # NOQA
ModuleNotFoundError: No module named 'django.db.migrations.migration'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 2699, cores: 1)

Here if you see in details, error is with the migrations. In above screen error is ModuleNotFoundError: No module named 'django.db.migrations.migration' common solution is to reinstall django. Here fortunately we are using virtualenv so I can remove and reinstall it without impacting other applications

$rm -rf venvft
$virtualenv venvft
created virtual environment CPython3.6.4.final.0-64 in 453ms
creator CPython3Posix(dest=/var/www/yourapp/venvft, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/user/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$source venvft/bin/activate
$pip install django 
$pip install psycopg2
pip install requests
deactivate

once its done I rerun the following

sudo uwsgi --http 0.0.0.0:8400 --home /var/www/mywebsite1/venvft --chdir /var/www/mywebsite1/mysite07 --wsgi-file /var/www/mywebsite1/mysite07/wsgi.py

Here again I got below error

sFatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f7b5b79d700 (most recent call first):
Aborted (core dumped)

however when I tried to access my website, it was working properly. so for now, I decided to ignore the error

django QuerySet examples

If you want to fetch whole table or if you want to fetch n number of rows (lets try 30 in current case)

context = MyModel.objects.all()
context = MyModel.objects.all()[:30]
## You can iterate over the output as well 
for e in MyModel.objects.all():
      print(e.headline)

if you want to fetch data in specific order. – sign indicates descending

context = MyModel.objects.all().order_by('pub_date')
context = MyModel.objects.all().order_by('-pub_date')

Filtering

(name__startswith='Beatles')
#like clause
filter(name__contains='Smith')
Product.objects.filter(name__search='Shiny')

exluding

 context = MyModel.objects.exclude(pub_date__gt=datetime.date(2005, 1, 3)).exclude(headline='Hello')

if you want to fetch distinct values. Please note you can combineorder_by and distinct

context = MyModel.objects.all().distinct('blog')
context = MyModel.objects.all().order_by('pub_date').distinct('blog')

 

.values()

Returns a QuerySet that returns dictionaries, rather than model instances, when used as an iterable.

context = MyModel.objects.all().distinct('blog').values()

 

Calling one python script from another

Ideally should should import requited python file using import into another and call required function from other programs but there could be some instances where you would need to trigger one python script from another.

This  is fairly simple.

Lets say I have following two program located in same folder one.py and one_sub.py. Let us try calling one_sub.py from one.py is fairly simple

one.py

import os

os.system('python3.6 ' + 'one_sub.py')

This code will trigger one_sub.py.

Passing parameters to calling file

This needs minor changes. Whatever you need to pass, just mention that value or variable after a space.
one.py

import os

os.system('python3.6 ' + 'one_sub.py 11 ')

one_sub.py

import sys
print("---This is inside script 2")

input_value = int(sys.argv[1])
print(" 0 ", sys.argv[0], )
print(" 1 ", sys.argv[1], type(sys.argv[0]))

Please note, 0th parameter is always the script name, you can pass multiple parameters.

Also, received files are always string, you need to change to required datatype using datatype conversion operators e.g. int(sys.argv[1])

Trigger file located at different directory

import os

file_path = '/home/user/code/quant/source/library/'
os.system('python3.6 ' + file_path + 'one_sub.py 11 22')

Please note the training '/' in file_path