Simple Git Auto Deployment to Digital Ocean or any other VPS

I have been using Digital Ocean VPS for couple of years now. I am always editing my websites even their technology stack has changed back and forth multiple times. I have always used FileZilla to transfer my code from local to remote repository. This is not efficient way and while searching for better way of autodeployment I came across an article by Digital Ocean itself for using git for deployment. This article recommended using beta directory for testing and then deploying to your primary directory. I thought it is not required for small and single developer websites so I thought to create a simple structure.

Here we will have only two repositories

  • Local repository where I will be making changes
  • Remote repository on Digital Ocean droplet from where my live website will run

Step#1 Install git on local and remote machine.

Installing git

$sudo apt-get install git

Step#2 Initiate Git repo

On Local machine cd to folder having your code and then initiate git repo as below

$git init

On remote server, cd to directory which will host your code and then initiate git repo

$git init --bare

Please note the difference : A bare repository is a git repository without a working copy, therefore the content of .git is top-level for that directory.

Use a non-bare repository to work locally and a bare repository as a central server/hub For example, when you create a repository on github.com, it is created as a bare repository.

To conclude, the repository on the server side is going to get commits via pull and push, and not by you editing files and then committing them in the server machine, therefore it is a bare repository.

You can push your changes from local to remote by using following command

Step#3 Add remote repository

You can add remote repository using below command.

$git remote add live ssh://user@domain.com/var/www/foldername

Here instead of domain.com you can use your droplets ip as well.

Once this is added you can check your remotes using following command

$ git remote -v
live ssh://user@mydomain.com/var/www/foldername (fetch)
live ssh://user@mydomain.com/var/www/foldername (push)

Step#4 Deploying the changes.

Once your changes are ready, you can deploy it using following commands

git add .  
git commit -m "1st commit"  
git push live master  

Step#5 Verify the changes

You can log in to your droplet and check if your code is updated. You can do this by simply verifying the files.

Troubleshooting

I faced an issue that my local repo was pushing files successfully but remote git repo was not updated.  To check if git changes were actually pushed or not run following command

$git log

This will show list of changes pushed to remote. If you see your changes are pushed but still not reflected, you need to run following manually

GIT_WORK_TREE=/var/www/foldername git checkout -f

Obviously, you should not run above command manually every time you make the change as this will loose the whole purpose of deployment automation.

cd to hooks folder, create post-receive file and add above line into file

cd hooks/
sudo nano post-receive

Once file is created, change the file permissions as below


sudo chmod a+x post-receive

And now you are all set. Going forward, any change that you push will be reflected on your remote repository.

Important Note: Depending on your changes and the type of application that you are hosting, you might have to restart the server to have these changes reflected on your website.

Django : ProgrammingError: column “id” does not exist

Whichever database you use as your backend, django needs every table to have a primary key.

If you have specified primary key in your model definition, you will never get this error.

If you have not defined primary key then django automatically creates a column named id (auto increment) and treat this as primary key.

If django takes care of both the situation, why are you getting this error ?

Well, most likely you have not created table using django migrate command. Your table already existed and you want to use this in django,

How to fix this issue ?

Simple, add a primary key to table and update your model to reflect the same.

How to handle media files in django application

Unlike static files, media files are the files (images , pdf or any other documents etc) uploaded by user.

For managing media files, you need to setup MEDIA_ROOT and MEDIA_URL .

MEDIA_ROOT

Media root is the directory where media files are stored

MEDIA_URL

URL that handles the media served from MEDIA_ROOT, used for managing stored files. It must end in a slash if set to a non-empty value.

Step#1

Add following lines at the end of settings.py

MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

Step#2

Create a folder called as media at the base directory. Directory structure looks like as below
├── static
│   ├── css
│   ├── img
│   └── js
├── templates
│   ├── components
│   └── blog
├── media

Step#3

we need to configure main urls.py file as below

from django.contrib import admin
from django.urls import path, include

from . import settings                                              #add this for media
from django.contrib.staticfiles.urls import static                  #add this for media
from django.contrib.staticfiles.urls import staticfiles_urlpatterns #add this for media

urlpatterns = [
    path('admin/', admin.site.urls),
    path('blog/', include('blog.urls')),# Add this line for new app
]

urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)  #add this for media

Step#4

You also need to define a field which will accept media file.

class Post(models.Model):
.
.
.

    featured_image = models.ImageField(upload_to='img', blank=True, null=True)
.
.

 

Please note the “upload_to” option. Here, “img” folder will get created (you dont have to create it manually) and files will be stored and served from here.

Once these changes are done , don’t forget to run following commands

$ python3.6 manage.py makemigrations blog
$ python3.6 manage.py migrate

Now start the server using following command and start using media files.

python3.6 manage.py runserver

If you run into any issue, please let me know

Building Blog application using Django

In this post we will build a blog application using Django, before going thru this application, you should have basic knowledge of Django. If you want to review basics, please go thru following posts.

Let us create project

$django-admin startproject blogapp
$cd blogapp
/blogapp$ python3.6 manage.py runserver

Now our basic project  is running. Let us create an app now.

$python3.6 manage.py startapp blog

urls.py is not created inside app directory, we need to create blog/urls.py manually.

from django.urls import path
from . import views

app_name = 'blog'
urlpatterns = [
    path('', views.index, name='index'),  
]

To have application urls accessible from main project, we need to add this urls.py with main project

from django.contrib import admin
from django.urls import include, path # Add include here

urlpatterns = [
    path('admin/', admin.site.urls),
    path('blog/', include('todo.urls')),# Add this line
]

before doing further changes let us create sample view in blog/views.py we will edit this file later for details but we need to create this view to avoid any error while running intermediate command.

from django.shortcuts import render, get_object_or_404, redirect

def index(request):
    context = "temp"
    return render(request, 'blog/index.html',{'context':context})  

Changes  in settings.py file

Add application in settings file

INSTALLED_APPS = [
    'blog.apps.BlogConfig', # add this statement
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Now let us define templates directory as below

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            BASE_DIR + '/templates/', # add this line
        ],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

and finally define static directory at the end of settings.py file.

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
]
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

We need to manually create static and templates directories. Directory structure created looks as below:

├── static
│   ├── css
│   ├── img
│   └── js
├── templates
│   ├── components
│   └── blog
├── media
│   ├── img

Creating model

Let us first create todoapp database and database user

$sudo su - postgres
[sudo] password for conquistadorjd: 
postgres@inspiron-3542:~$ psql
psql (10.10 (Ubuntu 10.10-0ubuntu0.18.04.1))
Type "help" for help.
postgres=# CREATE DATABASE blogapp;
CREATE DATABASE
postgres=# CREATE USER blogappuser WITH PASSWORD 'password';  
CREATE ROLE
postgres=# ALTER ROLE blogappuser SET client_encoding TO 'utf8';
ALTER ROLE
postgres=# ALTER ROLE blogappuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE
postgres=# ALTER ROLE blogappuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE
postgres=# ALTER ROLE blogappuser SET timezone TO 'UTC';
ALTER ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE blogapp TO blogappuser;
GRANT
postgres=# \q
postgres@inspiron-3542:~$ exit
logout

Now we need to configure this database in our application

DATABASES = {
    'default': {
        # 'ENGINE': 'django.db.backends.sqlite3',
        # 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'blogapp',
        'USER': 'blogappuser',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '',              
    }
}

Now let us create a model

from django.db import models
from django.contrib.auth.models import User

STATUS = (
    (0,"Draft"),
    (1,"Publish")
)

class Category(models.Model):
    # created_at = models.DateTimeField(auto_now_add=True, verbose_name="Created at")
    # updated_at = models.DateTimeField(auto_now=True, verbose_name="Updated at")
    title = models.CharField(max_length=255, verbose_name="Title")
    # slug = models.CharField(max_length=20, unique=True)

    class Meta:
        verbose_name = "Category"
        verbose_name_plural = "Categories"
        ordering = ['title']

    def __str__(self):
        return self.title

class Tag(models.Model):
    name = models.CharField(max_length=20, unique=True)
    # slug = models.CharField(max_length=40, unique=True)

    def __str__(self):
        return self.name

class Post(models.Model):
    title = models.CharField(max_length=200, unique=True)
    slug = models.SlugField(max_length=200, unique=True)
    author = models.ForeignKey(User, on_delete= models.CASCADE,related_name='blog_posts')
    content = models.TextField()
    created_on = models.DateTimeField(auto_now_add=True)
    status = models.IntegerField(choices=STATUS, default=0)
    category = models.ForeignKey(Category, on_delete = models.CASCADE,verbose_name="Category",default=None,blank=True)
    updated_on = models.DateTimeField(auto_now= True)
    tags = models.ManyToManyField(Tag, related_name='rel_posts',default=None,blank=True)
    featured_image = models.ImageField(upload_to='img', blank=True, null=True)

    class Meta:
        ordering = ['-created_on']

    def __str__(self):
        return self.title

class Comment(models.Model):
    blog_post = models.ForeignKey(Post,on_delete = models.CASCADE,verbose_name = "blog_post",related_name="comments")
    comment_author = models.CharField(max_length = 50)
    comment_author_email = models.EmailField()
    comment_content = models.CharField(max_length = 200)
    comment_date = models.DateTimeField(auto_now_add=True)
    def __str__(self):
        return self.comment_content
    class Meta:
        ordering = ['-comment_date']

Once this is done, we need to run following two commands

$ python3.6 manage.py makemigrations blog
$ python3.6 manage.py migrate

Let us have this model accessible from admin and create some dummy data for development and unit testing.

from django.contrib import admin

from .models import Post,Comment,Category,Tag

admin.site.register(Post)
admin.site.register(Comment)
admin.site.register(Category)
admin.site.register(Tag)

Now create a admin user using following command

python3.6 manage.py createsuperuser

Let us run the server and login to admin from http://127.0.0.1:8000/admin and create some dummy data.

We had created url.py file under blogs directory in previous steps. Now update code as below:

from django.urls import path
from . import views

app_name = 'blog'
urlpatterns = [
    path('', views.index, name='index'),  
    path('<str:category>/<str:slug>/', views.details, name='details'),
]

We need to update main application urls.py file to enable media upload.

from django.contrib import admin
from django.urls import path, include

from . import settings
from django.contrib.staticfiles.urls import static                  #add this for media
from django.contrib.staticfiles.urls import staticfiles_urlpatterns #add this for media

urlpatterns = [
    path('admin/', admin.site.urls),
    path('blog/', include('blog.urls')),# Add this line for new app
]

urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)  #add this for media

blog/views.py

from django.shortcuts import render, get_object_or_404, redirect
from .models import Post

def index(request):
    context = Post.objects.all()
    return render(request, 'blog/index.html',{'context':context})      

def details(request, category, slug):
    blog_post = Post.objects.get(slug=slug)  
    comments = blog_post.comments.all()
    print('comments', comments)
    return render(request, 'blog/detail.html',{'context':blog_post,'comments':comments}) 

index.html

{% extends "components/base.html" %}

{% block content %}
<div class="container-fluid"/>
    <div class="row"/>
        <div class="col-1"/></div/>
        <div class="col-8"/> 
        {% if context %}
            <div class="list-group"/>
                {% for post  in context %}
                <a href="{{post.category}}/{{post.slug}}" class="list-group-item list-group-item-action flex-column align-items-start"/>
                    <div class="d-flex w-100 justify-content-between"/>
                      <h1 class="mb-1"/>{{post.created_on|date:"jS F Y" }} by {{post.title}} </h1/>
                      <small/>{{post.created_on|date:"jS F Y" }}</small/>
                    </div/>
                    <p class="mb-1"/>{{post.content|linebreaks|slice:":200"}}</p/>
                    <small/>more ...</small/>
                </a/><br/>                <br/><br/>
                {% endfor %}
            </div/>
        {% else %}
            <p/> Nothing here :) . Finally you found it</p/>
        {% endif %}
        </div/>
        <div class="col-3"/></div/>
    </div/>
</div/>
{% endblock %}

details.html

{% extends "components/base.html" %}
{% load static %}

{% block content %}
{% if context %}
<div class="container-fluid"/>
    <div class="row"/>
        <div class="col-1"/></div/>
        <div class="col-8"/>
            <h2/>{{context.title|linebreaks}}</h2/>
            <small/> {{context.created_on|date:"jS F Y"}} by {{context.author}}</small/><br/>
            
            <img src="{{context.featured_image.url}}" alt="Smiley face" class="rounded mx-auto d-block"  height="250" width="250"/> 
            <div id="postdetails"/>{{context.content|linebreaks}}</div/>
            <br/>
                {% if comments %}
                    <h2/>Comments</h2/>
                    <div class="list-group"/>
                        {% for comment in comments  %}
                            <!-- <a href="#" class="list-group-item list-group-item-action "/> --/>
                            <div class="d-flex w-100 justify-content-between"/>
                                <h6 class="mb-1"/>by {{comment.comment_author}}</h6/>
                                <small/>{{comment.comment_date|date:"jS F Y"}}</small/>
                            </div/>
                            <p class="mb-1"/>{{comment.comment_content}}</p/>

                        {% endfor %}
                    </div/>
                {% endif %}         
        </div/>
        <div class="col-3"/></div/>
    </div/>   
{% else %}
    <p/>No tasks are available.</p/>
{% endif %}
</div/> 
{% endblock %}

Here template files are very basic ones, you can update it as per your needs. Once you have core functionality ready, its up to you, how you want to display it.

If you run into any issue while working on this code, please do let me know.

Using raw Query to Fetch data from PostgreSQL in Django

Instead of using ORM, if you want to hard code a query in django, you can refer to below code

def getprices(request):
    print("hello backend")
    conn = psycopg2.connect(database="ecommpedia1", user="admin", password="admin", host="127.0.0.1", port="5432")
    cur = conn.cursor()   
    productid = "CAME747PNEUYMSHT"
    cur.execute("SELECT * FROM ecomm_time WHERE  flipkartproductid = '"+productid+"';") 
    mobile_records = cur.fetchall()
    print("mobile_records : ", mobile_records)
    print(type(mobile_records))
    return HttpResponse(mobile_records)

How to update or insert field in Elasticsearch using Python

Update or insert is done by API. Most useful fact is that its upsert meaning, we use the same API. If field exists, it will get updated and if it does not exist, it will be inserted.
My first record is as below

import requests
import json

uri='http://localhost:9200/book/_doc/1'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing document before update : ", response.text)

Terminal output

Accessing document before update :  {
      "isbn": "9781593275846",
      "title": "Eloquent JavaScript, Second Edition",
      "subtitle": "A Modern Introduction to Programming",
      "author": "Marijn Haverbeke",
      "published": "2014-12-14T00:00:00.000Z",
      "publisher": "No Starch Press",
      "pages": 472,
      "description": "JavaScript lies at the heart of almost every modern web application, from social apps to the newest browser-based games. Though simple for beginners to pick up and play with, JavaScript is a flexible, complex language that you can use to build full-scale applications.",
      "website": "http://eloquentjavascript.net/"
    }

Now let us look at the code to update and insert the field in document.  Please note that there are multiple ways of doing update / insert.

import requests
import json
####################################################################### Insert one field in document
uri='http://localhost:9200/book/_update/1'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps(
    {
    "script" : "ctx._source.price = 365.65"
    }
)
response = requests.post(uri,headers=headers1,data=query)
print("*************************************")
print(" Output", response.text)

####################################################################### Insert two fields in document
uri='http://localhost:9200/book/_update/1'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps(
{
  "doc": {
    "summary":"summary3",
    "coauth":"co author name"
  }
}
)
response = requests.post(uri,headers=headers1,data=query)
print("*************************************")
print(" Output", response.text)

####################################################################### Insert nested field in document
uri='http://localhost:9200/book/_update/1'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps(
    {
    "doc": {
        "country":{
        "continent":"Asia",
        "code" : 91
        },
        "coauthor_two":"Another co-author"
    }
    }
)
response = requests.post(uri,headers=headers1,data=query)
print("*************************************")
print(" Output", response.text)

####################################################################### Update field
uri='http://localhost:9200/book/_update/1'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps(
{
  "doc": {
    "title":"Eloquent JavaScript, Second Edition - Updated using API from Python Script"
  }
}
)
response = requests.post(uri,headers=headers1,data=query)
print("*************************************")
print(" Output", response.text)

Once we have run above script, let us look at the document again by running below program or checking kibana.

import requests
import json
uri='http://localhost:9200/book/_source/1'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing document After update : ", response.text)

Output

Accessing document After update : {
    "isbn" : "9781593275846",
    "title" : "Eloquent JavaScript, Second Edition - Updated using API from Python Script",
    "subtitle" : "A Modern Introduction to Programming",
    "author" : "Marijn Haverbeke",
    "published" : "2014-12-14T00:00:00.000Z",
    "publisher" : "No Starch Press",
    "pages" : 472,
    "description" : "JavaScript lies at the heart of almost every modern web application, from social apps to the newest browser-based games. Though simple for beginners to pick up and play with, JavaScript is a flexible, complex language that you can use to build full-scale applications.",
    "website" : "http://eloquentjavascript.net/",
    "price" : 365.65,
    "summary" : "summary3",
    "coauth" : "co author name",
    "coauthor_two" : "Another co-author",
    "country" : {
      "continent" : "Asia",
      "code" : 91
    }

Kibana output

Creating GRID structure using bootstrap and Django

In this article we will write a code to display list of items as below

Here is the code for this.

{% extends "components/base.html" %}

{% block content %}
<div class="container-fluid">
{% if context %}
    <div class="row">
      {% for product  in context %}
          <div class="col rounded border border-light mt-3 ml-3 shadow">
              <img src={{product.productBaseInfoV1.imageUrls.400x400}} class="mt-3 mb-3" style="max-width: 200px;max-height: 200px;width: auto;height: auto;margin: auto;">
                <p class="card-text">{{product.productBaseInfoV1.title}}</p>
                <a href="{% url 'ecomm:details' product.productBaseInfoV1.productId %}" >Go to Details page</a>
          </div>
          {% if forloop.counter|divisibleby:4 %}
            </div>
            <div class="row">
          {% endif %}
      {% endfor %}
  </div
{% else %}
    <p>No tasks are available.</p>
{% endif %} 
</div>
{% endblock %}

This code is written for django but bootstrap and html part can be used for any other framework without any challenge.

You can remove following part without any impact.

          
{% if forloop.counter|divisibleby:4 %}
            </div>
            <div class="row">
          {% endif %}
{% endfor %}

My requirement was to have 4 item in a row and hence I had kept it but its not necessary and code will work flawlessly.

Elasticsearch API Search document queries

Elasticsearch has very detailed search API but its bit different for someone with RDBMS and SQL query background. Here are some of the sample queries.

search all

GET /ecomm/_search
{
    "query": {
        "match_all": {}
    }
}

Search for specific key

GET /twitter/_search
{
    "query": {
        "match": { "user":"ssss"}
    }
}

Search using URL

GET /ecomm/_search?q=Apple

Detailed search for nested value

GET /ecomm/_search
{
    "query": {
        "match" : {
            "productBaseInfoV1.productId": "MOBFKCTSYAPWYFJ5"
        }
    }
}

Search with more parameters

POST /ecomm/_search
{
    "query": {
        "match" : {
            "productBaseInfoV1.productId": "MOBFKCTSYAPWYFJ5"
        }
    },
    "size": 1,
    "from": 0,
    "_source": [ "productBaseInfoV1.productId", "productBaseInfoV1.title", "imageUrls","productDescription" ]
}

Accessing Elasticsearch API from Python Script

Elasticsearch provides easy to use API and it can be access from kibana, postman, browser and curl.  You can read here how to access elasticsearch API from these options.

In this post we will look at very simple example of accessing elasticsearch API from python. Here is simple example along with results

import requests
import json

uri='http://localhost:9200/_cat/indices'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("this is : ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing document before creating : ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
query = json.dumps({
    "query": {
            "user" : "NewUser",
            "post_date" : "2009-11-15T14:12:12",
            "message" : "trying out Elasticsearch from python"
            }
        })
response = requests.put(uri,headers=headers1,data=query)
print("*************************************")
print("Document is created ", response.text)

uri='http://localhost:9200/twitter/_doc/7'
headers1 ={'Content-Type': 'application/json'}
response = requests.get(uri,headers=headers1)
print("*************************************")
print("Accessing newly created document: ", response.text)

Here is output


$ python3 second.py 
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.7) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
this is :  yellow open ecommpediatest               R6WypBc2RtCS_ITA40_rFw 1 1    0 0    283b    283b
yellow open test-index                   OeILxjmFRhODhztu4GgE_w 1 1    1 0   4.5kb   4.5kb
yellow open twitter                      BUmjGbLpTNCbnDIPxht9vA 1 1   13 4  24.8kb  24.8kb
yellow open bookindex                    RKq8oJKvRb2HQHxuPZdEbw 1 1    1 0   5.4kb   5.4kb
green  open .kibana_task_manager_1       pc_LXegKQWu9690vT1Z-pA 1 0    2 1  16.9kb  16.9kb
green  open kibana_sample_data_ecommerce FkD1obNSSK6mfLDsy9ILPQ 1 0 4675 0   4.9mb   4.9mb
yellow open facebook                     4Wzax6UhThm5rXi03PiG7w 1 1    2 0     7kb     7kb
green  open .apm-agent-configuration     sxoXMmsoS4mRPyjcaPByzw 1 0    0 0    283b    283b
green  open .kibana_1                    pK7pO-SCSBORonZLDi8Vew 1 0   60 2 945.9kb 945.9kb
yellow open twitter1                     3iMzcYoJR3qY9JiM2sY8_g 1 1    1 0   4.5kb   4.5kb

*************************************
Accessing document before creating :  {"_index":"twitter","_type":"_doc","_id":"7","found":false}
*************************************
Document is created  {"_index":"twitter","_type":"_doc","_id":"7","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":28,"_primary_term":2}
*************************************
Accessing newly created document:  {"_index":"twitter","_type":"_doc","_id":"7","_version":1,"_seq_no":28,"_primary_term":2,"found":true,"_source":{"query": {"user": "NewUser", "post_date": "2009-11-15T14:12:12", "message": "trying out Elasticsearch from python"}}}

We will have a look at complex queries from python at later stages.

How to use Elasticsearch API

Elasticsearch exposes REST APIs that are used by the UI components and can be called directly to configure and access Elasticsearch features.

Show current indices

http://localhost:9200/_cat/indices

Output

yellow open test-index                   OeILxjmFRhODhztu4GgE_w 1 1    1 0   4.5kb   4.5kb
yellow open twitter                      BUmjGbLpTNCbnDIPxht9vA 1 1    1 0   3.7kb   3.7kb
yellow open bookindex                    RKq8oJKvRb2HQHxuPZdEbw 1 1    1 0   5.4kb   5.4kb
green  open kibana_sample_data_ecommerce FkD1obNSSK6mfLDsy9ILPQ 1 0 4675 0   4.9mb   4.9mb
green  open .kibana_task_manager_1       pc_LXegKQWu9690vT1Z-pA 1 0    2 1  16.9kb  16.9kb
yellow open facebook                     4Wzax6UhThm5rXi03PiG7w 1 1    1 0   3.5kb   3.5kb
green  open .apm-agent-configuration     sxoXMmsoS4mRPyjcaPByzw 1 0    0 0    283b    283b
green  open .kibana_1                    pK7pO-SCSBORonZLDi8Vew 1 0   60 2 949.5kb 949.5kb

This and below mentioned commands can be run from kibana, postman, browser or cURL

Curl

$ curl "localhost:9200/_cat/indices"
yellow open ecommpediatest               R6WypBc2RtCS_ITA40_rFw 1 1    0 0    283b    283b
yellow open test-index                   OeILxjmFRhODhztu4GgE_w 1 1    1 0   4.5kb   4.5kb
yellow open twitter                      BUmjGbLpTNCbnDIPxht9vA 1 1   11 5  26.9kb  26.9kb
yellow open bookindex                    RKq8oJKvRb2HQHxuPZdEbw 1 1    1 0   5.4kb   5.4kb
green  open .kibana_task_manager_1       pc_LXegKQWu9690vT1Z-pA 1 0    2 1  16.9kb  16.9kb
green  open kibana_sample_data_ecommerce FkD1obNSSK6mfLDsy9ILPQ 1 0 4675 0   4.9mb   4.9mb
green  open .apm-agent-configuration     sxoXMmsoS4mRPyjcaPByzw 1 0    0 0    283b    283b
yellow open facebook                     4Wzax6UhThm5rXi03PiG7w 1 1    2 0     7kb     7kb
green  open .kibana_1                    pK7pO-SCSBORonZLDi8Vew 1 0   60 2 945.9kb 945.9kb
yellow open twitter1                     3iMzcYoJR3qY9JiM2sY8_g 1 1    1 0   4.5kb   4.5kb

Kibana

 

Postman

 

Browser

How to create new index

PUT /ecommpediatest

same can e achieved by create new document command

PUT /twitter1/_doc/1
{
"user" : "kimchy updated",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}

Creating or updating record

------------------- Get records
GET twitter/_doc/1
HEAD twitter/_doc/1
GET twitter/_source/10
HEAD twitter/_source/1
-------------------update API
PUT /twitter/_doc/1
{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}

POST /twitter/_doc/2
{
"user" : "tom",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
PUT /twitter/_create/2
{
"user" : "Jerry",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
POST /twitter/_create/
{
"user" : "cartoon",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
-------------------delete
DELETE /twitter/_doc/1
-------------------- create
PUT /twitter/_doc/1
{
"user" : "one",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}

POST /twitter/_doc/
{
"user" : "one",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}

PUT /twitter/_create/aa
{
"user" : "xxx",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
POST /twitter/_create/ss
{
"user" : "ssss",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
------------------------------------------------------search
GET /_search
{
"query": {
"ids" : {
"values" : ["1", "ss", "100"]
}
}
}
GET /twitter/_search
{
"query": {
"ids" : {
"values" : [11,"ss"]
}
}
}
GET /twitter/_search
{
"query": {
"match_all": {}
}
}

GET /twitter/_search
{
"query": {
"match": { "user":"ssss"}
}
}