Hands-on with Redis and Django

Caching data, scheduling tasks with the queue, and publishing messages using Redis in Django.

Hello python programmer. In this blog, we will see the multiple sides of Redis in Django. My goal is to explain how Redis is used in different scenarios with an example of building a to-do list rest API and tracking all hit requests.

1. Caching data with Redis.
2. Scheduling task with Redis queue.
3. Publish/subscribe service using Redis.

What is Redis?

Redis is an in memory data structure store used as a database, cache and message broker.

It is a simple key value database store with faster execution time, along with a ttl- (Time to live) functionality.

Redis supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries.

It stores everything in primary memory.

Without any further ado, let’s quickly check how it's done.

Problem statement?

According to the diagram, the client will hit a request to the middleware. The middleware will first add the request in the queue i.e increase the count by 1 of the “request method” key in Redis cache memory. Then middleware will hit the server with the request and complete the tasks. The server sends a response to the middleware. It sends messages to all subscribers(pubsub) about the request and a response to the client.

Now, here we have one parallel process of queues by middleware. Redis-rq, one by one, dequeue the request and executes it. The job of Redis-rq is to count the call and the number of times that call was hit. Then, it will take the call method and increase it by 1 in Redis cache storage.

let’s, start the installation and complete it with code.

1. Make virtualenv

First, we make a virtual environment for the Django project.

INSTALL  - pip install virtualenv
CREATE   - virtualenv virtualenv_name
START    - source virtualenv_name/bin/activate

2. Install Django and Create a Project

Now, jump to the folder of virtualenv, install Django and create the project name as “todoproject”.

For this project we required PostgreSQL. For installation follow this link.

(virtualenv_name)home~/$ cd virtualenv_name
(virtualenv_name)home~/virtualenv_name$ pip install django 
(virtualenv_name)home~/virtualenv_name$ pip install psycopg2-binary
(virtualenv_name)home~/virtualenv_name$ pip install djangorestframework
(virtualenv_name)home~/virtualenv_name$ django-admin startproject todoproject
(virtualenv_name)home~/virtualenv_name$ cd todoproject

3. Create App

Create an app in our “todoproject”. Let’s name it “todo”. Now you can see the directory structure as shown below.

(virtualenv_nam)home~/virtualenv_name/todoproject$python3 manage.py startapp todo

todoproject directory

todo app directory

4. Add Todo app config in settings.py

Now add todo app and rest_framework in settings.py of “todoproject”.

We add to app one rest_framework and todo app config

5. Change database config in settings.py

Add your PostgreSQL database config in settings.py.

6. Create a model for todo

Create a models.py in the todo directory. Here we make a model for our schema.

7. Migrate the Database

Now migrate your database using the following command.

(virtualenv_nam)home~/virtualenv_name/todoproject$ 
   ~ python3 manage.py makemigrations
(virtualenv_nam)home~/virtualenv_name/todoproject$
   ~ python3 manage.py migrate

8. Create Serializers

What is a serializer?

The representation for our model in JSON format and convert an object to a more transferable format is know as serializer.

The primary purpose of a serializer is to convert a Django model or rather a database table into a format that can be transferred over the internet such as a JSON output or an XML output.

Create a serializers.py in the todo directory. Here we create a todo serializer for our schema.

9. Redis Cache Setup

Now, let's start with the Redis cache. First, we install all dependencies and add configuration in settings.py for Redis cache.

(virtualenv_nam)home~/virtualenv_name/todoproject$
    pip3 install django-redis

Add configuration in settings.py for Redis cache

In this, we are taking the default client configurations and have set default cache ttl ( Time to live) = “60 * 1500”.

Now, create a new file, cache_function.py, for defining cache operation for cache data. For more information, you can refer to this link.

Here we define all functions used to evaluate cache data.

Create a new file in the todo app “cache_function.py”.

10. Redis Queue Setup ( A Background Job handler )

Now that we have set up the Redis cache, we can start with Django-rq (Redis-queue).

First, install Django-rq and add the configuration for Django-rq in settings.py.

pip3 install django-rq

11. Pubsub Setup ( publisher/subscriber )

What is Pubsub?

It is a form of asynchronous service to service communication used in serverless and microservices architectures.

In this publisher publish message and connected subscriber will receive message.

Create file pubsub.py, create a connection for the Redis server and a function to publish messages.

While publishing data on channels, we just have to provide channel name and data as the argument of function “publish_data_on_redis”.

12. Add Middleware

What is middleware?

Middleware is something that acts as a bridge between two parts of a program that enables communication between them.

It is a lightweight plugin that processes during request and response execution.

Middleware is used to perform a function in the application. The functions can be a security, session, csrf protection, authentication, etc.

Here we completed all installations of Redis. Now let’s connect all of them in Django middleware one by one.

Create middleware.py in the todo app.

Here we initialize the cache key-value pair in middleware. On every call, we enqueue tasks to update call details and schedule the call updater to publish a message.

In this we initialize cache key value for REST API -: GET, POST, PUT, DELETE. Now when anyone hits API, the Redis-rq schedules an update task for a call. The publisher publishes a message to all subscribers who subscribe to “notify” that which API call we receive.

Add custom middleware config “todo.middleware.middleware” in “settings.py”.

Here we add all config for middleware. you can see in line number 9 where we added our middleware

13. Add Rest API

Create tasks.py where we define whole logic for rest operation and interaction with the database.

In this, we are defining our REST API. Here we have a CRUD operation. which talks with the database.

Now, edit view.py to define a route for CRUD operation and get Redis cache storage data using API.

Create url.py in the todo app to define the todo route.

./todo/urls.py

Edit the urls.py file in “todoproject” and add the route for the todo app as shown below.

path("", include("todo.url")),
path("django-rq/", include("django_rq.urls"))

Now it's time to start the app.

Start Redis server

(virtualenv_nam)home~/virtualenv_name/todoproject$ redis-server

Start Redis-rq

(virtualenv_nam)home~/virtualenv_name/todoproject$
      python3 manage.py rqworker default

Subscribe to notify the publisher to see messages

(virtualenv_nam)home~/virtualenv_name/todoproject$ redis-cli
(virtualenv_nam)home~/virtualenv_name/todoproject$ psubscribe notify

Start the Django server

(virtualenv_nam)home~/virtualenv_name/todoproject$ 
      python3 manage.py runserver

Finally, our project is up and running. Now we open postman and start hitting API.

Hit GET request in postman to get todo list- http://127.0.0.1:8000/todo.

In this, we hit get request to get all todo list.

You can see in this image it schedules a job to update the call count.

In this subscribers get messages from publishers.

Conclusion

With this blog, we have successfully seen caching data, queuing jobs, and how to subscribe/publish (pubsub) a service using Redis in Django.

I hope this article has helped you. Thanks for reading!