Django optimizing Memcache and Nginx

I have a django application which complete and ready to deploy. One of the requirement are :

• It will run 24x7 with minimum interupption
• Down Time will be only 3 hours every monday
• User will access concurrently around 17 users

So with this requirement, i want to have my django app high performance without glitch and fast. What already implemented in code are using some method which are :

• Using Postgresql Database in a separate machine.
• Using memcached to do caching
• Using django-compressor to compress static files
• Using Nginx web server to server static files
• using Gunicorn to server the Django application

So that all basic optimization for Django application or Web based application. The application is a simple single table query and an insert to database.

So the application will have insert to table when external user hit the web apps. Then the client will have pool the server for the input every 2 seconds. so the access will be all the time.

So I put some testing of the performance. First using single machine hosting a memcache and postgresql. The setup are :

• Cache using memcache localy
• Session using Memcache
• Nginx server static files

 and here are the results :

Test 1 with single machine setup

Then i try other setup, which difference are put session in postgresql and here the results :

Test 2 Django session using Postgresql

And then last one, i setup a load balanced setup with 2 django app server, here are the setup :

• Memcached setup as shared memcache between 2 Django app
• Nginx as a single entry load balance the application request to 2 Django app server
• Postgresql is still separated machine.
• Django Compressor is still used.

And here are the result of the test :

Django app load balanced with Nginx in front of application

As you see, the result is better if we horizontal scale the python serving django applications. The load balance in Nginx is a standard setup with 2 upstream point to django app server.

Even the Nginx server still hosting 1 django app, and other machine hosting only django app,with memcache in each server as shared memcache pool, and the postgresql is a single machine.

The improvement is huge. From 40 Req/sec to 117 Req/sec , 2x improvements. This still not optimize the database cache, as i using Django 1.6 and johny cache still not available at this time to be use with Django 1.6

On problem to be note, the django-compressor got problem when setup as multiple python app server, because the key to generate key cache prefix is using the host name, so in one another of the server, there will be missing static compressed file. So we just generate the compressed static in 1 machine, and copied to the other machine and this solved.

Read More

PDF generation in Django Apps

One of the beautiful thing when using Django is the rich set of application ready to use as a plugin in your project. Is it use at is or modify/extend with a little effort.

One of my project need to generate a PDF and the time is come to get some Django PDF application plugin. So Django documentation officially told to use reportlab. I try the reportlab, but got error with first try from the code in Django documentation stated here.

Then look further, i found pisa is very neat to generate a PDF for you with a common Django way. How does pisa do the job? Well it still need reportlab, and html5lib , and it just convert your HTML output into pdf output using your django view.

So we already have the html output in Django app, then just feed it to pisa, and the pdf will generated.
You can download pisa3.0.33 in here .

But also notice that pisa3.0.33 is got problem which cause it cannot run with reportlab>3.0 .

To fix it go to the installation of pisa , and find sx/pisa3/pisa_util.py :

if not (reportlab.Version[0] == "2" and reportlab.Version[2] >= "1"):

   raise ImportError("Reportlab Version 2.1+ is needed!")

REPORTLAB22 = (reportlab.Version[0] == "2" and reportlab.Version[2] >= "2")

This will make pisa fail when the import processed by python. To resolve this replace it with this code :

if not (reportlab.Version[:3]>="2.1"):

    raise ImportError("Reportlab Version 2.1+ is needed!")

REPORTLAB22 = (reportlab.Version[:3]>="2.1")

Then that's it, your pisa will work with Django.

The next post , i will give the code to generate the pdf with pisa in django.

Cheers

Read More

Display disk usage in graphical way

Recently, i got my disk usage 100% in my home folder. With using only du in linux, we got only text output about the disk usage. I need to see which folder occupy most of the disk space in a quick way, in a glance and in an interesting way, using graphical representation.

So i search for the application to do this in linux, and i found Baobab. This application name not imply its usage, but very neat. We can see the disk usage in a graphical way, using a ring chart , and also have the list of subdirectory we scan so we can got the directory which consume much of the disk space.

Baobab application screen shoot

With this, it is faster and quicker way to find the folder which occupy most of the space, even with remote folder. This feature i found most interesting.

For the website source and developer can go to baobab website

Nice to share this. Hope you got benefit from this share.

Read More