Django Test Improvement Speed

Testing in software development is a tedious task, and some programmers skip it. But many start to do the right thing from the beginning, Test Driven Development (TDD) movement make software developer more aware on delivering properly tested battle proven software. I also using TDD with my Django development.

One of factor when your test getting more and more in quantity, the test running slow as the test increased as your software getting more complex. I notice that my test is 135 cases, and need to wait 25 seconds to finish all the test. The test way run using py.test and using Django test also. Of course the test using SQLite on memory database to speed up the test. Also Pytest give the test speed an increase.

Then i read on documentation that for testing, Django still using default setup on other thing in the framework which can be removed in testing environment, like no middleware, no add on or things that slowing down. 1 i try are change the password hashed in django. The password hasher used by Django authentication was PBKDF2 algorith with a SHA256 hash, a password strecthing mechanism recomended by NIST. This should be sufficient for most users. It's quite secure, requiring massive amounts of computing time to break. This was from Django documentation. Sure it is save, and sure need more time and processing power to encrypt it. So can we change it in testing environment, to a faster password hasher. Especially when in testing, we have a lot of login and logout session, and it will make the testing longer.

You see in the documentation also state that django have other hasher, one is md5 which is common used for hasing a file and fast enough. Try to use that in the test environment and change the password hasher settings and see the improvement gains. Of course you need to have a test.py settings file used for your testing environment database and other config. Add this entries for hashing using md5.

#Use faster password hasher for testing
PASSWORD_HASHERS = (
    'django.contrib.auth.hashers.MD5PasswordHasher',
)

I try run my test, the improvement was really fast. Here are the screenshoot of my test without the md5 password hasher which using default hasher.

Raw Django Test

Then below the result when using md5 password hasher.

Optimized Django Test

So the improvement was 4 times the speed on testing. Total of 135 test case. I am happy to write my test again as much as possible to increase the quality of my code. Make testing a habit. No more worries when change something on the code will break other part of the code. Test suite will come to the rescue.

Happy Testing and Happy Coding. Testing and Coding should be fun with python.

Read More

Upgrading Django-1-6 to 1-7 part 2

This is part 2 of upgrading Django-1.6 to Django-1.7 journey. For part 1 you can go Here.

First error was my application using a module that was deprecated since Django-1.6 which already give me warning to fix it, but i ignore it. And i remember that, so i remove it and update with other lib.
The lib i use was django.utils.simplejson.

from django.utils import simplejson 

I change it to using json lib.

import json

Next is the testing give a warning that i am using settings  from django-1.5 which should be updated. I found out it was the test runner should be stated in the settings.py. So addition of the settings was added :

TEST_RUNNER = 'django.test.runner.DiscoverRunner'

Then test suite not give any warning message again, and the test all success.

Next is the django-debug-toolbar, i was using old version which are 1.0.1, and need to upgrade to django-debug-toolbar-1.4 because the import is giving error when using old version of django-debug-toolbar.

Next is the Django-compressor , i was using django-compressor-1.4 , and need to upgrade todjango-compressor-1.6.

And also change in HttpResponse , if you use mimetype , you should change it to content_type.

    return HttpResponse(json.dumps(dataxx), mimetype='application/json')

Update to this :

    return HttpResponse(json.dumps(dataxx), content_type='application/json')

And last one is the migration which already using south. If the app is newly developed and no db migration, no migration at all will give no problem. What if you have already migration in south running and deployed on the production, now you start develop using new django version, the migration will be need to use django build in migration. Well as in Django-1.7 migration already build in and use south,as the author said it will support on django-1.7.
Below is from south documentation.

This is the last major release of South, as migrations have been rolled into Django itself as part of the 1.7 release, and my efforts are now concentrated there. South will continue to receive security updates but no feature updates will be provided.

So for existing migration files, move the migrations folder to south_migrations, as django will use migrations folder for it migration. Then you need upgrade to south-1.0.2 which will first look in south_migrations directory, and fallback to migrations directory if not found.

And of course, south will not run on django-1.7.11. How we going to do with previous south migration created? we can still use django-1.6 with virtualenv, and run the south migration from there. But it is not recommended as in production will be to hard to maintain 2 virtualenv and also 2 migration in the long run. So when upgrade finish, better make django migration hold all the migration.

Next is in django model forms. You must specify the field you will include in the form creation. Django-1.7 give warning that it will removed in Django-1.8. So better clean up before next update will give you complete errors.

So hope my journey here will give other Djangonouts a heads up on the update bump you might encountered.

Read More

Upgrading Django-1-6 to 1-7 part 1

Currently Django already in version 1.9 the latest. And my Django app developed on Django version 1.6 since my last update. As the support and update will not delivered to version 1.6, better to upgrade it to the next version, which is 1.7 .

In upgrading a Python web framework like Django, the path not easy especially if you already build and using many library. Some will support the new version, some just not available for the new version of Django. As 1.7 is long enough, i will try my upgrade with virtualenv just like my previous upgrade from 1.5 to 1.6 in here.

So why need to upgrade. Actually, the why is have different answers on each case. For me, i want to have the update and feature Django new version offers, but provide stable framework for my development and app. Because so many component already in use and to make sure the app not break is the first priority. Some library i use were django-compressor, django-debug-toolbar, pytest for testing, south for database migration.

Some of the library like south migration, in django-1.7 already have the migration in core, so will need to remove south and use default migration shipped with django.

I will make this blog as an upgrade process log which i run step by step. Some notes are currently i am upgrading from Django-1.6.5 . If you still running Django-1.5.x you need upgrade it to django-1.6.x first.
As what best practice to do on every upgrade, one should read about the change logs in the new version. The new change in django-1.7 was :

1. Migration in core , which can replace south-migration lib if you use it.
2. Some deprecated lib like django.utils.simplejson. You can use json in new django-1.7
3. Test runner setup that changes
4. Support Python > 1.7

For details release notes can go to Django-1.7 Release notes

Let's start the migration path on new Django-1.7.11 from Django-1.6. First of all, we will using virtualenv in python for the upgrade. First we have the running old django in virtualenv, we need create new virtualenv and install the new Django-1.7.11

# cd /home/masterman/
# makevirtualenv dj-1.7.11
# workon dj-1.7.11

Then we can now install all the package we need for the webapp. I assume you will have requirements files to be feed to pip for installation. If you have the django requirement, change the version of django to Django==1.7.11

Then run the installation using pip :
# pip install -r requirements.txt

It will install all the requirement lib of the project you have, including django-1.7.11

I assume you project still intact, and in my case is /home/masterman/super-ecommerce/
Because i have test build in my app, and no changes, i can test it using django-1.7.11.

#py.test

This will test all my test file and give error if encountered. Thats why we need a test in every application we build so this things help us in migration or updates. Test is worth the effort.

And then voila, if there is errors you will spot where the errors are. And for me, i found some errors regarding the application library i use.

Continue on next part >>>

Read More

Postgresql Database basic Tune up

Postgresql is a robust database. it can server thousand of request per minute, if tune properly.

Some basic config you can do for tune the database are documented below.

Memory use :

shared_buffers = 2GB
# RAM/4 up to 8 GB

work_mem = 32MB
#Non shared memory use for sort , etc
# 8 MB to 32 Mb: web
# 128MB to 1 GB: reporting
# limit : Ram/(max_connection/2)

effetive_cache_sie = 6GB
# 3/4 of RAM

wal_buffers = 64Mb
# just set it

maintenance_work_mem = 512MB
# RAM/32
# more for reporting , use by analyze and autovacuum

checkpoint_segments = 64
# make WAL bigger
# space / 32 MB

checkpoint_completion_target = 0.9

stats_temp_directory = '/mnt/ramdisk'
#help with latency

random_page_cost = 1.5
# for AWS, SSD , decision for using index / io

effective_io_concurrency = 4
#for AWS, SSD, RAID

Logging Part , can use with pgBadger to read all the logs and view statistics :

log_connections = on
log_disconnections = on
log_temp_files = 1kB
log_lock_waits = on
log_checkpoints = on
log_min_duration_statement = 0

What we can do for optimize postgresql :

- Do less querying
- fix resoiurce-hungry requests
- get adequate hardware
- scale your infrastructure
- tune the config
- do caching on your application

source :
https://www.youtube.com/watch?v=dBeXS5aFLNc&list=PLE7tQUdRKcyaRCK5zIQFW-5XcPZOE-y9t

Read More