Just a quick announcement for all the Houston and surrounding area Python lovers out there. There is a Texas Python Regional Unconference being held in a couple weeks (Sept. 15 - 16) at the University of Houston main campus. Registration is free and as simple as adding your name to the registration wiki page. I’ll be there to talk or hack Django with anyone interested. See you there!

A short while ago, I went on a mission to find a better way to hack Django. I needed a way to keep my patches organized and updated. So, in this article I discuss my experiences with a tool I found to help me achieve these goals…

Enter Bazaar, one of the several distributed version control systems that have been getting some hype recently. With so many to choose from, why did I choose Bazaar? Well, maybe it was because of the cool name, or maybe it was the pretty website, or maybe just because it’s written in the language I love — Python. If you are familiar with Subversion, then you will feel right at home with Bazaar. This is because Bazaar uses most of the same commands as Subversion (e.g. svn ci = bzr ci, svn st = bzr st, etc.).

Following this guide for hacking an open source project using bzr, I created a django directory to stash my django hacks. After using svn to check out the latest Django code to a directory named upstream, and turning upstream into a bzr repository too, it was time to start the real work. To re-iterate in command line speak, here is what I did:

Update: The steps below have been updated to reflect changes with bzr commands (versions >=0.15rc2).

mkdir django
cd django
bzr init-repo .            # use "bzr init-repo --trees ." here instead if using a version <0.15rc2
svn co http://code.djangoproject.com/svn/django/trunk/ upstream
cd upstream
bzr init
echo ".svn" > .bzrignore   # do not store svn metadata in our bzr repo
bzr add
bzr ci -m 'Initial import of Django.'

Usually, when you want to add a new feature to some code, you create a branch. The purpose of these, so called, feature branches is to keep the features separated. Trying to create multiple features in the same branch would just be a big mess. Creating branches in a Subversion repository is easy enough, you just svn cp the code to a new location. It’s the keeping your branch in sync with the trunk that’s not as easy.

With Subversion, you have to keep track of the revisions that have already been merged to your branch and make sure to only merge the changes that haven’t already been merged. Not so with Bazaar; Bazaar keeps track of this for you. A simple bzr merge will pull in upstream changes that have not been merged yet.

So, when I wanted to start working on a patch for ticket #2473, I created a branch:

bzr branch upstream in-and-empty-list-2473

After cd‘ing into my new branch and committing my changes, I created a diff to attach to the ticket with the command:

bzr diff -r branch:../upstream

So far, working with Bazaar has been a breeze. Without it, my Django holiday hacking wouldn’t have been nearly as fun.

A warning for those who might not of noticed the change. About a month ago (in [3360]), an is_authenticated method was added to the User and AnonymousUser classes. These are the classes used for Django’s default authentication system.

Previously, the template code used for displaying content based on whether or not a user had authenticated went something like:

{% if not user.is_anonymous %}
Content for logged in users.
{% else %}
Content for non-logged in users.
{% endif %}

The problem with this code is that if, somehow, the user template variable did not get populated, your template would treat the requesting user as if they were logged in.

Notice that this sort of behavior will be seen for all negative if statements. The template code {% if not variable %} will always evaluate to True if variable doesn’t exist in the context. This is because Django uses the settings.TEMPLATE_STRING_IF_INVALID value for non-existent template variables; by default, TEMPLATE_STRING_IF_INVALID is '', which evaluates to False (and not False is True).

Notice also that even using the template code:

{% if user.is_anonymous %}
Content for non-logged in users.
{% else %}
Content for logged in users.
{% endif %}

(without the "not" ) will not work in this case because if the user variable is non-existent, the requesting user will still be treated as if they were authenticated.

The new recommended way for checking that a requesting user has been authenticated in your templates is:

{% if user.is_authenticated %}
Content for logged in users.
{% else %}
Content for non-logged in users
{% endif %}

Here, if the user template variable were to not exist your template would treat the requesting user as non-authenticated, just as we would want.

Let’s face it, humans are not well adapted to memorizing strings of random characters; and hence, the average computer user is not very good at creating secure passwords. Most users create passwords made up of easy-to-remember words, like the name of a favorite sports team or maybe the name of a significant other. In this article I will show you how you can make Django require better passwords from your users.

Here are the packages you will need:

• CrackLib
• python-crack

Installing the required packages

Step 1: Install CrackLib.
CrackLib is a library for checking that passwords are not easily crackable, or in other words, it makes sure that a password is not based on a simple character pattern or on a dictionary word. CrackLib is a common package that you should be able to find in your distribution’s package manager (or, quite possibly, it could already be installed). Alternatively, you could download the source and follow the installation instructions in the README file. By default, CrackLib installs a python package named cracklib, but it does not have as many features as python-crack. (Redhat does not include the cracklib python package in its cracklib package.)

Step 2 (optional, but recommended): Install extra word dictionaries.
Packaged with CrackLib is a file name cracklib-small. (Some distributions, like RedHat, don’t include this file in their cracklib package, in which case keep reading…) This file is a dictionary of words, simply a long list of words with one word per line. These 50,000 words are a good start, but we can do better. On the CrackLib download page, there is also a package named cracklib-words. After downloading and extracting the package, you will have a single file containing 1,648,379 “words”. Many distributions also have a cracklib-dicts or cracklib-words package that maybe the same or similar to the cracklib-words file on the CrackLib website.

For even more dictionaries, take a look at these word lists. It might also be a good idea to create your own list of words. For example, if you work for a company in the financial industry, it would be a good idea to make a list of words and slang specific to that industry. If you will be dealing with non-english speaking users, it would be a good idea to find some dictionaries in other languages.

Step 3: Create the word indexes that get used by CrackLib.
Now that we have our dictionaries, we need to compile them into an index that CrackLib uses. So, put all your dictionaries in one directory (/usr/share/dict/ is a common place). Depending on your distribution, there will be different commands to run. On gentoo (or if you installed from source), the following (as root) should do the trick:

create-cracklib-dict /usr/share/dict/*

And on RedHat:

mkdict /usr/share/dict/* | packer /usr/lib/cracklib_dict

These commands should create the following files:

/usr/lib/cracklib_dict.hwm
/usr/lib/cracklib_dict.pwd
/usr/lib/cracklib_dict.pwi

Step 4: Install python-crack
If your distribution’s package manager doesn’t include python-crack, then you will have to download and install it yourself. Neither Gentoo nor RedHat include this package, so I will walk you through the install. Download and unpack python-crack. Now, cd into the directory and run

./configure

By default, configure will setup python-crack’s files to be installed to /usr/local. This means that you will have to add /usr/local/lib/python2.4/site-packages to your PYTHONPATH. Alternatively, you could add a --prefix=/usr to the end of the configure command to install python-crack’s files to the usual /usr/lib/python2.4/site-packages directory.

If you get this error:

checking for prefix of the default cracklib dictionary database… unknown
configure: error: crack.h does not define CRACKLIB_DICTPATH. Please use DEFAULT_DICTPATH, see ./configure –help

then set the DEFAULT_DICTPATH variable when running configure, like so

./configure DEFAULT_DICTPATH=/usr/lib/cracklib_dict

Now run

make
make install

Now that we are finished installing, lets test out python-crack

$ python
Python 2.4.3 (#1, Jun 28 2006, 23:41:37)
[GCC 3.4.6 (Gentoo 3.4.6-r1, ssp-3.4.5-1.0, pie-8.7.9)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import crack
>>> crack.VeryFascistCheck(’foo’)
[snip]
ValueError: it is WAY too short
>>> crack.VeryFascistCheck(’foobar’)
[snip]
ValueError: it is based on a dictionary word
>>> crack.VeryFascistCheck(’3#hsad2U>u2u’)
‘3#hsad2U>u2u’

VeryFascistCheck() raises a value error and prints an explaination if the password is not strong enough, otherwise it echos back the password.

Creating the Django Manipulator

from django import forms

class NewUserForm(forms.Manipulator):
    def __init__(self):
        self.fields = [
            forms.TextField(field_name="username", length=20, maxlength=20, is_required=True),
            forms.PasswordField(field_name="password", length=20, maxlength=50, is_required=True,
                validator_list=[isStrongPassword]),
            forms.PasswordField(field_name="confirm_password", length=20, maxlength=50, is_required=True,
                validator_list=[forms.validators.RequiredIfOtherFieldGiven("password", "You must confirm password"),
                                forms.validators.AlwaysMatchesOtherField("password", "Passwords did not match"),]),
        ]

def isStrongPassword(field_data, all_data):
    """Test the password with cracklib to make sure it is strong."""

    import crack
    # Increase the number of credits required from the default of 8 if you want.
    #crack.min_length = 11
    try:
        crack.VeryFascistCheck(field_data)
    except ValueError, message:
        raise forms.validators.ValidationError, "Password %s." % str(message)[3:]

Notice that I added isStrongPassword to the password field’s validator_list. Now you are ready to put the NewUserForm manipulator to use in your you views.py. After running the manipulator’s get_validation_errors(), bad passwords will generate errors like “Password is based on a dictionary word” or “Password does not contain enough DIFFERENT characters.” For help on using manipulators in your view, see the Forms, fields, and manipulators documentation.

If you want to make Django’s built-in authentication require stronger passwords, then you could add validator_list=[isStrongPassword] to the password field of django.contrib.auth.models.User.

Now, go forth and require strong passwords.

Adrian’s recent post on the dev mailing list about the code within django.db.models got me remembering some other “monstrous” functions I’ve seen while browsing the Django source. Guess no more as to the most monstrous functions of Django because below are the results of my previously posted complexity.py script when run on the Django source.

Adjust your window until you can see all of the following line:

#===============================================================================================================#

Here are all 80 of the functions and methods of Django that are of moderate risk or higher:

Showing functions/methods with complexity greater than or equal to 11:

Filename                                    Function/Method                              Lines of Code Complexity
./core/management.py                        get_validation_errors                                  146         78
./db/models/manipulators.py                 AutomaticManipulator.save                               97         43
./contrib/comments/templatetags/comments.py DoCommentForm.__call__                                  51         31
./views/i18n.py                             javascript_catalog                                      68         28
./contrib/admin/views/main.py               change_stage                                            75         28
./utils/translation/trans_real.py           translation                                             62         26
./bin/make-messages.py                      make_messages                                          114         26
./contrib/admin/views/main.py               _get_deleted_objects                                    65         25
./core/handlers/base.py                     BaseHandler.get_response                                50         24
./views/defaults.py                         shortcut                                                40         23
./contrib/auth/create_superuser.py          createsuperuser                                         64         23
./core/management.py                        inspectdb                                               84         23
./contrib/admin/views/doc.py                model_detail                                            64         22
./templatetags/i18n.py                      do_block_translate                                      41         21
./core/management.py                        syncdb                                                  61         21
./core/management.py                        execute_from_command_line                               74         21
./views/generic/create_update.py            update_object                                           54         20
./contrib/admin/views/decorators.py         staff_member_required                                   44         20
./utils/simplejson/decoder.py               scanstring                                              41         19
./contrib/admin/views/main.py               add_stage                                               51         19
./core/validators.py                        RelaxNGCompact.__call__                                 51         19
./db/models/query.py                        lookup_inner                                           107         19
./db/models/query.py                        delete_objects                                          48         19
./template/__init__.py                      TokenParser.value                                       61         18
./utils/simplejson/decoder.py               JSONObject                                              37         18
./utils/decorators.py                       decorator_from_middleware                               25         18
./contrib/comments/templatetags/comments.py DoGetCommentList.__call__                               31         18
./contrib/comments/views/comments.py        post_comment                                            66         18
./utils/translation/trans_real.py           get_language_from_request                               40         17
./core/mail.py                              send_mass_mail                                          31         17
./views/generic/create_update.py            create_object                                           38         16
./views/static.py                           serve                                                   28         16
./core/servers/fastcgi.py                   runfastcgi                                              56         16
./core/management.py                        get_sql_delete                                          49         16
./db/models/query.py                        QuerySet.__getitem__                                    38         16
./utils/simplejson/encoder.py               floatstr                                                17         15
./utils/simplejson/encoder.py               JSONEncoder._iterencode_dict                            49         15
./views/generic/create_update.py            delete_object                                           36         15
./contrib/comments/templatetags/comments.py DoCommentCount.__call__                                 25         15
./db/models/fields/related.py               create_many_related_manager                             73         15
./db/models/query.py                        QuerySet._get_sql_clause                                54         15
./middleware/common.py                      CommonMiddleware.process_request                        22         14
./template/__init__.py                      Parser.parse                                            35         14
./template/defaulttags.py                   do_if                                                   33         14
./template/defaulttags.py                   cycle                                                   25         14
./views/generic/date_based.py               object_detail                                           43         14
./contrib/comments/views/comments.py        post_free_comment                                       41         14
./contrib/admin/templatetags/admin_list.py  items_for_result                                        59         14
./contrib/admin/views/main.py               get_javascript_imports                                  23         14
./contrib/admin/views/main.py               ChangeList.get_query_set                                39         14
./core/handlers/base.py                     BaseHandler.load_middleware                             32         14
./db/models/base.py                         Model.__init__                                          33         14
./template/defaulttags.py                   SsiNode.render                                          21         13
./views/generic/list_detail.py              object_list                                             46         13
./contrib/auth/handlers/modpython.py        authenhandler                                           28         13
./db/models/fields/__init__.py              Field.get_manipulator_fields                            24         13
./db/models/base.py                         ModelBase.__new__                                       26         13
./dispatch/saferef.py                       safeRef                                                 65         12
./dispatch/dispatcher.py                    _removeOldBackRefs                                      20         12
./template/loader.py                        find_template_source                                    26         12
./template/defaulttags.py                   IfNode.render                                           21         12
./utils/timesince.py                        timesince                                               34         12
./views/generic/date_based.py               archive_month                                           36         12
./core/management.py                        _get_sql_model_create                                   44         12
./conf/__init__.py                          Settings.__init__                                       26         12
./db/models/fields/__init__.py              DateTimeField.to_python                                 14         12
./db/models/related.py                      RelatedObject.get_list                                  21         12
./db/models/options.py                      Options.has_field_type                                  16         12
./dispatch/dispatcher.py                    connect                                                 34         11
./template/defaultfilters.py                pluralize                                               18         11
./utils/simplejson/decoder.py               JSONArray                                               21         11
./views/generic/list_detail.py              object_detail                                           33         11
./views/generic/date_based.py               archive_week                                            31         11
./views/generic/date_based.py               archive_day                                             34         11
./contrib/redirects/middleware.py           RedirectFallbackMiddleware.process_response             17         11
./contrib/admin/templatetags/log.py         DoGetAdminLog.__call__                                  11         11
./db/backends/postgresql/base.py            DatabaseWrapper.cursor                                  21         11
./db/backends/postgresql_psycopg2/base.py   DatabaseWrapper.cursor                                  21         11
./db/models/fields/related.py               ReverseSingleRelatedObjectDescriptor.__get__            19         11
./db/models/query.py                        get_where_clause                                        18         11

I sure wish there were news sites in my city that didn’t suck. How happy I would be if I could have sites such as the Django-powered 49abcnews or LJWorld to give my attention to each evening. Instead, I’m stuck with crappy news site after crappy news site after crappy news site. Popups, popovers, registrations, and URLs that make you cringe…

http://sports.statesman.com/merge/tsnform.aspx?c=statesman&page=mlb/news/index.aspx?sport=AA
http://www.kvue.com/news/top/stories/051906cckrKvurescape.2b620a4.html

Oh my!

To be fair, Statesman.com (and its Austin360.com entertainment site) is now much more pleasing on the eye due to a major redesign last December. The redesign seems to have paid off, with the Statesman’s web site duo recently receiving the top ranking by the NAA for the greatest local reach among large newspaper sites. With a monthly readership of 31 percent of Austinites, it edged out The Washington Post (27 percent) and the San Diego Union-Tribune (23 percent).

In my mind though, Statesman.com fails to deliver the news. All those headlines on the home page are just a tease to get me to register. I just want to be able to read the news and find Rolexes in supermarkets without getting hassled. Is that so much to ask?

LJWorld gets it. There’s plenty of content here just begging to be Django-ed. With Austin being the state’s capital, land of the silicon hills, a smart place to live, home to one of the largest universities in the U.S. (with plenty of college girls making out (04:43)), and then there’s always the Live Music Capital of the World bit. Austin news sites don’t get it.

I needed to make a form that had two inputs, and wanted to require that one or both fields be filled in. While poking around in django.core.formfields, I noticed the following check:

 if field.is_required or new_data.get(field.field_name, False) or hasattr(validator, 'always_test'):

Ah ha! What is this always_test I see here? The devs must have already thought of a situation like this, but just failed to mention it in the documentation. Anyway, on to the to the custom form:

from django.core import formfields, validators

class ItemAddForm(formfields.Manipulator):
    def __init__(self):
        self.fields = [
            formfields.TextField(field_name="name", length=50, maxlength=100,
                validator_list=[self.oneIsRequried]),
            formfields.URLField(field_name="url",
                validator_list=[self.oneIsRequried]),
        ]

    def oneIsRequried(self, field_data, all_data):
        if not all_data['name'] and not all_data['url']:
            raise validators.ValidationError, "One of either name or URL must be filled in."
    oneIsRequried.always_test = True

After defining the oneIsRequired method, I set its always_test attribute to True. You see, normally, a field’s validators will not get run if the field is left blank. Here, the two fields will always get tested against the oneIsRequired validator method, blank or not.

Note: Technically, I did not need to put the oneIsRequired in both fields’ validator_list parameters, but by listing it in both, each field will get the error message in the form.errors dict.