Learning Django by Example(6): AJAX File Upload

Web October 4th, 2007

In the last post, I just added the entities in the database, but did not bind the meta data with eBook files. This post will demonstrate how to upload the file in AJAX flavor:

The idea is inspired by this tutorial , using dojo.io.iframe, we could submit a form with file upload asynchronously. In client side, a hidden field meta is added to store the marshaled JSON string. dojo.io.iframe.send shares the same defer concept as dojo.xhrPost:

var meta = dojo.query(“input[@name=meta]“, form)[0];
    meta.value = dojo.toJson(cache[i]);

    // submit the form
    dojo.io.iframe.send({
        url: form.action,
        method: “post”,
        handleAs: “json”,
        form: form,
    }).addCallback(function(ret) {
    … …

NOTE: the handleAs is json, so the argument ret is JavaScript object.

In server side, the JSON string needs to be wrapped by textarea, this is the hard requirement of dojo.iframe.send.

return HttpResponse(“<textarea>%s</textarea>” % simplejson.dumps(retset) , mimetype=‘text/html’);

Conceptually, add-by-search will issue INSERT operation to the database, the result may fall in the following categories:

  • Success
  • FileAlreadyExist: the eBook file exists, so the uploaded file is discarded
  • UnexpectedError: an 500 error happens in server side for unknown reason.

Check r34 for the implementation.

Learning Django by Example(5): Time to Attack

Web September 19th, 2007

In the last four section(1, 2, 3, 4) ,we have been familiar with the environment, it is time to do some real work.

I will add more fresh to models.py , linking eBook file and its meta data, thumbnail of the cover, regular size cover etc. Two other models are added for eBook:

class FileType(models.Model):
    type = models.CharField(maxlength=10)

class File(models.Model):
    type = models.ForeignKey(FileType)
    handle = models.FileField(upload_to=“files”)
    meta = models.ForeignKey(Book)

Book to File is one-to-many as one book may have more than one formats, a PDF version for print, a CHM version for reference.

To server the images of book covers, , we need to setup the MEDIA_ROOT and MEDIA_URL:

MEDIA_ROOT = ‘/home/share/eBooks/’
MEDIA_URL = ‘http://localhost:8000/repo/’

and setup the url mapping as well:

(r’^repo/(?P .*)$’, ‘django.views.static.serve’, {’document_root’: ‘/home/share/eBooks’}),

To save an image file, we can easily call save_FOO_file method, it works with two drawbacks: first, you need to read the image into the memory; second, it is synchronous function call aka it is blocked until the file is fetched, read and written. urlretrive is preferred in this way:

from django.conf import settings
from urllib import urlopen, unquote, urlretrieve
from urlparse import urlparse
from os import path
… …
                thumb = unquote(urlparse(item[‘thumburl’])[2].split(‘/’)[-1])
                # using hard-coded image temporary
                urlretrieve(item[‘thumburl’], path.join(settings.MEDIA_ROOT, ‘images’, thumb))
                book.thumb = path.join(‘images’, thumb);
                book.save()

A further optimization may introduce Twisted or asyncore.

More work needs to be done in the UI side: the Book objects are populated in brief or detail modes. Brief mode just highlight the title, cover, authors and ISBN of the book, commonly used in general listing, search results, my favorite and so on; while detail mode shows more detailed information like editorial review, related book, tags etc. It is worthy extracting the code snippet of brief mode to a template, so we can reuse it later by {% include book_item_brief.html %}. Detail view would fetch the editorial view from Amazon using the same AJAX hack as discussed before.

With stylesheet, it looks much better, Brief view:

Breif view



and Detail view:

Detail view



Check r30.

Learning Django by Example(4): First user authenticated

Web September 18th, 2007

In the last post, we succeeded to update the database, in this post we would add more functionalities to the view of normal users.

In r23, I re-organize the file directory in the repository, media is added to the repository for your convenience.

Though Gelman is designed for personal use, the multi-user functionality is added just in case you want to share you collection with family members and friends later.The built-in authentication is quite neat, if you deploy Gelman into an enterprise environment, you may consider this. Now, just follow the documentation, and add the very first reader to Gelman by running python manage.py shell:

>>> from django.contrib.auth.models import User
>>> u = User.objects.create_user(‘python’, , ’spam’)
>>> u.save()

Now, let’s build the gateway for the user to login. Copy the basic login form in the doc to templates/login.html, The default success login action is to redirect to user’s profile, we may override it to the index first:

<input type=“hidden” name=“next” value=“/library/” />

add the URL mapping in urls.py:

(r‘^accounts/login/$’, ‘django.contrib.auth.views.login’, {‘template_name’: ‘login.html’, }),
       (r‘^accounts/logout/$’, ‘gelman.library.views.logout_view’, ),

For logout action, just logout the user and redirect the user to the front page, in library/views.py:

def logout_view(request):
    logout(request)
    return HttpResponseRedirect(‘/library’)

We would greet the users in the top-left corner of the page just as Wordpress’s dashboard does. Add the following code snippet to base_generic.html:

{% if user.is_authenticated %}
Howdy, {{ user.username }} [Logout, My Profile]
{% else %}
[Login or Register]
{% endif %}

as the following(left: logged-in user, right: logged-out user):

Login and Logout demo



Check out r25 for the implementation details.

Learning Django by Example(3): Just works

Web September 7th, 2007

In the last post, we succeeded to post the Javascript objects via JSON to the server, the data is to be serialized to the database in this section.

Before we move on, let’s remove some bumps in our way, the database scheme is modified in this way:

class Book(models.Model):
isbn = models.CharField(maxlength=10, primary_key=True)
pub_date = models.DateField(’data published’)

The reasoning lies in that AWS returns ISBN-10 as ASIN, we may take advantage of this. Another fix is to replace DateTimeField by DateField for pub_date. Django does not support seamless database scheme refactory so far. So we just dropped the database and re-generate it as described in the tutorial. The username/password for new database is set as admin/gelman as common practice.

It is quite annoying that a 500 error returned without any reason, Django server ate the exception due to the nature of Web server. We could wrap the suspicious buggy code with try/except block like this:

try :
    // database access
except :
    print sys.exc_info()[0]

Well, InterfaceError or IntegrityError pinned the bug in database insertion, but where? Interactive shells are our best friends, you can either use pdb or python manage.py shell.

Oh, my fault, I forgot the return value of get_or_create is a tuple, instead of the object; and for many to many relation, we could not simply set a list as its value, we need to use add method after the object is constructed:

for item in simplejson.loads(request.POST[‘items’]):
        publisher, created = Publisher.objects.get_or_create(name=item[‘publisher’])
        authors = [Author.objects.get_or_create(name=au) for au in item[‘authors’]]
        authors = [author for (author, created) in authors];
        y,m,d = [int(x) for x in item[‘pub_date’].split(‘-’)]
        pub_date = date(y, m, d)
        book, created = Book.objects.get_or_create(isbn=item[‘isbn’], name=item[‘title’],
            pages=300, pub_date=pub_date, publisher=publisher)
        book.authors.add(*authors)
        #print sys.exc_info()[0]
    return HttpResponseRedirect(‘/admin/library/book/’)

At the exit, we just redirect the browser to a new URL.

It seems the database is updated, but the page is not redirected. Since the POST operation is launched in an AJAX request, we need to send JSON object as the response. In server side, simplejson.dump is used:

xhr = {’succeed’: [], ‘failed’: []};
       for item in simplejson.loads(request.POST[‘items’]):
               # … ….
               try :
                       book, created = Book.objects.get_or_create(isbn=item[‘isbn’], name=item[‘title’],
                               pages=300, pub_date=pub_date, publisher=publisher)
                       book.authors.add(*authors)
               except :
                       xhr[‘failed’].append(item[‘title’])
               xhr[’succeed’].append(item[‘title’])

       return HttpResponse(simplejson.dumps(xhr), mimetype=‘text/javascript’);

In client side, we evaluate the JSON string then populate the DOM nodes:

dojo.xhrPost({
                url: dojo.byId(“book_form”).action,
                content: { items: dojo.toJson(formdata) },
            }).addCallback(function(response) {
                var result = dojo.fromJson(response);
                var messages = dojo.query(“ul[@class=messagelist]“)[0]
                dojo.forEach(result.failed, function(item) {
                    var msg = document.createElement(“li”);
                    msg.innerHTML = “<li><em>” + item + “</em> <strong>failed</strong> to add.</li>”
                    messages.appendChild(msg);
                });
                dojo.forEach(result.succeed, function(item) {
                    var msg = document.createElement(“li”);
                    msg.innerHTML = “<em>” + item + “</em> successfully added.”
                    messages.appendChild(msg);
                });
            });

The final result looks like this:

Add books with feedback



Check r19 for details.

Learning Django by Example(2): Show me your data

Web September 5th, 2007

In last post, a quick-and-dirty prototype is developed to make Gelman running. In this post, I would tail the XSLT for our needs.

If you have not tried XSLT, consider the XSLT (for dummies and official reference). It is a powerful tool, if you only take interest data extraction, You could compare the efforts to parse XML, for example, PyAWS. The xsl is not in the repository since we need a URI to access.

Let’s keep it simpler and more stupid. Since AWS support both ISBN query and keyword search, we could combine the two input box as one and send different requests to AWS based on the knowledge of ISBN validation. Check r12 for the implementation, here is the screenshot:

Add book by search



In r14, we decided to take a try of dojo.query, an alternative of jQuery; it is really cool to reference the object using CSS syntax.

dojo.forEach(dojo.query(“#book_form input[@class=incoming]“), function(item) {
                               if (item.value != “”) {
                                       formdata.push(cache[i]);
                               }
                               i++;
                       });

In the above code snippet, cache holds the JSON objects parsed from AWS request, if an eBook file is attached to incoming(upload does not work now, just use it as the mark), the corresponding meta data is collected in formdata. The last question is how to POST the it to the server?

JSON is supported in both client and server sides, i.e dojo.json and django.utils.simplejson. In client side, we could serialize the JavaScript to JSON string:

dojo.xhrPost({
                url: dojo.byId(“book_form”).action,
                content: { items: dojo.toJson(formdata) },
            }).addCallback(function(response) {

And in server side, just loads the JSON string to Python objects. Check r16 for implementation.

items = simplejson.loads(request.POST[‘items’])

Next section, we would discuss how to manipulate the database using Django’s database API.