Yet another locale problem

Gentoo January 27th, 2008

In the last post, we manage to leverage eye3D for standardizing the ID3 tag. But we still got messy code when manually manipulating the tags in the command line. It may result in either wrong arguments or encoding bug.

Further investigation focused on eyeD3’s __init__.py:

LOCAL_ENCODING = locale.getpreferredencoding(do_setlocale=False);
if not LOCAL_ENCODING or LOCAL_ENCODING == "ANSI_X3.4-1968":
    LOCAL_ENCODING = ‘latin1′;

Either LOCAL_ENCODING is None or mysterious ANSI_X3.4-1968, the encoding is assumed as latin1. In my Gentoo box, with do_setlocale set False, getpreferredencoding just returns ANSI_X3.4-1968 though the locale is en_US.UTF-8 instead.

According to the documentation:

On some systems, it is necessary to invoke setlocale to obtain the user preferences, so this function is not thread-safe. If invoking setlocale is not necessary or desired, do_setlocale should be set to False.

I need to dig into whether Linux belongs to some systems. Right now, just apply the patch to eyeD3’s __init__.py:

37c37
< LOCAL_ENCODING = locale.getpreferredencoding(do_setlocale=False);
---
> LOCAL_ENCODING = locale.getpreferredencoding(do_setlocale=True);

And remember to specify the encoding of tags using –set-encoding , RTFM for more details.

Learning Django by Example(8) Tag it in place

Web January 21st, 2008

The main challenge of tagging is the usability: tags are usually used for navigation, but they are subject to updating at any time. Therefore, tags needs to switch back and forth in display mode and edit mode in a non-intrusive way. Dojo’s dijit.InlineEditBox seems a good candidate for this purpose.

I failed to declare a dijit widget inside the HTML file, maybe the mechanism of dojo.parser prevents me to do so? So I went back to a local copy of dojo-svn and declared FilteredInlineEditBox in dijit namespace:

dojo.provide("dijit.FilteredInlineEditBox");
dojo.require("dijit.InlineEditBox");

dojo.declare("dijit.FilteredInlineEditBox",
        dijit.InlineEditBox,
        {
        postMixInProperties: function() {
                this.inherited(‘postMixInProperties’, arguments);
                this.disabled = true;
        },

        setValue: function(val) {
                this.value = this.filterIn(val);
                this.displayNode.innerHTML = dojo.trim(val) || this.noValueIndicator;
        },

        filterIn: function(val) {
                return val.replace(/<[^>]+>/g, );
        },
});

No extra functionalities added, the inherited functions just manipulate the status for different behaviors. First, the dijit widget is disable by default to block any mouse event.

In the client side, the edit control tailgates the FilteredInlineEditBox:

<p>Tags: <span id="tags" dojoType="dijit.FilteredInlineEditBox" editorParams="{lowercase:true, trim:true,}" onChange="updateTags">
                        {{ object.tags|popuptags|safe }}</span>
                        (<a href="javascript:void(0);" id="edit">edit</a>)

and its onclick is hooked to invoke _edit:

dojo.connect(dojo.byId("edit"), ‘onclick’, function() {
                dijit.byId("tags")._edit();
        });

Notice that _edit never references/modifies disabled, so when the FilteredInlineEditBox exits edit mode, it is still disabled. This is the exact behavior we ask for!

A xhrPost request is used to update the tags asynchronously when onChange is invoked:

def tag_update(request):
        newTag = "none";
        book = Book.objects.get(isbn=request["isbn"])
        if book:
                book.tags = request["tags"]
                book.save()
                tags = [ x for x in re.split("\W+", request["tags"]) if x ]
                newTag = " ".join(['<a href="/bookshelf/tags/%s">%s</a>' % (t, t) for t in tags ])
        return HttpResponse(newTag)

Now in our hack of setValue, the displayNode is still rendered as HTML links, but all the HTML elements are filtered by filterIn, so only the raw text is passed to value, which is used to construct the edit widget later. Therefore, the tags are displayed to users as HTML links, but modified as raw text.

Learning Django by Example(7) Attach a tag

Python, Web January 20th, 2008

Tag is probably the most distinguish feature of Web 2.0 applications that differentiate them from the traditional hierarchy categories. I want to attach a Web 2.0 tag to Gelman, so the user could simply click the tag, and find the related books that may arouse his/her interest.

django-taggingis a generic tag application to simplify the backend development, all you need to do is just add the TagField to the Book model:
class Book(models.Model):
……
tags = TagField()

And in the book_detail.html template, refer it as object.tags as this:

{{ object.tags|popuptags|safe }}

popuptags is a custom tag that decorates the tags as a list of HTML links, and join them:

@register.filter
def popuptags(value):
    tags = value.split(" ")
    return " ".join(['<a href="/bookshelf/tags/%s">%s</a>' % (x, x) for x in tags])

So tag foo is linked with /bookshelf/tags/foo, so just redirect the request to the view:

(r'^tags/(?P<tag_name>[\w-]+)/$', views.by_tag)

And handle it in views.py:

@login_required
def by_tag(request, tag_name):
    tag = Tag.objects.get(name=tag_name)
    return list_detail.object_list(
        request,
        queryset=TaggedItem.objects.get_by_model(Book, tag),
        template_name="bookshelf/book_list.html",
        extra_context={'title': 'Tagged by %s' % tag_name},
    )

All the tedious work has been handled by django-tagging: we first get a tag object by name, and then build a QuerySet using get_by_model method; the rest is handled by the generic view, done. Salute to django-tagging developers!

We would discuss how to add a tag in the next post, that is the magic of Dojo.

Python/SOAP: second encounter

Python, Web January 15th, 2008

It looks like my experience in Python/SOAP programming no better than the first encounter. Here is the full story:

I am working on a client application to consume Microsoft SharePoint server web service recently. Since the client does not support scripting, I decided to develop a Python console application to ease my routine job.

The first two candidate are SOAPpy and ZSI. SOAPpy is the official library used in the DiveIntoPython, and ZSI is the succeeder of SOAPpy.

The first problem I have met in Python 2.5 environment is ZSI-2.0 depends on PyXML, which is not compatible to Python 2.5 though, so I migrated to ZSI-2.1_a1, and eventually get ZSI successfully imported. ZSI supports two approaches to use WSDL, I must admit I won’t consider any web service without WSDL.

  • ServiceProxy: dynamically build the method proxy in the run time
  • wsdl2py: generate the python code in compile time for later use

Neither of them works even for a simple web service from xmethods.com. Some type error or attribute exception is thrown when calling service method.

So I went back to the traditional SOAPpy. Though it is a little annoying to build fpconst, (thanks to easy_install to make it less painful), I really love the SOAPpy’s API, simple and stupid:

from SOAPpy import WSDL          
wsdlFile = ‘http://www.xmethods.net/sd/2001/TemperatureService.wsdl’)
server = WSDL.Proxy(wsdlFile)
server.foo(‘bar’)

However, the SOAPpy generate wrong SOAP message. Once the type is declared as sequence, the SOAPpy would insert tag as the container, that failed the service provider.

Don’t forget we have not touched the authentication, NTLM, the default certification policy used in most of the SharePoint server in the Intranet. This is really a long way to go.

UPDATE: Moved to Web category, so it won’t pollute the Gentoo Planet.

UPDATE: Thanks to Lawrence’s advice, I did try soaplib tonight last night. It is really a promising library for SOAP, the parser is based upon cElementTree, so it is supposed fast and memory-friendly. Once you build the service provider, Bang, you already get the consumer application. The only problem is there is no official support for WSDL. wsdl2py in ext package seems promising, it is under active development and should not be used in a production scenario, and checked-in 1 year ago. No wonder there are tons of errors in WSDL parsing. I just wonder optio developers may have a private code repository for daily check-in, and they may sync their work to the public from time to time.

Once I finish WSDL specification, I may refactor the wsdl2py module.

Learning Django by Example(6): Search

Web January 4th, 2008

Search is one of the must-have functionalities in Gelman. Here is an SearchQuerySet based upon MySQL full text search extension. It is really cool and neat, but

  • first, I don’t want to build my application against specific database extension, even though MySQL is universally picked up in the Web application.
  • second, I still prefer more flexible and powerful search syntax other than what MySQL provides
  • Last but not the least, I may still need Lucene or Xapian to index, search the PDF, CHM eBooks

So I home-brew the search using PLY, the Python Lex Yacc toolchain. You could check the code here, most of parser.py is just boilerplate, the interesting part is to build django.db.models.Q:

def p_expression_term(t):
    ‘expression : TERM’
    t[0] = Q(**{‘title__icontains’:t[1]})

The semantics is quite straightforward: AND(the default), OR operations are supported directly from Q; and only field title is searched. We may extend the syntax using author: like Google does later, so stay tune.