Anything Else

Sunday, July 29, 2007

dbviews in Django

Came across this comment:

Julian: what we did first was implementing just normal models in the models.py, but that always created the tables in the DB upon “syncdb” (which we deleted then and created views for). Actually I even wrote a patch for django where you can mark a model as “create_table=False” but the patch never made it in.

So we went another actually much better way, after discussing it a lot in the team. We simply did not create the models in models.py, which are used for syncdb but we create a file dbviews.py where we put the views’ models. This is very nice separation of code too.
The next step is writing the view itself, which we just did in pure SQL of course. I.e. if we have the model Forum and we want some specialized ForumActivity-view then we created a view “CREATE VIEW core_forumactivityview” (we are on mysql). We then fired that onto our DB and the model that matched it (make sure to use the same column names as the view does!!!) simply looks like this:

class ForumActiviy(models.Model):
    ….. all the fields
    class Meta:
        db_table = “core_forumactivityview”

now you can simply do

import project.core.dbviews

and they just look like models :-).
Depending on how you wrote the view you might even be able to update the data.

Smart! Learn about MySQL views here.

Labels: Python Programming Django


Thursday, July 26, 2007

Django: Extending User Model

One of first issue one faces with starting developing with django is extending User. There are three approaches various people are using. 

 User Profile

Django has a concept of UserProfile model. This is the recommended way to extend User model in django, recommended by the django book and django's official documentation. Lets say the name of your project is myproj, you create an app to manage the accounts user registration etc, lets call it accounts. After "startapp accounts" go to the accounts/models.py, and create a model:

# extending user # {{{
class UserProfile(models.Model):
    user = models.ForeignKey(User)
    openid = models.URLField(blank=True)

    def _get_comments(self):
        return Comment.objects.get(user=self, is_public=True)
        public_comments = property(_get_comments)
    del _get_comments

    class Admin: pass
# }}}

While this is enough to store more information about user, django adds a shortcut to access this user profile model more conveniently by calling user.get_profile() if in your settings.py file you add the following line:

AUTH_PROFILE_MODULE = 'accounts.UserProfile'

While this is easy to get started, and "official" way of doing things, it has one major drawback: you put your relations in user profile model. What I mean is, if for example you had a concept of friends, you will say:

# extending user # {{{
class UserProfile(models.Model):
    user = models.ForeignKey(User)
    openid = models.URLField(blank=True)
    friends = models.ManyToManyField('self', symmetrical=True)
    ...

# }}}

and to access it you will have to use something like:

# show_friends # {{{
@login_required
def show_friends(request, userid):
    user = get_object_or_404(User, id=userid)
    friends = user.get_profile().friends
    # do other things with friends and return
    return HttpResposne(
        ",".join(
            [
                friend.get_profile().username
                for friend in user.get_profile().friends
            ]
    )
# }}}

See the problem? You have to remember to call get_profile() at appropriate times, and this sounds a little less right. Some variable are user.XYZ where as some are user.get_profile().ABC, and you are friends with "user", and not his profile! Keeps me wanting a neater solution. Then there is the hassle of making sure that one to one relationship between User and UserProfile is maintained, which on deleting User, you have to delete UserProfile and vice versa.  

The replaces_module Method

This is an incredibly hacky but works with issues method that is on django's wiki on the same topic.

Lost-Theories Solution

I came across another solution for it via Jeff Craft 's lost-theories.com source code; that I am actually using for my current project. In this case he created a LostUser that has a foreignkey to django's User model, and uses LostUser throughout the code. 

class LostUser(models.Model):
    user = models.ForeignKey(User, unique=True)
    # Location info
    city = models.CharField(maxlength=200, blank=True)
    state = models.CharField(maxlength=200, blank=True)
    country = models.CharField(maxlength=200, blank=True)
    ...

The primary problem I faced was the lack of request.user in views and user object in template niceties. I solved that by writing the following middleware:

from vakow.accounts.models import MyUser

class MyUserMiddleware(object):
    def process_request(self, request):
        if request.user.is_authenticated():
            request.vuser = MyUser.objects.get(duser=request.user)
        else: request.vuser = None

and the following context-processor:

# context_processor # {{{
def context_processor(request):
    d = {
       'media_url': settings.MEDIA_URL,
    }
    d['duser'] = request.user
    if request.user.is_authenticated():
        d['vuser'] = request.vuser
    else: d['vuser'] = None
    return d
# }}}

This has the drawback of having to deal with two objects about a user, duser and vuser, where duser is an instance of django's User models, and vuser in my project's user extension. Such naming conventions helped in disambiguating what instance I am talking about, and since almost all aspect of my code worked with my user class, this was not really a problem. I had added a few properties in my derived class for username, email to query it from self.duser.username when self.username is requested. Life was good, and the only time I had any issues was when dealing with django's comment framework, as comment objects contained a foreignkey to duser, and not vuser. I hacked my comment to added a property vuser in Comment model. 

The Final Solution 

This is partially derived from the cool tip I got on Pythoneer. Here is what you can do:

# Extending User # {{{
User.add_to_class("openid", models.URLField(blank=True))
User._meta.admin.fields += (
    ("AmitCom Extensions", { 'fields': ('openid', ) }),
)

class UserExtension(object):
    def _get_comments(self):
        return Comment.objects.get(user=self, is_public=True)
    public_comments = property(_get_comments)
    del _get_comments

User.__bases__ = User.__bases__ + ( UserExtension, )
# }}}

This sounds the best solution to me. All fields, relations and custom methods are on User model and no multiple Models to keep track of.

PS: If you are curios about all the "# {{{" and "# }}}" in the code above, they are code folding markers, learn about code folding for python in vim here

Labels: Python Programming Invented Here Django


Django Master Class

If I have to call a piece of documentation: "mind blowing", this has to be it. You do any kind of web programming, you must read this. If you do django, you will know what all you can do, if not you will know what all can be done in a thought-out framework, and start considering switching or implementing part of it in your own. Read it now.

Labels: Python Programming Django Tips n Tricks


Monday, July 23, 2007

Reverse Pagination

You are all aware of object pagination, search results, your photo on flickr, stories on reddit, all have a next page/previous page paradigm. Django makes it trivially easy to create such pages by providing object_list generic view. There are some problems with the current implementation of pagination that we see around, page no 0/1 is assigned to the latest objects in the list. While in search result this makes little difference, in other cases it has a few consequences.

In reddit for example, you are on the main page, you see 25 stories there, you take 10 minutes to go through all of them, and click next. There is a good chance that 3-5 stories on the first page would have moved to now second page, and you will see them again. Not a good experience, but acceptable. You spend next 3-4 hours working, and click back to see the stories on the first page, you are taken to page 0, and there are good chances you have missed 12 stories that moved from first page to second, as your current page labelled second page became third. This is all quite confusing if you think about it. [I am assuming from this discussion the stories changing their relative rankings for simplicity, you can take example of flickr group photos, they also change rapidly, and order does not change there].

Another problem is, I am page on 26 of this flickr group, and I see some fellow has posted 8 nice photos to the group. I bookmark the page, and come back later/email it to a friend, and by the time the page is visited again, 100s of new photos has been added and the content of page 26 is now on page 29 or so, and I don't find what I was looking for.

The last consequence of this is caching difficulty. If a group has 5 thousand pages worth 30 photos each, and one more photo gets added, either the page number of the photos in each page will change for 5 thousand of those pages. This will happen on each photo being added, and there for the page can hardly be ever cached.

I propose a solution to this problem, I call it reverse pagination, and this blog is currently using a patched django to demonstrate it. In reverse pagination, page no 0/1 is assigned to the older page ever. When on reddit home page, and click next, you will not go to page 2, you will jump to page 20566 or something like that. The content of page 20566 will never change, only the content of latest page would be changing while new items are being added. This means all pages other than the main page can be cached for the rest of the life span of the website. And user will not face the other two problems I listed above.

Only downside is on the main page/latest page, you will upto 19 items if you paginate by 10 items per page.

Here is the patch for django. Enjoy!

Labels: Python Programming Invented Here Django


Friday, July 13, 2007

Featured Django Project: i18ndynamic

I am a huge fan of django. And after their recent unicode improvements, they have become really good option of i18n enabled sites, or sites that are available in multiple languages. So now the interface can be in more than one language, django will select the default website language based on browser settings, and allows user to select their language preferences if they want. Only missing piece was translating data. While django's translation utilities support translating data in database, but something was missing, the capability to for example, store description of a book, for example, in different languages. i18ndynamic is exactly what was needed.

Labels: Python Programming Django


Saturday, June 23, 2007

wikidBASE = Wiki X Database

Came across this interesting project, wikidBASE.

Wikidbase is a powerful and highly-flexible combination of two structural extremes of data management systems: a wiki and a database. As such, wikidbase has:

  • all of the flexibility of a wiki (e.g. any kind of unstructured data can be stored)
  • and the structural data capabilities of a database (e.g. data may be modelled similarly to database fields, tables and relations such that structural reports can me made of that data).

The functionality of a wiki and a database is combined in such a way that this general-purpose data management system may be shaped easily (without the need of expensive database experts) into a custom data management application (e.g. a contact relation management (CRM) system, a knowledge base, a shared calendar system, a project management system, etc.)

Read the overview for further details.

I have always been a fan of keynote, for my personal note taking, but there are times I wanted to have some way to store structural data. Things like "personal movie/book log" are rather structural, title of the book/movie, date-time I watched/read it, my notes about the book/movie if any, rating? etc. I am still not willing to give up Keyonote's trees and tabs and easy formatting abilities to move to a web based wiki, but this tool has made me think about it.

Labels: Programming Django


Tuesday, June 19, 2007

Django Data-Browse

Data browse was recently added to django. Databrowse is an application that creates browse user interface for your database. You don't have to write any code, just have to register the models/tables you want to allow browsing for. It was not obvious to me how it would look like, so I enabled it on my blog, you can browse this blog's post and categories etc here. As you can see it is pretty comprehensive view of your database. What about django's admin application? Admin is not very suitable for browsing purpose, and is primarily meant for administration only.

Label: Django


Monday, June 11, 2007

Django Image Bundle

One of the most important factor in a users percieved performance of a website is the initial page load time. Initial load time depends on backend performance, time it took for the backend to generate the HTML page, and front end performance, the time it takes for the browser to download and render the HTML and all its dependencies. And it turns out that browsers typically take much longer to download dependencies than the original HTML page, due to browser pipelining. Look at the chart below:

As one can see, most of the time is spent in downloading the images, especially problematic for image heavy sites. Read Yahoo! UI Blog's entries on this topic for futher details.

Google recently released GWT 1.4, and one of the features introduced in this release is ImageBundle. The basic idea is to bundle all the images into one at the server side, and use CSS sprite technique to render them. Inspired by it, I just finished an implementation of "Image Bundling" for Django. Here is a demo of the same. The original template to generate the page is here. Look at the bundled image. And the image bundle template tag library can be downloaded from djangosnippets.com. As an added bonus the size of bundled image is about 70% of the total size of individual images, so one can save both total bandwidth, and number of http requests.

This is still the first cut solution. Both this and Google's ImageBundle face an issue when dealing with images with padding, google suggests to avoid padding, or put a wrapper div around the image and put the padding on it. Another issue is images in CSS. They are to be handled slightly differently, but to still keep the output image count to one, some kind of bundle naming is to be done. Google's approach here is creating bundles and using them as two separate steps, I have tried to combine the two together for greater flexibility.

Labels: Python Programming Google Invented Here Django Tips n Tricks


Sunday, June 10, 2007

labelr: Closure

This is a very old news for all directly affected. About a year and a half back, I started this project to allow category support to blogger hosted blogs. It went well in intial testing, and near the end I had about 300 blogs using it. It was a django powered website, and hosted on dreamhost, and for some reason dreamhost was just not reliable enough for this service. One of the specialty of this service was if it goes down, it takes down all those 300 blogs with it. After a series or reported incidents I was forced to shut it down, for lack of any technical solution, as I did not have time and money to get a new hosting service. So here is an apology for all the people using labelr, for its shutting down abruptly.

The culprit here was my stern requirements, "not a minute downtime", especially difficult for shared hosting services. Dreamhost should not really be faulted, as they do not officially support django, and are ideally suited for PHP based and static websites. They are still my favorite for running mediawiki and keeping backups [they give huge amount of space] and are really cheap. There support staff is usually good, so please do not look at this take down as to indicate too badly on them.

Off late I am using webfaction for hosting my blog, and other django applications. They are really seamless to start, I had the website running in less than 20 minutes since I got the account activated, including all learning, configuration etc. They give you your own instance of http, which is really cool. Trac is much easier to install with webfaction than dreamhost for example. If you are looking for a place to host django website, I highly recomment webfaction, for mediawiki, enormous amount of disk to archieve all your mails, or for running wordpress/mediawiki, go for dreamhost [they have the best backup solution I have seen implemented, snapshots].

I intend to release the source code of labelr, but its right now in terrible shape, may be I will write a few articles about the specific things worth taking away from that work.

So dear labelr users, sorry for the shutdown, and thanks for the enormous amount of support and interest I recieved from you guys, I still have more than thousand unread labelr beta request in my mail box.

And comments are not working as of now, so please mail as of now.

Labels: labelr Invented Here Django


Thursday, June 7, 2007

Blog Reborn

After using Blogger for years, finally moved to home brewed blogging system in django. The biggest challenge was to make sure all links keep on working, and data migration. BeautifulSoup is your friend, if you are thinking of getting data from arbitrary services.

Lots left to do, archive page is giving some problems, comments are not there yet, feed is not working still; but am glad to be back to blogging.

Label: Django


Tuesday, January 31, 2006

Introducing labelr

If you notice carefully, you will see something different in this blog.

Presenting you

: organize your www.blogger.com blog.

Currently under beta as every self respecting web 2.0 application has to be :-P. There still may be some stuff left to debug, so you will have to hold off for the moment, my day job takes lot of time of mine. In the meanwhile if you want to beta test it, please drop a comment here, or mail me.

Enjoy!

Labels: Python Programming Google labelr Invented Here Django Tips n Tricks


Sunday, January 15, 2006

Django Tips

Here are a few bits I discovered while using django in my projects of late:

  • Authentication framework documentation does not mention whats happening behind the scene. To log in a user you can set request.session[users.SESSION_KEY] to user_id of the user. To logout, delete users.SESSION_KEY from request.session. They should mention this in their documentation.
  • If you find login_required decorator enough for your needs, but do not like the /accounts/login/ url, you can edit django/views/auth/login.py and set them what you want. They should make it configurable through setting.py
  • login_required is convenient but django.views.auth.login lacking , if you modify the "/accounts/login/" line as discussed above, to some of your view own you can avoid django.views.auth.login views.
  • render_to_response documentation, well actually this is just a blog post entry and not supposed to be updated as and when required, and they don't mention what is it in the tutorial when they are using it, nor in the right place where they should. Anyways, when using render_to_response I could not find how to user DjangoContext, and giving up such nice shortcut just because of this was a bit sad. Well it so happens that it is supported, use something like render_to_response('template', context_instance=DjangoContext(request))
  • get_absolute_url model method: well the only documented use of this method is in the admin UI, but admin prefers to ignore it, and uses some redirects instead. Well actually they do not ignore it, and redirect takes you to the right place, but for that to work you will have put (r'^r/', include('django.conf.urls.shortcut')), in your URLConf. Not documented.
  • But this may not solve all your problems with admin site taking you to the right place when you click on "View on Site" in admin. You will have to make sure the sites model contains the right site, which includes the domain name and port number, but without http:// part. Not documented.
  • One of the first thing that comes to your mind when you read django's authentication framework, is wow its cool, but what if I need more fields than they support? Well given I just leared the super cool model framework, I am tempted to create a model and define a one-to-one relationship on it with User. Well bad idea. For some reason this won't work, and the preferred way to do such extensions is Subclassing models.

Best django tip other than "use django, it is cool" would be: join mailing lists, esp django-developers, people are very friendly there, and you get prompt help.

Labels: Python Programming Django Tips n Tricks


Sunday, September 25, 2005

SOAPed Django

After XMLRPC, I have added SOAP support for everyone's favourite web framework: django. Yet another thing out of my ToDo list, looks like Gera's advise is working good for me :-).

SOPA implementation is derived[copied] heavily from Server.py from SOAPpy distribution, so API's are the same as well as every configuration setting is done as usual with SOAPpy's Config class. Though it works, I do not like it, for the simple reason that NO SOAP server/client implementation should bypass WSDL for compatibility's sake, which our beloved Microsoft has made the biggest problem in SOAP world. Whats really annoying is PHP's SOAP implementation by PEAR supports WSDL by server but SOPApy does not, [tho the REDAME hints on the contrary]. Obviously type reason as mentioned by Charles Moad doesn't hold as 1) PHP beaing a weakly typed language can support it so why can't python which is strongly typed 2) SOAP implementation has to know about the type or else it can not convert them to SOAP packets, now if they know why can't they varify it against some WSDL? Trying to generate WSDL programmatically is evil, you go in that direction and the first thing you find is WSDL's generated by Visual Studio and your soul cringes trying to make everything compatible with them, bad bad bad. Ruby does it, but I won't like to see it in python world.

Ofcourse all this has little to do with django per se, its just the state of affairs with SOAP and python right now, as far as I can see. Assuming SOAPpy is good, SOAP support for django, available here for now, ticket for adding it to django is pending, is acceptable. Its implemented as a callable view object, and implements the SOAPServerBase for registering functions and instances. Map url to the view and you are good to go.

Labels: Python Programming Invented Here Django


Friday, September 23, 2005

XMLRPC Support For Django

Just completed a long standing entry in my todo, "XMLRPC support for django". View ticket, here is the patch, or just get the file from here, and place it in django/contrib directory, [after renaming it to .py, my server for some weird reasons is not serving .py files].

Enjoy!

Labels: Python Programming Invented Here Django