Dev411 Blog

I played with Catalyst::Plugin::I18N today to test returning text according to the user's language. This is called internationalization (I18N) or localization (L10N). I prefer the term localization when customizing output for users for this (and internationalization for storing multi-lingual data in the backend) but the definition is subject to debate and both are used. It seems there are two stages to L10N: locale detection and the localization.

C::P::I18N can do dection based on an explicit setting:

$c->languages( ['de'] );

or automatic browser detection based on the browser's HTTP_ACCEPT_LANGUAGE. C::P::I18N does the localization portion using string substitution. To set this up put your string definitions in the your mo/po or Maketext classes and then localize in the View, for example using TT:

[% c.loc('Hello World') %]

Also make sure to set the Vary header to assist with caching:

# Covers one or more Vary headers
$c->res->headers->push_header(
    Vary => 'Accept-Language'
);

# If you only have one Vary header
$c->res->header( Vary => 'Accept-Language' );

The Vary header should theoretically handle proxy caching though these servers are typically out of your control and cannot be guaranteed to recognize and process the header.

The other popular detection option seems to be to put the locale in the URI, either in the domain name (e.g. http://www.mysite.cn) or in the path / query string, as opposed to reading a HTTP request parameter.

Cal Henderson's Building Scalable Websites O'Reilly book talks about Localization and mentions three methods:

1. string substitution
2. multiple template sets
3. multiple frontends

Cal mentions all three methods are hard and doesn't give an outright recommendation. He does, however, lists problems with string substitution and multple template sets but not multiple frontends so perhaps he's implicitly recommending the latter. He doesn't talk about detection methods. At first I was very excited to get Cal's book because I thought he would make outright recommendations. I'm starting to realize it's more of a primer and you have to tease out the best option.

Do you L10N or I18N? If so, how do you do it?

Over the weekend I got to use Python for the first time to add channel categories to Planet which can be seen in action at Planet MVC. My first impressions of Python is that the resulting code looks very clean. The syntax was also easy to pick up primarily due to the excellent docs site and because it reminded me of JavaScript.

The first problem I ran into was using the regex object's re.match which definately does not follow the Python philosophy of "There should be one -- and preferably only one -- obvious way to do it" because there's more than one way and re.match is not obvious because it works different than other languages. Python's re.match automatically starts from the begining of the string unlike regex engines for other languages (grep|sed|awk|perl|ruby). This results in the following behavior:

# matches - pattern is at start of string
m = re.match('b', 'bc')

# does not match - pattern is not at start of string
m = re.match('b', 'abc')

# matches - add the .* to match any leading characters
# but .* isn't needed after '.*b' to match 'c'
m = re.match('.*b', 'abc')

# matches - second .* matches 'c' but isn't needed
m = re.match('.*b.*', 'abc')

I was sticking to re.match because I needed the MatchObject to capture groups but it didn't DWIW. After posting to a forum, I learned that I should use re.search instead which will match anywhere in the string. re.search also returns a MatchObject but I didn't expect this at first because search != match. I get the impression re.match and MatchObject were created first and re.search was added later. Perhaps the MatchObject should be called something generic, like ResultObject, in the docs. I'm sort of curious but when is re.match desired over re.search? It seems like re.match is useful only in a limited number of situations and then it's always easy to implement it with re.search.

The open issue for me is Python's dependency on indenting. Lately I've been using tabs exclusively for indentation because I like to see the tickmarks may editors show for tabs. This lets me quickly know what level I'm at. This python-list thread was titled Tab Wars and argued in favor of spaces over tabs. Planet is indented with spaces and that's what I used for my hacking. Given my preference for tabs, the more time I spend with Python the more likely I'll switch to tabs. Do I need to convert the entire script to tabs or can I mix tabs with spaces? What do people with a preference for tabs do with Python code that uses spaces?

I've been thinking about releasing my enhanced version of PlanetPlanet which allows one planet site to run multiple planets but now I'm thinking I want to do more enhancements to make it even more user friendly. Let me know how Planet MVC is working for you or if I can add any additional blogs. I've already found it pretty useful to find out about things I otherwise wouldn't have.

Here are some notes on getting Planet running.

I just wanted to report and let everyone know with Prototype 1.5.x, Dojo Toolkit and Prototype should now be interoperable. Earlier versions of prototype.js extended the JavaScript Object prototype that would break third-party JavaScript but this is no longer being done. A few people, including myself, have been using them together without any problems. The prototype website still offers 1.4.0 as the current version, however 1.5.0 rc0 is included with Scriptaculous 1.6.1. I'm currently using prototype 1.5.0 rc0 with Dojo 0.3.1 and Scriptaculous 1.5 rc4. Although Scriptaculous 1.6.1 is available, it breaks autocomplete for me and I haven't had time to track down what changed yet. Hopefully soon.

UPDATE: The Scriptaculous 1.6.1 issue has been reported to RoR as ticket #5673. This has since been fixed with Scriptaculous 1.6.2. I'm now using Dojo 0.3.1, Prototype 1.5.0 rc0 and Scriptaculous 1.6.2.

UPDATE 2: When running dojo.js and scriptaculous.js together, make sure you load dojo.js first. Loading scriptaculous.js first will cause Dojo to fail.

HTML::Prototype is by far the most painless way to get started with AJAX that I've found. Simply put, you do not need to know or write any JavaScript! HTML::Prototype is a Perl module on CPAN that wraps the prototype AJAX library and Script.aculo.us effects library with Perl helper methods so you don't need to write a single line of JavaScript to get some great effects. There are also a number of modules that wrap HTML::Prototype for integration with Catalyst, CGI::Application, Template Toolkit and others.

There are a number of convenient methods such as link_to_remote() and submit_to_remote() that create a link and form button to make an AJAX call and populate the innerHTML of a specified DOM element with the response body. Syntactic sugar to be sure as the JS generated by HTML::Prototype is very simple once you look at it, but the beauty is that you never have to.

To truly appreciate HTML::Prototype you need to use some of Scriptaculous' more advanced widgets such as autocomplete. Sebastian Riedel put together a screencast using Catalyst that demonstrates how easy it is to use. Links to the screencast are available on the Catalyst wiki movies page:

http://dev.catalyst.perl.org/wiki/Movies

Another great thing about having HTML::Prototype generate the JS syntax for you is that you can learn prototype syntax just by View Source. I picked up enough of prototype's JS syntax this way that I haven't looked at the docs yet.

I've recently removed HTML::Prototype from a project in favor of using prototype.js and scriptaculous.js directly and I'm evaluating the Dojo Toolkit but HTML::Prototype let me get started with very effective, painless AJAX functionality. I used it with Catalyst::Plugin::Prototype and thought 'this is how frameworks make you productive.' Thanks to all the contributors.

IMO, one of the major limitations of Ruby on Rails compared to other frameworks is its ORM, ActiveRecord. ActiveRecord is a fairly early ORM (object-relational mapper) that has made some questionable design decisions and doesn't support some very basic relational database concepts. These issues have been discussed on Joel on Software and elsewhere. Here are some limitations I wish were fixed:

I've been happily developing with Prototype and Scriptaculous for a while now but was recently convinced to look at Dojo. To start, I was pretty happy with dojo.io.bind but disappointed that I wasn't able to get the Dojo scripts to load from a remote server. Having gotten dojo.io.bind to work, I was off to test my first widget, Dojo's ComboBox, which offers functionality similar to the Scriptaculous autocomplete widget. I've been using Dojo 0.3.0. The Dojo ComboBox Nightly Test is often mentioned as a good place to start so I went there first. Eventually I also spent some time reading and implementing the code at Java.net's AJAX with Dojo and JSON page. In the end I came away disappointed due to several bugs and strange behaviors listed below:

1. Default value: Dojo does not let you set a default value via the HTML or any other way. To get a default value to show up, I've heard you can write it in with JavaScript but I'm not sure where or when yet due to the DOM onLoad transform issue described below.
2. Single option bug: when the a resulting list only has one item, hitting the 'enter' key will not close the list. With autocomplete, the single entry will already be populated in the box but not selectable with 'enter'. You need to select the single option in the list with the down key and then hit the 'enter' key. To try this out, type in 'am' for American Samoa in the first input on the nightly test and click 'enter' to continue. I haven't found out whether this is being worked on or not.
3. DOM onLoad transform: Unlike other libraries, after the page has been rendered Dojo will go back and transform the DOM, replacing dojoTypes with HTML templates. This can cause some timing problems when one wants to use DOM to modify the resulting HTML templates. Dojo provides a dojo.addOnLoad capability to specify a callback function however the new DOM elements don't seem to be available yet. Perhaps something to the effect of dojo.afterDojoOnLoad is needed? This issue can be seen when trying to remove the ComboBox downArrowNode. The ComboBox comes with a down arrow graphic by default and is assigned dojoattachpoint="downArrowNode". With some lists, the reply possibilities are enormous and a default list isn't desirable. When nothing has been typed, it may also give the impression that the list provided is a complete list like a standard select element. I haven't found a good way to disable this yet. Setting the display property on the downArrowNode isn't a good solution because it only works after the page has been rendered. It can't be set immediately rendering the ComboBox input tag because Dojo hasn't been run yet. Setting the display property in the dojo.addOnLoad callback is also too early, i.e. the widget doesn't exist yet. Here's the JS I've been working with:

   var ac = dojo.widget.byId("ac");
   ac.downArrowNode.style.display="none";

   I've also tried setting downArrow and downArrowNode attributes to 0, -1 and false. So far, the only solution I found that works is to delete or comment out the downArrowNode HTML in the following file:

   dojo/src/widget/templates/HtmlComboBox.html

   Not exactly a great solution. Is there a better way to remove the arrow?
4. Client-side result filtering: It seems like the Dojo javascript does it's own filtering. You give it a list in the format of an Array of Arrays where each inner array has two elements. Dojo then applies it's own filtering. This is nice if you are working off a local list such as a JS file in the Nightly Test example but if you are retrieving a list from a server, the server should have filtered the list already so there's no reason to filter it a second time. This may be partially at fault for the choppy rendering. Is there any way to disable this?
5. Aribitrary location match: Because Dojo does client-side result filtering, the user isn't guaranteed to see what your server delivers. One area this shows up is that it's not clear how to match any part of the string instead of just the beginning. On the nightly test page, the only results presented are ones where the input matches the start of the string so if someone types a last name and the strings have the format "$first_name $last_name" there would not be a match. The DocTool on dojo.org will match inside the string but the rendering seems choppy and I can't seem to locate it any more. I haven't gotten around to seeing how the DocTool does it. The SVN copy of the doctool.html file shows the following which does not result in the desired non-start match for me:

   <input
       dojoType="combobox"
       id="search"
       autoComplete="false"
       value=""
       style="width: 300px;"
   >

6. HTML result rendering: Often it's nice to show the user which part of the string matched, especially if the match is not at the beginning of the string. This is done by bolding the matching letters a la Gmail. If HTML is included in the array, it is rendered literally instead of interpreted as HTML. Because Dojo does client-side filtering, HTML mark up will also cause matching to fail. With Scriptaculous, you can add HTML mark-up to the results and it will render properly. Perhaps with Dojo, the HTML needs to be added by client-side JavaScript?

These are my initial observations of Dojo's ComboBox. It seems very difficult to get it to DWIW and very different than the Scriptaculous Autocomplete which just works and has been working. Scriptaculous autocomplete has been working like champ for me since prototype 1.3.1 and it's now on 1.5.0 rc0. Please let me know of any ways to accomplish the above Dojo ComboBox issues. Also let me know if you're using Dojo's ComboBox in production and / or your experiences with it.

I recently looked into Dojo Toolkit 0.3.0's dojo.io.bind to perform AJAX requests because of it's good handling of memory leaks. I checked out the dojotoolkit.org intro page which was okay but didn't cover how to pass parameters or how to process a JSON response so I thought I'd put some info together here. Here are a few things worth knowing:

1. A dojo response has a type that is typically set to load or error, though sometimes it may be neither. With dojo, type=="load" means success.
2. Dojo.io.bind has a few handlers to process the response. The load handler is executed when the response type=="load" and the error handler is executed when the response type=="error". There is also a catch-all handle handler that can accept all response types so you can process a response that has a type that's neither load nor error.
3. If request parameters are set as a hashref using content, dojo.io.bind will automatically turn them into form post parameters
4. The response mimetype (or content-type) is set in the client JavaScript. Dojo uses this to determine how to process the response. Dojo ignores content-types set by the server

Here are two examples to help put the above together. The examples cover a plain HTML fragment response as well as a JSON response. Both examples submit a hashref. Dojo can also submit an entire form automatically. This is covered on the Dojo.IO Intro link above and isn't covered here, for now. Thanks to Toby Corkindale who posted his saveHandler function to the catalyst mailing list.

Example 1: HTML processed by load
This first example retrieves a single fragment of HTML in the response body and processes it using the load handler.

dojo.io.bind({
    url: "/binduri",
    method: "post",
    /* optional content. If the content is in the
     * form of a hashref they are converted to
     * post paramters */
    content: {
        username: "George",
        password: "curious"
    },
    load: function(type,data,evt) {
        if (data) alert('Response Data: '+data);
    },
    mimetype:'text/html'
});

Example 2: JSON processed by handle
This second example processes a JSON response using the all-in-one handle handler. The handle is typically used in special situations where load and error are not appropriate such as a progress bar. It is shown here just to provide the syntax. To process a JSON response, set the dojo.io.bind mimetype to text/json and return the JSON object in the response body.

dojo.io.bind({
    url: "/binduri",
    method: "post",
    /* optional content. If the content is in the
     * form of a hashref they are converted to
     * post paramters */
    content: {
        username: "George",
        password: "curious"
    },
    handle: function(type,data,evt) {
        if (type=='load') {
            alert( 'Successful response' );
            if (data.myItem)
                alert('Response JSON Item: '
                +data.myItem);
        } else if (type=='error') {
            alert('A error has occurred');
        } else {
            alert('Some other event has occurred');
        }
    },
    mimetype:'text/json'
});

Conclusion
Dojo.io has a nice interface that has a few conveniences including auto-conversion of a hashref to request form parameters and auto-eval of JSON set in the response body. Along with the good memory leak handling for IE, a very nice implementation.

Dojo Toolkit 0.3.0 seems to have a problem loading it's javascript libraries remotely, i.e. when the Dojo scripts are hosted on a server different than where the HTML is. For example, if the page http://www.mysite.com/index.html includes http://www.mystatic.com/dojo/dojo.js. The main Dojo file, dojo.js, runs fine when loaded remotely, however the following fatal errors occur when it attempts to load its external libraries:

FATAL: Could not load 'dojo.io';
   last tried '__package__.js'
FATAL: Could not load 'dojo.widget.ComboBox';
   last tried '__package__.js'

It was suggested this could be due to XSS protections. Scriptaculous' scriptaculous.js, however, has a way around this since it can be loaded remotely and access it's external files (builder.js, controls.js, etc.). Often it's nice to have static files hosted on a separate server. Is there a way to fix this issue for Dojo?

UPDATE 1: For now, I'm going to stick with dojo.io.bind and Scriptaculous autocomplete. A nice thing about this set up is I can load a remote dojo.js browserio build w/o issues since it doesn't need call any external libraries for just bind.

UPDATE 2: Alex Russell provided a uri for cross domain package loading. You can turn on cross domain support by using a custom build and setting the dojoLoader option to xdomain option:

ant -Dprofile=ajax -DdojoLoader=xdomain
  -Ddocless=true clean release intern-strings

If you don't want to build it yourself, James Burke has done it and made it available as dojo-0.3.1-ajax-xd.tar.gz. This page has some more info.

A while back the Catalyst framework open source community was working on a mascot for their project. The idea was to have a flying camel made up of Lego-like building blocks. This would convey key characteristics of the project including (a) Catalyst would make your project fly, (b) Catalyst is very modular and (c) Catalyst is written in Perl. There was a lot of excitement and Tim Gould was kind enough to make the following camel for the group:

Unfortunately, the idea did not get the approval of O'Reilly Media, Inc. which holds a trademark on the camel image. Their policies are described on their Perl Camel Usage and Trademark Information page. Tim O'Reilly mentioned their Perl Camel policies in his response to the recent "Web 2.0" service mark controversy which caused me to wonder if O'Reilly Media, Inc.'s commercial trademark policies are in the spirit of open source.

In the open source world, you can typically take someone's copyrighted information and modify with attribution, especially for a non-commercial use. If that standard were applied to logos, especially mascot-type logos, then it would seem that the spirit of open source is in line with modifying and giving attribution for mascot logos as well. Two popular mascots that have been altered and published include Tux for Linux and Mozilla originally for Netscape. These alterations have arguably made Tux and Mozilla even more popular and recognizable, thus strengthening the brand, than they would otherwise be. Unlike Netscape and Linux, O'Reilly has chosen to use a strong hand in preventing modification of the Camel logo for use by open source projects such as the Catalyst. Is O'Reilly's policy too stringent for the open source community or acceptable business practices?

In the area of Perl ORM (object relational mappers), there are essentially three players: Class::DBI, DBIx::Class and Rose::DB::Object. Of the three, Rose::DB::Object (a.k.a. RDBO) is known to be the fastest, running just 2-4 times the speed of straight DBI in many tests (albeit using an unloaded database). DBIx::Class is known as the most extensible and flexible. Class::DBI is considered outdated (e.g. it doesn't support JOINs) and to be avoided by many. RDBO is the fastest because speed is it's primary design consideration, implementing many perl and DBI-level optimizations. The design philosophy is described as:

RDBO's implementation accomplishes this feat in several ways:

1. aggressive code inlining: method calls can be very expensive and RDBO goes to great lenghts to avoid this. The get_objects method in Rose::DB::Object:Manager is 2129 lines long.
2. extensive caching of "derived metadata", e.g. commonly used SQL query strings, column lists, etc.
3. extensive use of "the 'fast path' in DBI, such as always using bind_col() and fetch() to retrieve data instead of fetchrow_array(), unbound calls to fetchrow_arrayref(), or fetchrow_hashref()."
4. "compiling down" accessor and mutator methods, both with and without triggers, to an optimized form in order to shorten the code path.

Achieving this high performance has resulted in some compromises. They include:

1. the code is harder to maintain and extend, resulting in a smaller community of contributors.
2. aggregate functions and other queries that do not return table rows (e.g. GROUP BY, HAVING, etc.) are not natively supported yet. A previous entry, Rose::DB::Object - aggregates, describes how to implement aggregates with RDBO.

Perl has a TIMTOWTDI tradition and ORMs are no exception. The primary alternative, DBIx::Class, is designed with extensibility as it's primary consideration and addresses the compromises RDBO has made. The implementation gives up some perl execution speed to RDBO to gain this extensibility while generating roughly equivalent SQL for a similar database load. Both RDBO and DBIC are much faster than Class::DBI. The extensibility philosophy has resulted in an alternative ORM that is more flexible, supports aggregate functions and has a larger community of committers relative to RDBO.

RDBO should be a strong contender if you want the fastest Perl ORM and can accept the design decisions it has made to achieve that performance. If you would like some extra capability and can give up some perl execution speed in your web nodes, it may be worthwhile to evaluate some alternatives.