Dev411 Blog

Whenever popular Perl sites are discussed, a number of large websites are mentioned with the usual suspects being Amazon, IMDB, LiveJournal, Slashdot and others. While this is a good list, what often doesn't get mentioned is the new Web 2.0 sites that Perl 5 is powering. Perl 5 web frameworks (such as Catalyst and Jifty), OO models (Moose) and other techniques (PAR) dramatically improve Perl for web development and large-scale projects making many complaints about the language irrelevant. These, when combined with CPAN, continue to make Perl 5 a very attractive language for new web apps.

Examples of Perl-powered Web 2.0 sites include del.icio.us, (which uses the Mason templating system / mini-framework), EditGrid (an AJAX-enhanced online spreadsheet running on the Catalyst framework) and Vox (a multi-user blogging site by Six Apart, maker of LiveJournal, MovableType and TypePad). Other sites include HiveMinder, IUseThis, and MighTyV. Of these six sites, four are running on the Catalyst framework. HiveMinder is built on the Jifty framework. Yahoo is even interested in developers with Perl/Catalyst skills along with C and Java for this Senior Software Engineering position (screenshot from jobs.perl.org). Screenshots and links of active sites are provided below.

del.icio.us Description: Social Bookmarking Perl Framework: Mason

EditGrid Description: Online Spreadsheets Perl Framework: Catalyst

Bill Odom, President of The Perl Foundation, recently posted a blog entry called TPF RFC. My comment to that blog entry wasn't approved for some reason so I figured I'd post some easy-to-accomplish recommendations here.

Lucene is a Java-based fulltext indexing and search solution run under the Apache Foundation. It is arguably the most popular of many fulltext solutions now but its use of Java makes it a secondary choice for many non-Java projects. To improve the speed of Lucene, there are several C/C++ ports including:

• Lucene4c: Apache Foundation project
• Lucy: Apache Foundation project
• CLucene: Sourceforge project

I've been looking for a program that will take full screen shots of web pages even when the web page is larger than the window size on my physical screen, requiring scrolling. This morning I found such a program in Petr Šmejkal's Win32::CaptureIE when it was mentioned by Displeaser on DevShed Forums in the "Screenshot of webpage" thread. It uses ImageMagick for image manipulation.

The YAPC::NA conference doesn't have a blog badge yet so I edited one from another event. It's not official and just for fun:

You can add it to your blog with the following:

In your HTML:

<div id="yapc_badge">
<a href="http://www.yapcchicago.org/" title="YAPC::NA">
<img src="http://www.dev411.com/images/logos/  \
  badge_yapcna2006.gif" alt="YAPC::NA badge" />
</a>
</div>

In your CSS:

#yapc_badge {
  position: absolute;
  top: 0; /* 10px works for Blogger */
  right: 0;
}

UPDATE 1: This is now on the YAPC::NA Chicago homepage and Planet YAPC Chicago. Show your support :)

UPDATE 2: This is also seen in Jonathan Rockway's Catalyst Weblog Software presentation.

I played with Catalyst::Plugin::I18N today to test returning text according to the user's language. This is called internationalization (I18N) or localization (L10N). I prefer the term localization when customizing output for users for this (and internationalization for storing multi-lingual data in the backend) but the definition is subject to debate and both are used. It seems there are two stages to L10N: locale detection and the localization.

C::P::I18N can do dection based on an explicit setting:

$c->languages( ['de'] );

or automatic browser detection based on the browser's HTTP_ACCEPT_LANGUAGE. C::P::I18N does the localization portion using string substitution. To set this up put your string definitions in the your mo/po or Maketext classes and then localize in the View, for example using TT:

[% c.loc('Hello World') %]

Also make sure to set the Vary header to assist with caching:

# Covers one or more Vary headers
$c->res->headers->push_header(
    Vary => 'Accept-Language'
);

# If you only have one Vary header
$c->res->header( Vary => 'Accept-Language' );

The Vary header should theoretically handle proxy caching though these servers are typically out of your control and cannot be guaranteed to recognize and process the header.

The other popular detection option seems to be to put the locale in the URI, either in the domain name (e.g. http://www.mysite.cn) or in the path / query string, as opposed to reading a HTTP request parameter.

Cal Henderson's Building Scalable Websites O'Reilly book talks about Localization and mentions three methods:

1. string substitution
2. multiple template sets
3. multiple frontends

Cal mentions all three methods are hard and doesn't give an outright recommendation. He does, however, lists problems with string substitution and multple template sets but not multiple frontends so perhaps he's implicitly recommending the latter. He doesn't talk about detection methods. At first I was very excited to get Cal's book because I thought he would make outright recommendations. I'm starting to realize it's more of a primer and you have to tease out the best option.

Do you L10N or I18N? If so, how do you do it?

HTML::Prototype is by far the most painless way to get started with AJAX that I've found. Simply put, you do not need to know or write any JavaScript! HTML::Prototype is a Perl module on CPAN that wraps the prototype AJAX library and Script.aculo.us effects library with Perl helper methods so you don't need to write a single line of JavaScript to get some great effects. There are also a number of modules that wrap HTML::Prototype for integration with Catalyst, CGI::Application, Template Toolkit and others.

There are a number of convenient methods such as link_to_remote() and submit_to_remote() that create a link and form button to make an AJAX call and populate the innerHTML of a specified DOM element with the response body. Syntactic sugar to be sure as the JS generated by HTML::Prototype is very simple once you look at it, but the beauty is that you never have to.

To truly appreciate HTML::Prototype you need to use some of Scriptaculous' more advanced widgets such as autocomplete. Sebastian Riedel put together a screencast using Catalyst that demonstrates how easy it is to use. Links to the screencast are available on the Catalyst wiki movies page:

http://dev.catalyst.perl.org/wiki/Movies

Another great thing about having HTML::Prototype generate the JS syntax for you is that you can learn prototype syntax just by View Source. I picked up enough of prototype's JS syntax this way that I haven't looked at the docs yet.

I've recently removed HTML::Prototype from a project in favor of using prototype.js and scriptaculous.js directly and I'm evaluating the Dojo Toolkit but HTML::Prototype let me get started with very effective, painless AJAX functionality. I used it with Catalyst::Plugin::Prototype and thought 'this is how frameworks make you productive.' Thanks to all the contributors.

In the area of Perl ORM (object relational mappers), there are essentially three players: Class::DBI, DBIx::Class and Rose::DB::Object. Of the three, Rose::DB::Object (a.k.a. RDBO) is known to be the fastest, running just 2-4 times the speed of straight DBI in many tests (albeit using an unloaded database). DBIx::Class is known as the most extensible and flexible. Class::DBI is considered outdated (e.g. it doesn't support JOINs) and to be avoided by many. RDBO is the fastest because speed is it's primary design consideration, implementing many perl and DBI-level optimizations. The design philosophy is described as:

RDBO's implementation accomplishes this feat in several ways:

1. aggressive code inlining: method calls can be very expensive and RDBO goes to great lenghts to avoid this. The get_objects method in Rose::DB::Object:Manager is 2129 lines long.
2. extensive caching of "derived metadata", e.g. commonly used SQL query strings, column lists, etc.
3. extensive use of "the 'fast path' in DBI, such as always using bind_col() and fetch() to retrieve data instead of fetchrow_array(), unbound calls to fetchrow_arrayref(), or fetchrow_hashref()."
4. "compiling down" accessor and mutator methods, both with and without triggers, to an optimized form in order to shorten the code path.

Achieving this high performance has resulted in some compromises. They include:

1. the code is harder to maintain and extend, resulting in a smaller community of contributors.
2. aggregate functions and other queries that do not return table rows (e.g. GROUP BY, HAVING, etc.) are not natively supported yet. A previous entry, Rose::DB::Object - aggregates, describes how to implement aggregates with RDBO.

Perl has a TIMTOWTDI tradition and ORMs are no exception. The primary alternative, DBIx::Class, is designed with extensibility as it's primary consideration and addresses the compromises RDBO has made. The implementation gives up some perl execution speed to RDBO to gain this extensibility while generating roughly equivalent SQL for a similar database load. Both RDBO and DBIC are much faster than Class::DBI. The extensibility philosophy has resulted in an alternative ORM that is more flexible, supports aggregate functions and has a larger community of committers relative to RDBO.

RDBO should be a strong contender if you want the fastest Perl ORM and can accept the design decisions it has made to achieve that performance. If you would like some extra capability and can give up some perl execution speed in your web nodes, it may be worthwhile to evaluate some alternatives.

Catalyst 5.66 introduced the Root controller (e.g. MyApp::Controller::Root) as a best practice. It's purpose is to remove the need for the Application Class (a.k.a. App Class; e.g. MyApp.pm) to have actions or be a controller. Previously, Local top-level actions (e.g. /login) would be put in the App Class; the Root controller replaces this because it's base path is /. I asked mst why the Root controller is important and he told me the following.

Having the Application Class be a controller makes it a controller on a class-based level that can lead to several problems. One problem is name collision between App Class actions and $c methods, for example an App Class login action will cause problems with $c->login used by Catalyst::Plugin::Authentication. This and other problems can now be avoided entirely.

A typical App Class includes:

use Catalyst qw/.../;

This makes the App Class isa Catalyst::Controller because "use Catalyst" automatically injects Catalyst and Catalyst::Controller as base classes if the calling class isn't isa Catalyst. This can be avoided by making the App Class isa Catalyst before calling "use Catalyst" with the following:

use base qw/Catalyst/;
use Catalyst qw/.../;

Once the App Class is no longer isa Catalyst::Controller, it should no longer have any actions or subroutines with attributes, including the auto, default and end subroutines.

The Catalyst helper script will create a MyApp::Controller::Root controller for you, but the name can actually be anything. There's only one line that distinguishes the Root controller from any other one and it's a config setting that sets the name space to ''. You can make any controller the root by simply add the following line, just be sure the action attributes are updated if necessary:

__PACKAGE__->config->{namespace} = '';

Previously, you could avoid actions in the App Class by using Global actions, now you can with Local actions as well.

I use the prototype.js library (aka prototype) for AJAX and recently moved to using JSON to return multiple elements instead of a single one. I had to rearchitect my client request code and my server code to handle JSON but it's worth it to minimize the number of HTTP requests. I use Catalyst and JSON::Syck which make generating the JSON response on the server-side very easy.

Prototype.js is interesting in that it auto-evals a JSON object that's placed in the X-JSON header instead of the response body like other libaries such as Dojo Toolkit. This allows you to send HTML in the response body but I'm not convinced this is a good implementation since it supports only one HTML fragment in the response body. I think it would be better to simply put the JSON object in the response body and have all the HTML fragments in the JSON object. I just seems cleaner to put all the HTML fragments in JSON instead of fragment 1 in the response body and fragments 2-n in JSON. Although I have the X-JSON header working, I may just move to the more standard JSON in response body approach and eval the JSON string myself. To read the auto-evaled JSON response, the following example expects a JSON associative array and displays the value of the myItem key (more code is available in the wiki link below):

onComplete: function( request, json ) {
    alert( json.myItem );
}

On the server-side, I use the Catalyst framework to set the header and JSON::Syck to transform Perl data structures to JSON. JSON::Syck is a wrapper around the C libsyck library and very fast. There is a view, Catalyst::View::JSON, however it puts the JSON string in the response body because that is what Dojo and other libraries expect. Since AJAX libraries don't idnetify themselves in anyway to the server (e.g. user agent), the Catalyst View would probably need a patch to with a configuration switch to populate X-JSON instead of the response body. For now, I'm using JSON::Syck directly. The following are the minimal calls to use. If you have UTF-8 output, see C::V::JSON for UTF-8 encoding.

$c->res->header(
    'X-JSON' => JSON::Syck::Dump( \%args );
);

I've put together some example code that I thought would be better placed on the wiki. It can be found at: Prototype.js, JSON and Catalyst.

UPDATE: After running into the IE max header length issue, I've decided to put the JSON object in the response body and manually eval request.responseText on the client.