Dev411 Blog: Category perl

Comparing CPAN Modules with YUI DataTable

John Wang — Fri, 09 Nov 2007 00:09:00 -0600

There is a lot of choice on the CPAN for open source Perl libraries and sometimes it's difficult to get an idea of how modules compare to each other. CPAN Ratings is a good source of reviews but it's not convenient to compare one module with another. To provide a partial solution, I whipped up a quick CPAN Compare page which will pull the CPAN Ratings from a number of modules and summarize them for you.

I decided to use YUI DataTable for this. I've heard good things about YUI so I decided to give it a try. Getting the example code to work off of the Yahoo website almost as straight forward as say using Scriptaculous demos but it was faster than working with Dojo in the early days. The nice thing about the DataTable is that it takes a JavaScript array which can be populated using server-side JSON generated code. I used JSON::XS for this.

YUI DataTable has a nice sorting feature and it can sort on text, numbers, dates, etc. However, it does not seem to be able to sort on visual information only so if you include HTML markup, that will be used for sorting as well. To get around this I used standard text sorting and customized the title fields to assist in the sorting. For example, in a link, I start with <a title=" instead of <a href=" because title is arbitrary and can be used to mirror the InnerHTML. For numbers a text sort will have 25 come before 4 so I added leading zeros to numbers using sprintf and put them in the title attribute as well.

A few Perl modules and the Logo Creator website made this easy to set up. YUI DataTable has a nice default CSS so I just left that as is.

Note: YUI DataTable is convenient if you just drop in a Perl data structure and have it generate the HTML and JS for you. This script uses 3 DataTables (ratings, popular and recent) so I wrote a Perl wrapper for YUI which takes a hashref and generates the client code, extracting the fields from the column definitions. This works because YUI does not require the HTML table to be built beforehand. By not having an underlying table, it's faster to get running but also won't fallback as nicely for people who aren't running JS (8% of users?). As an alternative, jQuery has a couple of add ons which work by enhancing an existing HTML table. jQuery has some nice syntax but I haven't gotten around to using it yet. Perhaps it's worth a look.

Database Abstraction - code vs infrastructure

John Wang — Tue, 04 Sep 2007 23:38:00 -0500

I've worked on a number of database-driven projects and no matter how much people want database abstraction, it was always difficult to code and maintain. I was recently reminded of this when I read this Drupal article on dropping PostgreSQL support. Not only can it be difficult to maintain support for multiple databases, but it may be difficult to find developers.

One solution of modern programming is to move database abstraction from the code to the infrastructure using a ORM (Object-Relational Mapper) or Data Mapper. A ORM and Data Mapper abstracts the database for you so you no longer have to do tie db abstraction to each app. Not only does it let you code once for multiple databases it lets your users migrate their data from one database to another. This blog runs Typo which is based on Ruby on Rails and ActiveRecord. I've been contemplating migrating Typo from MySQL to PostgreSQL and I've been told that it would be as simple as exporting the data with YAML, updating the database.yml file and importing the data. I haven't gotten around to doing it yet but it is a powerful idea. ActiveRecord is a data mapper and isn't as flexible as a full blown ORM but it gets the job done for the most part. For a full-blown ORM, I think of Perl's DBIx::Class which provides a full OO interface to the RDBMS allowing you to code just once for multiple DBs without limiting you when you want to use some esoteric database-specific SQL. DBIx::Class is often used with the Catalyst Framework but is also used by itself.

There are PHP frameworks out there like Symfony and Cake but do any of them have stand-alone ORMs? If so, could Drupal move to something like that and solve their maintainership problems once and for all? Drupal is part of the Go PHP5 effort so there should be no issue using PHP 5 OO. Something to think about for the Drupal folks if a PHP ORM is available.

Perl, MySQL and UTF-8

John Wang — Mon, 02 Oct 2006 10:35:00 -0500

One of the mysteries of Perl to me is that why, as of yet, is there no UTF-8 support in DBD::mysql although this issue has been discussed on the msql-mysql-modules list since at least 2003 (using the MARC archives). This is also given that MySQL does have UTF-8 support itself.

When I first looked into this I found a few articles on this:

Pedro's article mentions that the reason this hasn't been done for DBD::mysql is that the DBI and DBD::mysql folks cannot decide where to put UTF-8 implementation, i.e. in DBI itself or the DBD drivers. Because, there is still no built-in support. To get around this, there have been numerous patches produced. Andrew Forrest even put together UTF-8 versions of DBI and CGI.pm (link seems broken atm). However, some of these patches seem to have problems and are non-standard.

If you prefer to use an ORM, DBIx::Class and Class::DBI get around this by implementing UTF-8 support in their own libraries with DBIx::Class::UTF8Columns and Class::DBI::utf8 respectively. I'd recommend DBIx::Class over Class::DBI since it has more functionality (e.g. built-in JOIN support) and is supposed to generate more efficient SQL.

The intersting thing is that DBD::Pg for PostgreSQL has had built-in UTF-8 support for some time. While not an issue specific to the MySQL database, the UTF-8 perl driver issue is something to consider when choosing MySQL or PostgreSQL.

Update: Thanks to Dominic Mitchell for mentioning the latest developer release, DBD::mysql 3.0007_1 released on 8 Sep 2006, has integrated UTF-8 support. It's a developer release but good things are finally happening!

Encoding Hashed UIDs: Base64 vs. Hex vs. Base32

John Wang — Mon, 02 Oct 2006 03:08:00 -0500

I recently looked at using various encodings for hashed UIDs, e.g. UIDs generated by a crytographic hash algorithm such as SHA-1 or MD5. These are often useful when the UID does not need to have human meaning but should exhibit some uniformity, such as character set and length. I considered Base64 and hexadecimal first because they are commonly used by crypto libraries and then decided on Base64 and Base32 where appropriate. Base36 is actually the most compact case insensitive encoding (using Arabic numbers and Roman letters) but is not an option for me at the moment because there's no Perl module for it that will take arbitrary text and binary input at the moment. Math::Base36 exists but only handles numbers.

Base64 has the advantage of generating shorter representations because it uses a 64 character set instead of hexadecimal's 16. While Base 64 can use any 64 characters, many implementations follow IETF PEM RFC 1421 which specifies A-Z, a-z, 0-9, / and + while using = for padding. Because + and / need to be URI escaped, Wikipedia mentions a variant that uses * and - in place of + and / respectively and removes line breaks as well as padding. I used this for a while but unfortunate * cannot be used in DOM ids. Because of this I've switched to using _- instead.
Hexadecimal uses 0-9 and A-F and is case insensitive. The representations are longer due to the smaller character set, but case insensitivy can be advantageous in certain situations such as full text indexing with Xapian which is case sensitive, and MySQL which is case insensitive by default. It's common practices to lowercase all the indexed terms using Xapian while reserving uppercase characters for boolean and probabilistic term prefixes. After thinking about how the query parser works for a bit, my conclusion is that lower casing all the inexed terms will make life with Xapian a lot easier. To enable case sensitivity in MySQL, the table definition needs to be changed.
Base32 is the middle ground between the 16 character hex and 64 character Base64 encodings, but does not get included in crypto libraries. For UIDs, however, it is especially attractive because it uses a larger character set than hexadecimal resulting in shorter representations and is case insensitive for applications that work better with it. Base32, as defined in IETF RFC 3548, uses the characters A-Z and 2-7. Since Base32 isn't included in many crypto libraries, an extra encoding is needed, however, creating hashed UIDs only happens once per entry so it seems reasonable.

There are some modules on CPAN for Base32 including MIME::Base32 and Convert::Base32. For Base32, I'm currently using a patched copy of MIME::Base32 because the CPAN version only handles uppercase. For Base64, I'm using _ and - as the two non-alphanumeric characters.

Perl - Strictify utf8 to UTF-8

John Wang — Fri, 29 Sep 2006 13:21:00 -0500

Perl has two UTF-8 encodings, utf8 which is Perl's liberal version and UTF-8 which is a strict interpretation, aka utf-8-strict. The liberal version allows for encoded characters outside the UTF-8 character set, however you can run into problems when interoperating with applications that expect utf-8-strict, such as PostgreSQL. Here's a function I wrote to strictify utf8 to UTF-8 using the Encode core module:

use Encode;

sub strictify_utf8 {
    my $data = shift;
    if (Encode::is_utf8($data) && !Encode::is_utf8($data,1)) {
        Encode::_utf8_off($data);
        Encode::from_to($data, 'utf8', 'UTF-8');
        Encode::_utf8_on($data);
    }
    return $data;
}

Perl - Getting a Unicode Character's Hex Codepoint

John Wang — Fri, 29 Sep 2006 13:00:00 -0500

I recently responded to someone asking how to get a Unicode hex codepoint from a Unicode literal on DevShed Forums. Since I think it may be more generally useful, here's my solution. The following function takes a unicode literal, converts it to a decimal representation using unpack and then converts it to hex usning sprintf:

sub codepoint_hex {
    if (my $char = shift) {
        return sprintf '%2.2x', unpack('U0U*', $char);
    }
}

my $cp = codepoint_hex('カ'); # eq '30ab'

Using unpack is a nice solution here because the U0 option will require the literal to be strict UTF-8, aka utf-8-strict and not Perl's liberal utf8 version. It throws a warning if the literal is not valid UTF-8. The other nice thing about using unpack is it works whether the UTF-8 flag is on or not, removing one check.

An alternate way to get the decimal representation from the literal is to use ord instead of unpack, but ord requires the UTF-8 flag to be on. This can be turned on with Encode::_utf_on but that just turns the flag on without checking for strict UTF-8 so you'll have to do that with another check.

Update: miyagawa just showed me how to use ord with Encode:

use Encode;

sub codepoint_hex {
    sprintf "%04x", ord Encode::decode("UTF-8", shift);
}

Perl 5 - Powering Web 2.0

John Wang — Tue, 05 Sep 2006 10:07:00 -0500

Whenever popular Perl sites are discussed, a number of large websites are mentioned with the usual suspects being Amazon, IMDB, LiveJournal, Slashdot and others. While this is a good list, what often doesn't get mentioned is the new Web 2.0 sites that Perl 5 is powering. Perl 5 web frameworks (such as Catalyst and Jifty), OO models (Moose) and other techniques (PAR) dramatically improve Perl for web development and large-scale projects making many complaints about the language irrelevant. These, when combined with CPAN, continue to make Perl 5 a very attractive language for new web apps.

Examples of Perl-powered Web 2.0 sites include del.icio.us, (which uses the Mason templating system / mini-framework), EditGrid (an AJAX-enhanced online spreadsheet running on the Catalyst framework) and Vox (a multi-user blogging site by Six Apart, maker of LiveJournal, MovableType and TypePad). Other sites include HiveMinder, IUseThis, and MighTyV. Of these six sites, four are running on the Catalyst framework. HiveMinder is built on the Jifty framework. Yahoo is even interested in developers with Perl/Catalyst skills along with C and Java for this Senior Software Engineering position (screenshot from jobs.perl.org). Screenshots and links of active sites are provided below.

del.icio.us
Description: Social Bookmarking
Perl Framework: Mason

EditGrid
Description: Online Spreadsheets
Perl Framework: Catalyst

HiveMinder
Description: To Do Lists
Perl Framework: Jifty

IUseThis
Description: App Lists
Perl Framework: Catalyst

MighTyV
Description: TV/Radio Listings
Perl Framework: Catalyst

Vox
Description: Blogging site by Six Apart
Perl Framework: Catalyst

These sites show that Perl 5 is alive and well as a language of choice for Web 2.0. Although Perl 5 is over 10 years old, the combination of new frameworks and techniques, combined with CPAN's 10,000+ modules, continue to make Perl an attractive choice. The key is to use make sure you are familiar with modern Perl programming techniques.

Recommendations for The Perl Foundation

John Wang — Sat, 26 Aug 2006 21:28:00 -0500

Bill Odom, President of The Perl Foundation, recently posted a blog entry called TPF RFC. My comment to that blog entry wasn't approved for some reason so I figured I'd post some easy-to-accomplish recommendations here.

Community events calendar: Perl should adopt something like PHP.net's events calendar sidebar for the perl.org and pm.org sites. There is a lot of activity in the Perl community including YAPC conferences, Perl Monger user group meetings (often monthly), Perl Hackathons (Chicago Perl Hackathon is upcoming) and other events such as the Pittsburgh Perl Workshop (also upcoming). Right now, there's no central place to find a list of these events. Having a calendar on the perl.org and pm.org homepages will highlight the activity level of and events happening in the Perl community.
Use Perl-based software for TPF sites: The Planet Perl, Planet Perl Six and Planet Parrot websites run by TPF use the Python-based Planet (aka PlanetPlanet) Atom/RSS aggregator software and say "powered by python" and/or "powered by planetplanet." While it's nice to use FOSS, Plagger is a perfectly fine Perl-based Atom/RSS aggregator. In fact, Plagger has a much nicer and more extensible architecture than Planet. TPF should support Perl by running software written in Perl.

I'm not sure how good TPF is at communicating with the community but the impression I've gotten is that they don't communicate very well. I think it would be nice if:

TPF talked about future plans, like open source projects do
TPF officers responded to emails
TPF blog allowed unmoderated comments or had a more liberal filter. There are very few comments on the blog so I'm guessing that many are not approved. Either that or the moderation has discouraged people from commenting. I would normally expect more comments on something like TPF blog.
there was a TPF IRC channel that can be attended by community at large, say #tpf at irc.perl.or

Just a few observations. In any event, I'd love to see an events calendar on the perl.org and pm.org sites as well as TPF sites actually using Perl. Better communication would be a bonus.

Lucene on CPAN

John Wang — Mon, 21 Aug 2006 20:15:00 -0500

Lucene is a Java-based fulltext indexing and search solution run under the Apache Foundation. It is arguably the most popular of many fulltext solutions now but its use of Java makes it a secondary choice for many non-Java projects. To improve the speed of Lucene, there are several C/C++ ports including:

Lucene4c: Apache Foundation project
Lucy: Apache Foundation project
CLucene: Sourceforge project

For a long time Perl users who wanted fulltext search capabilities chose between Plucene (a Perl-port of Lucene), Xapian (a C++ library with many language bindings) and several others. Plucene has a variety of performance problems so Xapian became the choice for many projects. Recently, the Lucene module was added to CPAN as a wrapper around CLucene. The naming gets funny with all the ports because the Lucene Perl-module uses CLucene, not Lucene, and the CLucene project doesn't seem to be run under the Lucene project at the Apache Foundation, which runs the Lucene4c and Lucy C ports.

I've been using Xapian and the Search::Xapian Perl-bindings for a while now and will continue to do so. It will be interesting to see if there's a shake out with the various C/C++ Lucene ports and if any of them will gain popularity. It will also be interesting to see if any of the C/C++ ports grow beyond being just a port of Lucene.

Automated web screen shots with Perl

John Wang — Fri, 21 Jul 2006 10:02:00 -0500

I've been looking for a program that will take full screen shots of web pages even when the web page is larger than the window size on my physical screen, requiring scrolling. This morning I found such a program in Petr Šmejkal's Win32::CaptureIE when it was mentioned by Displeaser on DevShed Forums in the "Screenshot of webpage" thread. It uses ImageMagick for image manipulation.

From reading the Win32::CaptureIE POD, the CapturePage function does exactly what I want:

CapturePage ( ) Captures whole page currently loaded in the Internet Explorer window. Only the page content will be captured - no window, no scrollbars. If the page is smaller than the window only the occupied part of the window will be captured. If the page is longer (scrollbars are active) the function will capture the whole page step by step by scrolling the window content (in all directions) and will return a complete image of the page.

After installing ImageMagick, Image::Magick and Win32::CaptureIE on my Windows / ActiveState Perl system, I generated this screen shot with the following short program using no additional processing:

#!perl
use strict; use warnings;
use Win32::CaptureIE;

StartIE( width => 900 );
Navigate( 'http://www.dev411.com/blog/' );

my $img = CapturePage();
$img->Write( 'capture.png' );
QuitIE;

Perl and CPAN continue to amaze me with their treasure trove of functionality. Are there similar tools for using Firefox, Linux, other image libraries or languages?

UPDATE: ishnid has found two programs with CLIs (posted to the same thread):

khtml2png on SourceForge. This is a command-line program that looks like it can be run without a browser. It uses libkhtml (used by Konqueror) and ImageMagick's convertn.
Pearl Crescent Page Saver, a commercial app but available in a free version. This is a Firefox extension and requires the browser.

UPDATE 2:: I recently tried Win32::CaptureIE with ImageMagick 6.3.0 and it doesn't work. Apparently there used to be a link to "PerlMagick" in older versions of ImageMagick that may not exist anymore. Unfortunately Win32::CaptureIE relies on PerlMagick.

UPDATE 3:: I just tried the free version of Pearl Crescent with Firefox 1.5.0.7 which it says it should support but I get a "Download error" with pageserverbasic-1.3.xpi.