Dev411 Blog: Category mysql

Database Abstraction - code vs infrastructure

2007-09-04T23:38:00-05:00

I've worked on a number of database-driven projects and no matter how much people want database abstraction, it was always difficult to code and maintain. I was recently reminded of this when I read this Drupal article on dropping PostgreSQL support. Not only can it be difficult to maintain support for multiple databases, but it may be difficult to find developers.

One solution of modern programming is to move database abstraction from the code to the infrastructure using a ORM (Object-Relational Mapper) or Data Mapper. A ORM and Data Mapper abstracts the database for you so you no longer have to do tie db abstraction to each app. Not only does it let you code once for multiple databases it lets your users migrate their data from one database to another. This blog runs Typo which is based on Ruby on Rails and ActiveRecord. I've been contemplating migrating Typo from MySQL to PostgreSQL and I've been told that it would be as simple as exporting the data with YAML, updating the database.yml file and importing the data. I haven't gotten around to doing it yet but it is a powerful idea. ActiveRecord is a data mapper and isn't as flexible as a full blown ORM but it gets the job done for the most part. For a full-blown ORM, I think of Perl's DBIx::Class which provides a full OO interface to the RDBMS allowing you to code just once for multiple DBs without limiting you when you want to use some esoteric database-specific SQL. DBIx::Class is often used with the Catalyst Framework but is also used by itself.

There are PHP frameworks out there like Symfony and Cake but do any of them have stand-alone ORMs? If so, could Drupal move to something like that and solve their maintainership problems once and for all? Drupal is part of the Go PHP5 effort so there should be no issue using PHP 5 OO. Something to think about for the Drupal folks if a PHP ORM is available.

MySQL Deployment Presentations

2006-10-05T22:07:00-05:00

I just ran across the MySQL Web 2.0 page which lists a number of their users including the following:

The most interesting thing from that page, however, is links to various presentations given by those sites on how they architected their sites to scale with MySQL, some of them scaling up to hundreds of MySQL servers.

I've included a list of the presentations below because you have to do some page hopping to find which customer pages actually have presentations. The presentations are in PDF and PPT which I've indicated. None of the presentations are in XUL nor do any use the Takahashi method, both of which are becoming more popular.

eVite 2005-04-19 pdf
Flickr 2005-10-18 ppt
Flickr 2006-03-14 pdf: ~25,000 db transactions/second peak
LiveJournal 2004-04-01 pdf
LiveJournal 2005-08-04 pdf
Mixi 2006-04-26 pdf: More than 100 MySQL servers; Add more than 10 servers/month
Technorati 2006-04-26 ppt
TicketMaster 2005-09-05 pdf
Wikipedia 2006-04-26 pdf: >25000 SQL requests per second

I wonder if there's anything similar for PostgreSQL deployments at high traffic sites. Let me know if there are other deployment presentations for MySQL, PosgreSQL or other databases at well known, high traffic sites.

Update: Lukas Kahwe Smith gave me a link to OmniTI's PostgreSQL deployment presentation which discusses their migration from Oracle.

OmniTI pdf

Update 2: Some additional information on MySQL and UUID is available on articles by FeedLounge and Brad Fitzpatrick (chief architect for SixApart). Interesting discussion on how InnoDB clusters by PK.

Update 3: Robert Ficcaglia gave me this link to the Wikimedia Servers page at meta.wikimedia.org. Looks like good reading!

Perl, MySQL and UTF-8

2006-10-02T10:35:00-05:00

One of the mysteries of Perl to me is that why, as of yet, is there no UTF-8 support in DBD::mysql although this issue has been discussed on the msql-mysql-modules list since at least 2003 (using the MARC archives). This is also given that MySQL does have UTF-8 support itself.

When I first looked into this I found a few articles on this:

Pedro's article mentions that the reason this hasn't been done for DBD::mysql is that the DBI and DBD::mysql folks cannot decide where to put UTF-8 implementation, i.e. in DBI itself or the DBD drivers. Because, there is still no built-in support. To get around this, there have been numerous patches produced. Andrew Forrest even put together UTF-8 versions of DBI and CGI.pm (link seems broken atm). However, some of these patches seem to have problems and are non-standard.

If you prefer to use an ORM, DBIx::Class and Class::DBI get around this by implementing UTF-8 support in their own libraries with DBIx::Class::UTF8Columns and Class::DBI::utf8 respectively. I'd recommend DBIx::Class over Class::DBI since it has more functionality (e.g. built-in JOIN support) and is supposed to generate more efficient SQL.

The intersting thing is that DBD::Pg for PostgreSQL has had built-in UTF-8 support for some time. While not an issue specific to the MySQL database, the UTF-8 perl driver issue is something to consider when choosing MySQL or PostgreSQL.

Update: Thanks to Dominic Mitchell for mentioning the latest developer release, DBD::mysql 3.0007_1 released on 8 Sep 2006, has integrated UTF-8 support. It's a developer release but good things are finally happening!

Encoding Hashed UIDs: Base64 vs. Hex vs. Base32

2006-10-02T03:08:00-05:00

I recently looked at using various encodings for hashed UIDs, e.g. UIDs generated by a crytographic hash algorithm such as SHA-1 or MD5. These are often useful when the UID does not need to have human meaning but should exhibit some uniformity, such as character set and length. I considered Base64 and hexadecimal first because they are commonly used by crypto libraries and then decided on Base64 and Base32 where appropriate. Base36 is actually the most compact case insensitive encoding (using Arabic numbers and Roman letters) but is not an option for me at the moment because there's no Perl module for it that will take arbitrary text and binary input at the moment. Math::Base36 exists but only handles numbers.

Base64 has the advantage of generating shorter representations because it uses a 64 character set instead of hexadecimal's 16. While Base 64 can use any 64 characters, many implementations follow IETF PEM RFC 1421 which specifies A-Z, a-z, 0-9, / and + while using = for padding. Because + and / need to be URI escaped, Wikipedia mentions a variant that uses * and - in place of + and / respectively and removes line breaks as well as padding. I used this for a while but unfortunate * cannot be used in DOM ids. Because of this I've switched to using _- instead.
Hexadecimal uses 0-9 and A-F and is case insensitive. The representations are longer due to the smaller character set, but case insensitivy can be advantageous in certain situations such as full text indexing with Xapian which is case sensitive, and MySQL which is case insensitive by default. It's common practices to lowercase all the indexed terms using Xapian while reserving uppercase characters for boolean and probabilistic term prefixes. After thinking about how the query parser works for a bit, my conclusion is that lower casing all the inexed terms will make life with Xapian a lot easier. To enable case sensitivity in MySQL, the table definition needs to be changed.
Base32 is the middle ground between the 16 character hex and 64 character Base64 encodings, but does not get included in crypto libraries. For UIDs, however, it is especially attractive because it uses a larger character set than hexadecimal resulting in shorter representations and is case insensitive for applications that work better with it. Base32, as defined in IETF RFC 3548, uses the characters A-Z and 2-7. Since Base32 isn't included in many crypto libraries, an extra encoding is needed, however, creating hashed UIDs only happens once per entry so it seems reasonable.

There are some modules on CPAN for Base32 including MIME::Base32 and Convert::Base32. For Base32, I'm currently using a patched copy of MIME::Base32 because the CPAN version only handles uppercase. For Base64, I'm using _ and - as the two non-alphanumeric characters.

Lessons from MySQL

2006-06-26T16:36:00-05:00

I picked up a copy for Fortune Magazine's Secrets of Greatness: Teamwork! issue for the flight to YAPC::NA. There was an insightful article on MySQL. MySQL has 350 workers in 25 countries and 70% of them telecommute. This structure has led to some hiring practices contrary to the norm in other organizations.

Employee characteristics: Given the decentralized nature of its organization, MySQL for skills, not the ability to play nice with a team.
Finding talent: MySQL finds some of its talent (50+ employees) through open source cntributions to its code base. Others also look for developers among those that actively check in good code.
Work schedule: CEO Marten Mickos says:
The brightest engineers like the calmness and coolness of the night
and likes hearing the following from prospective employees:
question: How do you plan your day?
answer: I always sleep until 11 A.M., and then I start working

While working late at night seemed common when I was in college, when I got to the workplace many developers seemed to work a regular schedule leaving around 5-6pm, albeit in a regular office environment as opposed to MySQL's telecommuting environment. Do you like to work at night and is it common to come in "late" and work late at places other than MySQL? Also, in a non-telecommunting situation, is this mostly for developers without kids? If you do have a start late / work late schedule, why do you like it and what time to you tend to start work?

Birth of a Typo Blog

2006-06-21T17:04:00-05:00

I'm giving Typo a shot after hearing some good things about it and having performance problems at a hosted blog service. It's been pretty good so far. Let's see how it runs in action.

I'm currently using the Lucid theme. So far I've played with Azure (the default), Lucid and Phokus. The default font size seems smaller than normal on other sites. Let me know if it's too small or there are other issues.

Up until now, my wiki has been the only app at this domain. I'm leaning towards putting commentary, questions, short summaries and the like in the blog while the wiki would still contain reference type information. To that end, I put up some detailed Typo install instructions covering MySQL, Apache, lighttpd and FastCGI on the wiki. Right now it just covers installation as I'm just getting into customizing the template. Let me know what you think.