Reflections on YAPC::NA 2009

I had a lot of fun at YAPC::NA this year. I met some forest friends that didn't make it last year, such as ilmari, Nick Perez, and Dylan Hardison. I also laughed until it hurt. A few times.

I was a hermit on Sunday and Monday nights because I had not finished my slides. That was very unfortunate, so I will have future talks ready well in advance. Jitting slides is so not my style.

Moose Course

The eight hour Moose course went very well. I was a passable Rolsky-substitute. The feedback I recieved was (nearly) unanimously good, so I'm thrilled that people enjoyed it. There were some minor complaints, so if I ever teach a course again I'll be sure to keep them in mind. This year, YAPC::NA is running surveys on talks and courses, so I expect I'll learn more ways that the class could be improved. If you attended the class, please do not worry about offending Dave or me if you disliked something.

I was under-prepared for the class, but the course material was very good. I got the impression that the exercises were extremely helpful. In particular, having a good test suite (powered by the metaobject protocol!) was very useful. The exercises were also useful for me because they let me sit down. Speaking on your feet for so many hours is tough.

Like the first version of anything, the course material contained some typos, inconsistencies, and bugs. It was pleasant to fix these and push them back to the source repository while people worked on the exercises. For some reason, we could not get public (i.e. noncommitter) access to the repository working, perhaps due to DNS issues. It would have been useful to have people pull my changes to fix the exercises. Instead, I settled for announcing what the problems were and how to fix them.

Context

Early in the course I had a vitriolic rant about Perl's notion of context. On the whole I think context is a nice solution to a certain class of problems, but it definitely has some edge cases where it really bites programmers. For example, propagating context is very fiddly. You must propagate context if you wrap a function to do work after it is called without affecting its return value.

sub method {
    my $self = shift;

    # call SUPER::method in the right context.
    # not handling void context is a BUG!
    my @ret;
    if (wantarray) {
        @ret = $self->SUPER::method(@_);
    } elsif (defined wantarray) {
        $ret[0] = $self->SUPER::method(@_);
    } else {
        $self->SUPER::method(@_);
    }

    $self->called_method($self->called_method + 1);

    # return what the caller asked for.
    # void context doesn't need to be explicitly handled here
    # .. or does it? maybe!
    return @ret if wantarray;
    return $ret[0];
}

I don't recall mentioning in the class that Moose does all of this for us in the after method modifier:

after method => sub {
    my $self = shift;
    $self->called_method($self->called_method + 1);
};

The other painful detail about context is in returning a list in scalar context. Most people know that an array in scalar context evaluates to the number of elements:

sub users {
    my $self = shift;
    my @users = $self->objects('users');
    return @users;
}

my $users = $game->users; # 51

However, a lot of people don't know that there is an irritating amount of subtlety in evaluating a list in scalar context:

sub administrators {
    my $self = shift;
    return ('Sartak', 'autarch');
}

my $admins = $game->administrators; # autarch

Instead of receiving the count of administrators, we get the last element of the list. The problem is actually the comma. Due to C's strong influence on Perl 5, the comma is not just a list separator. In scalar context it acts as a sequencing operator. We're evaluating the string 'Sartak', throwing it away, then evaluating the string 'autarch'. There is no list at all here!

Binary "," is the comma operator. In scalar context it evaluates its left argument, throws that value away, then evaluates its right argument and returns that value. This is just like C's comma operator.

– perldoc perlop

Almost everywhere else in the language, arrays and lists are interchangeable. I wish this inconsistency could be excised. Thanks to do BLOCK, we do not need C's comma operator at all.

Extending Moose

On Tuesday morning I presented Extending Moose for Applications (keynote source). There were maybe 20 attendees. I blame the 8AM start time. Oh well.

I'm told that at one point someone walked out saying the talk got "too abstract". It certainly is a very abstract topic, but I hope the examples showed that there is value to be had in using the ideas I presented.

Some good questions were asked, and some people did get it. I conveyed the point I wanted to make, which is that the metaobject protocol uses concepts you already know well.

I encourage you to read the slides. There are notes which is basically what I spoke when I was presenting, so they should be pretty readable. Please let me know how I did. I've received very little feedback on this talk.

TAEB

One of my good friends Jesse Luehrs (doy) gave a lightning talk about our NetHack bot, TAEB.

The line "The code I'm working on isn't really a NetHack bot; it's more of a framework for writing NetHack bots" got quite a few laughs. It's completely true, too. The topic of the talk was writing a new AI for this bot framework.

The bitrotted web interface seemed to please people. It's too bad they saw how it was when I first wrote it. Since I took those screenshots, I improved it quite a bit, making it use NetHack's colors and providing an AJAXy interface. Oh well, I should really get that working again.

This lightning talk was a hit. Several people cheered! Great job, Jesse.

Others' Talks

My favorite talk, by far, was Brock Wilcox's CGI::Inspect talk. As I said during the talk, when I wrote Carp::REPL, I was actually aiming for exactly what CGI::Inspect is. And I meant it! About two years ago, it was mentioned to me how when there's an error in your Django application, it opens an AJAXy REPL for you. I figured out how to do some of that - REPL on error - but I never actually hooked it up to Jifty. Now I don't have to. Bravo, Brock!

Another talk I enjoyed was Ricardo Signes's Git is Easy talk. I knew most of the material presented, but since seeing that talk I have had a lot more confidence in using git. In particular, remotes now make sense to me. I also now use GitX to visualize where other developers are relative to me.

Scott Walter's Perl in Vegas talk was fascinating. He drew an interesting parallel between the incredible strictures of Vegas slot machines versus America's completely opaque and mostly unregulated voting machines. One strange requirement of the slot machines is that code must be compiled. Though Perl is an interpreted language, it provides a perlcc utility to "compile" the code.

Hans Dieter Pearcey's Dist::Zilla - Automating quality since 2008 reminded me of its excellent plugin-centric design:

This slide directly caused me to submit a talk about API Design to YAPC::Asia 2009.

Finally, miyagawa's Build a desktop application with Perl, HTTP::Engine, SQLite and jQuery talk was exciting. I've been working toward using his perl-app-builder for TAEB since he blogged about it. The talk showed how we can write standalone, web-based, GUI apps, in Perl, without inflicting CPAN on our users. Because Perl has been a web language ever since the web has had languages, the tools for generating HTML, serving pages, etc. are very good. The traditional GUI tools are mediocre at best. They are why my applications have been strictly web- and curses-based.

Thanks!

Thank you Robert Blackwell, Casey West, Tom Moertel, Dan Wright, and anyone else who helped! This YAPC was fantastic.

If you have never been to YAPC, you should seriously consider going to the next one!

Shawn M Moose at YAPC

YAPC::NA 2009 begins Monday; it's not too late to sign up! Don't miss the affectionately-titled Moose track.

Extending Moose talk

I'm giving a 50 minute talk about Extending Moose for Applications. The content will be similar to the Moose's MOP series, though focusing more on how all the metaprogramming features work together in harmony. I've been writing the articles so I can learn how to teach the concept and utility of a metaobject protocol.

I'm still writing the talk, but you can have a look at what I have so far: [pdf] [keynote source]. Suggestions for improvement or clarification would be very welcome. I have the "script" included as notes at the bottom of each slide, so you can learn even without me there speaking at you. If you intend to attend the talk (or the Moose course on Sunday) you should probably not read it. I stick to the script, terrible jokes and all. When it is done and presented, I'll post about the final version.

Moose class

Due to health problems, Dave Rolsky will sadly not be attending YAPC this year. He was scheduled to give an eight hour class about Moose on Sunday. It would be a lot of hassle for organizers and disappointment for students to cancel the class, so I stepped up to present the bulk of it. Jon Rockway will be helping as well. I don't make great use of Moose's types, but he does, so we are forcing him to teach the types portion of the class.

I will also be presenting my Extending Moose talk at the end of the class (if we turn out to be short on time). This way you can see Matt Trout's Future of DBIx::Class or Christopher Nehren's CLI apps don't have to suck. Both will probably be Moose-heavy and certainly worth seeing. Or.. you can sleep in!

Please let me know if you have any questions about the talk or class. I've given a few talks before, but never an extended class, so it will certainly be interesting!

I look forward to seeing many of you next week. I'll try to squeeze in a post or eight about talks and all the other good stuff at YAPCs.

Moose's MOP: Schematize

Last time, we met Moose::Meta::Attribute. This meta-level class tracks information about Moose attributes. As Moose users know, attributes can have a lot of information associated with them, such as accessors, type constraint, predicate, delegation, default value, and more. Today we will explore such information to generate a simplistic SQLite schema for a Moose class.

Since Moose adds the concept of attributes to Perl, plain classes do not contain enough useful information to generate a schema. We have nothing to inspect since all of the attribute metadata is locked up in opaque constructors and accessors. If you are playing along at home, you will need to use a real Moose class. You have plenty handy, right?

As always, please follow along on github and try it for yourself! Experimentation is a great way to learn a system.

Prelude

use strict;
use warnings;
use Moose ();
use DBI;

All Perl code should, of course, enforce strictures and warnings. We need Moose to inspect classes. Finally, we need DBI, Perl's database abstraction layer, to safely quote strings.

my $class = shift or die "usage: $0 Class::Name\n";
Class::MOP::load_class($class);

my $meta = Moose::Meta::Class->initialize($class);

The user passes in as a command-line argument the class name for which she wants a schema. We load it up, then get the class metaobject. The class metaobject is the gateway into meta-land. From the class metaobject we can access all of the attribute metaobjects, as we will do next.

print header($meta);

print join ",\n",
    "id INTEGER PRIMARY KEY AUTOINCREMENT",
    map { attribute($_) } $meta->get_all_attributes;

print footer();

The header function begins the CREATE TABLE statement. footer ends it.

The middle bit is where it starts to get interesting. We hardcode an integer id primary key column for the table. This is not great, but it serves us for the toy example. If you are interested in something high-tech, see Sam Vilain's comment on the previous article, pointing to his design of marrying Perl 6's metaobject protocol to a database.

After the id column, we loop over all of the class's attributes and serialize them with the attribute function. get_all_attributes will include superclass attributes (use get_attributes if you only want local attributes). I am not going to design object-oriented tables; if I wanted an object-oriented database I would use KiokuDB!

Create Table

sub class_to_table {
    my $class = shift;
    my $table = lc $class;
    $table =~ s/::/_/g;
    return $table;
}

class_to_table is a helper function for transforming class name Foo::Bar to table name foo_bar. Though whether table names should be singular or plural is apparently a hotly-debated topic, we're going to sidestep the issue by just using the class name.

sub header {
    my $meta = shift;

    my $name = class_to_table($meta->name);

    return "CREATE TABLE $name (\n";
}

The header function does just start the CREATE TABLE statement. Recall that $meta->name is the name of class that $meta represents. We could have just passed in the class name to this function, but we may want to declare table constraints in the future, such as compound primary keys.

sub footer { "\n);\n" }

footer closes the CREATE TABLE that header opened. Now onto the good stuff: attributes → columns.

Columns

sub attribute {
    my $attribute = shift;
    my @constraints;

    push @constraints, type_of($attribute);
    push @constraints, 'NOT NULL' if $attribute->is_required;
    push @constraints, default_of($attribute);
    push @constraints, foreign_key_of($attribute);

    return join ' ', $attribute->name, @constraints;
}

The attribute function takes an attribute metaobject and returns a string suitable for creating a column in the schema. We generate a list of constraints (type, default, requiredness, foreign key) from the values in the attribute. These correspond to the Moose attributes that we can usefully translate to the relational database.

If the attribute requires a value in the Perl OO side ($attribute->is_required), then it should require a value in the SQLite relational side. We can enforce this by declaring the column to be NOT NULL.

Once we have all the constraints, we return them along with the column name as a plain string.

sub type_of {
    my $attribute = shift;

    return if !$attribute->has_type_constraint;
    my $tc = $attribute->type_constraint;

    my @sqlite_types = (
        [Int => 'INTEGER'],
        [Num => 'REAL'],
        [Str => 'TEXT'],
    );

    for (@sqlite_types) {
        my ($moose_type, $sqlite_type) = @$_;
        return $sqlite_type
            if $tc->is_a_type_of($moose_type);
    }

    return;
}

type_of maps a Moose type constraint to a SQLite data type. If there is no type constraint, then we do not need to specify one in the schema.

SQLite has a dearth of data types, so there are only three mappings. If we had swapped the order of the Int and Num checks, then Moose Ints would map to REALs in the database. Moose's types are hierarchical, and Int is a type of Num. We need to check most-specific type constraints first.

In a few weeks, we will see how we can get more power out of the type constraint system by extending it to provide arbitrary check constraints.

sub default_of {
    my $attribute = shift;

    return unless $attribute->has_default
               && !$attribute->is_default_a_coderef;

    my $default = $attribute->default;

    if ($default =~ /^\d+$/) {
        return ('DEFAULT', $default);
    }

    return ('DEFAULT', DBD::_::db->quote($default));
}

default_of returns a suitable default value for the table column.

Default values in Moose are interesting because there are two very different kinds. There are plain defaults, which are numbers and strings. Those are interesting to us.

There are also subroutine (coderef) defaults. Moose calls the provided subroutine to generate a default value for each instance of a class. Since each instance of the class can get a different default, there is no sane default for the database. Subroutines in Perl are (effectively) opaque. One of the great benefits of Moose is that it adds transparency to classes by making them declarative. As I lamented in the beginning of this post, Perl's default OO is almost completely opaque, which harshly limits its second-order utility. Second-order utility happens to be the theme of this series of articles!

Anyway, if the default value is an integer, we pass it through to the CREATE TABLE without escaping it. This could be relaxed a little, perhaps to allow a decimal point. Or scientific notation. Down that path lies Scalar::Util's looks_like_number. I doubt Perl's notion of numerality exactly matches SQLite's, so I remain conservative. SQLite does support what it calls manifest typing, so there is little need to get the distinction exactly right. Just like Perl.

The DBD::_::db->quote method quotes string default values for the database. I am not so concerned about SQL injection in this code as I am about syntax errors. I apologize for the expression's vulgarity, but Perl decided to elide the builtin mysql_real_escape_string. Usually when interacting with DBI, you have a database handle for calling quote. Or better yet, placeholders. I did not want to have to set up a database just to generate a correct schema, but when you are writing an application you will probably have a handle handy.

sub foreign_key_of {
    my $attribute = shift;

    return if !$attribute->has_type_constraint;
    my $tc = $attribute->type_constraint;

    return if !$tc->isa('Moose::Meta::TypeConstraint::Class');
    my $table = class_to_table($tc->class);

    return ('REFERENCES', $table, '(id)');
}

foreign_key_of is used to generate foreign key constraints when an attribute contains an object. Though foreign_key_of inspects the attribute's type constraint, it is a different kind of constraint in the database, so it exists here as a separate function.

Moose::Meta::TypeConstraint::Class is a type constraint that accepts object of a particular class. This type constraint metaobject has a class accessor for determining what the class is.

Obviously we are optimistic in assuming that any delegate object's class will exist in the database. If you were building up a MOP-aware ORM, you might inspect the delegate to see if it does a particular role, one that requests the delegate be serialized as a string in the database. DateTime is an obvious choice here, serializing to a timestamp. Another option (beyond us for a few weeks, but sit tight!) would be to tag the attribute, indicating that the attribute should not exist as a column in the database.

Example

As an example, I'll generate a schema of Wallace Reis's Business::CardInfo, version 0.02.

package Business::CardInfo;
use Moose;
use Moose::Util::TypeConstraints;

subtype 'CardNumber'
  => as 'Int'
   => where { validate($_) };

coerce 'CardNumber'
  => from 'Str'
   => via {
    my $cc = shift;
    $cc =~ s/\s//g;
    return $cc;
   };

no Moose::Util::TypeConstraints;

has 'country' => (
  isa => 'Str',
  is  => 'rw',
  default => 'UK'
  );

has 'number' => (
  isa => 'CardNumber',
  is  => 'rw',
  required => 1,
  coerce => 1,
  trigger => sub { shift->clear_type }
);

has 'type' => (
  isa => 'Str',
  is  => 'rw',
  lazy_build => 1,
);

sub _build_type { ... }

sub _search { ... }

sub validate { ... }

This class generates the following schema.

CREATE TABLE business_cardinfo (
id INTEGER PRIMARY KEY AUTOINCREMENT,
number INTEGER NOT NULL,
country TEXT DEFAULT 'UK',
type TEXT
);

One note of interest is that the number attribute was a user-defined subtype of Int, but we still sussed out the correct data type.

Conclusion

Though there were several instances where I deferred features for later articles, I think we built a usable, if simplistic, system. Its output could serve as a first pass for writing a real, production schema. With any luck, it may inspire someone to write the mythical Moose-based-ORM-for-SQL-haters. Moose is already the basis for an ORM for SQL lovers.

A particularly nice feature of this system is that it generates a schema without touching the class's code. The class is just an ordinary Moose affair that uses no extensions and no special semantics.

One way to extend this system using still more vanilla Moose inspection would be to transform ArrayRef[Class::Name] type constraints into many-to-many relationships. Some of the interesting MOP questions include:

• What is the class of a ArrayRef[Class::Name] type constraint metaobject?
• How do you extract Class::Name from that type constraint metaobject? You probably need to dive into Moose's code. One incorrect answer is: with a regular expression.
• How do you load a type constraint metaobject by name?

Next time we will see that we can do much more than inspect metaobjects with a MOP!