Chris Cruft

I am a complete Perl novice.  But there are occasions when I need to install perl modules.  I have learned that

perl -MCPAN -e ‘install (module-name)’

will allow you to install a perl module from the CPAN respository.

So I have been developing MemoryMiner (as my introduction to Rails) for almost six months now, and performance issues are starting to become recognizable. In a desperate attempt to kill two birds with one stone, I upgraded my Linux server (bmw) to Fedora Core 5 this past weekend. It was not uneventful but I finished with MySQL 5.0.22 installed and running for my Rails apps. I also now have ruby 1.8.5 and Edge Rails.   Unfortunately, the performance is still inadequate, and while I’m perfectly willing to suffer due to limited hardware, I’m pretty sure there is some bloat that needs to be trimmed -particularly in the area of database query tuning.

The first order of business was to get a handle on where my application was spending time. The classic solution, a profiler, applies to Rails as well. I tried the built-in profiler (script/performance/profiler) but was not blown away. Then I found ruby-prof and its graph profiles. I installed the gem and added a real simple around filter in the controller to generate an HTML Graph Profile. Cool -I could exercise a method/URL in a controller and immediately (in another browser) see the results in an easy-to-digest format.

Unfortunately, I could not find a smoking gun.
References:

1. http://ruby-prof.rubyforge.org/graph.txt
2. http://ruby-prof.rubyforge.org/
   
3. http://glu.ttono.us/articles/2006/06/23/stefen-kaes-optimizing-rails
4. http://www.thoughtstoblog.com/articles/2006/10/24/rails-performance-tool-box
5. http://blog.kovyrin.net/2006/08/28/ruby-performance-results/

1. Good example of benchmarking -with useful server-vs-server results.

Git is da bomb, written by The Man, all the kewl kidz sing its praises.  The Rails community has dropped Subversion like penicillin on clap.   And I’m really looking forward to taking advantage of the distributed nature of Git.

But I have existing projects that use Subversion, including projects that represent Rails plugins.  Plugins are indispensable when working with Rails and probably a bajillion other frameworks, tool sets and such use them to compose independently developed code.

Too bad, then, that Git’s support for composed projects sucks.  It cannot match the simplistic yet effective behavior of svn:externals.  At least for me, Subversion’s externals do the following Good Things:

• Records my project’s dependency on external plugins, either to a specific version or the latest version (edge) of that plugin.
• Supports transparent commits to my project’s code with no regard to fact that there are external plugin dependencies.
• Supports transparent commits to a plugin’s code with the same tools and commands as changes to my project’s code -this can be valuable if you use plugins to factor your own common functionality into a self-maintained plugin or if you are a contributor to plugin project.
• Transparently promulgates macro operations on my project code base to the plugins, i.e. one can checkout my project and get updates on my project with no concern that it is composed of plugins.

Git can do some of this through submodules -but there are holes.  I’m far from competent with Git, but just trying to setup a Git project with submodules seems to have these issues:

It’s not possible to track the latest version of a submodule.

And while there are tools that attempt to address the problem, I’m left with the nagging feeling that this is a fundamental design flaw or a limitation from the distributed nature of Git.  Anyone?  Anyone?

So I’ve got a couple of projects that I like to claim run on “Edge Rails”.  But the reality is that I let them slip for a month or two before playing catch up.  Today I decided to update an application that had somehow gotten behind 2-3-stable by hundreds of commits.  Being bold, I jumped straight to HEAD and was met with a shower of sparks running rake.  Where to start?

In a word or two: git bisect

I started the bisection run and told git that head was bad and my current commit was good.  Note: I use a git submodule to track Rails…

[…vendor/rails]# git bisect start

[…vendor/rails]# git bisect good

[…vendor/rails]# git bisect bad 2-3-stable

(and in another terminal window)

[myapp] # rake

Then it was just a matter of marking each auto-selected commit as good or bad with git bisect good or git bisect bad.  Worked great.  Next time, I’m going to automate the whole thing with git bisect run.

Somehow, someway my gem installation became subtly corrupt.  Specifically, many of my gems are not available in the cache directory.  That’s not a very intuitive erroneous situation, but there are gem commands that clearly expect a gem file in the cache directory -and they can’t recover the situation if the file is missing.  Doesn’t sound like my definition of a cache, but it’s time to move on.

Symptom:
1. whiny errors while attempting to unpack a gem like:

[cch1@bimota:~/Documents/Development/t]$ gem unpack exifr
ERROR:  While executing gem … (Gem::Exception)
Cannot load gem at [/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/cache/exifr-0.10.7.gem] in /Users/cch1/Documents/Development/t

or

[cch1@bimota:~/Documents/Development/t]$ gem pristine exifr
Restoring gem(s) to pristine condition…
ERROR:  Cached gem for exifr-0.10.7 not found, use `gem install` to restore

2. Indeed, no .gem file exists in the referenced cache directory.

Note that in OSX Leopard, gems are typically installed in several locations:

[cch1@bimota:~/Documents/Development/t]$ gem env paths
/Users/cch1/.gem/ruby/1.8:/Library/Ruby/Gems/1.8:/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8

My theory is that earlier versions of gem (such as the one I got with my distribution of Leopard in early 2008) had problems installing and updating gems other than those originally packaged with the system.  Using the gems by requiring the library file typically worked fine.  But by not building the cache file (or perhaps building it in the wrong one of the multiple gem directories), problems crept up months later trying to unpack the gem.

Sadly, the only solution I can find is to re-install the gem:

[cch1@bimota:~/Documents/Development/t]$ gem list exifr

*** LOCAL GEMS ***

exifr (0.10.7)
[cch1@bimota:~/Documents/Development/t]$ sudo gem install exifr –version ‘=0.10.7′
Password:
Successfully installed exifr-0.10.7
1 gem installed
Installing ri documentation for exifr-0.10.7…
Installing RDoc documentation for exifr-0.10.7…

Now the commands that depend on the “cached” version will work:

[cch1@bimota:~/Documents/Development/t]$ gem unpack exifr
Unpacked gem: ‘/Users/cch1/Documents/Development/t/exifr-0.10.7′

Today I ran into a problem with a factory pattern for an ActiveRecord inheritance hierarchy and scoping: has_many association members were being created without references to the “parent” model instance.  For example, my parent class looks something like this:

class Post < ActiveRecord::Base
  has_many :attachments, :as => :attachee
end

In my AttachmentsController, I was trying to create an attachment like this:

@post.attachments.create(attributes)

This approach is bog standard Rails and works fine.  The next iteration involved subclassing the Attachment model class and making Attachment#new method be a smart factory returning an instance of a subclass based on the initialization attributes:

class Attachment < ActiveRecord::Base
  validates_presence_of :attachee_id

  def new(attributes, &block)
    if self == Attachment
      return AttachmentURL if attributes[:url]

      return AttachmentBlob if (attributes[:blob] or attributes[:file])
    end

    super
  end
end

class AttachmentURL < Attachment
end

class AttachmentBlob < Attachment
end

Exercising this code in the console showed the right behavior:

Attachment.new({:url => ‘http://cho.hapgoods.com’})   # => AttachmentURL

But things started looking a bit squirrely when I started to use associations: my validation for the presence of the attachee_id reference was failing:

@post.attachments.create({:url => ‘http://cho.hapgoods.com’})

How could Rails be leaving out the critical association reference?!?  It turns out that association methods like create use :create scoping to set attribute values.  In ActiveRecord::Base.initialize, the :create scope is examined and all scoped attributes are assigned values from the :create scope.  My problem was that the :create scope was being lost when I switched to a subclass in the Attachments#new constructor.  This is the Important Fact:  A scoped ActiveRecord model does not imply scoping for subclasses.

I solved the problem by “transferring” the scope from the base class to the subclasses.  It really wasn’t that complicated and I ended with a really spiffy pattern for subclassed AR model factories.  Here is what my factory method looks like now:

class Attachment < ActiveRecord::Base
  def self.new(attributes, &block)
    if self == Attachment
      klass = AttachmentURL if attributes[:url]
      klass = AttachmentBlob if attributes[:blob] or attributes[:file]
      raise ‘Unknown Attachment type’ unless klass
      klass.send(:with_scope, :create => (scope(:create) || {})) do  # Scope subclass to same create attributes as this class.
        return klass.new(attributes, &block)
      end
    end
    super
  end
end

nota bene: It so happens that I was using Single Table Inheritance (STI) for this hierarchy, but I suspect the problem and solution would be identical with true abstract classes and arbitrary table mapping.  Also, the fact that I was using a polymorphic has_many association is almost certainly irrelevant.

Technorati Tags: STI, Rails, factory, with_scope

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes is the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses these abstract resources to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our abstract resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is a new, unsaved, AR model instance.? Every controller action that handles a REST request starts with the identification of the resource.? For the following examples, let’s define some resources:

map.resources users do |user|
? user.resources groups, :controller => ‘groups’ do |group|
??? group.resources tags, :controller => ‘tags’
? end
end

map.resources groups do |group|
? group.resources users, :controller => ‘users’ do |user|
??? user.resources tags, :controller => ‘tags’
? end
end

map.resources vehicles do |vehicle|
? vehicle.resources wheels, :controller => ‘wheels’
end

map.resources unicycles do |unicycle|
? vehicle.resource wheel, :controller => ‘wheels’
end

As these resource definitions show, sometimes resource identification is trivial and sometimes it’s not…

1. /vehicles/567/wheels
2. /unicycle/234/wheel
3. /users/17/groups/2/tags
4. /groups/2/users/17/tags

In example #1, wheels is a has_many association of a vehicle instance.? It would be nice if Rails helped us find the vehicle instance and pointed us towards its has_many association.? What we get is the params hash, which indicates the vehicle instance, and the invocation of WheelsController.index, which implies the wheels collection association.? Indirect, but adequate.

In example #2, wheel is a has_one association of a unicycle instance.? Rails should help us find the vehicle instance and point us towards the association.? Much as in the first example, Rails gives us a params hash and invokes the WheelsController.show method.? Again, indirect but adequate.?

Examples one and two together illustrate the first ugly problem with resource identification:

Problem One: The arity of resources is implied by the invoked controller method instead of being explicit.

In example #3, tags is a has_many association of a Group instance.? In example #4, tags is a has_many association of a User instance.? The params hash and invoked method? are identical for these two requests.? Rails really lets us down here: short of parsing the request path, there is nothing to distinguish these two requests.

Problem Two: The hierarchy of the request’s resource chain is not preserved.

I believe these two problems stem from the outdated perception of the URL only as a means of triggering a specific controller action and providing some unordered key-value parameters.? In the RESTful world, however, the URL has become a resource specifier.? And that means hierarchy matters and parameters are not limited to key-value pairs.? In a nutshell, Rails needs to help us match a concrete request with our abstract resource model.? Until it does, identification of RESTful resources will continue to be a pain in the ass.

In the meantime, I have taken two steps to help the programmer match an abstract resource to the concrete request by providing some extra information to the controller.? First, I have made the matched route available to the controller as an attribute of the request by monkey patching ActionController::Routing::RouteSet and ActionController::AbstractRequest.? Now I can see the specific components of the request in an abstract way instead of as a string.

But once I had made that change, I realized I was getting very close to the holy grail of the actual abstract resource definition (from routes.rb) used to generate the route.? It was only a couple of more monkey patches before I had the source ActionController::Resources::Resource instance available to the controller as well.

Now in my controllers I can see which of my abstract resources (as defined in routes.rb) matches the incoming request.? That goes a long way towards getting the matching AR model instances instantiated.?

My only remaining beef is that the abstract route definitions (in routes.rb) do not store any meaningful hierarchy information.? In my example above, the subordinate collection resource beneath a group and called ‘users’ is only aware of its parent resource (a group) though a couple of strings (name_prefix and one other).? That means that even when I know the abstract resource that matches the incoming request, I can’t really see its enclosing resources with inferring them from string pattern matching.? Rails should store the parent resource for any nested resources.

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing
??? Discussion of some of these issues from a different angle

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes is the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses it to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is a new, unsaved, AR model instance.? Every controller action that handles a REST request starts with the identification of the resource.? For the following examples, let’s define some resources:

map.resources users do |user|
? user.resources groups, :controller => ‘groups’ do |group|
??? group.resources tags, :controller => ‘tags’
? end
end

map.resources groups do |group|
? group.resources users, :controller => ‘users’ do |user|
??? user.resources tags, :controller => ‘tags’
? end
end

map.resources vehicles do |vehicle|
? vehicle.resources wheels, :controller => ‘wheels’
end

map.resources unicycles do |unicycle|
? vehicle.resource wheel, :controller => ‘wheels’
end

As these resource definitions show, sometimes resource identification is trivial and sometimes it’s not…

1. /vehicles/567/wheels
2. /unicycle/234/wheel
3. /users/17/groups/2/tags
4. /groups/2/users/17/tags

In example #1, wheels is a has_many association of a vehicle instance.? It would be nice if Rails helped us find the vehicle instance and pointed us towards its has_many association.? What we get is the params hash, which indicates the vehicle instance, and the invocation of WheelsController.index, which implies the wheels collection association.? Indirect, but adequate.

In example #2, wheel is a has_one association of a unicycle instance.? Rails should help us find the vehicle instance and point us towards the association.? Much as in the first example, Rails gives us a params hash and invokes the WheelsController.show method.? Again, indirect but adequate.?

Examples one and two together illustrate the first ugly problem with resource identification:

Problem One: The arity of resources is implied by the invoked controller method instead of being explicit.

In example #3, tags is a has_many association of a Group instance.? In example #4, tags is a has_many association of a User instance.? The params hash and invoked method? are identical for these two requests.? Rails really lets us down here: short of parsing the request path, there is nothing to distinguish these two requests.

Problem Two: The hierarchy of the request’s resource chain is not preserved.

I believe these two problems stem from a common failing in Rails: the outdated perception of the URL as a means of triggering a specific controller action and providing some unordered key-value parameters.? In the RESTful world, however, the URL has become a resource specifier.? And that means hierarchy counts and parameters are not limited to key-value pairs.? Until Rails embraces those realities, identification of RESTful resources will continue to be a pain in the ass.

I

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing
??? Discussion of some of these issues from a different angle

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes is the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses it to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is a new, unsaved, AR model instance.? Every controller action that handles a REST request starts with the identification of the resource.? For the following examples, let’s define some resources:

map.resource account do |account|
? account.resource vehicle, :controller => ‘vehicles’
end

map.resources users do |user|
? user.resources groups, :controller => ‘groups’ do |group|
??? group.resources tags, :controller => ‘tags’
? end
end

map.resources groups do |group|
? group.resources users, :controller => ‘users’ do |user|
??? user.resources tags, :controller => ‘tags’
? end
end

map.resources vehicles do |vehicle|
? vehicle.resources wheels, :controller => ‘wheels’
end

map.resources unicycles do |unicycle|
? vehicle.resource wheel, :controller => ‘wheels’
end

As these resource definitions show, sometimes resource identification is trivial and sometimes it’s not…

1. /vehicles/567/wheels
2. /unicycle/234/wheel
3. /users/17/groups/2/tags
4. /groups/2/users/17/tags

In example #1, wheels is a has_many association of a vehicle instance.? It would be nice if Rails helped us find the vehicle instance and pointed us towards its has_many association.? What we get is the params hash, which indicates the vehicle instance, and the invocation of WheelsController.index, which implies the wheels collection association.? Indirect, but adequate.

In example #2, wheel is a has_one association of a unicycle instance.? Rails should help us find the vehicle instance and point us towards the association.? Much as in the first example, Rails gives us a params hash and invokes the WheelsController.show method.? Again, indirect but adequate.?

Examples one and two together illustrate the first ugly problem with resource identification:

Problem One: The arity of child resources is implied by the invoked controller method instead of being explicit.

In example #3, tags is a has_many association of a Group instance.? In example #4, tags is a has_many association of a User instance.? The params hash and invoked method? are identical for these two requests.? Rails really lets us down here: short of parsing the request path, there is nothing to distinguish these two requests.

Problem Two: The hierarchy of the request’s resource chain is not preserved.

a relationship between a parent resource (vehicle or unicycle instance
in this example) and a child resource (wheels or wheel) is not
available.? relationship of resources is .? To do so will require a
more explicit representation of the parent resource (unicycle instance
or vehicle instance) and the child resource (wheel or wheels).

Does Rails exploit our resource modelling effort to help us identify the request’s resource?? No!? Are going to take that insult lying down?? No!? We’re going to carp and bitch!? Okay, I got a bit ahead of myself…but I maintain that Rails does a poor job of helping the programmer identify the resources in

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes are the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses it to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is an unsaved AR model instance.? Every controller action that handles a REST request must start with the identification of the resource.? Sometimes the resource identification is trivial…

/vehicles/234

sometimes it’s not…

/vehicles/567/wheels
/vehicles/234/wheel
/account/vehicle

Does Rails exploit our resource modelling effort to help us identify the request’s resource?

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing