Chris Cruft

I am a complete Perl novice.  But there are occasions when I need to install perl modules.  I have learned that

perl -MCPAN -e ‘install (module-name)’

will allow you to install a perl module from the CPAN respository.

So I have been developing MemoryMiner (as my introduction to Rails) for almost six months now, and performance issues are starting to become recognizable. In a desperate attempt to kill two birds with one stone, I upgraded my Linux server (bmw) to Fedora Core 5 this past weekend. It was not uneventful but I finished with MySQL 5.0.22 installed and running for my Rails apps. I also now have ruby 1.8.5 and Edge Rails.   Unfortunately, the performance is still inadequate, and while I’m perfectly willing to suffer due to limited hardware, I’m pretty sure there is some bloat that needs to be trimmed -particularly in the area of database query tuning.

The first order of business was to get a handle on where my application was spending time. The classic solution, a profiler, applies to Rails as well. I tried the built-in profiler (script/performance/profiler) but was not blown away. Then I found ruby-prof and its graph profiles. I installed the gem and added a real simple around filter in the controller to generate an HTML Graph Profile. Cool -I could exercise a method/URL in a controller and immediately (in another browser) see the results in an easy-to-digest format.

Unfortunately, I could not find a smoking gun.
References:

1. http://ruby-prof.rubyforge.org/graph.txt
2. http://ruby-prof.rubyforge.org/
   
3. http://glu.ttono.us/articles/2006/06/23/stefen-kaes-optimizing-rails
4. http://www.thoughtstoblog.com/articles/2006/10/24/rails-performance-tool-box
5. http://blog.kovyrin.net/2006/08/28/ruby-performance-results/

1. Good example of benchmarking -with useful server-vs-server results.

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes is the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses these abstract resources to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our abstract resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is a new, unsaved, AR model instance.? Every controller action that handles a REST request starts with the identification of the resource.? For the following examples, let’s define some resources:

map.resources users do |user|
? user.resources groups, :controller => ‘groups’ do |group|
??? group.resources tags, :controller => ‘tags’
? end
end

map.resources groups do |group|
? group.resources users, :controller => ‘users’ do |user|
??? user.resources tags, :controller => ‘tags’
? end
end

map.resources vehicles do |vehicle|
? vehicle.resources wheels, :controller => ‘wheels’
end

map.resources unicycles do |unicycle|
? vehicle.resource wheel, :controller => ‘wheels’
end

As these resource definitions show, sometimes resource identification is trivial and sometimes it’s not…

1. /vehicles/567/wheels
2. /unicycle/234/wheel
3. /users/17/groups/2/tags
4. /groups/2/users/17/tags

In example #1, wheels is a has_many association of a vehicle instance.? It would be nice if Rails helped us find the vehicle instance and pointed us towards its has_many association.? What we get is the params hash, which indicates the vehicle instance, and the invocation of WheelsController.index, which implies the wheels collection association.? Indirect, but adequate.

In example #2, wheel is a has_one association of a unicycle instance.? Rails should help us find the vehicle instance and point us towards the association.? Much as in the first example, Rails gives us a params hash and invokes the WheelsController.show method.? Again, indirect but adequate.?

Examples one and two together illustrate the first ugly problem with resource identification:

Problem One: The arity of resources is implied by the invoked controller method instead of being explicit.

In example #3, tags is a has_many association of a Group instance.? In example #4, tags is a has_many association of a User instance.? The params hash and invoked method? are identical for these two requests.? Rails really lets us down here: short of parsing the request path, there is nothing to distinguish these two requests.

Problem Two: The hierarchy of the request’s resource chain is not preserved.

I believe these two problems stem from the outdated perception of the URL only as a means of triggering a specific controller action and providing some unordered key-value parameters.? In the RESTful world, however, the URL has become a resource specifier.? And that means hierarchy matters and parameters are not limited to key-value pairs.? In a nutshell, Rails needs to help us match a concrete request with our abstract resource model.? Until it does, identification of RESTful resources will continue to be a pain in the ass.

In the meantime, I have taken two steps to help the programmer match an abstract resource to the concrete request by providing some extra information to the controller.? First, I have made the matched route available to the controller as an attribute of the request by monkey patching ActionController::Routing::RouteSet and ActionController::AbstractRequest.? Now I can see the specific components of the request in an abstract way instead of as a string.

But once I had made that change, I realized I was getting very close to the holy grail of the actual abstract resource definition (from routes.rb) used to generate the route.? It was only a couple of more monkey patches before I had the source ActionController::Resources::Resource instance available to the controller as well.

Now in my controllers I can see which of my abstract resources (as defined in routes.rb) matches the incoming request.? That goes a long way towards getting the matching AR model instances instantiated.?

My only remaining beef is that the abstract route definitions (in routes.rb) do not store any meaningful hierarchy information.? In my example above, the subordinate collection resource beneath a group and called ‘users’ is only aware of its parent resource (a group) though a couple of strings (name_prefix and one other).? That means that even when I know the abstract resource that matches the incoming request, I can’t really see its enclosing resources with inferring them from string pattern matching.? Rails should store the parent resource for any nested resources.

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing
??? Discussion of some of these issues from a different angle

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes is the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses it to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is a new, unsaved, AR model instance.? Every controller action that handles a REST request starts with the identification of the resource.? For the following examples, let’s define some resources:

map.resources users do |user|
? user.resources groups, :controller => ‘groups’ do |group|
??? group.resources tags, :controller => ‘tags’
? end
end

map.resources groups do |group|
? group.resources users, :controller => ‘users’ do |user|
??? user.resources tags, :controller => ‘tags’
? end
end

map.resources vehicles do |vehicle|
? vehicle.resources wheels, :controller => ‘wheels’
end

map.resources unicycles do |unicycle|
? vehicle.resource wheel, :controller => ‘wheels’
end

As these resource definitions show, sometimes resource identification is trivial and sometimes it’s not…

1. /vehicles/567/wheels
2. /unicycle/234/wheel
3. /users/17/groups/2/tags
4. /groups/2/users/17/tags

In example #1, wheels is a has_many association of a vehicle instance.? It would be nice if Rails helped us find the vehicle instance and pointed us towards its has_many association.? What we get is the params hash, which indicates the vehicle instance, and the invocation of WheelsController.index, which implies the wheels collection association.? Indirect, but adequate.

In example #2, wheel is a has_one association of a unicycle instance.? Rails should help us find the vehicle instance and point us towards the association.? Much as in the first example, Rails gives us a params hash and invokes the WheelsController.show method.? Again, indirect but adequate.?

Examples one and two together illustrate the first ugly problem with resource identification:

Problem One: The arity of resources is implied by the invoked controller method instead of being explicit.

In example #3, tags is a has_many association of a Group instance.? In example #4, tags is a has_many association of a User instance.? The params hash and invoked method? are identical for these two requests.? Rails really lets us down here: short of parsing the request path, there is nothing to distinguish these two requests.

Problem Two: The hierarchy of the request’s resource chain is not preserved.

I believe these two problems stem from a common failing in Rails: the outdated perception of the URL as a means of triggering a specific controller action and providing some unordered key-value parameters.? In the RESTful world, however, the URL has become a resource specifier.? And that means hierarchy counts and parameters are not limited to key-value pairs.? Until Rails embraces those realities, identification of RESTful resources will continue to be a pain in the ass.

I

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing
??? Discussion of some of these issues from a different angle

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes is the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses it to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is a new, unsaved, AR model instance.? Every controller action that handles a REST request starts with the identification of the resource.? For the following examples, let’s define some resources:

map.resource account do |account|
? account.resource vehicle, :controller => ‘vehicles’
end

map.resources users do |user|
? user.resources groups, :controller => ‘groups’ do |group|
??? group.resources tags, :controller => ‘tags’
? end
end

map.resources groups do |group|
? group.resources users, :controller => ‘users’ do |user|
??? user.resources tags, :controller => ‘tags’
? end
end

map.resources vehicles do |vehicle|
? vehicle.resources wheels, :controller => ‘wheels’
end

map.resources unicycles do |unicycle|
? vehicle.resource wheel, :controller => ‘wheels’
end

As these resource definitions show, sometimes resource identification is trivial and sometimes it’s not…

1. /vehicles/567/wheels
2. /unicycle/234/wheel
3. /users/17/groups/2/tags
4. /groups/2/users/17/tags

In example #1, wheels is a has_many association of a vehicle instance.? It would be nice if Rails helped us find the vehicle instance and pointed us towards its has_many association.? What we get is the params hash, which indicates the vehicle instance, and the invocation of WheelsController.index, which implies the wheels collection association.? Indirect, but adequate.

In example #2, wheel is a has_one association of a unicycle instance.? Rails should help us find the vehicle instance and point us towards the association.? Much as in the first example, Rails gives us a params hash and invokes the WheelsController.show method.? Again, indirect but adequate.?

Examples one and two together illustrate the first ugly problem with resource identification:

Problem One: The arity of child resources is implied by the invoked controller method instead of being explicit.

In example #3, tags is a has_many association of a Group instance.? In example #4, tags is a has_many association of a User instance.? The params hash and invoked method? are identical for these two requests.? Rails really lets us down here: short of parsing the request path, there is nothing to distinguish these two requests.

Problem Two: The hierarchy of the request’s resource chain is not preserved.

a relationship between a parent resource (vehicle or unicycle instance
in this example) and a child resource (wheels or wheel) is not
available.? relationship of resources is .? To do so will require a
more explicit representation of the parent resource (unicycle instance
or vehicle instance) and the child resource (wheel or wheels).

Does Rails exploit our resource modelling effort to help us identify the request’s resource?? No!? Are going to take that insult lying down?? No!? We’re going to carp and bitch!? Okay, I got a bit ahead of myself…but I maintain that Rails does a poor job of helping the programmer identify the resources in

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing

The support that Rails has provided for developing RESTful applications is nothing short of amazing considering it was added on long after Rails had an established URL-mapping mechanism (Routes.draw).? But I want more.? Specifically, I want my resources to have a more complete life.

Warning: I am not an expert in Rails or REST.

Background: in the current (late summer 2007) Edge Rails, RESTful resources (ActionController::Resources) are defined primarily for the purpose of creating named routes with a DSL (map.resource …).? A very important by-product of generating named routes are the helper methods available to generate routes in views and controllers.? The net result is a tidy means of generating and recognizing RESTful URIs.? A typical exploitation of these features looks like this:

1. Define Routes per Resource
2. Recognize Incoming Request’s Route
3. (process request in controller)
4. Generate view, typically with one or more links

Repeat steps 2 through 4 as required.

In step one, the Rails programmer does the hard work: modeling the resources, the interface to those resources, the mapping of resources to controllers, etc.? Rails honors our work and uses it to recognize routes (in step two) and to help up generate routes (in step four).? But what about step three?? Can we exploit our resource model here?? Let’s recast the typical methods in a controller that backs a resource:

def index
? identify resource
? manipulate resource
end

def destroy
? identify resource
? manipulate resource
end

def create
? identify resource
? manipulate resource
end

For the index action, the identified resource is a collection represented by a Class or an association.? For the destroy (and show and edit) actions, the identified resource is a member represented by an ActiveRecord model instance.? For the create (and new) actions, the resource is an unsaved AR model instance.? Every controller action that handles a REST request must start with the identification of the resource.? Sometimes the resource identification is trivial…

/vehicles/234

sometimes it’s not…

/vehicles/567/wheels
/vehicles/234/wheel
/account/vehicle

Does Rails exploit our resource modelling effort to help us identify the request’s resource?

Resources
??? Jamis Buck’s awesome tutorial on Rails Routing

NUT (Network UPS Tools) has been disappointing me pretty much since the start (two years ago) due to strange shutdowns. Today I decided to do a full analysis.

First, let me describe the configuration: I have two Compaq Proliant 3000 servers, named triumph and bmw, both running Fedora Core 5 with kernel 2.6.20-1.2300.fc5smp. I have two APC Smart-UPS, named cchSU2200 and cchSU1250. They are an APC Smart-UPS 2200 and an APC Smart-UPS 1250, respectively.

Server triumph has two power supplies, each one connected to a different UPS. Server bmw has one power supply, connected to UPS cchSU2200. Both UPSs have their signalling cables connected to serial ports on server triumph. Both UPSs have fresh batteries and “official” APC serial cables.

I won’t go into the details of my upsmon configuration (but will if you ask). Suffice to say that each server requires a minimum of one power supply and monitors all other UPSs (there is actually a third UPS several hundred miles away that is also monitored).

Here are the results and narrative of my testing:

The syslog on triumph:

Mar 22 10:28:26 My Action: Start UPS services
Mar 22 10:28:27 triumph upsd[6734]: Connected to UPS [cchSU2200]: apcsmart-ttyS0
Mar 22 10:28:27 triumph upsd[6734]: Connected to UPS [cchSU1250]: apcsmart-ttyS1
Mar 22 10:28:29 triumph upsd[6735]: Startup successful
Mar 22 10:28:29 triumph upsmon[6738]: Startup successful
Mar 22 10:28:29 triumph upsd[6735]: Connection from 127.0.0.1
Mar 22 10:28:29 triumph upsd[6735]: Client monuser@127.0.0.1 logged into UPS [cchSU2200]
Mar 22 10:28:29 triumph upsd[6735]: Connection from 127.0.0.1
Mar 22 10:28:29 triumph upsd[6735]: Client monuser@127.0.0.1 logged into UPS [cchSU1250]
Mar 22 10:28:36 triumph upsd[6735]: Connection from 192.168.1.10
Mar 22 10:28:36 triumph upsd[6735]: Client monuser@192.168.1.10 logged into UPS [cchSU2200]
Mar 22 10:28:36 triumph upsd[6735]: Connection from 192.168.1.10
Mar 22 10:29:09 My Action: Disconnect power to cchSU1250
Mar 22 10:29:10 triumph upsmon[6739]: UPS cchSU1250@localhost on battery
Mar 22 10:29:11 My Action: Connect power to cchSU1250
Mar 22 10:30:30 triumph upsmon[6739]: UPS cchSU1250@localhost on line power
Mar 22 10:30:54 My Action: Disconnect power to cchSU2200
Mar 22 10:30:55 triumph upsmon[6739]: UPS cchSU2200@localhost on battery
Mar 22 10:30:56 My Action: Connect power to cchSU2200
Mar 22 10:31:03 triumph upsmon[6739]: UPS cchSU2200@localhost on line power
Mar 22 10:34:41 My Action: Disconnect power to cchSU1250 and cchSU2200
Mar 22 10:34:42 triumph upsmon[6739]: UPS cchSU2200@localhost on battery
Mar 22 10:34:42 triumph upsmon[6739]: UPS cchSU1250@localhost on battery
Mar 22 10:34:43 My Action: Connect power to cchSU1250 and cchSU2200
Mar 22 10:35:56 triumph upsd[6735]: Client monuser@127.0.0.1 set FSD on UPS [cchSU2200]
Mar 22 10:35:56 triumph upsd[6735]: Client monuser@127.0.0.1 set FSD on UPS [cchSU1250]
Mar 22 10:36:02 triumph upsd[6735]: Host 192.168.1.10 disconnected (read failure)
Mar 22 10:36:02 triumph upsd[6735]: Host 192.168.1.10 disconnected (read failure)
Mar 22 10:36:02 triumph upsmon[6739]: Executing automatic power-fail shutdown
Mar 22 10:36:02 triumph upsmon[6739]: Auto logout and shutdown proceeding
Mar 22 10:36:07 triumph upsd[6735]: Host 127.0.0.1 disconnected (read failure)
Mar 22 10:36:07 triumph upsd[6735]: Host 127.0.0.1 disconnected (read failure)
Mar 22 10:36:07 triumph logger: upsmon.says.shutdwn

The syslog on bmw:

Mar 22 10:28:35 My Action: Start UPS services
Mar 22 10:28:36 bmw upsmon[9927]: Startup successful
Mar 22 10:29:12 bmw upsmon[9928]: UPS cchSU1250@triumph.hapgoods.com on battery
Mar 22 10:30:32 bmw upsmon[9928]: Giving up on the master for UPS [cchSU1250@triumph.hapgoods.com]
Mar 22 10:30:33 bmw upsmon[9928]: UPS cchSU1250@triumph.hapgoods.com on line power
Mar 22 10:30:58 bmw upsmon[9928]: UPS cchSU2200@triumph.hapgoods.com on battery
Mar 22 10:31:04 bmw upsmon[9928]: UPS cchSU2200@triumph.hapgoods.com on line power
Mar 22 10:34:43 bmw upsmon[9928]: UPS cchSU2200@triumph.hapgoods.com on battery
Mar 22 10:34:43 bmw upsmon[9928]: UPS cchSU1250@triumph.hapgoods.com on battery
Mar 22 10:35:57 bmw upsmon[9928]: Giving up on the master for UPS [cchSU2200@triumph.hapgoods.com]
Mar 22 10:35:57 bmw upsmon[9928]: Giving up on the master for UPS [cchSU1250@triumph.hapgoods.com]
Mar 22 10:35:57 bmw upsmon[9928]: Executing automatic power-fail shutdown
Mar 22 10:35:57 bmw upsmon[9928]: Auto logout and shutdown proceeding
Mar 22 10:36:02 bmw logger: ups.says.shutdown

And now the analysis of the results…

• On both servers, everything starts normally. The upsmon on bmw is clearly seen to connect to triumph.
• At 10:29:09, I pulled the power on cchSU1250 and immediately plugged it back in. Immediately, the master upsmon (on triumph) notes the on-battery condition, but I was disappointed to see the amount of time it took to recognize the return to on-line status (over a minute). On bmw, it is worth noting that upsmon “gives up” on the master for cchSU1250 (triumph) right at the end of this wait. This may or may not be a contributing factor to the hideous behavior we are going to observe in a few moments…(ominous background music starts…)
• At 10:30:54, I pulled the power on cchSU2200 and immediately plugged it back in. The behavior on both triumph (the master) and bmw (a slave) seems perfectly normal, with timely observation of both the on-batter and on-line condition.
• (ominous background music builds…) At 10:34:41, I pulled the power to both cchSU2200 & cchSU1250 and immediately plugged it back in to both. The master recognize the on-battery conditions, and it propagates them to the slave nicely. But the master never recognizes the immediate return to an on-line condition. The inevitable results appear about a minute later when the master starts shutting down and the slave follows suit (ominous background music reaches a crescendo, and then… silence).

I should point out that this problem has happened numerous times over the past two years. A simple 2 second power glitch will provoke the shutdown of bmw and sometimes even triumph. During that time, I have kept up-to-date with nut through the Fedora yum distribution and upgraded from Fedora Core 4 to Fedora Core 5. Currently, I am running nut version 2.0.3, release 0.1.fc5. The APC Smart protocol driver claims to be version 1.99.7 with a command table version of 2.0.

Any ideas? I ran some more auxiliary experiments and found that while upsmon seems to take a long time to recognize a return to on-line in cchSU1250, upslog reports the OL condition immediately. So I think the communication between the UPS and upsdrvctl is working.

I decided to take a detour and look at benchmarking my Rails application. Some background: I am running on a Compaq Proliant 3000 2×550 Pentium III Xeon with 2GB of RAM and a RAID 5 array using the old-style (white) Compaq drives. Software is Fedora Core 5 (Linux kernel 2.6.18-1.2200.fc5smp) with Apache 2.2.2 balancing across a mongrel cluster (v0.2.1) of two mongrel servers (v0.13.4). I am running on Edge Rails (r5875), and the database is MySQL 5.0.27.

To start, I used standard apache benchmark tools to get a feel for performance: