I am a complete Perl novice. But there are occasions when I need to install perl modules. I have learned that
perl -MCPAN -e ‘install (module-name)’
will allow you to install a perl module from the CPAN respository.
So I have been developing MemoryMiner (as my introduction to Rails) for almost six months now, and performance issues are starting to become recognizable. In a desperate attempt to kill two birds with one stone, I upgraded my Linux server (bmw) to Fedora Core 5 this past weekend. It was not uneventful but I finished with MySQL 5.0.22 installed and running for my Rails apps. I also now have ruby 1.8.5 and Edge Rails. Â Unfortunately, the performance is still inadequate, and while I’m perfectly willing to suffer due to limited hardware, I’m pretty sure there is some bloat that needs to be trimmed -particularly in the area of database query tuning.
The first order of business was to get a handle on where my application was spending time. The classic solution, a profiler, applies to Rails as well. I tried the built-in profiler (script/performance/profiler) but was not blown away. Then I found ruby-prof and its graph profiles. I installed the gem and added a real simple around filter in the controller to generate an HTML Graph Profile. Cool -I could exercise a method/URL in a controller and immediately (in another browser) see the results in an easy-to-digest format.
Unfortunately, I could not find a smoking gun.
References:
For the second time this year, I struggled debugging the dreaded “A copy of XXX has been removed from the module tree but is still active”. Last time, I learned the lesson of letting Rails autoload wherever possible and to not try to manage the loading myself. Kind of a Zen thing.
But there are limitations with that approach. For starters, we all know this only applies to Rails, not Ruby in general. And so when writing gems, require is your friend. An edge case is plugins. The auto-dependency mechanism does not reload plugin code (at least not by default), but it will autoload it initially. That’s a code smell, in my opinion, that leads to the error noted in the first paragraph when “locked in” code references autoloaded code. It kinda works like this as best I can tell: don’t let Rails autoload code that it is not also reloading.
Given that plugins are (normally) not be reloaded, you need to avoid having Rails grab const_missing and autoload your code. So, in plugin code, use your normal arsenal of Ruby-standard load/require to load stuff. Here’s a neat trick to find where you’ve failed to make the grade: start the console and run
>> print ActiveSupport::Dependencies.history.grep(/plugin/).join(”\n”)
Any plugin code listed is probably counting on Rails’ dependency resolution/autoloader. You may need to muck around with your code a bit to trigger stuff getting autoloaded, but in my case there was plenty to work on just post-initialization.
I switched the test database for one of my plugins to SQLite3 today -I like the idea of not requiring a database setup step for plugin tests and the in-memory option for SQLite3 nicely sidesteps that requirement. Immediately my tests all failed with:
ActiveRecord::StatementInvalid: SQLite3::SQLException: unrecognized token: <INSERT statement mixed in with a bunch of binary gobbledygook>
The weak link here is Ruby. Think about this chain of custody for my binary data:
The second custodian (prior to 1.9 only, I hope) loses the “binariness” of my data and thus fails to properly quote it for insertion into SQLite3 (interestingly, MySQL does not appear to require special quoting). Normally ActiveRecord knows how to restore the binariness of the data on insert by looking at the destination column type (you can see this code in fixtures.rb). But it gets this column information in a strange way: despite knowing the table into which the fixture is to be inserted, ActiveRecord gets the column data by referencing the model class associated with the table. The model class is guessed from the name of the fixture’s DB table -which fails to find a match whenever the model class is not in the root namespace. Head spinning yet?
<fixture file name> => <database table> => <model class> => <column details>
(The irony is that a model class knows its column types because it inspected the table during initialization. Why doesn’t the Fixtures class (which also knows which table to inspect) do the same thing? Like this:
<fixture file name> => <database table> => <column details>
Who knows?)Anyway, without a model class and thus column type hints, ActiveRecord happily inserts fixtures with generic quoting. Ruby strings, for example, are assumed to be destined for plain string columns. For blob columns in SQLite3, this causes a shower of sparks.All hope is not lost, however. Rails provides the #set_fixture_class method which allows you to explicitly link the fixture file to a model class. For example:
set_fixture_class {:widgets => My::Deeply::Nested:Widget, :thingys => My::Deeply::Nested::Thingy}
Problem solved.
Recently I decided to jack up my profiling abilities for my Rails apps. At some point, looking at call graphs started to hurt my brain so I went to install kcachegrind on my Snow Leopard machine. To make a long story short, I ended up doing an across-the-board upgrade of Mac Ports (sudo port selfupdate followed by sudo port upgrade). Despite still having some lingering problems with kdeinit whining about not being able to bind to the socket (claiming the address was already in use), I was able to use kcachegrind to study the profiler output from Rails’ spiffy profiler tests (ActionController::PerformanceTest). Whoopee!
Too bad that having upgraded all my ports sent me directly to the next step in dependency chain hell. First problem: rmagick won’t load. Hoping for the best, I fired off an update of the rmagick gem (sudo gem update rmagick). YeeHaa! I’m on a roll. The only thing left is a tiny little warning when running my test suite about “WARNING: Nokogiri was built against LibXML version 2.7.6, but has dynamically loaded 2.7.7″ and how hard could a little warning be? After all, my test suite passed just fine. But, I’m a bit to OCD to let that warning ride, so I started digging.
First, let’s evaluate that message: nokogiri is complaining about LibXML being too new. OK, that’s easy, let’s get a newer version of Nokogiri that’s more comfortable with that newer version of LibXML (which probably came along with Mac Ports). Easy peasy upgrade (sudo gem update nokogiri) from 1.3.3 to 1.4.1. Unfortunately, the warning is still there. On a hunch, I open my Rails console…
# ./script/console>> Nokogiri::VERSION=> “1.3.3″
Uh, excuse me, who is loading the outdated version of Nokogiri? I check my environment files (I’m not yet on Rails 3) to confirm that my app is not tied to a specific version… nope. Now I have to hunt down the offending party. Here’s how I fingered him…Remove the config.gem line for nokogiri and see if anybody complains during the test suite. Sure enough, we have a smoking gun:
in `activate’: can’t activate nokogiri (~> 1.3.3, runtime) for [”sanitize-1.1.0″], already activated nokogiri-1.4.1 for [”webrat-0.5.3″]
To make a long story short, that little tilde (’~') in the error message above has a very specific meaning. The above error message means that sanitize wants nokogiri version 1.3.3 or later, but only as long as it shares the same major (1) and minor (3) versions. So “1.3.956″ works, but “1.4.0″ does not. Sadly, I can’t find this interpretation of the tilde in gem dependency specs documented anywhere. Strange, considering its impact. I have found a couple of blogs and such that point out that the tilde has this role, but nothing formal in the docs.Solution: update sanitize (and thankfully the sanitize author had updated it to be compatible with nokogiri 1.4).Afterthought: the gem dependency command shows clearly the dependency of sanitize 1.1.0 on nokogiri 1.3.x:# gem dependency -R nokogiri
Gem nokogiri-1.3.3
racc (>= 0, development)
rexical (>= 0, development)
rake-compiler (>= 0, development)
hoe (>= 2.3.2, development)
Used by
sanitize-1.1.0 (nokogiri (~> 1.3.3, runtime))
webrat-0.5.3 (nokogiri (>= 1.2.0, runtime))
mime-types-1.16 (nokogiri (~> 1.2, development))
I’ve got a family of (ARes) resources that are available through an external web service. These resources are used pervasively throughout my site -they appear on about 75% of my site’s pages. When testing, of course I don’t want to hit the web service and endure the vagaries of performance and availability. The nice people developing ActiveResource seem to have accommodated this desire by providing the HttpMock class.
But… HttpMock is very exacting in its demands for matching requests -it’s all or nothing. That means that if I make fifty different requests, all identical save for a tiny change in the URL params, I need to make fifty different mocks. Ouch.
So I decided to mock the mocker. No, really. Using mocha, I mock the HttpMock::Request#== method to always return true. Now, regardless of the request path, HttpMock always sees a match and returns my mock data.
The denouement, using the Rails docs as a starting point:
def setup
@matz = { :id => 1, :name => "Matz" }.to_xml(:root => "person")
ActiveResource::HttpMock.respond_to do |mock|
mock.post "/people.xml", {}, @matz, 201, "Location" => "/people/1.xml"
mock.get "/people/1.xml", {}, @matz
mock.put "/people/1.xml", {}, nil, 204
mock.delete "/people/1.xml", {}, nil, 200
req_res = mock.get "/", {}, "Constant Data Goes Here", 200
req_res.last.first.stubs(:==).returns(true)
end
end
Since moving to more recent versions of Rails, I’ve noticed that the auto-generated Rails tests assume that the myapp/test directory is on the load path. And the bundled Rake testing tasks do indeed take care of that requirement. But the Aptana RadRails plugins for Eclipse don’t. The result is that whenever you run a test, either with the test buttons or the Run As | Test::UnitTest context menu, you get something like the following:
./test/unit/user_test.rb:1:in `require': no such file to load -- test_helper (LoadError)
There is an open issue on the Aptana issue tracker, but it doesn’t seem to be resolved yet. Some people have suggested prefixing your requires with the test directory when requiring the test_helper file at the start of your test files -but I don’t think that’s a good solution, especially if you’re working with multiple developers. Frustrated by not getting my daily green bar fix, I found this work-around:
You should now be able to run your tests without having to edit each one to include the test directory. The downside is that you now have an ‘extra’ directory in your load path which will give you a (tiny) performance hit to all your ruby ops -not just tests. Conceivably this could also introduce an incompatibility if you have some conflicting stuff in the test directory -extremely unlikely if you only have test_helper.rb in there.
Tested against Edge Rails (d37e6413366c9a3fafa02c4298a2946dc8327a42), Aptana RadRails 1.0.3.200807071913NGT, Ruby 1.8.6 patchlevel 114 running under Darwin 9.4.0
Like a lot of Rails developers who have been spoiled by the excellent Eclipse plugins for Subversion, I was disappointed by the lack of equivalent plugins for Git. In fact, the only one that I found was Egit. Unfortunately, its documentation is a bit weak. And for someone who is not a Java guru, Eclipse guru and/or a mind reader, the installation instructions are really weak. But I was desperate. And once I dove in, it turned out to not be that difficult.
Assumptions (most directly from Egit’s sparse installation guide):
The installation steps are pretty straightforward:
You’re done.
Add Git features to an existing project from the “Team” menu.
Issues:
I’m pretty new to Macintosh development but I’ve been working in Ruby for a couple of years under Win32 and Linux. I was excited by the concept of Ruby and Rails being supported “out-of-the-box” on Leopard with the installation of XCode 3.0. But it didn’t take long for the luster to wear off.I first knew things were not going to be easy when I tried to install MySQL. Since I have a recent MacBook Pro with a Core 2 Duo processor, I went for the x86_64 disk image package (ominous background music starts now). It installed without too many difficulties (but not painless w.r.t the system preference pane). I then went to install the C-based ruby mysql bindings gem:
bimota:lib cch1$ sudo gem install mysql
Building native extensions. This could take a while...
ERROR: Error installing mysql: ERROR: Failed to build gem native extension.
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb install mysql
checking for mysql_query() in -lmysqlclient... nochecking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... nochecking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... nochecking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... nochecking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
Gem files will remain installed in /Library/Ruby/Gems/1.8/gems/mysql-2.7 for inspection.
Results logged to /Library/Ruby/Gems/1.8/gems/mysql-2.7/gem_make.out
…and thus began the disenchantment.The failure above is caused by the gem command attempting to compile the native MySQL extensions but not being able to find the necessary library and header files. This can be caused by several more fundamental problems:
Both of the above problems can apparently be solved nicely with one command line option that helps the configurator learn the appropriate build options directly from the MySQL installation:
bimota:mysql cch1$ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
Building native extensions. This could take a while...
Successfully installed mysql-2.7
1 gem installed
But you would be foolish to believe that it actually works. Run the same command with the verbose option enabled, and you can see that there is trouble on the horizon despite the apparent success:
bimota:mysql cch1$ sudo gem install -V mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
Installing gem mysql-2.7/Library/Ruby/Gems/1.8/gems/mysql-2.7/COPYING/Library/Ruby/Gems/1.8/gems/mysql-2.7/COPYING.ja/Library/Ruby/Gems/1.8/gems/mysql-2.7/README.html/Library/Ruby/Gems/1.8/gems/mysql-2.7/README_ja.html/Library/Ruby/Gems/1.8/gems/mysql-2.7/extconf.rb/Library/Ruby/Gems/1.8/gems/mysql-2.7/mysql.c.in/Library/Ruby/Gems/1.8/gems/mysql-2.7/test.rb/Library/Ruby/Gems/1.8/gems/mysql-2.7/tommy.css/Library/Ruby/Gems/1.8/gems/mysql-2.7/mysql.gemspec
Building native extensions. This could take a while...
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb install -V mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
checking for mysql_ssl_set()... no
checking for mysql.h... yes
creating Makefile
makegcc -I. -I. -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin9.0 -I. -DHAVE_MYSQL_H -I/usr/local/mysql/include -Os -arch x86_64 -fno-common -fno-common -arch ppc -arch i386 -Os -pipe -fno-common -c mysql.ccc -arch ppc -arch i386 -pipe -bundle -undefined dynamic_lookup -o mysql.bundle mysql.o -L"." -L"/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib" -L. -arch ppc -arch i386 -lruby -L/usr/local/mysql/lib -lmysqlclient -lz -lm -lpthread -ldl -lm
ld: warning in /usr/local/mysql/lib/libmysqlclient.dylib, file is not of required architecture
ld: warning in /usr/local/mysql/lib/libmysqlclient.dylib, file is not of required architecture
make install/usr/bin/install -c -m 0755 mysql.bundle /Library/Ruby/Gems/1.8/gems/mysql-2.7/libSuccessfully installed mysql-2.71 gem installed
Indeed, when you go to use (not just ‘require’) the gem, you are likely to see this nasty error:
bimota:mmweb cch1$ rake db:version(in /Users/cch1/Documents/Development/mmweb)dyld: lazy symbol binding failed: Symbol not found: _mysql_init Referenced from: /Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle Expected in: dynamic lookup
dyld: Symbol not found: _mysql_init Referenced from: /Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle Expected in: dynamic lookup
Trace/BPT trap
I’m still have no idea what causes this specific failure, but I suspect it’s related to the compiler options suggested by mysql_config being ignored/munged by the configurator. The gcc command above tries to build for three different architectures (ppc, i386, x86_64) despite mysql_config’s “recommendation” of just x86_64:
bimota:mysql cch1$ /usr/local/mysql/bin/mysql_config
Usage: /usr/local/mysql/bin/mysql_config [OPTIONS]
Options:
--cflags [-I/usr/local/mysql/include -Os -arch x86_64 -fno-common]
--include [-I/usr/local/mysql/include]
--libs [-L/usr/local/mysql/lib -lmysqlclient -lz -lm]
--libs_r [-L/usr/local/mysql/lib -lmysqlclient_r -lz -lm]
--socket [/tmp/mysql.sock]
--port [3306]
--version [5.0.51a]
--libmysqld-libs [-L/usr/local/mysql/lib -lmysqld -lz -lm]
I then try forcing the issue by setting the environment variable as well:
bimota:lib cch1$ sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_configBuilding native extensions. This could take a while...Successfully installed mysql-2.71 gem installed
Again, it looks promising. But now when Rails asks Ruby (1.8.6 from the default XCode 3.0 install) to load the mysql gem, the OS generates a load error due to some kind of a mismatch:
bimota:mmweb cch1$ script/console
Loading development environment (Rails 2.0.2)
>> require_library_or_gem 'mysql'
LoadError: dlopen(/Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle, 9): no suitable image found. Did find:
/Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle: mach-o, but wrong architecture - /Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle
from /Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:32:in `require'
from /Users/cch1/Documents/Development/mmweb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:496:in `require'
from /Users/cch1/Documents/Development/mmweb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in'
from /Users/cch1/Documents/Development/mmweb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:496:in `require'
from /Users/cch1/Documents/Development/mmweb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/kernel/requires.rb:7:in `require_library_or_gem'
from /Users/cch1/Documents/Development/mmweb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/kernel/reporting.rb:11:in `silence_warnings'
from /Users/cch1/Documents/Development/mmweb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/kernel/requires.rb:5:in `require_library_or_gem'
from (irb):1
>> ^Dbimota:mmweb cch1$
Why the mismatch? Some googling led me to this post that notes that the Ruby interpreter bundled in XCode 3.0 is in fact only compiled as a 32-bit i386 executable. That’s right: the latest and greatest Macs (with Intel Core 2 Duo processors) running the latest and greatest OS (Leopard 10.5.2) are shipping with a neutered Ruby interpreter.At this point, I see two solutions:1. Install the i386/32-bit only version of MySQL (boo!). I’ve confirmed that this allows a nice working mysql binding to be built with the following command:
bimota:mysql cch1$ sudo env ARCHFLAGS="-arch i386" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
Building native extensions. This could take a while...Successfully installed mysql-2.71 gem installedbimota:mysql cch1$
And not only does it compile, it actually works.2. Recompile Ruby with x86_64 support. That sounds like a bigger job -but there are web sites that show you how to get started. For now though, I’m leaving XCode alone until I know what works -that way, when it breaks, I can assign the failure to the 64-bit ruby interpreter.
I read Eric’s blog on creating an ActiveRecord validation for an exactly-one-of predicate. I myself had just recently gone through a similar exercise and found a straightforward way of expressing it using the validate method:
validate :exactly_one_of?
def exactly_one_of?
attrs = [:url, :file, :blob]
returning (attrs.inject(false){|m, o| m^self[o]}) do |bool|
self.errors.add(:base, “Exactly one of #{attrs} must be set.”) unless bool
end
end
While this worked nicely for me, Eric went to the trouble of building a semantically clean validation method that mimics the options of the standard validations. I prefer using Ruby’s Exclusive OR operator combined with inject instead of the looping in Eric’s method, but I like the semantic style he defined. Combining his semantics with my logic yields something slick and DRY like this:
def self.validate_exactly_one_of(*attrs)
options = attrs.last.is_a?(Hash) ? attrs.pop : {}
configuration = { :message => “one of #{attrs.to_sentence :connector => ‘or’} must be set”, :on => :save }
configuration.update(options)send(validation_method(options[:on] || :save)) do |record|
returning (attrs.inject(false){|m, o| m^record[o]}) do |bool|
self.errors.add(:base, configuration[:message]) unless bool
end
end
end
I also fiddled the options logic because I think otherwise it would have been possible to have an attribute listed in the error message list that was in fact an option.