for the longest time, we've been hunting down an annoying bug here at swivel. in our test cases, while generating graphs, we would hit deadlocks, surly ones with stink eyes. weirdly, it was around assigning random (but nicely saturated) colors to new values or columns. we couldn't reproduce the problem in development nor in production, so we just kept working around it, 'cuz well, you know.. we're lazy sometimes. but in our recent push towards 96% test coverage, the deadlocks came back with a frightening vengeance. so i got to take ruby-debug for a good spin to finally figure out what was going on.
turns out, it was a combination of erb, activerecord transactions, and ruby's safe-level taint checking. first, everyone (?) knows that rails is a multithread-UNfriendly. this is related, sadly. in our graph generation pipeline, we use erb to create a ploticus template. trying to be $SAFE, we call
ERB.new template, 0, '%<>'
this causes erb to evaluate the template in a seperate thread. during erb evaluation, some methods call back to get some color information. and sometimes, to pick pretty colors for certain values or columns, we end up doing some database work (so that shale shade for how many shots of espresso ms. nic has drank stays shale-y). and it does all of that sexy database work within a sexy transaction. now, the o.g. thread that called for the graph to be generated is also most likely in a transaction. and if you delve into rails's transaction code, it uses a mutex to make sure that the begin and end of a transaction are demarcated correctly at the right time (handled differently in 1.2 rails, btw). this is totally fine as long as you're in the same thread, but try this the way we were trying it, and you hit a DEADLOCK OF SHAME. and we weep.
but fortunately, there's a quick fix for this. just call ERB.new with a nil safe-level and everything will stay in the same thread. and all will be ok for now.
as i mentioned, this only applies to old, crufty rails 1.1. if you happen to know how well rails 1.2 or edge rails plays with allow_concurrency nowadays, be sure to leave a comment..
Recent Comments