Tasty Data Goodies

The database needed some love

We're upgrading parts of the database right now and as always, it's taking longer than we expected.  It should be about half an hour before we get Swivel back up and running.

Offhand, for those of you engineering-inclined, we're also working on a much smaller (zero?) scheduled downtime window from now on for these types of big database upheavals.  If you happen to love migrating databases with zero downtime (and you're amazingly good at it), you know we're hiring, right?

A little maintenance

We're just making some quick tweaks here and there to the database for some speed gains.  Bear with us; we should be back up in about 30 minutes.

ERb, ActiveRecord, deadlocks, and you

for the longest time, we've been hunting down an annoying bug here at swivel.  in our test cases, while generating graphs, we would hit deadlocks, surly ones with stink eyes.  weirdly, it was around assigning random (but nicely saturated) colors to new values or columns.  we couldn't reproduce the problem in development nor in production, so we just kept working around it, 'cuz well, you know.. we're lazy sometimes.  but in our recent push towards 96% test coverage, the deadlocks came back with a frightening vengeance.  so i got to take ruby-debug for a good spin to finally figure out what was going on.

turns out, it was a combination of erb, activerecord transactions, and ruby's safe-level taint checking.  first, everyone (?) knows that rails is a multithread-UNfriendly.  this is related, sadly.  in our graph generation pipeline, we use erb to create a ploticus template.  trying to be $SAFE, we call

ERB.new template, 0, '%<>'

this causes erb to evaluate the template in a seperate thread.  during erb evaluation, some methods call back to get some color information.  and sometimes, to pick pretty colors for certain values or columns, we end up doing some database work (so that shale shade for how many shots of espresso ms. nic has drank stays shale-y).  and it does all of that sexy database work within a sexy transaction.  now, the o.g. thread that called for the graph to be generated is also most likely in a transaction.  and if you delve into rails's transaction code, it uses a mutex to make sure that the begin and end of a transaction are demarcated correctly at the right time (handled differently in 1.2 rails, btw).  this is totally fine as long as you're in the same thread, but try this the way we were trying it, and you hit a DEADLOCK OF SHAME.  and we weep.

but fortunately, there's a quick fix for this.  just call ERB.new with a nil safe-level and everything will stay in the same thread.  and all will be ok for now.

as i mentioned, this only applies to old, crufty rails 1.1.  if you happen to know how well rails 1.2 or edge rails plays with allow_concurrency nowadays, be sure to leave a comment..

Database Adjustment

We found an issue with Swivel that is causing some slowness on the site.  We're going to perform a quick fix for it. Our intention is that this will take less than 30 minutes.  Sorry for the interruption to your swiveling.

UPDATE: that was quicker than we thought.  The downtime was only 78 seconds according to huned.