Swivel is built using Ruby on Rails. This may be nice for developers, but for sys admins it's a nightmare. Rails is single threaded, so it can only process one request at a time. That means instead of running one app server process that can serve multiple requests at once, you have to run multiple rails app server processes bound to different ports. Seriously, WTF!
Since the app server is single threaded, a slow page can cause Mongrel, our rails app server, to stop serving any more requests until it's completed. Depending on how many Mongrel servers you have running, your site can stop responding if you are hit with multiple slow requests at the same time. Another effect of being single threaded is that each Mongrel process has to store it's own copy of the code. This means Mongrel eats memory, big time. I've seen a Mongrel process eat up 1-2GB on a stupid request before. That's just one of the many Mongrel processes running on a server. Mongrel dies a lot too.
Even with how much rails sucks, there is a nice tool out there to help you wrangle the rabid Mongrel. We use Monit to handle all of the tedious administration tasks. Monit is a "utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations." We use Monit to monitor each Mongrel process for these three reoccurring problems:
1. crashing
2. memory hogging
3. hung processing
Here's how you setup Monit to make administering a rails app less crappy.
Build Monit from source or grab the pre-built package from here.
Throw the config below into /etc/monitrc and start it up with the monit command. Make sure you go through the config and change it to match your setup. Here's a link to explain the config a little more.
set daemon 60
set logfile syslog facility log_daemon
set mail-format {
from: monit@swivel.com
subject: $SERVICE $EVENT at $DATE
}
set alert shitsdown@swivel.com # receive all alerts
set httpd port 2812 and
use address localhost
allow localhost
##### mongrel 3000 #####
check process app05-mongrel-3000 with pidfile /home/build/apps/swivel/current/tmp/pids/mongrel.3000.pid
start program = "/etc/init.d/mongrel_single restart 3000"
stop program = "/etc/init.d/mongrel_single stop 3000"
if memory is greater than 420.0 MB for 3 cycles then restart # eating up memory?
if cpu is greater than 50% for 3 cycles then alert # send an email to admin
if cpu is greater than 80% for 3 cycles then restart # hung process?
if loadavg(5min) greater than 10 for 8 cycles then restart # bad, bad, bad
if 3 restarts within 5 cycles then timeout # something is wrong, call the sys-admin
if failed port 3000 protocol http # check for response
with timeout 10 seconds
for 3 cycles
then restart
group dev
Now Monit will restart Mongrel individually if it crashes, restart when a process consumes 420MB of memory or is stuck on a long running process. ggg'yeah!
There are a couple more promising options in the near future. mod_rails is a rails module for Apache similiar to mod_php. With mod_rails Apache would handle the threads instead of stupid rails. Also there are plans for threaded support in the next 2.2 release of rails.
Comments