Supervisord

I’ve recently written a couple of projects that run as a python-based daemon. I want them to run as long-lived projects on a VPS, with automatic restart should they fail. I thought I was going to have to go through the tedium of writing init.d scripts for each.

Then I stumbled upon supervisord, which is already available in Debain. As packaged, it automatically starts as any other python daemon. However, the benefit for us as developers is that it offers a really straight-forward interface for turning our applications into long-lived daemons themselves.

$ apt-get install supervisor

This will create a configuration (albeit empty) in /etc/supervisor/conf.d ready for your individual project configurations, and then will auto-start supervisord itself as root.

So, let’s say this is the command line I run normally:

root:/home/deployed/myapp$ python main.pyc

I therefore add this to /etc/supervisor/conf.d/myapp.conf

[program:myapp]
directory=/home/deployed/myapp
command=python main.pyc

The defaults for supervisord are sensible. A program defined like this will auto-start; will auto-restart on unexpected stop.

Without doing anything else, we now access the supervisord terminal.

$ supervisorctl
supervisor> help

default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update 
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

We have it rescan the configuration:

supervisor> reread
myapp: available
supervisor> avail
myapp              avail     auto      999:999

and our application becomes available. It’s not actually running yet of course; it’s simply available to run. Before we do that though, let’s improve our configuration.

[program:myapp]
directory=/home/deployed/myapp
command=python main.pyc -d
user=deployed
redirect_stderr=true
stdout_logfile=/home/deployed/myapp/log/myapp.log
stdout_logfile_maxbytes=10MB

The app itself logs to stderr, when we run it as a daemon that means we won’t get to see the output. supervisord conveniently includes redirect and capture facilities so we can turn that stderr output into a file. Even better, it also includes automatic log rotation above a size threshold (which I’ve set here to 10MB). By default supervisord will pick a location (/var/log/supervisor by default) and will auto-delete any log file on exit. Setting it by hand causes it to live longer.

Back to supervisorctl: we could now start our daemon with the add command (after another reread), but better is to rescan and start/stop whatever needs doing in one simple command:

supervisor> update
myapp: added process group
supervisor> status
myapp              RUNNING    pid 23386, uptime 0:00:07

supervisord will detect if the application keep cyclically starting and stopping and will remove it from its process table. That prevents your badly behaved app from sucking all the cycles from your system. My first use happened to do exactly that; so very soon the status command gave:

supervisor> status
myapp              FATAL      Exited too quickly (process log may have details)

In my case, I’d forgotten to supply a password, easily discovered by reading the log file that supervisord created.

You can check for these runaway processes in /var/log/supervisor/supervisord.log, in case your crashes are happening infrequently and unpredictably. It’s not of use to me, but it can also manage FastCGI applications, sharing a single port,

All in all, I’m absolutely delighted. supervisord has literally just saved me at least a day of work, if not more. It’s also made it unnecessary to add a load of extra code to my applications just to make them daemonizable. Very nice.

This entry was posted in FussyLogic and tagged , , . Bookmark the permalink. Trackbacks are closed, but you can post a comment.

Post a Comment

You must be logged in to post a comment.