I’ve recently written a couple of projects that run as a python-based daemon. I want them to run as long-lived projects on a VPS, with automatic restart should they fail. I thought I was going to have to go through the tedium of writing
init.d scripts for each.
Then I stumbled upon supervisord, which is already available in Debain. As packaged, it automatically starts as any other python daemon. However, the benefit for us as developers is that it offers a really straight-forward interface for turning our applications into long-lived daemons themselves.
$ apt-get install supervisor
This will create a configuration (albeit empty) in
/etc/supervisor/conf.d ready for your individual project configurations, and then will auto-start
supervisord itself as
So, let’s say this is the command line I run normally:
root:/home/deployed/myapp$ python main.pyc
I therefore add this to
[program:myapp] directory=/home/deployed/myapp command=python main.pyc
The defaults for
supervisord are sensible. A program defined like this will auto-start; will auto-restart on unexpected stop.
Without doing anything else, we now access the
$ supervisorctl supervisor> help default commands (type help <topic>): ===================================== add clear fg open quit remove restart start stop update avail exit maintail pid reload reread shutdown status tail version
We have it rescan the configuration:
supervisor> reread myapp: available supervisor> avail myapp avail auto 999:999
and our application becomes available. It’s not actually running yet of course; it’s simply available to run. Before we do that though, let’s improve our configuration.
[program:myapp] directory=/home/deployed/myapp command=python main.pyc -d user=deployed redirect_stderr=true stdout_logfile=/home/deployed/myapp/log/myapp.log stdout_logfile_maxbytes=10MB
The app itself logs to stderr, when we run it as a daemon that means we won’t get to see the output.
supervisord conveniently includes redirect and capture facilities so we can turn that stderr output into a file. Even better, it also includes automatic log rotation above a size threshold (which I’ve set here to 10MB). By default
supervisord will pick a location (
/var/log/supervisor by default) and will auto-delete any log file on exit. Setting it by hand causes it to live longer.
supervisorctl: we could now start our daemon with the
add command (after another
reread), but better is to rescan and start/stop whatever needs doing in one simple command:
supervisor> update myapp: added process group supervisor> status myapp RUNNING pid 23386, uptime 0:00:07
supervisord will detect if the application keep cyclically starting and stopping and will remove it from its process table. That prevents your badly behaved app from sucking all the cycles from your system. My first use happened to do exactly that; so very soon the
status command gave:
supervisor> status myapp FATAL Exited too quickly (process log may have details)
In my case, I’d forgotten to supply a password, easily discovered by reading the log file that
You can check for these runaway processes in
/var/log/supervisor/supervisord.log, in case your crashes are happening infrequently and unpredictably. It’s not of use to me, but it can also manage FastCGI applications, sharing a single port,
All in all, I’m absolutely delighted.
supervisord has literally just saved me at least a day of work, if not more. It’s also made it unnecessary to add a load of extra code to my applications just to make them daemonizable. Very nice.