I’ve recently written a couple of projects that run as a python-based daemon. I want them to run as long-lived projects on a VPS, with automatic restart should they fail. I thought I was going to have to go through the tedium of writing init.d
scripts for each.
Then I stumbled upon supervisord, which is already available in Debain. As packaged, it automatically starts as any other python daemon. However, the benefit for us as developers is that it offers a really straight-forward interface for turning our applications into long-lived daemons themselves.
$ apt-get install supervisor
This will create a configuration (albeit empty) in /etc/supervisor/conf.d
ready for your individual project configurations, and then will auto-start supervisord
itself as root
.
So, let’s say this is the command line I run normally:
root:/home/deployed/myapp$ python main.pyc
I therefore add this to /etc/supervisor/conf.d/myapp.conf
[program:myapp]
directory=/home/deployed/myapp
command=python main.pyc
The defaults for supervisord
are sensible. A program defined like this will auto-start; will auto-restart on unexpected stop.
Without doing anything else, we now access the supervisord
terminal.
$ supervisorctl
supervisor> help
default commands (type help <topic>):
=====================================
add clear fg open quit remove restart start stop update
avail exit maintail pid reload reread shutdown status tail version
We have it rescan the configuration:
supervisor> reread
myapp: available
supervisor> avail
myapp avail auto 999:999
and our application becomes available. It’s not actually running yet of course; it’s simply available to run. Before we do that though, let’s improve our configuration.
[program:myapp]
directory=/home/deployed/myapp
command=python main.pyc -d
user=deployed
redirect_stderr=true
stdout_logfile=/home/deployed/myapp/log/myapp.log
stdout_logfile_maxbytes=10MB
The app itself logs to stderr, when we run it as a daemon that means we won’t get to see the output. supervisord
conveniently includes redirect and capture facilities so we can turn that stderr output into a file. Even better, it also includes automatic log rotation above a size threshold (which I’ve set here to 10MB). By default supervisord
will pick a location (/var/log/supervisor
by default) and will auto-delete any log file on exit. Setting it by hand causes it to live longer.
Back to supervisorctl
: we could now start our daemon with the add
command (after another reread
), but better is to rescan and start/stop whatever needs doing in one simple command:
supervisor> update
myapp: added process group
supervisor> status
myapp RUNNING pid 23386, uptime 0:00:07
supervisord
will detect if the application keep cyclically starting and stopping and will remove it from its process table. That prevents your badly behaved app from sucking all the cycles from your system. My first use happened to do exactly that; so very soon the status
command gave:
supervisor> status
myapp FATAL Exited too quickly (process log may have details)
In my case, I’d forgotten to supply a password, easily discovered by reading the log file that supervisord
created.
You can check for these runaway processes in /var/log/supervisor/supervisord.log
, in case your crashes are happening infrequently and unpredictably. It’s not of use to me, but it can also manage FastCGI applications, sharing a single port,
All in all, I’m absolutely delighted. supervisord
has literally just saved me at least a day of work, if not more. It’s also made it unnecessary to add a load of extra code to my applications just to make them daemonizable. Very nice.