Monitoring long running processes with supervisord

I’m modernising a 12+ year old PHP code base, and to improve the application performance, I decided to run some tasks asynchronously. It’s notoriously difficult to write asynchronous PHP code, so I took an alternative approach using Beanstalk queues, where a task execution is deferred to a later time. This approach has its pros and cons, but for now it’s good enough for this system.

A worker watches the queue for new jobs, and when those arrive, it processes them. The worker - a simple PHP script - must be run all the time, which can be done in many different ways. The easiest solution would be setting up a cronjob, but it has a few drawbacks (e.g. properly monitoring the tasks, handling script crashes, etc). I thought about using monit or Upstart/Systemd, but it was definitely overkill. Then I got to supervisord: it’s almost as simple as cron, but with many more features.

Installing supervisord

To install supervisord, you must have Python installed. You can install it either via pip or through a package manager, although these packages tend to be outdated.

pip install supervisor

If you install it using pip, you will also need to setup a init script. The official repository contains the scripts for the most popular Linux distros.

For Ubuntu, you need to copy the script to /etc/init.d/supervisord, and then update its permissions:

sudo chmod +x /etc/init.d/supervisord

Next, you need to enable the service:

sudo update-rc.d supervisord defaults
sudo service supervisord start

Configuration

To configure supervisord, you need to create a new configuration file. supervisord comes with the echo_supervisord_conf command, which bootstraps this process:

sudo echo_supervisord_conf > /etc/supervisord.conf

After that, append the following to /etc/supervisord.conf. This setting will inform supervisord to look for our monitors in the directory /etc/supervisord.d/.

[include]
files=/etc/supervisord.d/*.conf

Then, create the monitors directory:

mkdir -p /etc/supervisord.d/

Finally, create a directory to store the monitors’ logs:

mkdir -p /var/log/supervisord

Process control

Creating a new monitor is trivial - you only need to create a new configuration file and reload supervisord.

Below is a monitor with very basic settings:

[program:my-queue]
command=php /home/mordor/www/script.php
numprocs=1
numprocs_start=1
autostart=true
autorestart=true
stderr_logfile=/var/log/supervisor/queue.err.log
stdout_logfile=/var/log/supervisor/queue.out.log

• [program:my-queue]: the program name (it will be used as the display name in supervisord).
• command: the command supervisord will run.
• numprocs: the number of instances of the program that we want to run.
• numprocs_start: the number of instances of the program that we want to run at the start of the program.
• autostart: specifies if the process should be automatically started when supervisord starts.
• autorestart: specifies if the process should be automatically restarted if it exits with an exit code different than 0 or 2.
• stderr_logfile: the path of the stderr log.
• stdout_logfile: the path of the stdout log.

After creating your monitor, you must reload supervisord by running supervisorctl update.

Managing processes

supervisord also provides the supervisorctl command line utility. All the available commands can be found in the documentation.

Start/stop/restart all workers

supervisorctl [start|stop|restart] all

Start/stop/restart a single worker

supervisorctl [start|stop|restart] <worker>

Reread configs and restart supervisord

supervisorctl update

Monitor status

supervisorctl status