Supervisord: less server babysitting
By Robin Andeer. This article was published on March 28, 2015.
Supervisord is a simple monitoring tool. It consists of a server and an accompanying command line interface to manage and check the status of long running processes.
Motivations for adding complexity
Adding yet another component should always be weighted against the consequences of adding more complexity. A successful deployment of supervisord will do much of the babysitting involved in server maintenance. But we all know that it's never that easy and a simple Bash script in a crontab will likely take you 90% of the way.
Having said that, here are a few really nice benefits you get from using supervisord:
- A unified interface for starting, stopping, and reloading your processes
- Automatic restarts if your processes go down
- Ability to define startup scripts using intuitive yet powerful
Supervisord is really simple to install:
$ sudo apt-get install supervisor # or $ pip install supervisor
Note that I will assume you will be running supervisord as non-root throughout this tutorial.
Setup of config file
The first thing we need to do is write a config file where we specify all of our processes to run as well as setup some supervisord settings. This is a minimal example:
# ~/setup/supervisord.conf [supervisord] pidfile = /home/www/logs/supervisord.pid logfile = /home/www/logs/supervisord.log [program:flask_server] command = python app.py directory = /home/www/flask_server autostart = true
directorysetting will make sure supervisord cd's to that path before executing the
autostartwill start that process automatically as soon as supervisord is initialized.
Now there are a few additional settings that I had to include in the config file for the suite to work full out. Right now I'm not entirely sure what everything does - but it does make the whole thing tick.
[inet_http_server] port = 127.0.0.1:9001 [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface # ... [supervisorctl] # use an http:// url to specify an inet socket serverurli = http://127.0.0.1:9001
Fleshing out the program settings
Let's complete our supervisor config by adding some additional settings to the server to track logs as well as including the MongoDB server which the Flask server communicates with.
Note here that
mongodhandles logging on it's own. This is all defined in the
# ... [program:mongod] command = mongod --config /home/www/setup/mongo.conf autostart = true autorestart = true [program:flask_server] command = python app.py directory = /home/www/flask_server autostart = true autorestart = true stdout_logfile=/home/www/logs/flask_server.out.log stderr_logfile=/home/www/logs/flask_server.err.log # ...
autorestart = true we can make sure that if the server goes down for any unexpected reason it will be automatically restarted without any manual intervention! This can also be used in other clever ways like killing the server to issue a git pull before supervisord automatically restarts the Flask server again.
One thing I learned was that supervisord wants you to run long running processes. In case of
mongod, this means that you shouldn't let is fork the process to run in the background; let supervisord worry about that!
Now we only need to start the supervisord server to get everything up and running.
supervisord --config /home/www/setup/supervisord.conf
Next up we can manage processes using
supervisorctl. We can for example easily check the status of all processes (programs) in the config file.
$ supervisord --config /home/www/setup/supervisord.conf status mongod RUNNING pid 39188, uptime 23:27:51 flask_server RUNNING pid 39237, uptime 23:27:28
One thing I really like is how
supervisorctl can be used as a unified interface for starting, restarting, and stopping services which is really easy to define and share with your team.
$ supervisorctl --config /home/www/setup/supervisord.conf restart flask_server flask_server: stopped flask_server: started
That's it for my brief introduction to supervisord. I hope it's enough to get you up and running so that you can spend less time worrying and babysitting your servers.
I will continue to improve the supervisord config file as I go a long. Keep an eye out to find out more.
And for reference; the complete config file of course:
# ~/setup/supervisord.conf [inet_http_server] port = 127.0.0.1:9001 [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisord] pidfile = /home/www/logs/supervisord.pid logfile = /home/www/logs/supervisord.log [program:mongod] command = mongod --config /home/www/setup/mongo.conf autostart = true autorestart = true [program:flask_server] command = python app.py directory = /home/www/flask_server autostart = true autorestart = true stdout_logfile=/home/www/logs/flask_server.out.log stderr_logfile=/home/www/logs/flask_server.err.log [supervisorctl] # use an http:// url to specify an inet socket serverurli = http://127.0.0.1:9001