Starting Programs at Boot using Supervisor Process Control Utility

Introduction

In this article we look at how to automatically to automatically start a set of programs upon startup using ‘Supervisor’ process control tool for embedded

Typically in any embedded application we require a set of services to be running upon startup or boot into operating system.

sudo apt-get install supervisor

Supervisor is a client/server system that allows its users to control of processes on UNIX-like operating systems. It can be user to start/stop/restart/monitor a single or group of processes

Supervisor consists of following important components

• supervisord - Server
• supervisorctl - commandline client
• Web Server

Installing

If the Python interpreter you’re using has Setuptools installed, and the system has internet access, you can download and install supervisor in one step using easy_install.

easy_install supervisor

Depending on the permissions of your system’s Python, you might need to be the root user to install Supervisor successfully using easy_install.

Or you can download the supervisor package from http://pypi.python.org/pypi/supervisor and install it. After unpacking the downloaded software archive, run

python setup.py install

It will download and install all distributions depended upon by Supervisor and finally install Supervisor itself.

Starting supervisor upon startup

In Unix-based computer operating systems, init (short for initialization) is the first process started during booting of the computer system. Init is a daemon process that continues running until the system is shut down

Init scripts are the scripts located in /etc/init.d. These scripts are part of the bootup sequence of Ubuntu.

During boot, they are not called directly, but through a structure of symbolic links which manage the services which are to be started in a particular runlevel. The scripts which are symlinked from /etc/rcS.d are executed first. Then the scripts in /etc/rcN.d/ are executed.

The naming convention of symlinks are /etc/rc[L].d/[S/K][NN]name

• L is the chosen runlevel (default 2)

  runlevel S is one of the runlevels supported by init, namely, ` 0123456789S`

  Runlevel 0 is used to halt the system and 6 to reboot the system and 2 is default runlevel ,1 is runlevel to boot into single user more

• NN is the two-digit sequence number that determines where in the sequence init will run the scripts.

• The K links are responsible for killing services and the S link for starting services upon entering the runlevel.

After installation we can find the file supervisor in the /etc/init.d/ directory indicating that it can be accessed as a service

root@prasad-TA790GX-A3:/media/sda2# ls -alrt /etc/rc2.d/S20supervisor
lrwxrwxrwx 1 root root 20 Dec  5 13:01 /etc/rc2.d/S20supervisor -> ../init.d/supervisor
root@prasad-TA790GX-A3:/media/sda2# 

we can also see that symlink for supervisor is present in rc2.d directory.Runlevel 2 is default runlevel in Ubuntu OS.Indicating that the service will start upon boot.And through supervisor we can control various processes being managed by supervisor.This include starting a set of processes at boot and stopping a set of processes at shutdown/reboot.

####Creating a Configuration File Once the Supervisor installation has completed, run echo_supervisord_conf. This will print a “sample” Supervisor configuration file to your terminal’s stdout.

Once you see the file echoed to your terminal

echo_supervisord_conf > supervisord.conf.

Once you have a configuration file on your filesystem, you can begin modifying it to your liking.

####Adding a Program

To add a program, you’ll need to edit the supervisord.conf file. The section relevant to present article is ` [program:x] Section ` of configuration file .

The program section will define a program that is run and managed when you invoke the supervisord command.

The configuration file must contain one or more program sections in order for supervisord to know which programs it should start and control

The header value is composite value. It is the word “program”, followed directly by a colon, then the program name

[program:foo]
command="/bin/echo %(program_name)s_%(process_num)02d "
numprocs=1 
process_name=%(program_name)s
redirect_stderr: true
autorestart: true

In the above example a header value of [program:foo] describes a program with the logical name of “foo”

Command The command section value will contain the command that will be run when this program is started. command line arguments can be passed to program .In the above configuration the command line arguments are predefined string expressions supported by supervisor

Now starting the supervisor service will create a file in the tmp directory

foo-stdout---supervisor-XXX.log and foo-stderr---supervisor-a35d4Z.log

The contents of /tmp/foo-stdout---supervisor-XXX.log will contain the output of echo command which is pushed onto the stdout

logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log) specified in the supervisord section of the configuration file defines where the log files will be created

foo_00
foo_00
foo_00
foo_00

Thus we can see that logical program name is foo and value assigned to process_num is a 2 digit decimal value 00 due to %(process_num)02d string literal expression.

The section also include autorestart parameter which If set to true will cause the process to be unconditionally restarted when it exits, without regard to its exit code.We can see that it will restart the above program a fixed number of times .However if the program has some fatal error due to which it restarts too many times,it will stop the restart process .

Since echo command terminates immediately,it will stop after some attempts.In present case after 4 attempts.We can observe the following lines in /tmp/supervisord.log file which indicate why program was terminated

2014-12-05 14:39:43,153 CRIT Supervisor running as root (no user in config file)
2014-12-05 14:39:43,155 INFO daemonizing the supervisord process
2014-12-05 14:39:43,157 INFO supervisord started with pid 5344
2014-12-05 14:39:44,161 INFO spawned: 'foo' with pid 5345
2014-12-05 14:39:44,172 INFO exited: foo (exit status 0; not expected)
2014-12-05 14:39:45,176 INFO spawned: 'foo' with pid 5346
2014-12-05 14:39:45,188 INFO exited: foo (exit status 0; not expected)
2014-12-05 14:39:47,193 INFO spawned: 'foo' with pid 5347
2014-12-05 14:39:47,206 INFO exited: foo (exit status 0; not expected)
2014-12-05 14:39:50,211 INFO spawned: 'foo' with pid 5349
2014-12-05 14:39:50,221 INFO exited: foo (exit status 0; not expected)
2014-12-05 14:39:51,222 INFO gave up: foo entered FATAL state, too many start retries too quickly

Multiprocess applications

A [program:x] section can also be used to represents a “homogeneous process group” to supervisor (as of 3.0). The members of the group are defined by the combination of the numprocs and process_name parameters in the configuration

[program:foo1]
command=/bin/echo %(program_name)s_%(process_num)02d 
numprocs=3 
process_name=%(program_name)s_%(process_num)02d
redirect_stderr: true
autorestart: false
startretries=0  

numprocs Supervisor will start as many instances of this program as named by numprocs.This can be used in case of multiprocess applications.

process_name A Python string expression that is used to compose the supervisor process name for this process. This is mandatory if numproc section value >1.

The default value is :%(program_name)s

It can contain string expression values : group_name, host_node_name, process_num, program_name

Example

But for instance, if you have a [program:foo1] section with a numprocs of 3 and a process_name expression of %(program_name)s_%(process_num)02d, the “foo1” group will contain three processes, named foo1_00, foo1_01, and foo1_02 as seen below

-rw-------  1 root    root       16 Dec  5 15:08 foo1_02-stdout---supervisor-AkgbT1.log
-rw-------  1 root    root       16 Dec  5 15:08 foo1_01-stdout---supervisor-LYqA26.log
-rw-------  1 root    root       16 Dec  5 15:08 foo1_00-stdout---supervisor-xRoT5o.log

The startretries=0 and autorestart=false cause the program to not be restarted upon exit.

This makes it possible to start a multiprocess application from a single program configuration

####Running supervisord

To start supervisord, run supervisord. The resulting process will daemonize itself and detach from the terminal. It keeps an operations log at supervisor.log file in the directory specified by configuration files

sudo supervisord -c "$HOME/supervisord.conf"

While starting the supervisord service using init.d scripts the command line arguments cannot be passed.In which case it searches the configuration file in default location /etc/supervisord.conf

Web Server

The various processes managed by the supervisor can be accessed by webservice.

[inet_http_server]         ; inet (TCP) server disabled by default
port=*:9001       		   ; (ip_address:port specifier, *:port for all iface)
username=user              ; (default is no username (open server))
password=123               ; (default is no password (open server))

The following section must be present in the supervisord.conf file so that webserver is started when supervisord service is started

The webservice can be accessed from the browser by accessing the url http://${hostname}:9001/ for example in the present case http://192.168.1.3:9001

The sample supervisord.conf files and sample script used in article can be found at tutorials/supervisor tree path of github pyVision repository