Restarting uvicorn Workers with the SIGHUP Signal

The most popular web server in the Python world is probably uvicorn, which plays nicely with many different deployment scenarios. Internally, uvicorn starts a predefined number of worker processes running below the main process, which manages the workers and distributes incoming requests to them. During runtime, the worker processes can be influenced by sending certain signals to the main process, which is the topic of today's article.

As it turns out, uvicorn ships with built-in handlers for various signals. This is useful in all deployment scenarios, but even more so if you work with deployments in environments requiring zero downtime. For the sake of this article, the ASGI application itself doesn't really matter so that we will be using an empty FastAPI application as a placeholder ...

from fastapi import FastAPI
import uvicorn

app = FastAPI()


if __name__ == "__main__":
    uvicorn.run(app="app:app", workers=2)

If you save this application to app.py, you can run it with python app.py. As you can see, it immediately logs that it has started two worker processes as requested in the configuration passed to the uvicorn.run() function.

$ python app.py
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started parent process [4147238]
INFO:     Started server process [4147241]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [4147240]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

You can use a tool like pstree to inspect the main process with PID 4147238 and its child processes.

$ pstree --hide-threads --compact-not --show-pids 4147238
python(4147238)─┬─python(4147239)
                ├─python(4147240)
                └─python(4147241)

Just for fun, we can terminate one of the worker processes and see if the main process correctly fulfills its responsibility by spawning a new worker process. To achieve this, I send a SIGINT signal to one of the worker processes.

$ kill -SIGTERM 4147240

Meanwhile, here is uvicorn's reaction as logged to the standard output.

INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [4147240]
INFO:     Waiting for child process [4147240]
INFO:     Child process [4147240] died
INFO:     Started server process [4147687]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

So, essentially, the main process detected that the worker with PID 4147240 died and immediately spawned a new worker with PID 4147687 to replace the old one.

$ pstree --hide-threads --compact-not --show-pids 4147238
python(4147238)─┬─python(4147239)
                ├─python(4147241)
                └─python(4147687)

Restarting all Workers with the SIGHUP Signal

By sending a SIGHUP signal to the main process, you can achieve the same behavior for all workers in a more controlled fashion. By behavior, I mean all workers are terminated and replaced with fresh worker processes. The main process keeps running at all times — its process ID does not change. Doing so doesn't result in any failed HTTP requests because uvicorn is waiting, for each worker process, until it has finished the HTTP request it is currently working on, before terminating it.

Continuing the example above, we can send the main process a SIGHUP signal.

$ kill -SIGHUP 4147238

And observe how uvicorn reacts ...

INFO:     Received SIGHUP, restarting processes.
INFO:     Terminated child process [4147241]
INFO:     Waiting for child process [4147241]
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [4147241]
INFO:     Terminated child process [4147687]
INFO:     Waiting for child process [4147687]
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [4147687]
INFO:     Started server process [4148369]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [4148370]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

As you can tell from their changed PIDs, the two old workers have been replaced with new worker processes.

$ pstree --hide-threads --compact-not --show-pids 4147238
python(4147238)─┬─python(4147239)
                ├─python(4148369)
                └─python(4148370)

Increasing the number of Workers with the SIGTTIN Signal

Another cool feature of uvicorn is that you can send a SIGTTIN signal to the main process to increment the number of worker processes by one. You can get by without restarting the main process, which may not be permissible if you work in an environment that requires uptime without interruptions.

Continuing the example above, we can use the SIGTTIN signal to scale up to three workers.

$ kill -SIGTTIN 4147238

Here is uvicorn's reaction and the updated pstree output.

INFO:     Received SIGTTIN, increasing the number of processes.
INFO:     Started server process [4158857]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

$ pstree --hide-threads --compact-not --show-pids 4147238
python(4147238)─┬─python(4147239)
                ├─python(4148369)
                ├─python(4148370)
                └─python(4158857)

Decreasing the number of Workers with the SIGTTOU Signal

Last but not least, you can also send it SIGTTOU to decrease the number of workers by one. This, too, works without restarting the main process and, therefore, without downtime. Note that you can scale down to a single worker, but you can't get any lower since that wouldn't make any sense.

Continuing the example above, we can use the SIGTTOU signal to scale back down to two workers.

$ kill -SIGTTOU 4147238

Here is uvicorn's reaction and the updated pstree output.

INFO:     Received SIGTTOU, decreasing number of processes.
INFO:     Terminated child process [4158857]
INFO:     Waiting for child process [4158857]
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [4158857]

$ pstree --hide-threads --compact-not --show-pids 4147238
python(4147238)─┬─python(4147239)
                ├─python(4148369)
                └─python(4148370)

Conclusion

I hope you found this article helpful and have learned something new. If so, please consider subscribing to my mailing list to receive updates about newly published articles on Web Development. Thank you, and see you soon!