Subscribe to the feed.
The recent server benchmark posted here comparing Erlang, Haskell, and Python understandably upset many people. Today we will move away from polarizing benchmarks and more about features.
Even though Erlang may not be the fastest language, it still has at least one great feature: hot swapping code without restarting the server. This is something that every long running server should have, but unfortunately it comes at the cost of performance, and some languages simply are not built to support it.
However, here is a way to emulate it for a web server, thanks to techniques borrowed from Nginx.
Here are the steps:
- Use the spawn family of system calls (ex. spawnl spawnvp) with the flag P_NOWAIT to execute itself. This new process has access to the open sockets.
- The new process should open a predetermined file containing the file descriptors of the sockets it needs to use saved by the old process.
- Start calling “accept” on the same listening sockets that the old process was accepting.
- Signal the old process to stop accepting, and to quit when all requests are finished.
While Python has a rudimentary reload, it may still be preferable to use this method as it greatly simplifies the upgrade logic. Upgrading functions piecewise may lead to corruption of logic and data.
Here is an example in Python that reloads itself in a loop using this method. It saves the file descriptors for 2 connected sockets which allows the new process to signal the old one at will. Simply run the program, and then edit the print statements below to watch it change.
import os import time import socket import sys import fcntl import _multiprocessing from multiprocessing import Pipe # global pipe variable pipe = None spare_pipe = None def PipeFromFD(fd): return _multiprocessing.Connection(fd) def recordPipe(p1, p2): with open('fd.txt', 'w') as f: f.write('%s,%s' % (p1.fileno(), p2.fileno())) def upgrade(): os.spawnlp(os.P_NOWAIT, 'python', 'python', sys.argv[0], '-upgrade') def upgradeBootstrap(): global pipe, spare_pipe with open('fd.txt', 'r') as f: p1, p2 = (int(fd) for fd in f.read().split(',')) pipe = PipeFromFD(p1) spare_pipe = PipeFromFD(p2) recordPipe(spare_pipe, pipe) pipe.send("I am upgraded!") if __name__ == '__main__': if '-upgrade' in sys.argv: upgradeBootstrap() else: pipe, spare_pipe = Pipe() recordPipe(spare_pipe, pipe) pid = os.getpid() print '%i: Version 1. Upgrade in 5 seconds.' % pid time.sleep(5) print '%i: Upgrading' % pid upgrade() print '%i: Message from new process: %s' % (pid, pipe.recv()) print '%i: Upgrade Complete. Exiting' % pid
Example Output
user@test:/mnt/shared$ python upgrade.py 22930: Version 1. Upgrade in 10 seconds. 22930: Upgrading 22971: Version 2. Upgrade in 10 seconds. 22930: Message from new process: I am upgraded! 22930: Upgrade Complete. Exiting user@test:/mnt/shared$ 22971: Upgrading 23025: Version 3. Upgrade in 10 seconds. 22971: Message from new process: I am upgraded! 22971: Upgrade Complete. Exiting killall python
Related posts:
- Debunking the Erlang and Haskell hype for servers
- Clearing Passwords in Memory with Python
- Host Your Own Email: Easy Disposable Addresses
- Make a Python JIT compiler without writing a single line of C or 3rd party library
- Patching a Program Without Source Code: How to be like the Skype Hacker for Newbies
I am not sure what is the point you want to make here. Spawning a new process always read the latest copy from the disk. That is not a discovery. What erlang provides is hot code loading into the currently running VM with no special code.
The point here is that it is possible to seamlessly transfer control of the sockets to a new process. Therefore to a client, it appears as though nothing has changed.
Nice work, interesting. I’m not very familiar with Python, are there any mechanisms for preserving the state of objects across the upgrade ? I think that is a critical component for this to be of use in a broader sense. i.e. if the server is long running then presumably it has state that needs preserving ; if it doesn’t have state then it’s likely you could just do a quick restart without impact to the application. In the case of Erlang OTP the upgrade inherently includes preservation of state, i.e. values of variables, without use of any work-arounds in application code ; and it does it very quickly and seamlessly.
Update : I just saw your earlier post on “A Better Python Reload”. I think that answers my question, viz : it can be done, but it’s not seamless, and has some drawbacks.
It certainly doesn’t match the way it’s done in Erlang.
I think it’s fair to say that a clean fast code reload mechanism is always going to be clumsy in mutable languages, or at least significantly harder than in immutables languages where their very nature makes it trivial.
The most important part about code reloading is maintaining an uninterrupted service to the user. This shows that it is possible to do even in C.
Obviously Erlang’s immutability makes this easier, but you will probably want to upgrade the VM from time to time, and I believe there is no mechanism to do that without restarting the process.
We were hot-reloading C for our chat server in 1998 or so. There’s really nothing new in it, it has been possible since dynamic loading and unloading of shared libraries was possible. Possible, but it was not very *convenient* to use and program for, and very easy to make mistakes, I think that’s where the new developments are centered in.
Sorry for commenting on an old article, but you can hot swap code if you use the twisted framework… We do this all the time on our server, I’ve never had any problems with it.