I was trying to make an http proxy using BaseHttpServer which is based on SocketServer
which got 2 asynchronous Mixins (ThreadingMixIn and ForkingMixIn)
the problem with those two that they work on each request (allocate a new thread or fork a new subprocess for each request)
is there a Mixin that utilize a pool of let's say 4 subprocesses and 40 threads in each
so requests get handled by those already created threads ?
because this would be a big performance gain and I guess it would save some resources.
You could use a pool from concurrent.futures
(in stdlib since Python 3.2):
from BaseHTTPServer import HTTPServer, test
from SimpleHTTPServer import SimpleHTTPRequestHandler
from SocketServer import ThreadingMixInfrom concurrent.futures import ThreadPoolExecutor # pip install futuresclass PoolMixIn(ThreadingMixIn):def process_request(self, request, client_address):self.pool.submit(self.process_request_thread, request, client_address)def main():class PoolHTTPServer(PoolMixIn, HTTPServer):pool = ThreadPoolExecutor(max_workers=40)test(HandlerClass=SimpleHTTPRequestHandler, ServerClass=PoolHTTPServer)if __name__=="__main__":main()
As you can see the implementation for a threading case is rather trivial.
If you save it to server.py
then you could run it as:
$ python -mserver
This command uses upto 40 threads to serve requests on http://your_host:8000/
.
The main use case of HTTPServer
is for testing purposes.