Python Distributed Computing (works)

2024/10/4 17:20:19

I'm using an old thread to post new code which attempts to solve the same problem. What constitutes a secure pickle? this?

sock.py

from socket import socket
from socket import AF_INET
from socket import SOCK_STREAM
from socket import gethostbyname
from socket import gethostnameclass SocketServer:def __init__(self, port):self.sock = socket(AF_INET, SOCK_STREAM)self.port = portdef listen(self, data):self.sock.bind(("127.0.0.1", self.port))self.sock.listen(len(data))while data:s = self.sock.accept()[0]siz, dat = data.pop()s.send(siz)s.send(dat)s.close()class Socket:def __init__(self, host, port):self.sock = socket(AF_INET, SOCK_STREAM)self.sock.connect((host, port))def recv(self, size):return self.sock.recv(size)

pack.py

#http://stackoverflow.com/questions/6234586/we-need-to-pickle-any-sort-of-callable
from marshal import dumps as marshal_dumps
from pickle import dumps as pickle_dumps
from struct import pack as struct_packclass packer:def __init__(self):self.f = []def pack(self, what):if type(what) is type(lambda:None):self.f = []self.f.append(marshal_dumps(what.func_code))self.f.append(pickle_dumps(what.func_name))self.f.append(pickle_dumps(what.func_defaults))self.f.append(pickle_dumps(what.func_closure))self.f = pickle_dumps(self.f)return (struct_pack('Q', len(self.f)), self.f)

unpack.py

from types import FunctionType
from pickle import loads as pickle_loads
from marshal import loads as marshal_loads
from struct import unpack as struct_unpack
from struct import calcsize#http://stackoverflow.com/questions/6234586/we-need-to-pickle-any-sort-of-callableclass unpacker:def __init__(self):self.f = []self.fcompiled = lambda:Noneself.sizeofsize = calcsize('Q')def unpack(self, sock):size = struct_unpack('Q', sock.recv(self.sizeofsize))[0]self.f = pickle_loads(sock.recv(size))a = marshal_loads(self.f[0])b = globals() ##c = pickle_loads(self.f[1])d = pickle_loads(self.f[2])e = pickle_loads(self.f[3])self.fcompiled = FunctionType(a, b, c, d, e)return self.fcompiled

test.py

from unpack import unpacker
from pack import packer
from sock import SocketServer
from sock import Socket
from threading import Thread
from time import sleepcount = 2
port = 4446def f():print 42def server():ss = SocketServer(port)pack = packer()functions = [pack.pack(f) for nothing in range(count)]ss.listen(functions)if __name__ == "__main__":Thread(target=server).start()sleep(1)unpack = unpacker()for nothing in range(count):print unpack.unpack(Socket("127.0.0.1", port))

output:

<function f at 0x12917d0>
<function f at 0x12915f0>
Answer

I don't think Process objects are designed to be sent over the network. Look at line 256 in multiprocessing/process.py.

# We subclass bytes to avoid accidental transmission of auth keys over network.

Sounds like there's a good reason to me. If you want to do distributed computing, maybe you should look into a library designed for that.

https://en.xdnf.cn/q/70581.html

Related Q&A

Django - Stream request from external site as received

How can Django be used to fetch data from an external API, triggered by a user request, and stream it directly back in the request cycle without (or with progressive/minimal) memory usage?BackgroundAs…

django rest framework - always INSERTs, never UPDATES

I want to be able to UPDATE a user record by POST. However, the id is always NULL. Even if I pass the id it seems to be ignoredView Code:JSON POSTED:{"id": 1, "name": "Craig Ch…

Copy fields from one instance to another in Django

I have the following code which takes an existing instance and copies, or archives it, in another model and then deletes it replacing it with the draft copy. Current Codedef archive_calc(self, rev_num,…

Python selenium get Developer Tools →Network→Media logs

I am trying to programmatically do something that necessarily involves getting the "developer tools"→network→media logs. I will spare you the details, long story short, I need to visit thou…

deep copy nested iterable (or improved itertools.tee for iterable of iterables)

PrefaceI have a test where Im working with nested iterables (by nested iterable I mean iterable with only iterables as elements). As a test cascade considerfrom itertools import tee from typing import …

ImportError: No module named cv2.cv

python 3.5 and windows 10I installed open cv using this command :pip install opencv_python-3.1.0-cp35-cp35m-win_amd64.whlThis command in python works fine :import cv2But when i want to import cv2.cv :i…

Why arent persistent connections supported by URLLib2?

After scanning the urllib2 source, it seems that connections are automatically closed even if you do specify keep-alive. Why is this?As it is now I just use httplib for my persistent connections... bu…

How to find accented characters in a string in Python?

I have a file with sentences, some of which are in Spanish and contain accented letters (e.g. ) or special characters (e.g. ). I have to be able to search for these characters in the sentence so I can…

Import error with PyQt5 : undefined symbol:_ZNSt12out_of_rangeC1EPKc,versionQt_5

I had watched a video on YouTube about making your own web browser using PyQt5. Link to video: https://youtu.be/z-5bZ8EoKu4, I found it interesting and decided to try it out on my system. Please note t…

Python MySQL module

Im developing a web application that needs to interface with a MySQL database, and I cant seem to find any really good modules out there for Python.Im specifically looking for fast module, capable of h…