How to find source of error in Python Pickle on massive object

2024/10/8 4:34:12

I've taken over somebody's code for a fairly large project. I'm trying to save program state, and there's one massive object which stores pretty much all the other objects. I'm trying to pickle this object, but I get this error:

pickle.PicklingError: Can't pickle : it's not found as builtin.module

From what I can find on google, this is because somewhere I'm importing something outside of python init, or that a class attribute is referencing a module. So, I've got a two questions:

  1. Can anybody confirm that that's why this error is being given? Am I looking for the right things in my code?

  2. Is there a way to find what line of code/object member is causing the difficulties in pickle? The traceback only gives the line in pickle where the error occurs, not the line of the object being pickled.

Answer

2) You can subclass pickle.Pickler and monkey-patch it to show a log of what it's pickling. This should make it easier to trace where the problem is.

import pickle
class MyPickler (pickle.Pickler):def save(self, obj):print 'pickling object', obj, 'of type', type(obj)pickle.Pickler.save(self, obj)

This will only work with the Python implementation of pickle.Pickler. In Python 3.x, the pickle module uses the C implementation by default, the pure-Python version of Pickler is called _Pickler.

# Python 3.x
import pickle
class MyPickler (pickle._Pickler):def save(self, obj):print ('pickling object  {0} of type {1}'.format(obj, type(obj))pickle._Pickler.save(self, obj)
https://en.xdnf.cn/q/70165.html

Related Q&A

What is the most idiomatic way to index an object with a boolean array in pandas?

I am particularly talking about Pandas version 0.11 as I am busy replacing my uses of .ix with either .loc or .iloc. I like the fact that differentiating between .loc and .iloc communicates whether I a…

Error: Command failed with rc=65536 python and mod_wsgi

im having this problem: im runing pythonbrew to get python2.7, and so i re-compiled mod_wsgi to use the 2.7 python. to that end, i followed this tutorial: code.google.com/p/modwsgi/wiki/QuickInstallati…

Why cant I freeze_panes on the xlsxwriter object pandas is creating for me?

I have a class whose objects contain pandas dataframes (self.before, and self.after below) and a save() method which uses xlsxwriter to export the data (which has two worksheets, "before" and…

Python multiprocessing pipe recv() doc unclear or did I miss anything?

I have been learning how to use the Python multiprocessing module recently, and reading the official doc. In 16.6.1.2. Exchanging objects between processes there is a simple example about using pipe t…

How would you unit test this SQLAlchemy Core query/function?

Im working on learning how to unit test properly. Given this function...def get_user_details(req_user_id):users = sa.Table(users, db.metadata, autoload=True)s = sa.select([users.c.username,users.c.favo…

Retrieve wall-time in Python using the standard library?

How can I retrieve wall-time in Python using the standard library?This question, and this question would suggest that something like clock_gettime(CLOCK_MONOTONIC_RAW) or /proc/uptime are most appropr…

NLTK: Package Errors? punkt and pickle?

Basically, I have no idea why Im getting this error. Just to have more than an image, here is a similar message in code format. As it is more recent, the answer of this thread has already been mentione…

Is there a bit-wise trick for checking the divisibility of a number by 2 or 3?

I am looking for a bit-wise test equivalent to (num%2) == 0 || (num%3) == 0.I can replace num%2 with num&1, but Im still stuck with num%3 and with the logical-or.This expression is also equivalent …

Check image urls using python-markdown

On a website Im creating Im using Python-Markdown to format news posts. To avoid issues with dead links and HTTP-content-on-HTTPS-page problems Im requiring editors to upload all images to the site and…

How to unittest command line arguments?

I am trying to supply command line arguments to Python unittest and facing some issues. I have searched on internet and found a way to supply arguments asunittest.main(argv=[myArg])The issue is this wo…