How to get unpickling to work with iPython?

2024/10/5 11:14:16

I'm trying to load pickled objects in iPython.

The error I'm getting is:

AttributeError: 'FakeModule' object has no attribute 'World'

Anybody know how to get it to work, or at least a workaround for loading objects in iPython in order to interactively browse them?

Thanks

edited to add:

I have a script called world.py that basically does:

import pickle
class World:""
if __name__ == '__main__':w = World()pickle.dump(w, open("file", "wb"))

Than in a REPL I do:

import pickle  
from world import World  
w = pickle.load(open("file", "rb"))

which works in the vanilla python REPL but not with iPython.

I'm using Python 2.6.5 and iPython 0.10 both from the Enthought Python Distribution but I was also having the problem with previous versions.

Answer

Looks like you've modified FakeModule between the time you pickled your data, and the time you're trying to unpickle it: specifically, you have removed from that module some top-level object named World (perhaps a class, perhaps a function).

Pickling serializes classes and function "by name", so they need to be names at their module's top level and that module must not be modified (at least not in such way to affect those names badly -- definitely not by removing those names from the module!) between pickling time and unpickling time.

Once you've identified exactly what change you've done that impedes the unpickling, it can often be hacked around if for other reasons you can't just revert the change. For example, if you've just moved World from FakeModule to CoolModule, do:

import FakeModule
import CoolModule
FakeModule.World = CoolModule.World

just before unpickling (and remember to pickle again with the new structure so you won't have to keep repeating these hacks every time you unpickle;-).

Edit: the OP's edit of the Q makes his error much easier to understand. Since he's now testing if __name__ equals '__main__', this makes it obvious that the pickle, when written, will be saving an object of class __main__.World. Since he's using ASCII pickles (a very bad choice for performance and disk space, by the way), it's trivial to check:

$ cat file
(i__main__
World
p0
(dp1

the module being looked up is (clearly and obviously) __main__. Now, without even bothering ipython but with a simple Python interactive interpreter:

$ py26
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import world
>>> import pickle
>>> pickle.load(open("file", "rb"))
Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1370, in loadreturn Unpickler(file).load()File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 858, in loaddispatch[key](self)File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1069, in load_instklass = self.find_class(module, name)File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1126, in find_classklass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'World'
>>> 

the error can be easily reproduced, and its reason is just as obvious: the module in which the class name's lookup is performed (that is, __main__) does indeed have no attribute named "World". Module world does have one, but the OP has not "connected the dots" as I explained in the previous part of the answer, putting a reference with the right name in the module in which the pickled file needs it. That is:

>>> World = world.World
>>> pickle.load(open("file", "rb"))
<world.World instance at 0xf5300>
>>> 

now this works just perfectly, of course (and as I'd said earlier). Perhaps the OP is not seeing this problem because he's using the form of import I detest, from world import World (importing directly a function or class from within a module, rather than the module itself).

The hack to work around the problem in ipython is exactly the same in terms of underlying Python architecture -- just requires a couple more lines of code because ipython, to supply all of its extra services, does not make module __main__ directly available to record directly what happens at the interactive command line, but rather interposes one (called FakeModule, as the OP found out from the error msg;-) and does black magic with it in order to be "cool" &c. Still, whenever you want to get directly to a module with a given name, it's pretty trivial in Python, of course:

In [1]: import worldIn [2]: import pickleIn [3]: import sysIn [4]: sys.modules['__main__'].World = world.WorldIn [5]: pickle.load(open("file", "rb"))
Out[5]: <world.World instance at 0x118fc10>In [6]: 

Lesson to retain, number one: avoid black magic, at least unless and until you're good enough as a sorcerer's apprentice to be able to spot and fix its occasional runaway situations (otherwise, those bucket-carrying brooms may end up flooding the world while you nap;-).

Or, alternative reading: to properly use a certain layer of abstraction (such as the "cool" ones ipython puts on top of Python) you need strong understanding of the underlying layer (here, Python itself and its core mechanisms such as pickling and sys.modules).

Lesson number two: that pickle file is essentially broken, due to the way you've written it, because it can be loaded only when module __main__ has a class by name Word, which of course it normally will not have without some hacks like the above. The pickle file should instead record the class as living in module world. If you absolutely feel you must produce the file on an if __name__ == '__main__': clause in world.py, then use some redundancy for the purpose:

import pickle
class World:""
if __name__ == '__main__':import worldw = world.World()pickle.dump(w, open("file", "wb"))

this works fine and without hacks (at least if you follow the Python best practice of never having any substantial code at module top level -- only imports, class, def, and trivial assignments -- everything else belongs in functions; if you haven't followed this best practice, then edit your code to do so, it will make you much happier in terms of both flexibility and performance).

https://en.xdnf.cn/q/70489.html

Related Q&A

Basic questions about nested blockmodel in graph-tool

Very briefly, two-three basic questions about the minimize_nested_blockmodel_dl function in graph-tool library. Is there a way to figure out which vertex falls onto which block? In other words, to ext…

How to get multiple parameters with same name from a URL in Pylons?

So unfortunately I find myself in the situation where I need to modify an existing Pylons application to handle URLs that provide multiple parameters with the same name. Something like the following...…

Kivy: Access configuration values from any widget

Im using kivy to create a small App for computer aided learning.At the moment I have some problems with accessing config values. I get the value withself.language = self.config.get(basicsettings, langu…

Multiprocessing with threading?

when I trying to make my script multi-threading, Ive found out multiprocessing,I wonder if there is a way to make multiprocessing work with threading?cpu 1 -> 3 threads(worker A,B,C) cpu 2 -> 3 …

Pandas Groupby Unique Multiple Columns

I have a dataframe.import pandas as pd df = pd.DataFrame( {number: [0,0,0,1,1,2,2,2,2], id1: [100,100,100,300,400,700,700,800,700], id2: [100,100,200,500,600,700,800,900,1000]})id1 id2 nu…

OpenCV Error: Assertion failed when using COLOR_BGR2GRAY function

Im having a weird issue with opencv. I have no issues when working in a jupyter notebook but do when trying to run this Sublime.The error is: OpenCV Error: Assertion failed (depth == CV_8U || depth == …

matplotlib 1.3.1 has requirement numpy=1.5, but youll have numpy 1.8.0rc1 which is incompatible

Im executing bellow command in Mac (High Sierra) as a part of getting started with pyAudioAnalysis.pip install numpy matplotlib scipy sklearn hmmlearn simplejson eyed3 pydub Im getting following error…

VS Code Debugger Immediately Exits

I use VS Code for a python project but recently whenever I launch the debugger it immediately exits. The debug UI will pop up for half a second then disappear. I cant hit a breakpoint no matter where i…

Sudoku Checker in Python

I am trying to create a sudoku checker in python:ill_formed = [[5,3,4,6,7,8,9,1,2],[6,7,2,1,9,5,3,4,8],[1,9,8,3,4,2,5,6,7],[8,5,9,7,6,1,4,2,3],[4,2,6,8,5,3,7,9], # <---[7,1,3,9,2,4,8,5,6],[9,6,1,5,…

Subclassing numpy scalar types

Im trying to subclass numpy.complex64 in order to make use of the way numpy stores the data, (contiguous, alternating real and imaginary part) but use my own __add__, __sub__, ... routines.My problem i…