I'm having trouble pickling a Cython class, but only when it's defined inside a package. This problem was noted previously online, but they didn't state how it was resolved. There are two components here: the Cython pickling using a __reduce__ method and a package error.

Cython Pickling Success

I'll first show how it works without the package part. This example works correctly.

Cython File

My Cython file is reudce.pyx:

cdef class Foo(object):cdef int ndef __init__(self, n):self.n = ndef __reduce__(self):return Foo, (self.n,)

Setup File

This may be compiled with a setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_extsetup(cmdclass = {'build_ext': build_ext},ext_modules = [Extension("reduce", ["reduce.pyx"])]
)

by executing python setup.py build && cp build/lib*/reduce.so .

Test Script

The test script is called test_reduce.py and is:

import reduce
import pickle
f = reduce.Foo(4)
print pickle.dumps(f)

Executing python test_reduce.py works fine.

Cython Pickling in Package Failure

However, once the reduce.pyx is put into a package, there is an error.

Package Creation

To reproduce this, first create a package called bar.

mkdir bar
mv reduce.so bar
echo "from reduce import Foo" > bar/__init__.py 

Test Script

Change the test_reduce.py file to be:

import bar
import pickle
f = bar.Foo(4)
print pickle.dumps(f)

Error Message

Running python test_reduce.py gives the following error:

File "/usr/lib/python2.7/pickle.py", line 286, in savef(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 748, in save_global(obj, module, name))
pickle.PicklingError: Can't pickle <type 'reduce.Foo'>: it's not found as reduce.Foo

There is a catch of errors which are all turned into a PicklingError in pickle.py After looking at that code, the specific error which is occuring is:

ImportError: No module named reduce

Sanity Test

To check that there is not some kind of scope or other issue, if I run the steps which the pickle module should execute, everything works:

f = bar.Foo(4)
call, args = f.__reduce__()
print call(*args)

So what's going on here?!

The problem was in the build script. The Pickle module uses the __module__ attribute of a function/class for pickling. That __module__ attribute comes from the first argument to the Extension() constructor in the setup.py script. Since I defined my constructor to be Extension('reduce', ['reduce.pyx']), the __module__ attribute is reduce. It should be bar/reduce though since it's now in a package.

Making setup.py look like:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_extsetup(cmdclass = {'build_ext': build_ext},ext_modules = [Extension('bar/reduce', ['reduce.pyx'])]
)

solves the problem.