Suppose I am embedding the CPython interpreter into a larger program, written in C. The C component of the program occasionally needs to call functions written in Python, supplying callback functions to them as arguments.

Using the CPython extending and embedding APIs, how do I construct a Python "callable" object that wraps a C pointer-to-function, so that I can pass that object to Python code and have the Python code successfully call back into the C code?

Note: this is a revised version of a question originally posted by user dhanasubbu, which I answered, but which was then deleted. I think it was actually a good question, so I have converted what I wrote into a self-answer to my own statement of the question. Alternative answers are welcome.

To define an extension type that is “callable” in the sense Python uses that term, you fill the tp_call slot of the type object, which is the C equivalent of the __call__ special method. The function that goes in that slot will be a glue routine that calls the actual C callback. Here’s code for the simplest case, when the C callback takes no arguments and returns nothing.

typedef struct {PyObject_HEAD/* Type-specific fields go here. */void (*cfun)(void);  /* or whatever parameters it actually takes */
} CallbackObj;static PyObject *Callback_call(PyObject *self, PyObject *args, PyObject *kw)
{/* check that no arguments were passed */const char no_kwargs[] = { 0 };if (!PyArg_ParseTupleAndKeywords(args, kw, "", no_kwargs))return 0;CallbackObj *cself = (CallbackObj *)self;cself->cfun();Py_RETURN_NONE;
}static PyTypeObject CallbackType = {PyVarObject_HEAD_INIT(NULL, 0).tp_name = "mymodule.Callback",.tp_doc = "Callback function passed to foo, bar, and baz.",.tp_basicsize = sizeof(CallbackObj),.tp_itemsize = 0,.tp_flags = Py_TPFLAGS_DEFAULT,.tp_new = PyType_GenericNew,.tp_call = Callback_call,
};

Instantiate the type object with PyType_Ready as usual. Don’t put it in any module visible to Python, though, because Python code can’t correctly create instances of this type. (Because of this, I haven’t bothered with a tp_init function; just make sure you always initialize ->cfun after creating instances from C, or Callback_call will crash.)

Now, suppose the actual function you need to call is named real_callback, and the Python function that you want to pass it to is named function_to_call. First you create one of the callback objects, by calling the type object, as usual, and initialize its ->cfun field:

    PyObject *args = PyTuple_New(0);CallbackObj *cb = (CallbackObj *)PyObject_CallObject((PyObject *)CallbackType, args);Py_DECREF(args);cb->cfun = real_callback;

Then you put cb into an argument tuple, and call the Python function object with that, as usual.

    args = Py_BuildValue("(O)", cb);PyObject *ret = PyObject_CallObject(function_to_call, args);Py_DECREF(args);Py_DECREF(cb);// do stuff with ret, here, perhapsPy_DECREF(ret);

Extending this to more complex cases, where the C callback needs to take arguments and/or return values and/or raise Python exceptions on error and/or receive “closure” information from the outer context, is left as an exercise.