Convert ctypes code to cython

2024/10/12 10:19:35

I'd like to convert some ctypes code to use cython instead, but I'm struggling. Essentially, the ctypes code:

  • copies the contents (floats) of two lists into C-compatible structs
  • sends the structs to my binary via FFI
  • receives the structs back (the length is never modified)
  • copies the contents to two new lists
  • sends the structs back across the FFI boundary so their memory can be freed

My ctypes code looks like this so far:

rlib.h

#ifndef RLIB_H
#define RLIB_Htypedef struct _FFIArray {void* data;size_t len;
} _FFIArray;typedef struct _Result_Tuple {_FFIArray e;_FFIArray n;
} _Result_Tuple;_Result_Tuple convert_to_bng_threaded(_FFIArray x, _FFIArray y);
void drop_float_array(_FFIArray x, _FFIArray y)#endif

mylib.pxd

cdef extern from "rlib.h":struct _FFIArray:void* datasize_t lenstruct _Result_Tuple:_FFIArray e_FFIArray ncdef _Result_Tuple convert_to_bng_threaded(_FFIArray x, _FFIArray y)cdef void drop_float_array(_FFIArray x, _FFIArray y)

util_cython.pyx

import cython
from mylib cimport _FFIArray, convert_to_bng_threaded, drop_float_arraydef call_convert_bng(_FFIArray x, _FFIArray y):return convert_to_bng_threaded(x, y)def call_drop_float_array(_FFIArray x, _FFIArray y):return drop_float_array(x, y)

setup.py

from setuptools import setup, Extension, find_packagesfrom Cython.Build import cythonize
from Cython.Distutils import build_extext = Extension('util_cython', sources=['util_cython.pyx'],libraries=['latlon_bng',],library_dirs=['.',],include_dirs=['.']
)extensions = [ext,]setup(name = "util_cython",ext_modules = cythonize(extensions),cmdclass={'build_ext': build_ext},
)

I have a few questions about how to proceed:

Firstly, the compilation step is currently failing:

python setup.py build_ext --inplace
Compiling util_cython.pyx because it changed.
[1/1] Cythonizing util_cython.pyxError compiling Cython file:
------------------------------------------------------------
...
import cython
from mylib cimport _FFIArray, convert_to_bng_threaded, drop_float_arraydef call_convert_bng(_FFIArray x, _FFIArray y):return convert_to_bng_threaded(x, y)^
------------------------------------------------------------util_cython.pyx:5:34: Cannot convert '_Result_Tuple' to Python object
Traceback (most recent call last):File "setup.py", line 17, in <module>ext_modules = cythonize(extensions),File "/Users/sth/dev/cythonize_test/venv/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 912, in cythonizecythonize_one(*args)File "/Users/sth/dev/cythonize_test/venv/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1034, in cythonize_oneraise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: util_cython.pyx

Why is Cython failing to convert _Result_tuple?

Secondly, how do I define a cython function to accept lists (or arrays; anything that supports __iter__), and copy their contents into _FFIArray structs, so I can call call_convert_bng?

Answer

The following is untested, because I don't have the library code and have had to take some guesses as to how it works. But it should give you an idea as to how you go about it.

I'd start by using a Python array type to store your input/output. Either the standard library array type, or numpy arrays. These store the arrays continuously in memory (like C would) and so provided the _FFIArray data attribute can just be pointed at this memory for input.

def convert_bng(double[::1] x, double[::1] y):# I'm assuming that the data is double, not float, but it's easy changed# here the [::1] promises it's continuous in memorycdef _FFIArray x_ffi, y_ffi# get a pointer to the data, and cast it to void*x_ffi.data = <void*>&x[0]x_ffi.len = x.shape[0] # possibly *sizeof(double) - depends on the C api# repeat for yy_ffi.data = <void*>&y[0]y_ffi.len = y.shape[0]cdef _Result_Tuple result = convert_to_bng_threaded(x_ffi, y_ffi)# get data pointers for the two result arrayscdef double* e_ptr = <double*>(result.e.data)cdef double* n_ptr = <double*>(result.n.data)# now view your output arrays using memoryviews# you need to tell it the length (this is how many doubles the contain)cdef double[::1] e = <double[:result.e.len:1]>e_ptrcdef double[::1] n = <double[:result.n.len:1]>n_ptr# create a numpy copy of the two arraysimport numpy as npe_numpy = np.copy(e)n_numpy = np.copy(n)# you can now free your two returned arrays# I assume this is done with drop_float_arraydrop_float_array(result.e,result.n)# return as tuple containing two arrays to pythonreturn e_numpy, n_numpy
https://en.xdnf.cn/q/118208.html

Related Q&A

Enable PyROOT Ubuntu 14.04

I downloaded madpgraph5, but when I run it I get the following error:ERROR: ROOT file called ROOT.py or ROOT.pyc is not foundERROR: Please check that ROOT is properly installed.When I try locate ROOT.p…

pygal on windows - cannot access classes from pygal

I have such short script:import pygal if __name__ == __main__:bar_chart = pygal.Bar()and following error: AttributeError: module object has no attribute BarDo you have any idea what is wrong? Shall I …

Parsing table for a link

Ive been able to isolate a row in a html table using Beautiful Soup in Python 2.7. Been a learning experience, but happy to get that far. Unfortunately Im a bit stuck on this next bit.I need to get t…

build matrix from blocks

I have an object which is described by two quantities, A and B (in real case they can be more than two). Objects are correlated depending on the value of A and B. In particular I know the correlation m…

How to convert string with UTC offset

I have date as In [1]: a = "Sun 10 May 2015 13:34:36 -0700"When I try to convert it using strptime, its giving error.In [3]: datetime.strptime(a, "%a %d %b %Y %H:%M:%S %Z"...: ) ---…

regex multiline not working on repeated patterns

I am trying to get a regex expression to match multiple patterns with multilines but it keeps matching everything. For instance I want to match two times this code:STDMETHOD(MyFunc)( D2D1_SIZE_U size, …

Django Rest Framework slug_field error

I have this serializer and model. I want to show in my API the field username of User model, but I receive this error.AttributeError at /api/v1/client_share_requests/1/Profile object has no attribute u…

Django - 500 internal server error no module named django

django return 500 internal server error (apache 2.4.10, ubuntu 15.04, django 1.9.6)apache log:[wsgi:warn] mod_wsgi: Compiled for Python/3.4.2. [wsgi:warn] mod_wsgi: Runtime using Python/3.4.3. [mpm_eve…

Unable to connect to Google Bigtable using HBase REST api

Following this example, running the test script "python put_get_with_client.py" results in a 400 error (Bad Request).Bad requestjava.lang.ClassCastException: org.apache.hadoop.hbase.client.Bi…

HTML form button to run PHP to execute Python script

I am building an HTML document that is meant to run locally. On it is a button that I would like to have run a Python script when clicked. Im trying to use a PHP-generated button. Theres no input or ou…