These docs have a nice example on how to compile some external C/C++ code and load this using ctypes. This works great with manual compilation (where i'm controlling the names of my library which later gets loaded).

Now how to prepare some code to be used within scipy with full incorporation into scipy's build-system? A more general github-issue does not provide the information i'm looking for (and sadly, there is not much action).

Also: i'm more interested in using ctypes (no real usage within scipy? despite numpy's explicit tools provided?), not cython which is used in all the other wrappers (i don't see any good reason for cython, except for it's common use there: my lib will only get loaded/called once and it's slow compared to potential call-overheads).

Let's assume i added a directory like:

scipy/optimize/_mylib/README
...
scipy/optimize/_mylib/src
scipy/optimize/_mylib/src/a.cpp
...
... bunch of cpp/h files

Let's touch scipy/optimize/setup.py:

from __future__ import division, print_function, absolute_importfrom os.path import joinfrom scipy._build_utils import numpy_nodepr_apidef configuration(parent_package='',top_path=None):from numpy.distutils.misc_util import Configurationfrom numpy.distutils.system_info import get_infoconfig = Configuration('optimize',parent_package, top_path)# MODIFICATION START# OTHER EXTENSIONS OMITTED # INSPIRED BY scipy.spatial: ckdtree/src & setup.pymylib_src = ['a.cpp', 'b.cpp']mylib_src = [join('_mylib', 'src', x) for x in mylib_src]mylib_headers = ['a.h', 'b.h']mylib_headers = [join('_mylib', 'src', x) for x in mylib_headers]mylib_dep = mylib_headers + mylib_srcconfig.add_extension('_mylib',sources=mylib_src,depends=mylib_dep,include_dirs=[join('_mylib', 'src')])# MODIFICATION END    return configif __name__ == '__main__':from numpy.distutils.core import setupsetup(**configuration(top_path='').todict())

Now this works in regards to:

# in scipy base-dir
python3 setup.py build_ext --inplace

which produces (within scipy.optimize):

_mylib.cpython-35m-x86_64-linux-gnu.so

and can be loaded in the interpreter (started from dir scipy.optimize)

import numpy.ctypeslib as ctl
lib = ctl.load_library('_mylib.cpython-35m-x86_64-linux-gnu.so', '.')
lib
# <CDLL '/home......._mylib.cpython-35m-x86_64-linux-gnu.so', handle X at X

Now the question

Assuming i want to add a new file:

scipy/optimize/new_alg.py

No matter what i try, i'm not able to load this one. E.g.:

# this is scipy/optimize/new_alg.py
import numpy.ctypeslib as ctl
mylib = ctl.load_library('_mylib', '.')
mylib = ctl.load_library('_mylib', '.optimize')
# ...

The second argument is probably expected to be a full-path (opposed to partials or wildcards). All my attempts failed with the general error:

OSError: no file with expexted extension

What should i do here? (goal: ready to be a real addition to scipy: multiple platforms).

Is the approach itself doomed? From a naive perspective it's promising: the expected downsides of ctypes are irrelevant for me and it's part of the standard. Additionally it got dedicated support in numpy and i control the C/C++-part, so that i can provide a simplified single-function wrapper to be called. Why isn't this used in other parts of scipy?

Inspired by numpy's sources, i'm now able to load the lib using:

import scipy as scp
import numpy.ctypeslib as ctl
lib = ctl.load_library('_mylib', scp.optimize.__file__)  # this is the magical line
myfunc = lib.myfunc

Well... at least in theory!

Against my prior assumption, my lib-build is not equivalent to a manual-compile and the function i want to load is not available (undefined symbol: myfunc). But the problem of locating the lib feels clean now and is solved for me.