[hynek]

They did (it's me :)) and unfortunately in this case rpath can't be used, because that particular Python driver uses ctypes (https://docs.python.org/3/library/ctypes.html) to open the binary drivers which oversimplified means that there is no binary top modify.

I hope I make it in the preamble clear that this is bad and one should not have to deal with this – but it happens in practice and I hope such a summary is useful.

For posterity: if you want to wrap / use a C library in Python, you should go for CFFI (Cython works too and is overall faster, but has other downsides). This PyCon US video is a great up-to-date summary: https://www.youtube.com/watch?v=gROGDQakzas

[kosinus]

Maybe it's possible to patch the dependency? For example, for Node.js / npm, there are automated ways to do this, like [patch-package](https://www.npmjs.com/package/patch-package). Does Python / pip have something similar?

[hynek]

You mean at the Python driver level? Unfortunately, that doesn't work with ctypes.

I've tried it by adding the DYLD_LIBRARY_PATH to os.environ before calling ctypes.LibraryLoader.LoadLibrary (https://docs.python.org/3/library/ctypes.html#ctypes.Library...) and it didn't work. I suspect ctypes gets somehow initialized much sooner and adding environment variables in your apps doesn't help.

TBH I didn't research it further, since the problems of the post are more general and it can happen that you trip into them regardless of runtime.

[orf]

Here’s the relevant source: https://github.com/python/cpython/blob/8dd2766d99f8f51ad62dc...

I believe you should just be able to replace the library name with an absolute path to the library, and remove the need for the lookup at all?

[hynek]

Right, but that would take some really deep patching of code I don't control, just so it works in development.

If this were a production issue, I would probably fork the driver and do what you're suggesting (it's not like it's actively maintained or something :|).

[orf]

Why not fork it? If only to have the code under your control in case it abruptly becomes inaccessible.

Then the fix is a single if statement. Seems worth it to me.

[throwaway09223]

You can still do it.

rpath doesn't just apply to executable binaries, it applies to all shared objects including libraries. So, what you can do is modify the dyld that the python/ctypes module will eventually dlopen(). You should be able to find it by rooting around in the files installed by the module in question.

rpath is evaluated not just at binary execution, but also upon any dynamic linking -- including dlopen(). So it will work for both ctypes and CFFI.

Happy to provide further pointers if you need!

[1vuio0pswjnm7]

Out of curiosity, what are the downsides of Cython?

[hynek]

Conceptually, Cython is mainly for accelerating Python code, and can _also_ access C code. Meanwhile CFFI is specifically for calling C code and nothing else. I recommend the video for the differences.

One concrete thing that pops to my mind is that Cython doesn't support Py_LIMITED_API which means that you need to ship a lot more binary wheels. At least the issue is still open (https://github.com/cython/cython/issues/2542) and Cython projects IME need new wheels for each minor Python release. Compare that to cffi projects that (musl & pypy aside) only have to ship wheels for one Python version / architecture: https://pypi.org/project/argon2-cffi-bindings/#files