Modules written in Cython usually comes with a setup.py script, compiling the Cython source codes into native shared libary. For whom not so familiar with Python’s packaging and distributing toolchains, such step is sometimes forbidding, and turns out to be a stumbling block for Cython freshmen. Moreover, the workflow, “run setup.py -> debug -> edit .pyx files -> run setup.py”, is also not convenient and troublesome for fast iterating projects.

pyximport is a handy tool from Cython official, to address the above problem. The module enables users to “directly import” .pyx files, with no explicit setup.py required. Let’s take an example here. Say we have two files residing in the same directory:

# main.py
import pyximport

pyximport.install(language_level=3)

import foo

print(foo.func(3))
# foo.pyx
cpdef int sqr(int x):
return x * x

The magically highlighted line, registers some import hooks to let Python recognize .pyx files. pyximport automatically compiles or re-compiles the .pyx files behind the scene, when they are imported for the first time, or modified in the future.

By default, the built shared libaries are stored at ~/.pyxbld/, which can be overrided with the build_dir argument of pyximport.install. One may also pass inplace=True, to place the libraries sibling to their source files. For example:

# main.py
import pyximport

pyximport.install(language_level=3, inplace=True)

import foo

print(foo.func(3))

will generate a .so (or .pyd in Windows) file in the same directory.

But there’s a problem with inplace building – the extension will not be automatically rebuilt. If you modify the .pyx file and re-run the program, the compilation process will not be triggered. The reason is that Python itself is able to recognize .so or .pyd modules, and their importers have higher priority than customized ones. Thus, if there’s a .so and a .pyx file reside in the same directory, the .so file will be imported instead of .pyx.

So if you want the compiled binaries be placed within your project directory, the best practice is to specify the build_dir argument. By setting build_dir to somewhere inside your project, compiled libraries will be kept in your project directory, and at the same time you can still enjoy the automatic rebuild service.

References