Auto Rebuild .pyx Files with pyximport

Modules written in Cython usually comes with a setup.py script that compiles Cython source codes into native shared libary. For whom not so familiar with Python’s packaging and distributing toolchains, such step is sometimes scary, and turns out to be a stumbling block for Cython freshmen. Moreover, the workflow, “run setup.py -> debug -> edit .pyx files -> run setup.py”, is also less convenient and troublesome for fast iterating projects.

pyximport is a handy tool from Cython official, provided to address the above problem. The module enables users to “directly import” .pyx files, with no explicit setup.py required. Let’s start from an example here. Say we have two files residing in the same directory:

# main.py
import pyximport

# hl: begin
pyximport.install(language_level=3)
# hl: end

import foo

print(foo.func(3))

# foo.pyx
cpdef int sqr(int x):
    return x * x

The magical highlighted line registers some import hooks to let Python recognize .pyx files. When the .pyx files imported for the first time or modified later, pyximport compiles or re-compiles them behind the scene automatically.

By default, the built shared libaries are placed at ~/.pyxbld/, which can be overrided with the build_dir argument of pyximport.install. One may also pass inplace=True to place the libraries sibling to their source files. For example:

# main.py
import pyximport

# hl: begin
pyximport.install(language_level=3, inplace=True)
# hl: end

import foo

print(foo.func(3))

will generate a .so (or .pyd in Windows) file in the same directory.

But there’s a problem with inplace building – the extension will not be automatically rebuilt. If you modify the .pyx file and re-run the program, the compilation process will not be triggered. The reason is that Python itself is able to recognize .so or .pyd modules, and their importers have higher priority than customized ones. Thus, if there’s an .so and a .pyx file with same module names reside in the same directory, the .so file will be imported instead of .pyx.

Therefore, if you want the compiled binaries be placed within your project directory, the best practice is to specify the build_dir argument. By setting build_dir to somewhere inside your project, compiled libraries will be kept within your project directory, but at the same time you can still enjoy the automatic rebuild service.

References

• https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#compiling-with-pyximport

Author: hsfzxjy.
Link: .
License: CC BY-NC-ND 4.0.
All rights reserved by the author.
Commercial use of this post in any form is NOT permitted.
Non-commercial use of this post should be attributed with this block of text.