How to make a Python package with compiled Rust (and why you should to increase performance)

Without going into too much detail on academic source referencing because this is a just a blog, I think it is fair to say that Python is at least one of the most commonly used programming languages in scientific application development. Python is great in that is is easy to learn, is very popular and widely supported, and has a vast range of libraries that can easily be attained through its pip package manager.

A problem that I encountered however while working with large data sets in biological settings is that it has some performance issues. I think many will be familiar with the fact that modules written in C and C++ are generally able to be faster as they are compiled to machine code, and don't have to go through the extra layer of the interpreter as Python does.

The problem with C and C++ though is that they can be quite difficult to master, and even when you do master them you can run into problems like segmentation faults because of memory management errors. Basically, Rust kind of does what C does but is safer to use and easier to debug (from what I hear and from my limited experience with C/C++), but that's a different story so I won't get into it, but definitely read some other articles if you're interested.

The main takeaway here is that you can compile your Rust library as a python package and install it locally on your machine, or you can simply compile a binary (.so file) and put it in your python project, and then if you're in a team like me where not everyone is as good at programming as everyone else and want to just stick with Python, then you can still get some performance increases from implementing modules in Rust. Bear in mind that a separate binary will have to be compiled for each platform (mac, linux, windows), so a package compiled on Linux won't work on Mac.

Anyway, I made a very simple example on Github using the pyo3 approach so the best is probably just to start with that, and just add whatever functions you need. I hope it can help someone, because I definitely spent a little bit of time getting it right the first time.

Leave a Reply

Your email address will not be published. Required fields are marked *