published on

Reduce python breakage

Recently I ran into an issue with a python project I was working on. A dependency of a dependency decided to do a breaking change, which broke my project even if I had everything in requirements.txt pinned (same issue as here).

As part of fixing it I learned a few new things which I’ll share here.

First, you can constraint pip to prevent a newer version of the package from installing. pip a flag -c which you can read more about documentation here.

You can add

markupsafe==2.0.1

to a file called constraints.txt and then do pip install -c constraints.txt project. And voilà, you pip install hopefully works now.

Making things future proof

Next we will take things a bit further, we’ll download every dependency we need and include it in our repo. That way we won’t be sad if the maintainers suddenly decide you can’t have an old version anymore.

First, on a working install, do a pip freeze to get a list of things you need. Put these into for example freeze.txt (you might be able to use your requirements.txt directly, but not if you use -e .).

Then run

pip download -d deps/ -r freeze.txt -c constraints.txt

And boom you will have a bunch of whl files in your deps/ folder.

Next we need to tell pip to use them and not rely on some index. Check that things work with

pip install -c constraints.txt --find-links ./deps/ --no-index .

You probably need to do this inside a clean docker or something to be really sure the project will install from a “clean” install.

Then commit the deps/ to the repo and never worry about missing or broken dependencies again.

Note that the .whl files are platform specific, so you might want to look at the --platform option to pip --download if you want to be really sure you have the right files when you need them. Another option is to download the source, but compiling python packages can be tricky (even the –help for pip download says so).