Package Dependencies
When developing your own python packages, you may run into instances where you want to use an existing library rather than create certain functionalities from scratch. Integrating modules and functions from other libraries into your own introduces package dependencies. There are two different types of package dependencies.
- Required Dependencies
- Optional Dependencies
Required Dependencies
Required dependencies occur when you build modules and functionalities around another package. If a user tries to run modules in your library without the necessary package installed, they will get an error. Within a python project you can specify required packages in the pyproject.toml file under the dependencies section (see the example below).
[project]
name = "polars-extensions"
version = "0.4.0"
description = "The Library of Polars Extensions"
readme = "README.md"
authors = [
{name = "Jayden Rasband"}
]
license = {text = "MIT"}
classifiers = []
dependencies = [
"polars",
]
This makes it so that when people install your package, all dependencies will be installed at the same time. That way, your library works right out the gate. Now there are a few drawbacks when it comes to relying on other packages.
- Dependent package functionalities can change.
- Dependent packages may create security risks.
Dependent Package Functionalities can change
Say you have version 1.9.0 of a package you depend on for one of your functions. Version 2.0.0 of that library comes along and introduces some breaking changes. This could in turn cause issues in your own library. In these cases, users of your package will likely submit an issue through github. But more on that later in the course.
Dependent Packages May Create Security Risks
The more dependencies you have, the more likely it is that vulnerabilities in someone else’s package can impact your own. Consider also that someone else’s package that you depend on may also have dependencies; creating a web of dependencies.
Optional Dependencies
Optional dependencies allow you to specify additional features or functionality that are not required for the core operation of your project. They give users the flexibility to install extra packages only if they need those features. You can add these to your pyproject file as follows:
[project]
name = "polars-extensions"
version = "0.4.0"
description = "The Library of Polars Extensions"
readme = "README.md"
authors = [
{name = "Jayden Rasband"}
]
license = {text = "MIT"}
classifiers = []
dependencies = [
"polars",
]
[project.optional-dependencies]
numpy = ["numpy >= 1.16.0"]
pandas = ["pandas", "polars[pyarrow]"]
pyarrow = ["pyarrow >= 7.0.0"]
pydantic = ["pydantic"]
Practical Advice for Dependencies
- If you need to use a required dependency consider locking the version.
- Reduce the number of dependencies you have.
- Rather than using “required dependencies” consider adding “optional dependencies”.
Locking Versions
Locking the package versions not only prevents breaking changes in one library from cascading to your own, but can also serve as a security measure. You can lock a dependency a few versions behind the current. This lets you keep an eye out for vulnerabilities that may be reported in those more recent versions. This is how ActiveState and other “security first” services approach code security in Python.
Reduce Dependencies
The most secure packages have very few required dependencies. When building out your package, you should strive for a solid core that’s mostly independent from other libraries. Ask yourself, do I really need another package or can I build it from scratch? Ask CoPilot, ChatGPT or other LLMs to help. Your goals is to reduce your dependencies.
Choosing Optional Over Required
Polars has a lot of functionalities that depend on other packages. But as of the time of this article they only have one package that’s truly required. The rest are labeled as optional dependencies. Their approach is very minimalist. They know that many functions will require additional packages, but they don’t force you to install them unless you actually need them. Its both space conscious and security conscious.
Conclusion
Want to start you’re own project, check out our article on project initiation with uv!