Blog Detail
Dependency management and virtual environments are integral to the Python ecosystem, yet the primary tools in use today are far from ideal. Some of the primary methods are:
- Dependencies management:
pipby way ofrequirements.txtis still the de facto solution for most of us. While this approach has worked in the past, there are limitations when it comes to guaranteeing that the same project will be consistently installed. - Virtual environments: a common setup is to use
virtualenvto create your virtual environment and manually activate it usingsource <path to venv>/activate. While this approach works, it requires the user to know which venv needs to be activated for each project and the command to execute can be lengthy. - Code packaging: (only applicable if you need to share your code), it is common to use
setuptoolsin asetup.pyfile, but this solution also has some shortcomings.
If you are using any or all of the methods described above, you should take a look at Poetry to help you manage your Python project(s). Poetry’s goal is to simplify the management of Python packaging and dependencies. Amongst other things, Poetry can help:
- Manage your dependencies by replacing requirements.txt
- Manage your virtualenv by simplifying the creation and activation of a virtualenv for your project
- Manage your Python package by replacing setup.py
- Publish your application to PyPi
- Turn Python functions into command line programs
- Ensure package integrity
It sounds like magic and too good to be true, but there is really nothing magical happening here. Poetry is just a modern tool implementing the best practices from Python and other tools to manage a project properly. Poetry is leveraging 2 main files:
pyproject.toml: As the main configuration file for your Python project, this file can be edited manually and Poetry also helps to manage the file. Thepyproject.tomlfile is not specific to Poetry and is meant to be the main configuration file for your Python project and all the tools surrounding it (Poetry, Black, etc.). It was introduced to the Python community in 2016 by PEP 518 to improve how to define Python packages, but its scope has increased year over year to become the default configuration file.poetry.lock: A lock file managed by Poetry, this file should never be edited manually. With thepoetry.lockfile, Poetry brings a much-needed feature to Python dependencies management where we can separately maintain the list of primary dependencies, the list of development dependencies, and the exact version of the libraries that should be installed on a system. This feature is common on other languages but it has been used infrequently in Python’ssetup.pyorrequirements.txtin the past. If you have ever generated your requirements.txt file withpip freeze > requirements.txtto ensure that you’ll always install the same version of your dependencies, you should be familiar with the problem the lock file solves. Whilepip freezeworks most of the time, it’s not a great solution and it’s prone to version conflicts between projects, which may require manual intervention.
If you are interested in reading more about the story behind
pyproject.toml, I recommend reading this blog from Brett Cannon.
Install Poetry
To install Poetry on Mac OS, Linux or Windows(bash) the recommended method is to use the below command on your system
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
For convenience, Poetry is also available via pip but it’s not the recommended method to install it. I usually reserve that for when I need to install it within a Docker container:
pip install poetry.
Manage Python dependencies and virtual environment with Poetry
Below is a simple pyproject.toml file to keep track of the dependencies for a project named mypythonproject.
This file can either be generated manually or Poetry can help you to generate it with poetry init.
[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]
[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
Taking a closer look at the file, the first section [tool.poetry] contains information about the project itself and the second section [tool.poetry.dependencies] defines the list of dependencies for the project, including both the Python version and the list of external dependencies that would usually be in a requirements.txt file.
The pyproject.toml file should be at the root of your project (here it’s the only file in my directory). Poetry will automatically install all the dependencies with poetry install (this replaces for pip install -r requirements.txt, python setup.py install, pip install ., or pip install -e .)
➜ mypythonproject# ll
total 8
-rw-r--r-- 1 damien staff 203B May 13 09:17 pyproject.toml
➜ mypythonproject#
➜ mypythonproject# poetry install
The currently activated Python version 2.7.16 is not supported by the project (^3.6).
Trying to find and use a compatible version.
Using python3 (3.7.7)
Creating virtualenv mypythonproject-0zMZkBqq-py3.7 in /Users/damien/Library/Caches/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.2s)
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
- Installing click (7.1.2)
➜ mypythonproject#
During the installation, Poetry automatically generates the poetry.lock file to track the exact version of the dependencies that have been install on my system. If the poetry.lock file was already present, it would have installed the exact version of click defined in the lock file, instead of trying to install the latest one from PyPi.
➜ mypythonproject# ll
total 16
-rw-r--r-- 1 damien staff 606B May 13 09:43 poetry.lock
-rw-r--r-- 1 damien staff 203B May 13 09:17 pyproject.toml
➜ mypythonproject# cat poetry.lock
[[package]]
category = "main"
description = "Composable command line interface toolkit"
name = "click"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
version = "7.1.2"
[metadata]
content-hash = "1876b927e070ae12d1e9090f5ea6bcdd2bb35f09269fc2182bcb9399c5e1be2a"
python-versions = "^3.6"
[metadata.files]
click = [
{file = "click-7.1.2-py2.py3-none-any.whl", hash = "sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc"},
{file = "click-7.1.2.tar.gz", hash = "sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a"},
]
Both the
pyproject.tomland thepoetry.lockshould be tracked in source control (git). Notice the hash values in the lock file, these values ensure the package installed locally is exactly the same as intended.
Also, during poetry install, Poetry created a new virtual environment for my project because it detected that no virtual environment was already associated with the project. Poetry is able to manage multiple environments per project and provides some commands to easily manage these virtual environments.
poetry env infoto list the existing envpoetry shellto activate the default virtualenv (replacessource <path to venv>/activate, orworkon <project>if you use virtualenvwrapper )poetry runto run a command within the default virtual environment without activating it
➜ mypythonproject# poetry env info
Virtualenv
Python: 3.7.7
Implementation: CPython
Path: /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7
Valid: True
System
Platform: darwin
OS: posix
Python: /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7
➜ mypythonproject# poetry shell
The currently activated Python version 2.7.16 is not supported by the project (^3.6).
Trying to find and use a compatible version.
Using python3 (3.7.7)
Spawning shell within /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7
➜ mypythonproject . /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7/bin/activate
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject#
It’s possible to disable the virtual environment management in Poetry with
poetry config virtualenvs.create falseif you want to manage your virtual environment on your own or if you don’t want to use a virtual environment at all.
Add a new dependency to your project
Poetry provides a method to easily install and track a new dependency for your project: poetry add <python package>
In the example below, I’m adding jinja2 as a dependency to my project:
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject# poetry add jinja2
Using version ^2.11.2 for jinja2
Updating dependencies
Resolving dependencies... (0.2s)
Writing lock file
Package operations: 2 installs, 0 updates, 0 removals
- Installing markupsafe (1.1.1)
- Installing jinja2 (2.11.2)
Poetry automatically updated the pyproject.toml and the poetry.lock file in the process:
[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]
[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
jinja2 = "^2.11.2"
Poetry can also maintain a list of dependencies specific to your development environment. To add a new dependency to the development dependencies list you need to add the option -D: poetry add -D pytest. This will create a new section [tool.poetry.dev-dependencies] in the pyproject.toml file.
[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
jinja2 = "^2.11.2"
[tool.poetry.dev-dependencies]
pytest = "^5.4.2"
Managing Python package with Poetry
As mentioned in the introduction, Poetry can also manage your Python package.
By default, Poetry will look for a directory with the name of the project and it will try to install it. In my example, since my project is named mypythonproject in the pyproject.toml, Poetry will automatically look for a directory with this name and install it.
I created a very simple file named cli.py in the directory mypythonproject
# mypythonproject/cli.py
def main():
print("hi there")
Here is how the project looks on my file system.
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject#
.
├── mypythonproject
│ └── cli.py
├── poetry.lock
└── pyproject.toml
Running poetry install again will automatically install the delta between the pyproject.toml file and my environment, here the only delta is the library mypythonproject itself.
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject# poetry install
Installing dependencies from lock file
No dependencies to install or update
- Installing mypythonproject (0.1.0)
Once installed, I can access my code from anywhere as long as I’m still within the same virtual environment. In the example below, I moved outside of the project directory and imported the function main() in Python with from mypythonproject.cli import main
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject# cd /
(mypythonproject-0zMZkBqq-py3.7) ➜ /
(mypythonproject-0zMZkBqq-py3.7) ➜ / python
Python 3.7.7 (default, Mar 10 2020, 15:43:33)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from mypythonproject.cli import main
>>> main()
hi there
We can also check the list of installed packages within the virtual environment with pip list:
(mypythonproject-0zMZkBqq-py3.7) ➜ / pip list | grep mypythonproject
mypythonproject 0.1.0 /Users/damien/projects/mypythonproject
If the name of your directory does not match the name of your project, you need to tell Poetry from which directory to install using packages key as part of the main [tool.poetry] section of the pyproject.toml:
[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]
packages = [
{ include = "mylibraryname" },
]
Creating command line programs with Poetry
Another feature that is extremely useful in Poetry is the ability to easily turn a Python function into an executable/program that will be available in your PATH.
Building on the previous example, I can convert my function main() into a CLI tool with if __name__ == "__main__":. At this point I can execute it as a script as long as I know its exact location.
def main():
print("hi there")
if __name__ == "__main__":
main()
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject# python mypythonproject/cli.py
hi there
By leveraging [tool.poetry.scripts] feature, I can automatically turn my function main() into an executable, here called myawesomecli:
[tool.poetry.scripts]
myawesomecli = "mypythonproject.cli:main"
After reinstalling the library with poetry install, I now have access to a new executable myawesomecli:
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject# myawesomecli
hi there
(mypythonproject-0zMZkBqq-py3.7) ➜ mypythonproject# which myawesomecli
/Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7/bin/myawesomecli
Conclusion
I hope this introduction to Poetry convinced you to give it a try, I know it’s hard to change our habits when it comes to tools and development environment sometimes. I wish I had tried Poetry a long time ago instead of waiting months before transitioning.
Poetry actually does even more than what we covered in this article, so I encourage you to check out the official documentation!
-Damien (@damgarros)
Does this all sound amazing? Want to know more about how Network to Code can help you do this, reach out to our sales team. If you want to help make this a reality for our clients, check out our careers page.
Share :
Contact Us to Learn More
Share details about yourself & someone from our team will reach out to you ASAP!
