Upgrade Your Python Project With Poetry

Dependency management and virtual environments are integral to the Python ecosystem, yet the primary tools in use today are far from ideal. Some of the primary methods are:

• Dependencies management: pip by way of requirements.txt is still the de facto solution for most of us. While this approach has worked in the past, there are limitations when it comes to guaranteeing that the same project will be consistently installed.
• Virtual environments: a common setup is to use virtualenv to create your virtual environment and manually activate it using source <path to venv>/activate. While this approach works, it requires the user to know which venv needs to be activated for each project and the command to execute can be lengthy.
• Code packaging: (only applicable if you need to share your code), it is common to use setuptools in a setup.py file, but this solution also has some shortcomings.

If you are using any or all of the methods described above, you should take a look at Poetry to help you manage your Python project(s). Poetry’s goal is to simplify the management of Python packaging and dependencies. Amongst other things, Poetry can help:

• Manage your dependencies by replacing requirements.txt
• Manage your virtualenv by simplifying the creation and activation of a virtualenv for your project
• Manage your Python package by replacing setup.py
• Publish your application to PyPi
• Turn Python functions into command line programs
• Ensure package integrity

It sounds like magic and too good to be true, but there is really nothing magical happening here. Poetry is just a modern tool implementing the best practices from Python and other tools to manage a project properly. Poetry is leveraging 2 main files:

• pyproject.toml: As the main configuration file for your Python project, this file can be edited manually and Poetry also helps to manage the file. The pyproject.toml file is not specific to Poetry and is meant to be the main configuration file for your Python project and all the tools surrounding it (Poetry, Black, etc.). It was introduced to the Python community in 2016 by PEP 518 to improve how to define Python packages, but its scope has increased year over year to become the default configuration file.
• poetry.lock: A lock file managed by Poetry, this file should never be edited manually. With the poetry.lock file, Poetry brings a much-needed feature to Python dependencies management where we can separately maintain the list of primary dependencies, the list of development dependencies, and the exact version of the libraries that should be installed on a system. This feature is common on other languages but it has been used infrequently in Python’s setup.py or requirements.txt in the past. If you have ever generated your requirements.txt file with pip freeze > requirements.txt to ensure that you’ll always install the same version of your dependencies, you should be familiar with the problem the lock file solves. While pip freeze works most of the time, it’s not a great solution and it’s prone to version conflicts between projects, which may require manual intervention.

If you are interested in reading more about the story behind pyproject.toml, I recommend reading this blog from Brett Cannon.

Install Poetry

To install Poetry on Mac OS, Linux or Windows(bash) the recommended method is to use the below command on your system

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

For convenience, Poetry is also available via pip but it’s not the recommended method to install it. I usually reserve that for when I need to install it within a Docker container: pip install poetry.

Manage Python dependencies and virtual environment with Poetry

Below is a simple pyproject.toml file to keep track of the dependencies for a project named mypythonproject.
This file can either be generated manually or Poetry can help you to generate it with poetry init.

[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]

[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"

Taking a closer look at the file, the first section [tool.poetry] contains information about the project itself and the second section [tool.poetry.dependencies] defines the list of dependencies for the project, including both the Python version and the list of external dependencies that would usually be in a requirements.txt file.

The pyproject.toml file should be at the root of your project (here it’s the only file in my directory). Poetry will automatically install all the dependencies with poetry install (this replaces for pip install -r requirements.txt, python setup.py install, pip install ., or pip install -e .)

➜  mypythonproject# ll
total 8
-rw-r--r--  1 damien  staff   203B May 13 09:17 pyproject.toml
➜  mypythonproject#
➜  mypythonproject# poetry install
The currently activated Python version 2.7.16 is not supported by the project (^3.6).
Trying to find and use a compatible version.
Using python3 (3.7.7)
Creating virtualenv mypythonproject-0zMZkBqq-py3.7 in /Users/damien/Library/Caches/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  - Installing click (7.1.2)
➜  mypythonproject#

During the installation, Poetry automatically generates the poetry.lock file to track the exact version of the dependencies that have been install on my system. If the poetry.lock file was already present, it would have installed the exact version of click defined in the lock file, instead of trying to install the latest one from PyPi.

➜  mypythonproject# ll
total 16
-rw-r--r--  1 damien  staff   606B May 13 09:43 poetry.lock
-rw-r--r--  1 damien  staff   203B May 13 09:17 pyproject.toml

➜  mypythonproject# cat poetry.lock
[[package]]
category = "main"
description = "Composable command line interface toolkit"
name = "click"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
version = "7.1.2"

[metadata]
content-hash = "1876b927e070ae12d1e9090f5ea6bcdd2bb35f09269fc2182bcb9399c5e1be2a"
python-versions = "^3.6"

[metadata.files]
click = [
    {file = "click-7.1.2-py2.py3-none-any.whl", hash = "sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc"},
    {file = "click-7.1.2.tar.gz", hash = "sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a"},
]

Both the pyproject.toml and the poetry.lock should be tracked in source control (git). Notice the hash values in the lock file, these values ensure the package installed locally is exactly the same as intended.

Also, during poetry install, Poetry created a new virtual environment for my project because it detected that no virtual environment was already associated with the project. Poetry is able to manage multiple environments per project and provides some commands to easily manage these virtual environments.

• poetry env info to list the existing env
• poetry shell to activate the default virtualenv (replaces source <path to venv>/activate, or workon <project> if you use virtualenvwrapper )
• poetry run to run a command within the default virtual environment without activating it

➜  mypythonproject# poetry env info
 
Virtualenv
Python:         3.7.7
Implementation: CPython
Path:           /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7

➜  mypythonproject# poetry shell
The currently activated Python version 2.7.16 is not supported by the project (^3.6).
Trying to find and use a compatible version.
Using python3 (3.7.7)
Spawning shell within /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7
➜  mypythonproject . /Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7/bin/activate
(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# 

It’s possible to disable the virtual environment management in Poetry with poetry config virtualenvs.create false if you want to manage your virtual environment on your own or if you don’t want to use a virtual environment at all.

Add a new dependency to your project

Poetry provides a method to easily install and track a new dependency for your project: poetry add <python package>

In the example below, I’m adding jinja2 as a dependency to my project:

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# poetry add jinja2
Using version ^2.11.2 for jinja2

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  - Installing markupsafe (1.1.1)
  - Installing jinja2 (2.11.2)

Poetry automatically updated the pyproject.toml and the poetry.lock file in the process:

[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]

[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
jinja2 = "^2.11.2"

Poetry can also maintain a list of dependencies specific to your development environment. To add a new dependency to the development dependencies list you need to add the option -D: poetry add -D pytest. This will create a new section [tool.poetry.dev-dependencies] in the pyproject.toml file.

[tool.poetry.dependencies]
python = "^3.6"
click = "^7.1.1"
jinja2 = "^2.11.2"

[tool.poetry.dev-dependencies]
pytest = "^5.4.2"

Managing Python package with Poetry

As mentioned in the introduction, Poetry can also manage your Python package.
By default, Poetry will look for a directory with the name of the project and it will try to install it. In my example, since my project is named mypythonproject in the pyproject.toml, Poetry will automatically look for a directory with this name and install it.

I created a very simple file named cli.py in the directory mypythonproject

# mypythonproject/cli.py 

def main():
    print("hi there")

Here is how the project looks on my file system.

(mypythonproject-0zMZkBqq-py3.7)  ➜  mypythonproject#
.
├── mypythonproject
│   └── cli.py
├── poetry.lock
└── pyproject.toml

Running poetry install again will automatically install the delta between the pyproject.toml file and my environment, here the only delta is the library mypythonproject itself.

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# poetry install
Installing dependencies from lock file

No dependencies to install or update

  - Installing mypythonproject (0.1.0)

Once installed, I can access my code from anywhere as long as I’m still within the same virtual environment. In the example below, I moved outside of the project directory and imported the function main() in Python with from mypythonproject.cli import main

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# cd /
(mypythonproject-0zMZkBqq-py3.7) ➜  /
(mypythonproject-0zMZkBqq-py3.7) ➜  / python
Python 3.7.7 (default, Mar 10 2020, 15:43:33)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from mypythonproject.cli import main
>>> main()
hi there

We can also check the list of installed packages within the virtual environment with pip list:

(mypythonproject-0zMZkBqq-py3.7) ➜  / pip list | grep mypythonproject
mypythonproject       0.1.0      /Users/damien/projects/mypythonproject

If the name of your directory does not match the name of your project, you need to tell Poetry from which directory to install using packages key as part of the main [tool.poetry] section of the pyproject.toml:

[tool.poetry]
name = "mypythonproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["NTC <info@networktocode.com>"]
packages = [
    { include = "mylibraryname" },
]

Creating command line programs with Poetry

Another feature that is extremely useful in Poetry is the ability to easily turn a Python function into an executable/program that will be available in your PATH.

Building on the previous example, I can convert my function main() into a CLI tool with if __name__ == "__main__":. At this point I can execute it as a script as long as I know its exact location.

def main():
    print("hi there")

if __name__ == "__main__":
    main()

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# python mypythonproject/cli.py
hi there

By leveraging [tool.poetry.scripts] feature, I can automatically turn my function main() into an executable, here called myawesomecli:

[tool.poetry.scripts]
myawesomecli = "mypythonproject.cli:main"

After reinstalling the library with poetry install, I now have access to a new executable myawesomecli:

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# myawesomecli
hi there

(mypythonproject-0zMZkBqq-py3.7) ➜  mypythonproject# which myawesomecli
/Users/damien/Library/Caches/pypoetry/virtualenvs/mypythonproject-0zMZkBqq-py3.7/bin/myawesomecli