Packaging in Python has been notoriously confusing for newcomers. How should I structure the directories? What is the difference between setup.py and setup.cfg? How do they relate to a requirements.txt? Why do I need a MANIFEST.in file? Do I need tox or nox, and what are they for? Why is flake8 ignoring my pyproject.toml file? What’s the deal with virtual environments?

Luckily, this has been simplified a lot in recent years, in substantial part thanks to uv. I finally (and hopefully permanently) converged on what I call a canonical Python project template. If you are new to Python or generally confused about Python packaging, this blog article is for you. Find the copier template at the end.

pyproject.toml

First of all, pyproject.toml was invented to unify all these scattered configuration files. Not only does it hold metadata about your package, like author, description, version, license, dependencies, etc., but adjacent tools like your linter, formatter, test runner, and so on, can also read their configuration from this file. It fully replaces setup.cfg, which was an attempt to replace setup.py with a completely declarative format.

The requirements.txt file is sometimes used, but often misunderstood. It’s a convenience feature to install a bunch of packages via pip, but if you are developing a proper Python package, dependencies are just one set of metadata that you need to declare, and metadata goes in pyproject.toml. For single-file scripts that you don’t want to package, I recommend putting the requirements in the inline metadata. uv can handle it. I basically never need a requirements.txt. Simply forget that it exists.

The pyproject.toml file is merely a convention, and some tools, e.g., the linter flake8, choose not to participate. I happen to prefer to store as much as possible in that file. Incidentally, flake8 is completely superseded by ruff, another tool by Astral, the company behind uv. It’s much faster, supports way more rules, and also supersedes the formatter black. For that reason, ruff is a dev dependency for all of my Python projects.

Virtual environments

Python looks for dependencies in several places. One of the first directories it looks in is ~/.local/lib/python3.12, a user-writable directory. Next, it looks in a system-wide directory like /usr/lib/python3 (the exact paths may depend on your operating system).

I used to put all dependencies in the user directory, and all my packages – either installed from somewhere or developed by me – used it. It’s only a matter of time until you get a version conflict. Let me give you an example: While it isn’t advisable, some authors define a dependency with an upper bound, say requests<2.14 (perhaps due to a conflict in version 2.14 with another package they didn’t want to deal with), while another package might require requests>=2.32 (because they might rely on a feature of that version). That situation is obviously impossible to satisfy in the same environment, and we get an error.

To avoid this, people use a separate virtual environment for each package they develop or each CLI tool they install. These are nothing but directories in which Python stores things like dependencies. pip even started to enforce this by preventing installations to the user directory unless you give it a special flag.

There used to be several third-party packages to create virtual environments. At some point, one of those was blessed by the Python Software Foundation and included in the standard library. Since then, you can run python3 -m venv <path/to/venv> to create a new directory.

However, in my opinion, it was an oversight not to have a sane default value for that path to the venv. Even the docs suggest three different conventions. uv does it better: run uv venv and it creates ./.venv, a hidden directory in your current working directory. (The actual dependencies are cached somewhere else and only linked to, which is why it is fast and space-efficient.) Not having to care how to name the venv, and being able to reasonably assume that all developers use the same name take a bit of mental load off, which is nice.

Even better: uv is incredibly fast compared to python3 -m venv. It uses smart algorithms as well as aggressive caching, and the fact that it’s been written in Rust probably gives it another boost. It makes venvs cheap and easy. You don’t even need to think much about venvs anymore; uv will simply create one whenever it’s needed, for example when running uv run my-python-app (which runs my-python-app in the project’s venv). Another relief for our mental load.

Linting, formatting and type checking

Python allows you to write a lot of questionable code. Some things are bad ideas even if they’re technically correct Python. A linter helps you stay focused and disciplined. Did you really want to assign a value to a variable which is never used? Did you really mean to declare a function whose second argument is never used? Or did you miss something? This not only helps you write correct, clear and sensible code, it also helps AI agents do the same. ruff enforces rules like that at blazing speed.

Back in the day, people had long discussions around how to format code. People had different preferences, and code written in inconsistent coding styles is confusing to read. These days, people mostly recognize that consistency is what matters more than anything else. ruff also acts as a formatter and comes with sensible defaults, and that’s what I use. One exception: I set the maximum line length to 120 (default: 88), but that’s a personal preference.

Type checking is also a relatively recent thing that came to Python. The big sell of Python as a learning language used to be “duck typing”: if the object has the method you need, that’s good enough. (“If it looks like a duck and quacks like a duck, it’s probably a duck.”) Turns out that this is super confusing in larger code bases, and Python matured quite a lot from a learning language to a very serious language people use to build huge projects. A type checker uses static code analysis to track an object’s type while it gets passed around in the program. Folks coming from Java or C++ are probably rolling their eyes now, but hey – better late than never, right?

I was very reluctant at first, because it’s a huge pain to retrofit an existing code base with type hints, especially if you used a framework that did nasty things, like Django, where you have dynamically created classes galore. Django projects are hard to type check. As far as I know, it can only be done using mypy with a Django-specific plugin.

But since then I’ve seen how nice it is to have my editor warn me about nonexistent attributes or objects which might be None even before saving the file. Even less mental load! For new Python projects, type hints are non-negotiable, if you ask me. Just like strict linting, type hints also help AI agents to find mistakes early and catch hallucinations.

Astral is also developing a type checker called ty. I’ve started to use that one, but pyright or mypy are fine, too.

I strongly recommend running these checks before committing and enforcing this via a pre-commit hook. prek is great for this. Needless to say, this also reminds AI agents that we have standards they need to adhere to.

Tests and docs

If you want others to use your software, you owe them a certain quality standard. Tests are a must, unless you want to constantly break parts of your code. And guess what, this is especially true for agentic coding. Isn’t it funny how using AI agents shows us how useful all these practices – which we should have been embracing all along – really are? A good test suite gives us confidence in our code and makes refactoring a lot less scary. In Python, pytest is the de facto standard. We could also use coverage to make sure all parts of our code are covered by tests, but that is for another time.

Tools like tox or nox can be used to run the test suite for different Python versions, but again: uv made that mostly obsolete. How to do that is a more advanced topic and also for another blog article.

Docs are similarly important depending on the size of your project. I like zensical, the successor to Material for MkDocs. However, sometimes a good README.md is good enough.

By the way, regarding the README, this is my advice: refrain from having AI generate the README. It’s the first thing many users see, and putting thought into what you personally want new users to see demonstrates respect and dedication. I’m clearly not opposed to AI and I don’t think code generated by AI is objectionable in general at all, but the README is like a business card. Not putting in even a minimum of effort shows that you don’t care about your users. A README that looks like all the other READMEs that have been AI-generated is a bit off-putting.

Versioning

Let’s say your little project actually gets popular and people start using it. Eventually, it will become important to know which version they are running so you can have productive discussions about issues, new features, usage, and so forth.

You can edit the version field of pyproject.toml for every release (or do that using uv version), but I like to version every commit. Sometimes I test my tools before a release and install them by running uv tool install -e ., so it’s good to know which version I’m running exactly. We can achieve this using dynamic versioning. This way, the abbreviated commit hash is appended to the version, such as 0.1.0.post0.dev0+713c391. This requires hatchling at build time.

Note that this makes things complicated when attempting to build from source without a git repository, but I find that such circumstances are rare.

Directory layout

There are two schools of thought in the Python world: placing the source directly in the root, or placing it in a directory called src/. I favor the latter (and that is what happens when running uv init). So my recommendation is this:

my-project/
├── docs/
│   └── index.md
├── frontend/              # only for apps with a web frontend
│   ├── deno.json
│   ├── greet.ts
│   ├── greet_test.ts
│   └── main.ts
├── src/
│   └── <package_name>/
│       ├── __init__.py
│       └── py.typed
├── tests/
│   └── test_<package_name>.py
├── .gitignore
├── .pre-commit-config.yaml
├── pyproject.toml
└── zensical.toml

You’ll note that it has a directory called frontend/. This is optional and only relevant for projects that serve an HTML frontend. It’s where the TypeScript and CSS files go. I’m not a big frontend guy, but I strongly dislike Node.js because I find the package manager and dependency management confusing. (I realize many people feel that way about Python, so I empathize.) I found deno to be a palatable alternative. It has a PyPI wrapper package, so installation is easy and self-contained. It does what I need (linting, formatting, type checking, bundling, etc.), is easy to use and fast. That’s all I have to say.

Task runner

In the past, I alternated between using make and a directory full of shell scripts to perform common tasks, like:

  • linting
  • formatting
  • type checking
  • running
  • version bumping
  • building
  • publishing

and more. Both work okay, but make wasn’t really made for this. Makefiles have weird syntax, require us to define “phony targets”, and just aren’t the right choice. Shell scripts have their own issues. There is something better: just.

The blog article Just! Stop using Makefiles along with my recent discovery that just is distributed by Debian Stable (and thus Ubuntu) finally convinced me to just use just. It reads my .env, has sane syntax, is fast, and is convenient.

It’s yet another reduction of my mental load to just enter a directory after months or even years of not thinking about it, and simply typing just test. It makes picking up old projects really easy.

copier template

Once I grokked how Python packaging works, I almost got addicted to it. One-off shell scripts or even one-off Python scripts feel clunky now. There is no question where to put Python packages, how to install them, or how to remove them. Installing one is as easy as installing dozens: uv tool install app1 app2 app3 .... Dependencies are pulled in automatically and everything is much more consistent.

It didn’t take long before I struggled to stay on top of all the packages I wrote, published or unpublished.

The advent of highly capable AI agents exacerbated this, because they make it orders of magnitude easier to quickly write small Python packages. Thus, having a canonical structure and being able to quickly and consistently instantiate a new project are becoming increasingly important (to me). That’s why I turned to copier.

One of the things uv is missing is the possibility to create new Python projects from a template. We can use uv init, and it has great defaults, but they aren’t customizable. So I created my own copier template.

Creating a new Python project is now as simple as running:


uv tool run copier copy https://github.com/AdrianVollmer/python-project-template.git my-project

Final remarks

This setup requires only uv to be installed, with just being a strong recommendation. Technically, you wouldn’t even need Python, since uv will download a suitable binary if necessary. Everything is as frictionless as possible and we’re ready to pump out Python packages at full speed!

As a side note, I can’t help but notice a lot of these tools were written in Rust:

  • uv
  • ruff
  • ty
  • zensical
  • deno
  • prek
  • just

I don’t know what it is, but the Rust community really seems to have a knack for good UI/UX when it comes to CLI applications. That isn’t something that type safety buys you. Unfortunately, Rust applications – like those written in many modern languages – take forever to be accepted in the Linux distribution of my choice: Debian. Even Debian Unstable. Modern software development decided that dynamically linking dependencies (or the equivalent in interpreted languages) is just too slow, so developers are “statically linking” everything while pinning dependencies, and maintainers of Linux distributions have a hard time keeping up. At least that’s my impression.

Anyway, this already got way longer than I originally intended. I hope you found it helpful!