Python projects often start with a small dependency list and become much more complicated as development, testing, optional features, and production requirements are added.
For example, a web application may need FastAPI in production, pytest for testing, a linter for development, and an optional database driver for a specific deployment environment. Managing these dependencies separately can become difficult when the team also needs repeatable builds.
A lock file helps capture the resolved dependency set. The pylock.toml format is designed for this type of dependency locking workflow, while extras provide a way to install optional dependency groups.
The important part for developers is understanding how these two concepts work together. An extra describes an optional dependency choice, while the lock information records the concrete packages selected for an environment.
What Is a Python Dependency Lock File?
A project's dependency declaration generally describes what the application needs.
For example:
[project]
dependencies = [
"fastapi>=0.116,<1.0",
"httpx>=0.28,<1.0"
]
These requirements do not necessarily identify every exact package version that will be installed.
The dependency resolver may need to select additional transitive dependencies.
A simplified dependency graph might look like this:
Application
|
+-- FastAPI
| |
| +-- Starlette
| +-- Pydantic
|
+-- HTTPX
|
+-- HTTP Core
A lock file records the resolved result of dependency resolution.
Conceptually:
Project requirements
|
v
Dependency resolution
|
v
Resolved dependency graph
|
v
pylock.toml
|
v
Reproducible installation
This distinction is important. The project configuration expresses intent, while the lock file represents a concrete resolution.
What Are Extras?
Extras allow a project to define optional dependency groups.
For example:
[project.optional-dependencies]
test = [
"pytest",
"pytest-cov"
]
lint = [
"ruff",
"mypy"
]
The main application dependencies remain separate from the optional groups.
A developer working on the project may need:
test
lint
while a production container may need only the default runtime dependencies.
This separation is useful because production does not always require development tooling.
Why Combine Extras With Locking?
Extras introduce different dependency combinations.
Consider a project with:
default dependencies
test dependencies
lint dependencies
The dependency environment for testing is different from the production environment.
If these environments are resolved independently, they can potentially select different versions of shared transitive dependencies.
For example:
Production
Package A 2.1
Package B 4.0
Testing
Package A 2.2
Package B 4.0
Even though both environments use the same project, their dependency graphs can differ.
A lock-file workflow can make those resolved combinations explicit and reproducible.
Define Extras in pyproject.toml
Start by declaring the optional dependency groups.
[project]
name = "orders-api"
version = "1.0.0"
requires-python = ">=3.11"
dependencies = [
"fastapi>=0.116,<1.0",
"httpx>=0.28,<1.0"
]
[project.optional-dependencies]
test = [
"pytest>=8,<9",
"pytest-cov>=6,<7"
]
dev = [
"ruff>=0.12,<1.0",
"mypy>=1,<2"
]
This gives the project a clear dependency structure.
The runtime dependencies are always part of the application environment, while test and dev are optional.
Think of Extras as Dependency Profiles
It is useful to think of extras as profiles.
Default
|
+-- Runtime dependencies
Test
|
+-- Runtime dependencies
+-- Testing dependencies
Dev
|
+-- Runtime dependencies
+-- Development dependencies
This becomes especially useful when the same project needs multiple environments.
For example:
Environment | Dependencies |
|---|
Production | Default |
Development | Default + dev |
Testing | Default + test |
CI | Default + test + dev |
The exact groups should reflect the needs of the project rather than being created simply because the feature exists.
Understand What the Lock File Represents
A lock file should be treated as part of the build input.
The important information captured by a lock workflow can include package versions, dependency relationships, and package distribution information needed for reproducible installation.
A simplified conceptual example might look like:
lock-version = "..."
[[packages]]
name = "fastapi"
version = "..."
[[packages]]
name = "httpx"
version = "..."
The exact structure should be generated and maintained by compatible tooling rather than manually written.
Do not edit generated lock information by hand unless the tool's documentation explicitly requires or supports it.
The lock file should be considered machine-managed configuration.
Generate Lock Information From a Clean Environment
Reproducibility begins with controlled resolution.
Use a clean virtual environment for testing the workflow:
python -m venv .venv
Activate it:
source .venv/bin/activate
On Windows PowerShell:
.venv\Scripts\Activate.ps1
Then use the supported pip workflow for generating or updating lock information.
The exact command should follow the pip version and lock-file workflow being used by the project.
The important principle is to avoid generating a lock file from an environment contaminated by unrelated globally installed packages.
Test the Default Environment
First validate the runtime dependency set.
The application should install without the development extras.
Conceptually:
Default environment
-------------------
FastAPI
HTTPX
Pydantic
Starlette
...
Then run the application test suite appropriate for runtime validation.
For example:
python -m pytest tests/runtime
The purpose is to confirm that the default dependency set contains everything required by the application.
Test the Test Extra
Next, validate the test profile.
The environment should contain the runtime dependencies plus the packages declared under the test extra.
For example:
Test environment
----------------
Runtime dependencies
+
pytest
pytest-cov
Then run:
python -m pytest
If tests import packages that are not included in the test extra, the project dependency declaration is incomplete.
That problem should be fixed in the project configuration rather than solved by manually installing packages on the CI runner.
Test Development Extras Separately
Development tools should also be validated.
For example:
Development environment
-----------------------
Runtime dependencies
+
ruff
+
mypy
Run the tools directly:
ruff check .
and:
mypy src/
This verifies that the development profile contains the tools required by contributors and CI.
Be Careful With Multiple Extras
Some projects define several extras:
[project.optional-dependencies]
test = ["pytest"]
docs = ["sphinx"]
lint = ["ruff"]
dev = ["pytest", "ruff", "mypy"]
This can create overlapping dependency definitions.
For example, dev may duplicate packages already declared by test and lint.
That is not necessarily wrong, but it increases maintenance.
A cleaner structure should minimize unnecessary duplication.
Think about the intended installation combinations before creating many extras.
Validate CI With the Same Dependency Definition
CI should reproduce the dependency environment that developers expect.
A common mistake is having CI install additional packages manually:
steps:
- run: pip install project
- run: pip install pytest
- run: pip install ruff
This can hide missing project dependencies.
A better approach is to define the required dependency profile in the project's configuration and make CI consume that profile.
Conceptually:
Project configuration
|
v
Defined dependency profile
|
v
Lock information
|
v
CI installation
|
v
Tests
This makes the dependency relationship visible and reviewable.
Test Reproducibility With a Second Environment
A lock file is only useful if another clean environment can reproduce the expected dependency state.
Create a second clean environment:
python -m venv .venv-clean
Install using the project's lock workflow.
Then compare the resulting installed packages:
python -m pip list
The goal is to verify that the same locked dependency set is being reproduced.
For stronger validation, perform this test in CI using a fresh runner or clean container.
Production Containers
Locking becomes especially useful when building containers.
A simplified Dockerfile might look like:
FROM python:3.11-slim
WORKDIR /app
COPY pyproject.toml .
COPY pylock.toml .
COPY src ./src
RUN python -m pip install .
CMD ["python", "-m", "src"]
The exact installation command depends on the project's packaging and pip workflow.
The important idea is that the container build should consume controlled dependency information rather than resolving an uncontrolled dependency graph during every build.
This can make builds easier to reproduce and investigate.
Common Mistakes
Treating Extras as Separate Applications
Extras are dependency groups, not independent projects.
The default runtime dependencies remain the foundation.
Installing Development Packages in Production
Installing every optional dependency into production increases the environment size and dependency surface.
Keep production dependencies focused on runtime requirements.
Editing Lock Files Manually
Generated lock information should normally be managed by the tooling that creates it.
Manual edits can create invalid or inconsistent dependency information.
Updating the Project Without Updating the Lock Information
If project dependencies change, the lock state may need to be regenerated.
Otherwise, the declared dependency configuration and resolved dependency state can drift apart.
Testing Only on One Machine
A lock workflow should be tested in a clean environment.
A developer's existing environment can hide missing or incorrectly declared dependencies.
Troubleshooting
The Lock Workflow Produces Unexpected Changes
Review the project dependency configuration first.
A seemingly small change to a version range or optional dependency can affect the resolved dependency graph.
CI Cannot Install a Locked Environment
Check:
Do not immediately regenerate the lock file without understanding why installation failed.
Test Dependencies Are Missing
Check whether the package is declared under the correct optional dependency group.
For example, if tests import pytest, the testing environment must explicitly include it.
Production Contains Development Tools
Review the installation command used by the container or deployment pipeline.
Make sure the production environment is not installing a development or test extra unintentionally.
Best Practices
Keep pyproject.toml as the source of project dependency intent.
Use extras for clearly defined optional dependency profiles.
Keep lock information under version control when the project requires reproducible builds.
Generate lock information using a clean and controlled environment.
Use the same dependency workflow in local development and CI where practical.
Validate default, test, and development profiles separately.
Rebuild production containers from clean environments.
Review lock-file changes during dependency updates.
Avoid unnecessary extras that duplicate large parts of the dependency configuration.
Document which dependency profile belongs to each environment.
Advantages and Disadvantages
Advantages | Disadvantages |
|---|
Makes dependency resolution more reproducible | Lock files require regular maintenance |
Separates runtime and optional dependencies | Multiple extras can become complicated |
Helps create consistent CI environments | Dependency updates may require lock regeneration |
Reduces accidental development dependencies in production | Teams need to understand the dependency workflow |
Makes dependency changes easier to review | Platform-specific dependencies can require additional testing |
Troubleshooting Checklist
When a locked Python environment behaves differently from the expected environment:
Verify the Python version.
Verify the pip version.
Inspect pyproject.toml.
Check the optional dependency group being installed.
Verify that the lock file is current.
Create a clean virtual environment.
Install using the documented lock workflow.
Compare installed packages.
Run application and integration tests.
Repeat the installation in CI or a clean container.
This helps distinguish dependency-definition problems from issues in the locking or installation process.
Summary
pylock.toml and optional dependency extras address two related but different problems in Python dependency management.
Extras let a project define optional dependency profiles such as testing, development, or documentation. Locking captures the resolved dependency state needed to reproduce an environment more consistently.
The practical approach is to keep runtime dependencies clearly defined, use extras for meaningful environment-specific requirements, and maintain lock information through the appropriate tooling.
Before adopting the workflow in production, test it from clean environments and make sure local development, CI, and container builds use the dependency definitions consistently. This turns dependency management from an environment-specific process into a repeatable part of the application's build pipeline.
Join the conversation! Your thoughts help the community grow.