A Python application may contain only a few source files, yet its Docker image can grow to hundreds of megabytes or even several gigabytes. This often happens because the image includes unnecessary dependencies, development tools, build artifacts, cached package files, or a base image that contains more software than the application needs.
Large Docker images create practical problems. They take longer to build, consume more storage in container registries, increase deployment time, and require more data to be transferred when deploying to new machines. They can also contain unnecessary packages that increase the maintenance and security burden.
Python dependencies are a common source of image growth. A package may install compiled libraries, native extensions, additional Python packages, or system-level components. A single dependency can introduce several layers of additional software.
The solution is not to remove packages randomly. First, identify which image layers and installed packages consume the most space. Then determine which dependencies are actually required at runtime and optimize the Docker build accordingly.
This article explains how to investigate Python dependency sizes, inspect Docker image layers, identify unnecessary packages, and build smaller production images without breaking the application.
Understand What Makes a Docker Image Large
A Docker image is assembled from a sequence of layers. Each instruction in a Dockerfile may create a new layer, depending on the instruction and how the build is structured.
For a Python application, image size can come from several sources:
- The operating system included in the base image.
- The Python runtime and supporting libraries.
- Installed Python packages.
- Native libraries required by packages such as scientific computing or image-processing tools.
- Compilers and development headers used to build native extensions.
- Package caches and temporary build files.
- Application source code, test data, and local development files.
Consider an application that uses FastAPI, PostgreSQL, and a machine-learning library. The application source code might be relatively small, but the machine-learning dependencies could require substantial numerical libraries and other supporting packages.
Before changing dependencies, determine whether the image is large because of the base image, Python packages, system packages, or files copied from the project directory.
Inspect the Docker Image Size
Start by listing the images available on your machine:
docker images
This command shows image names, tags, image identifiers, and reported sizes.
For a specific image, use:
docker image inspect my-python-app:latest
The output contains metadata about the image, including its configured environment and layer information.
Remember that the size shown by Docker is not necessarily the amount of additional disk space required by every deployment. Images can share layers, and the amount of storage used on disk depends on which layers are already present.
Examine Image Layers with Docker History
One of the simplest ways to investigate image growth is to inspect the history:
docker history my-python-app:latest
The output includes the size and corresponding Dockerfile instruction for each recorded layer.
For example, you might see a large layer associated with a command that installs Python dependencies:
RUN pip install -r requirements.txt
This indicates that the dependency installation step contributes significantly to the image size.
However, Docker history does not identify the exact size of every Python package installed in that layer. It helps locate the layer that introduced the growth, after which you can inspect the installed packages.
Also, a small or empty-looking instruction in the history does not guarantee that the instruction had no effect on the final image. Multi-stage builds, cached layers, and metadata instructions can make the output less straightforward to interpret.
Find Which Python Packages Consume the Most Space
Once you know that Python dependencies contribute significantly to the image, inspect the installed packages.
A basic command is:
python -m pip list
This lists installed Python packages and their versions. It does not report their disk usage, so it is useful for identifying what is installed rather than ranking packages by size.
To inspect package locations, use:
python -m pip show numpy
The output includes the installation location and package metadata.
For a more useful size comparison, inspect the installed package directories inside the container.
Measure Installed Package Directories
The following command measures the approximate sizes of installed Python package directories in a typical Linux-based image:
du -sh /usr/local/lib/python*/site-packages/* 2>/dev/null | sort -h
This displays directory sizes in a human-readable format and sorts the results by size.
The command assumes that Python packages are installed under /usr/local/lib/python.../site-packages, which is common in many official Python images. Other images may use a different path.
To see the largest entries first, use:
du -sh /usr/local/lib/python*/site-packages/* 2>/dev/null | sort -hr | head -20
This helps identify directories that occupy the most space.
For example, packages that include compiled extensions, large datasets, or numerical libraries may appear near the top.
There are two important limitations to remember:
- Directory sizes do not always correspond exactly to the amount of unique storage consumed by each package.
- One package may depend on another package, so removing a large package may affect other application features.
Use these results to decide what to investigate next, not as an automatic removal list.
Build a Temporary Inspection Image
If the final production image does not contain diagnostic tools such as du, build a temporary inspection target.
For example:
FROM python:3.12-slim AS inspect
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "--version"]
Build the inspection image:
docker build --target inspect -t my-python-app:inspect .
Then start a shell inside the image:
docker run --rm -it --entrypoint sh my-python-app:inspect
Inside the container, inspect the Python package directories and other large locations.
This approach lets you examine the installed environment without adding diagnostic utilities to the final production image.
The Python version in this example is illustrative. Use a version supported by your application and its dependencies.
Identify Unnecessary Dependencies
Finding a large package does not automatically mean it should be removed. First, establish whether the application needs it at runtime.
Python projects often install development and testing dependencies alongside production dependencies.
For example, a requirements file might contain:
fastapi
uvicorn
psycopg
pytest
black
mypy
The first three packages may be required to run the service, depending on how the application is configured. The others are commonly used for testing, formatting, or static type checking.
Installing all of them into the production image may be unnecessary.
Separate Runtime and Development Dependencies
A simple project structure might use two files:
requirements.txt:
fastapi
uvicorn
psycopg
requirements-dev.txt:
-r requirements.txt
pytest
black
mypy
Install only the production requirements in the runtime image:
RUN pip install --no-cache-dir -r requirements.txt
Install the development requirements in a development environment:
python -m pip install -r requirements-dev.txt
This is a straightforward approach for smaller projects. Larger projects may benefit from a dependency manager or lockfile workflow that explicitly defines production and development dependency groups.
Do not remove a package merely because it is absent from your application's direct imports. A package may be required by a framework, plugin, runtime feature, or dynamically loaded component.
After changing dependencies, run the application's tests and verify the features used in production.
Review Dependency Trees
Python packages frequently depend on other packages. A large dependency may introduce additional libraries that are not obvious from the project's requirements file.
Use pip to inspect the installed dependency relationships:
python -m pip show fastapi
For a broader dependency report, use:
python -m pip list
You can also use pip's dependency-checking facilities:
python -m pip check
This command checks whether installed packages have compatible declared dependencies. It does not identify unused packages or calculate package sizes.
For dependency-tree inspection, a tool such as pipdeptree can help explain why a package is installed. If you use such a tool, keep it in the inspection or development environment unless it is explicitly needed in production.
The goal is to identify packages that are unnecessary for the application's runtime behavior while preserving required dependencies.
Use a Smaller and More Appropriate Base Image
The base image can contribute substantially to the final size.
For example:
FROM python:3.12
The standard Python image includes more operating system components than the slim variant. A smaller alternative may be appropriate:
FROM python:3.12-slim
The slim image generally includes fewer system packages, but your application may need additional libraries to build or run certain dependencies.
A package that compiles native extensions might require a compiler and development headers during installation. A database driver or image-processing library may also require system-level runtime libraries.
Therefore, test the application after changing the base image.
Alpine-based images can be smaller in some cases, but they use musl libc rather than the glibc environment used by many common Linux distributions. This can complicate the installation of packages that rely on precompiled wheels or native dependencies.
A smaller base image is not automatically the best choice if it makes dependency installation more difficult or creates compatibility problems.
Use Multi-Stage Builds to Remove Build Tools
Some Python dependencies need compilers or development packages during installation. Those tools may not be required after the application has been built.
A multi-stage Dockerfile separates the build environment from the runtime environment.
For example:
FROM python:3.12-slim AS builder
WORKDIR /app
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
FROM python:3.12-slim AS runtime
WORKDIR /app
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY app.py .
CMD ["python", "app.py"]
The builder stage installs the dependencies into a virtual environment. The runtime stage copies that environment into a separate image.
This can prevent build-stage tools and temporary build files from becoming part of the final image.
However, the example assumes that the application can run using the copied virtual environment and the runtime libraries available in the second stage. Native extensions may require shared system libraries at runtime. Both stages should also use compatible operating system environments and Python versions.
If a dependency requires a compiler, install the compiler and development headers in the builder stage, then include only the necessary runtime libraries in the final stage.
Multi-stage builds reduce the final image size only when unnecessary files are actually excluded from the runtime stage.
Prevent Package Caches and Local Files from Entering the Image
Package managers and build contexts can introduce unnecessary files.
When installing Python packages, use:
RUN pip install --no-cache-dir -r requirements.txt
The --no-cache-dir option prevents pip from retaining its package download cache in the environment.
This is especially useful when the installation occurs in a layer that becomes part of the final image.
You should also review the files copied from the project directory. Without a suitable .dockerignore file, Docker may send local virtual environments, test data, logs, and other unnecessary files to the build context.
For example:
.venv/
__pycache__/
.pytest_cache/
.mypy_cache/
.git/
.env
*.log
tests/
Add entries only when those files are not required for your build. For example, a project may need its tests in a dedicated testing stage, even if they should not be included in the final runtime image.
A .dockerignore file reduces unnecessary build-context data and prevents excluded files from being copied by broad COPY instructions. It does not remove files that were already added by another build step or eliminate files installed by package managers.
Compare Common Image Optimization Approaches
Different optimization methods solve different problems.
| Approach | Main benefit | Limitation |
|---|
| Inspect Docker history | Identifies layers associated with image growth | Does not show exact package-level usage |
| Measure site-packages directories | Helps find large installed packages | Directory size does not establish whether a package is unnecessary |
| Remove development dependencies | Reduces unnecessary runtime packages | Requires clear dependency separation and testing |
| Use a slim base image | Reduces the base operating system footprint | May require additional runtime libraries |
| Use multi-stage builds | Excludes build tools and temporary files | Requires compatible build and runtime environments |
| Disable pip caching | Avoids retaining pip download caches | Does not remove installed package files |
| Add a .dockerignore file | Excludes unnecessary local files from the build context | Does not clean existing image layers |
| Replace a large dependency | May significantly reduce installed package size | Requires application changes and compatibility testing |
Common Mistakes to Avoid
Removing Packages Based Only on Size
A large package may provide essential functionality. Confirm its role and dependency relationships before removing it.
Assuming a Slim Image Will Work Without Changes
Some Python packages require native libraries that are absent from slim images. Test the application and its dependencies in the proposed runtime image.
Installing Build Tools in the Final Image
Compilers and development headers can increase image size unnecessarily. Use multi-stage builds when they can be safely excluded from the runtime environment.
Expecting .dockerignore to Shrink Existing Layers
A .dockerignore file controls which files enter the build context. It does not remove files already installed or copied into earlier layers. Rebuild the image after correcting the Dockerfile and build context.
Optimizing Image Size Without Testing Runtime Behavior
A smaller image is not useful if the application fails to start or a required feature no longer works. Test imports, application startup, health checks, database connectivity, and important runtime operations.
Comparing Image Sizes Without Considering Shared Layers
Docker can reuse layers across images. The apparent image size and the additional storage or transfer cost of a deployment are not always identical. Consider the deployment environment when evaluating the benefit of an optimization.
Troubleshooting Large Python Docker Images
When the image remains larger than expected, investigate the following areas:
- If one dependency layer is large, inspect the installed packages and native libraries.
- If the base image contributes most of the size, compare compatible base-image variants.
- If the image contains build tools, check whether a multi-stage build can exclude them.
- If local directories are unexpectedly included, review .dockerignore and COPY instructions.
- If the image grows after a dependency update, compare the dependency versions and newly installed packages.
- If the image becomes smaller but the application fails, inspect missing runtime libraries and dependency compatibility.
Make one major change at a time when practical. This makes it easier to identify which optimization produced the improvement and whether it introduced a new problem.
Summary
Finding which Python dependencies make a Docker image too large starts with identifying where the image growth occurs. Use docker history to locate large layers, inspect installed Python package directories, and review the dependency tree to understand which packages are required.
After identifying the source of unnecessary size, separate development dependencies from production requirements, choose an appropriate base image, use multi-stage builds where suitable, and prevent caches or local files from entering the final image.
The objective is not simply to achieve the smallest possible image. It is to produce a compact, secure, and reliable runtime image that contains everything the application needs and avoids carrying unnecessary tools and files into production.
Join the conversation! Your thoughts help the community grow.