Docker makes application deployment easier by packaging application code, dependencies, and runtime configuration into container images. It also provides a build cache that avoids repeating expensive operations when building an image.
However, developers sometimes notice an unexpected problem: changing a small dependency file causes Docker to reinstall every package, rebuild application layers, or repeat operations that were already completed successfully.
For example, a developer changes one package version in a project and runs docker build again. Instead of downloading only the changed dependency, Docker may execute several subsequent build instructions again.
This behavior is not necessarily a Docker bug. It is usually related to how Docker evaluates build cache and determines whether a previous layer can be reused.
Understanding cache invalidation is important because dependency installation, compilation, and asset generation can consume significant time during local development and CI/CD builds.
This article explains why small dependency changes trigger unexpected Docker rebuilds, how Docker evaluates cache entries, how to structure Dockerfiles for better cache reuse, and how to troubleshoot slow builds in production development pipelines.
Understanding How Docker Build Cache Works
When Docker builds an image, it processes the Dockerfile instructions in order.
Each instruction produces a filesystem change or build configuration that contributes to the resulting image.
Docker checks whether the result of an instruction can be reused from a previous build.
If the instruction and its relevant inputs match a cached result, Docker can reuse that result instead of executing the instruction again.
Consider this Dockerfile:
FROM node:22-alpine
WORKDIR /app
COPY . .
RUN npm install
RUN npm run build
CMD ["node", "server.js"]
During the first build, Docker performs the following operations:
- Pulls or reuses the Node.js base image.
- Creates the application working directory.
- Copies project files into the image.
- Installs dependencies.
- Builds the application.
- Configures the container startup command.
On a subsequent build, Docker checks whether each instruction can use an existing cached result.
If nothing relevant has changed, Docker can reuse most or all of the previous build results.
However, suppose you modify a source file or a dependency file.
The COPY instruction may no longer match the previous cache entry. Docker then invalidates that layer and the subsequent instructions that depend on it.
As a result, npm install and npm run build may execute again.
This is the basic reason why a small project change can trigger a much larger rebuild.
Why a Small Dependency Change Invalidates Multiple Layers
Docker's cache invalidation rules are based on the instructions and their relevant inputs.
Understanding these rules helps explain why some changes trigger rebuilds while others do not.
Changes to COPY instructions
Docker evaluates the files involved in COPY and ADD instructions when checking cache validity.
For example:
COPY . .
This instruction copies the entire build context into the image, except files excluded through .dockerignore.
If any relevant file changes, the cache for this instruction may be invalidated.
Consider this project:
my-application/
package.json
package-lock.json
server.js
Dockerfile
README.md
src/
app.js
With COPY . ., changing any included file can invalidate the cache.
Even updating README.md can cause the dependency installation step to run again if it appears after the COPY instruction.
The problem is not that README.md affects npm dependencies. The problem is that the COPY instruction includes both files.
A better approach is to copy only the files required for each build stage.
Changes to dependency manifests
Dependency manifests describe the packages required by an application.
Examples include:
Platform | Dependency manifest | Lockfile |
|---|
Node.js | package.json | package-lock.json |
.NET | Project file (.csproj) | packages.lock.json |
Python | requirements.txt or pyproject.toml | requirements lockfile |
Go | go.mod | go.sum |
Java with Maven | pom.xml | Dependency version information in the POM or associated lock mechanisms |
Java with Gradle | build.gradle | Gradle dependency locking information |
When a dependency manifest changes, rebuilding the dependency installation layer is usually necessary.
For example, adding a new package to package.json changes the inputs to npm ci.
Docker should invalidate the dependency installation layer because the required dependencies have changed.
However, changing application source code should not necessarily trigger dependency installation again.
That distinction is the foundation of an optimized Dockerfile.
Changes to RUN instructions
Docker also checks the command used in a RUN instruction.
For example:
RUN npm ci
If the command changes to:
RUN npm install
Docker considers it a different instruction and cannot reuse the previous result as though the commands were identical.
Another important detail is that Docker does not automatically inspect the external package repository every time it checks a cached RUN instruction.
For example:
RUN apk add curl
If the command and its preceding cache dependencies remain unchanged, Docker may reuse the existing layer even when a newer package version is available in the repository.
A rebuild does not automatically mean every external dependency has been refreshed.
Use explicit version pinning, controlled base image updates, and deliberate cache invalidation when reproducibility or security updates require it.
The Biggest Dockerfile Mistake: Copying Everything Before Installing Dependencies
One of the most common causes of unnecessary dependency installation is placing COPY . . before the dependency installation command.
Consider this Dockerfile:
FROM node:22-alpine
WORKDIR /app
COPY . .
RUN npm ci
RUN npm run build
CMD ["node", "server.js"]
Suppose you change only src/app.js.
The build process behaves like this:
Docker instruction | Result |
|---|
FROM node:22-alpine | Cached if the base image matches |
WORKDIR /app | Cached if its inputs match |
COPY . . | Cache invalidated |
RUN npm ci | Re-executed |
RUN npm run build | Re-executed |
CMD | Image configuration regenerated as needed |
The dependency installation runs again even though package.json and package-lock.json have not changed.
For a large application, this can add unnecessary time to every development build.
Optimize the Dockerfile by separating dependencies from source code
A better approach is to copy dependency manifests first.
FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
CMD ["node", "server.js"]
Now the dependency installation layer depends on package.json and package-lock.json rather than the entire application source.
The behavior changes:
Change | Dependency installation |
|---|
Modify src/app.js | Reused from cache |
Modify server.js | Reused from cache |
Modify README.md | Reused from cache |
Add a package to package.json | Re-executed |
Update package-lock.json | Re-executed |
This structure provides better cache reuse without sacrificing dependency consistency.
The same principle applies to other programming languages: copy the dependency manifests before copying frequently changing source files.
Why Lockfiles Matter During Docker Builds
Lockfiles are important because they describe the resolved dependency versions used by the application.
For example, package.json may contain:
{
"dependencies": {
"express": "^5.0.0",
"redis": "^5.0.0"
}
}
The version ranges allow compatible package updates.
However, the actual resolved versions can change depending on when dependencies are installed.
A package-lock.json file records the resolved dependency tree.
Using npm ci ensures that installation follows the lockfile rather than resolving a new dependency tree.
A typical Dockerfile should therefore use:
COPY package.json package-lock.json ./
RUN npm ci
Avoid relying on:
RUN npm install
when the intention is to reproduce an existing locked dependency tree in automated builds.
The equivalent approach depends on the package manager:
Package manager | Reproducible installation command |
|---|
npm | npm ci |
Yarn | yarn install --immutable |
pnpm | pnpm install --frozen-lockfile |
Go | go mod download with go.mod and go.sum |
.NET | dotnet restore with a committed packages.lock.json when lockfile enforcement is configured |
Python | Install from a pinned requirements or lock file |
The exact command and lockfile behavior depend on the package manager and project configuration.
A dependency update should intentionally invalidate the relevant dependency layer. An unrelated source change should not.
Using .dockerignore to Reduce Unnecessary Cache Invalidation
Another common problem is sending unnecessary files into the Docker build context.
When you run:
docker build .
Docker uses the current directory as the build context.
Files that are not needed during image construction can increase context size and affect COPY operations.
For example, a Node.js project may contain:
node_modules/
.git/
dist/
coverage/
logs/
.env
Copying these directories into the image can increase build time and create unnecessary cache dependencies.
Create a .dockerignore file:
node_modules
.git
dist
coverage
logs
.env
.env.*
*.log
This excludes the specified files from the build context.
There are important exceptions to consider.
For example, if the application requires a generated dist directory during the build, do not exclude it unless the Dockerfile generates it inside the build process.
Similarly, environment files containing secrets should not be copied into the image. Use runtime configuration or BuildKit secrets for sensitive build-time values.
Why .dockerignore improves cache performance
Suppose a developer changes a local log file.
If the file is included in the build context and copied using COPY . ., it may invalidate the corresponding cache layer.
If the file is excluded through .dockerignore, it does not affect that COPY instruction.
A carefully maintained .dockerignore file reduces context transfer, avoids unnecessary cache invalidation, and helps prevent accidental inclusion of local files.
Using BuildKit Cache Mounts for Faster Dependency Installation
Even after optimizing Dockerfile instruction order, dependency installation can still take time when the dependency layer must be rebuilt.
For example, changing package-lock.json correctly invalidates npm ci.
Docker must run the installation again because the dependency tree has changed.
However, it does not necessarily need to download every package from the internet again.
BuildKit provides cache mounts that allow package managers to reuse downloaded package data across builds.
For npm:
# syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
npm ci
COPY . .
RUN npm run build
CMD ["node", "server.js"]
The cache mount stores npm's downloaded package data separately from the resulting image layer.
When the dependency installation runs again, npm can reuse packages already present in its cache.
This is particularly useful when:
- A dependency version changes.
- A new package is added.
- The lockfile is updated.
- The dependency layer must be rebuilt for another reason.
The package manager still validates and installs the required dependency tree. The cache mount only helps avoid unnecessary downloads.
Cache mounts do not replace the Docker layer cache
These are two different caching mechanisms.
Cache type | What it stores | Main benefit |
|---|
Docker build layer cache | Results of Dockerfile instructions | Skips repeating unchanged build steps |
BuildKit cache mount | Package manager or compiler cache files | Speeds up commands that need to execute again |
External build cache | Build cache exported to or imported from another location | Reuses build results across different builders |
A cache mount should be treated as an optimization, not as a required source of build correctness.
The build must still work if the cache mount is empty or its contents have been removed.
Optimizing .NET Docker Builds
The same cache invalidation problem occurs in .NET applications.
Consider this Dockerfile:
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY . .
RUN dotnet restore
RUN dotnet publish -c Release -o /app/publish
FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY --from=build /app/publish .
ENTRYPOINT ["dotnet", "MyApplication.dll"]
This works, but every source change can invalidate COPY . ., which causes dotnet restore to run again.
For a large .NET solution with many projects and dependencies, repeated restore operations can increase build time.
Separate project files from application source
A more efficient approach is to copy project files first.
For example:
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY MyApplication/MyApplication.csproj MyApplication/
RUN dotnet restore MyApplication/MyApplication.csproj
COPY . .
RUN dotnet publish MyApplication/MyApplication.csproj \
-c Release \
-o /app/publish \
--no-restore
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .
ENTRYPOINT ["dotnet", "MyApplication.dll"]
Now the restore layer depends on the project file.
If application source code changes but the project file remains unchanged, Docker can reuse the restore layer.
The publish operation still runs because the application source has changed.
What about multi-project .NET solutions?
A solution may contain multiple projects:
MySolution/
MySolution.sln
src/
Api/
Api.csproj
Business/
Business.csproj
Data/
Data.csproj
Shared/
Shared.csproj
The API project may reference the Business, Data, and Shared projects.
In this case, copying only Api.csproj is insufficient if restore needs the referenced project files.
Copy all relevant project files before running restore:
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY MySolution.sln ./
COPY src/Api/Api.csproj src/Api/
COPY src/Business/Business.csproj src/Business/
COPY src/Data/Data.csproj src/Data/
COPY src/Shared/Shared.csproj src/Shared/
RUN dotnet restore MySolution.sln
COPY . .
RUN dotnet publish src/Api/Api.csproj \
-c Release \
-o /app/publish \
--no-restore
FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY --from=build /app/publish .
ENTRYPOINT ["dotnet", "Api.dll"]
If the solution uses Directory.Build.props, Directory.Build.targets, NuGet.config, central package management files, or other restore-related configuration, copy those files before the restore step as well.
For example, a change to Directory.Packages.props can affect dependency resolution even when the individual project files remain unchanged.
The rule is simple: every file that can influence dependency resolution must be included in the dependency layer's inputs.
Understanding Build Context and COPY Behavior
Docker cache optimization is not only about changing the order of instructions.
It is also about controlling which files are copied into each layer.
Consider:
COPY . .
This copies the complete permitted build context.
A more targeted approach is:
COPY package.json package-lock.json ./
RUN npm ci
COPY src/ ./src/
COPY server.js ./
Only the required files are copied at each stage.
This reduces unnecessary cache invalidation.
However, targeted COPY instructions should reflect the actual build requirements.
If the application uses configuration files, generated code, or additional assets, those files must also be included.
A Dockerfile that is optimized for caching but omits required inputs is not a correct build.
Understanding file metadata and cache checksums
Docker uses checksums of relevant file metadata to evaluate cache validity for COPY and ADD operations.
File modification time alone is not used in the cache checksum.
This means that simply touching a file without changing relevant content or metadata does not necessarily invalidate the cache.
However, changes to file contents, permissions, or other relevant metadata can invalidate the COPY cache.
Understanding this behavior helps distinguish actual cache invalidation from changes that do not affect the build.
Using Multi-Stage Builds Without Rebuilding Everything
Multi-stage builds separate application compilation from the final runtime image.
They are especially useful for .NET, Java, Go, Node.js, and other applications that require build tools or development dependencies.
For example:
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY MyApplication.csproj ./
RUN dotnet restore MyApplication.csproj
COPY . .
RUN dotnet publish MyApplication.csproj \
-c Release \
-o /app/publish \
--no-restore
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .
ENTRYPOINT ["dotnet", "MyApplication.dll"]
The SDK image contains the tools required to restore and publish the application.
The final runtime image contains only the published output and the runtime environment.
This reduces the final image size and separates build dependencies from runtime dependencies.
Using COPY --link
BuildKit also supports COPY --link.
For example:
COPY --link --from=build /app/publish .
This creates a linked copy layer that can be reused independently of certain changes to previous layers.
It can improve cache reuse in multi-stage builds, especially when copying artifacts from one stage to another.
However, COPY --link has specific filesystem semantics.
It cannot always be used when the destination depends on files or directories created by previous instructions.
Review the Dockerfile reference before applying it to complex build stages.
Why Builds Are Still Slow in CI/CD Pipelines
A Dockerfile may be optimized correctly on a developer's machine but still perform slowly in a CI/CD pipeline.
One reason is that local Docker build cache is not automatically available to every CI runner.
For example:
Developer Machine
|
v
Local Docker Cache
CI Runner 1
|
v
Fresh Builder
CI Runner 2
|
v
Another Fresh Builder
If every CI job starts with a fresh builder, previously created local layers may not be available.
This can cause dependency installation and compilation to run again.
Use external build cache
BuildKit supports exporting and importing build cache.
For example:
docker buildx build \
--cache-from=type=registry,ref=myregistry/myapp:buildcache \
--cache-to=type=registry,ref=myregistry/myapp:buildcache,mode=max \
-t myregistry/myapp:latest \
--push .
The cache can be stored in a registry and reused by subsequent builds.
The cache import and export configuration should be adapted to the CI environment and registry permissions.
External build cache is particularly useful when:
- CI runners are temporary.
- Multiple developers build the same application.
- A team uses separate build and deployment pipelines.
- Multi-stage builds contain expensive compilation steps.
Important limitation of cache mounts in CI
BuildKit cache mounts and exported build caches are not the same.
A cache mount is local to the builder's storage and is not automatically included in a registry cache export.
If every CI job creates a fresh builder, package manager cache mounts may start empty.
Therefore, CI pipelines need an appropriate strategy for preserving builder state or restoring package manager caches separately.
Do not assume that configuring cache mounts alone will make dependency downloads disappear across independent CI runners.
How to Troubleshoot Unexpected Docker Rebuilds
When Docker rebuilds more layers than expected, investigate the build output before deleting caches.
Start with:
docker build --progress=plain -t myapp .
The plain progress output makes it easier to identify which steps were reused and which were executed again.
Look for entries such as:
[+] Building 12.5s
[1/7] FROM node:22-alpine
CACHED
[2/7] WORKDIR /app
CACHED
[3/7] COPY package.json package-lock.json ./
CACHED
[4/7] RUN npm ci
CACHED
[5/7] COPY . .
DONE
[6/7] RUN npm run build
DONE
[7/7] CMD ["node", "server.js"]
DONE
This output shows that dependency installation was reused while the application build ran again.
If npm ci is unexpectedly executed, inspect the cache inputs:
- Did package.json change?
- Did package-lock.json change?
- Did the COPY instruction change?
- Did an earlier Dockerfile instruction change?
- Is the build using the same base image and build arguments?
- Is the CI builder able to access the previous cache?
Inspect image history
You can also inspect the image history:
docker history myapp:latest
This helps identify image layers and their sizes.
It is useful for understanding which operations contribute large layers, although it does not provide a complete explanation of every BuildKit cache decision.
Avoid clearing the cache as the first solution
Developers sometimes run:
docker builder prune
or:
docker build --no-cache .
These commands can be useful when troubleshooting corrupted or intentionally invalidated build caches.
However, they remove or bypass useful cached results.
If the Dockerfile has poor instruction ordering, clearing the cache does not solve the underlying problem.
First identify the invalidated instruction and its inputs.
Common Docker Build Cache Mistakes
Mistake | Why it causes problems | Better approach |
|---|
Using COPY . . before dependency installation | Source changes invalidate dependency layers | Copy dependency manifests first |
Ignoring lockfiles | Dependency resolution may vary between builds | Use committed lockfiles and reproducible install commands |
Copying node_modules from the host | Can introduce platform-specific dependencies | Exclude it using .dockerignore |
Including .git and logs in the build context | Unrelated changes may invalidate COPY operations | Maintain a focused .dockerignore |
Running docker build --no-cache routinely | Prevents useful layer reuse | Use only when a fresh execution is required |
Assuming RUN always checks external repositories | Cached commands may reuse older results | Use deliberate version and cache update strategies |
Relying only on local cache in CI | Fresh runners may not have previous layers | Configure external build cache |
Using cache mounts as persistent image data | Cache mounts are temporary build optimizations | Keep required application files in normal image layers |
Copying only one project file in a multi-project .NET solution | Restore may fail or use incomplete inputs | Copy all restore-related project and configuration files |
Over-optimizing COPY instructions | Required build inputs may be omitted | Validate the Dockerfile against the actual project |
Recommended Dockerfile Optimization Checklist
Before optimizing a Docker build, review the following areas:
- Place dependency installation after copying dependency manifests.
- Keep frequently changing source files out of dependency installation layers.
- Commit and use lockfiles.
- Use package-manager-specific reproducible installation commands.
- Maintain a focused .dockerignore file.
- Use BuildKit cache mounts for package manager and compiler caches.
- Use multi-stage builds to separate compilation and runtime environments.
- Consider COPY --link where its filesystem semantics are appropriate.
- Configure external build cache for temporary CI runners.
- Pin base image versions or digests when reproducibility is required.
- Review build output to identify unexpected cache misses.
- Avoid clearing the build cache without understanding why it was invalidated.
Conclusion
Small dependency and source changes can trigger unexpectedly large Docker rebuilds because Docker evaluates cache validity instruction by instruction.
When an instruction is invalidated, subsequent dependent instructions may need to run again.
The most common problem is copying the entire application before installing dependencies. A small source change then invalidates the dependency installation layer even when the dependency manifests have not changed.
Separate dependency manifests from application source, use lockfiles, maintain a focused .dockerignore file, and take advantage of BuildKit cache mounts.
For CI/CD environments, configure external build cache so temporary runners can reuse previous build results.
The objective is not to prevent every rebuild. Dependency changes should trigger dependency installation, and source changes should trigger application compilation. The goal is to ensure that Docker rebuilds only the parts of the application that actually depend on the changes.
Join the conversation! Your thoughts help the community grow.