Finding Slow Tests Before They Delay Your CI Pipeline

A test suite can contain thousands of passing tests and still create a serious problem for a development team: the pipeline takes too long to finish.

When a developer submits a pull request, automated tests are expected to provide quick feedback. If a small change triggers a test suite that takes 40 minutes, developers spend more time waiting for results and less time fixing problems. The delay becomes even more noticeable when several branches are being validated simultaneously.

The difficulty is that test execution time is rarely distributed evenly. A handful of tests may consume a large portion of the total runtime because they repeatedly access databases, start containers, launch browsers, wait for external services, or perform expensive setup operations.

Finding these tests requires more than checking the total duration of a pipeline. Teams need visibility into individual test execution times, setup and teardown costs, resource consumption, and changes in performance over multiple runs.

This article explains how to identify slow tests, investigate their actual cost, and improve CI execution without removing important test coverage.

Why Some Tests Become Slow

Not every test takes the same amount of time. A unit test that checks a calculation may complete in milliseconds, while an integration test that starts a database and sends HTTP requests can take several seconds.

The main causes usually fall into a few categories.

Expensive test setup

A test may spend more time preparing its environment than executing its actual assertions.

For example, creating a new database, inserting hundreds of records, or initializing a complex application context before every test can significantly increase execution time.

If several tests share the same setup requirements, repeated initialization may be unnecessary.

External service dependencies

Tests that communicate with real APIs, databases, message brokers, or cloud services depend on network availability and external response times.

A test that normally takes 200 milliseconds may take several seconds when an external service is slow.

These dependencies also make test execution less predictable.

Browser automation

End-to-end tests using Selenium, Playwright, or other browser automation tools often take longer than unit tests.

Launching a browser, navigating pages, waiting for elements, authenticating users, and interacting with the interface all contribute to execution time.

Unnecessary waiting

Fixed delays are another common source of slow tests.

For example:

import time

def test_order_confirmation():
    create_order()

    time.sleep(5)

    assert get_order_status() == "confirmed"

The test waits five seconds regardless of whether the order is confirmed after 100 milliseconds.

A condition-based wait is generally more appropriate:

import time

def wait_for_order_confirmation(order_id, timeout=5):
    deadline = time.monotonic() + timeout

    while time.monotonic() < deadline:
        if get_order_status(order_id) == "confirmed":
            return True

        time.sleep(0.1)

    return False

This example uses polling to check the expected condition. In production test suites, prefer the application's event mechanism or the testing framework's condition-based wait utilities where available.

The key difference is that the test continues as soon as the required condition is satisfied instead of waiting for an arbitrary fixed duration.

Identify the Slowest Tests First

Before changing the test suite, collect execution-time information.

For Python projects using pytest, the built-in duration reporting option provides a simple starting point.

Run:

pytest --durations=20

This displays the 20 slowest test execution phases.

To ignore very short entries and focus on tests taking at least one second:

pytest --durations=20 --durations-min=1.0

An example report might look like this:

============================= slowest 20 durations =============================

8.42s call     tests/test_orders.py::test_import_large_catalog
5.31s setup    tests/test_api.py::test_customer_search
4.18s call     tests/test_checkout.py::test_apply_coupon
3.75s teardown tests/test_browser.py::test_user_login
2.91s call     tests/test_reports.py::test_generate_monthly_report

============================= 125 passed in 42.6s ===============================

The report provides more than a list of test names. It identifies whether the time was spent during setup, execution, or teardown.

This distinction changes how the problem should be investigated.

For example:

A slow setup phase may indicate expensive fixtures or database initialization.
A slow call phase may indicate inefficient application code or excessive external requests.
A slow teardown phase may indicate incomplete resource cleanup or unnecessary shutdown operations.

A test that appears slow may not have an expensive assertion at all. Its setup or cleanup process may be responsible for most of the delay.

Track Test Duration Across CI Runs

A single test execution is not enough to determine whether a test is consistently slow.

Execution time can vary because of CPU availability, network conditions, shared infrastructure, database load, and parallel test execution.

A test that takes three seconds once may not need immediate optimization. A test that consistently takes eight seconds and accounts for a large portion of the pipeline deserves closer attention.

Store test execution results from every CI run.

Useful information includes:

Test name and unique identifier.
Commit SHA.
Branch name.
Test outcome.
Execution duration.
Setup and teardown duration, where available.
CI runner type.
Test environment.
Retry count.

For example, a historical report might look like this:

Test	Previous duration	Current duration	Change
test_import_large_catalog	3.2 sec	8.4 sec	+5.2 sec
test_customer_search	2.8 sec	5.3 sec	+2.5 sec
test_apply_coupon	4.0 sec	4.2 sec	+0.2 sec
test_user_login	3.5 sec	3.8 sec	+0.3 sec

The first two tests deserve investigation because their execution times increased significantly.

A useful monitoring system should also compare tests within similar environments. Comparing a local laptop result with a heavily loaded CI runner can produce misleading conclusions.

Use averages or medians across several runs rather than treating the latest duration as a reliable baseline. Keep maximum duration available as well because occasional spikes can reveal infrastructure problems.

Add Slow-Test Reporting to a CI Pipeline

Once test duration reporting works locally, integrate it into the automated pipeline.

For a GitHub Actions workflow using pytest:

name: Automated Tests

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install pytest

      - name: Run tests with duration reporting
        run: |
          pytest --durations=20 --durations-min=1.0 \
            --junitxml=test-results.xml

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results.xml

The workflow runs tests and generates a JUnit XML report that can be retained as a CI artifact.

The duration output helps developers identify slow tests directly in the build logs. The XML report provides structured test results that can be processed by reporting tools.

For a production implementation, retain the reports from multiple builds and associate each result with its commit and runner environment.

Also remember that the example assumes dependencies are available through requirements.txt. Projects using Poetry, uv, or another dependency manager should use their existing installation process.

Investigate Why a Test Is Slow

Identifying a slow test tells you where to investigate. It does not explain the underlying cause.

The next step is to profile the test or inspect its dependencies.

Consider this example:

def test_customer_report():
    customers = create_test_customers(1000)

    report = generate_customer_report(customers)

    assert report.total_customers == 1000

The test may take several seconds because creating 1,000 customers involves database inserts, validation, and related-object creation.

Possible improvements include:

Creating only the records required by the scenario.
Using bulk database operations when appropriate.
Reusing immutable reference data.
Avoiding unnecessary relationships.
Testing report calculations independently from database integration.

However, reducing test data must not remove important boundary conditions. If the purpose of the test is to verify performance with 1,000 customers, replacing that dataset with five records would change what the test validates.

A useful investigation separates the test into its major operations and measures each one.

For example:

import time

def test_customer_report():
    start = time.perf_counter()

    customers = create_test_customers(1000)
    setup_time = time.perf_counter() - start

    start = time.perf_counter()

    report = generate_customer_report(customers)
    report_time = time.perf_counter() - start

    print(f"Setup time: {setup_time:.3f}s")
    print(f"Report time: {report_time:.3f}s")

    assert report.total_customers == 1000

This basic instrumentation can help distinguish setup time from application execution time.

For deeper investigation, use a profiler that can identify expensive functions, database queries, fixture execution, and external calls.

Separate Tests by Execution Cost

Not every test belongs in the same CI stage.

A practical testing strategy separates tests according to their purpose and execution characteristics.

Test category	Typical purpose	Execution strategy
Unit tests	Validate individual functions and classes	Run on every relevant pull request
Component tests	Validate a component with controlled dependencies	Run in the main validation pipeline
Integration tests	Verify communication between services	Run in dedicated pipeline stages
End-to-end tests	Validate complete user workflows	Run on selected changes and release pipelines
Performance tests	Measure response time and resource consumption	Run in controlled performance environments

This does not mean slow tests should automatically be removed from pull-request validation.

A critical payment integration test may be slow but still need to run before a change is merged. A browser test covering a low-risk workflow may be suitable for a separate stage.

The decision should consider business risk, failure impact, execution cost, and the value of immediate feedback.

For pytest, custom markers can help organize test categories.

import pytest

@pytest.mark.slow
def test_full_customer_import():
    customers = import_large_customer_dataset()

    assert len(customers) > 0

[tool.pytest.ini_options]
markers = [
    "slow: tests that require additional execution time",
    "integration: tests that require external services"
]

Then run the faster test group separately:

pytest -m "not slow"

Run the slower group in another job:

pytest -m slow

This approach is useful only when the pipeline still runs the excluded tests at an appropriate point. Moving a test out of the pull-request stage should not mean abandoning it.

Use Parallel Execution Carefully

Parallel testing can reduce wall-clock time by distributing tests across multiple workers.

For pytest, pytest-xdist provides parallel execution:

pip install pytest-xdist

Run tests using four workers:

pytest -n 4

The framework distributes test execution across worker processes.

However, increasing the worker count does not always improve performance.

If tests share a database, modify common files, use global state, or depend on execution order, parallel execution can introduce failures.

It can also make a suite slower when workers compete for limited CPU, memory, database connections, or external service capacity.

Before increasing concurrency, check:

Whether tests are independent.
Whether each worker has isolated test data.
Whether shared resources are protected.
Whether the CI runner has sufficient CPU and memory.
Whether external services can handle parallel requests.

A parallel test suite should be both faster and reliable. Speed improvements that introduce nondeterministic failures are not useful.

Distinguish Slow Tests From Flaky Tests

Slow tests and flaky tests are related operational concerns, but they are not the same problem.

A slow test consistently takes longer than expected. A flaky test produces inconsistent results under the same or similar conditions.

For example:

A report-generation test consistently takes 12 seconds. It is slow.
A database test passes most of the time but occasionally fails because of shared state. It is flaky.
A browser test takes 30 seconds and sometimes times out. It may be both slow and flaky.

Flaky tests often depend on uncontrolled system state, test ordering, timing assumptions, or shared resources. Running tests in parallel can expose these problems.

Avoid using automatic retries as the only solution. Retries can hide the underlying failure and add more execution time. Track repeated failures, investigate the root cause, and use quarantine only as a temporary measure with an owner and a plan for resolution.

Set Performance Budgets for Test Suites

Once a team has reliable duration history, it can establish performance budgets.

A performance budget defines an acceptable execution-time boundary for a test suite, pipeline stage, or individual critical test.

For example:

Measurement	Example target
Unit test stage	Under 3 minutes
Integration test stage	Under 10 minutes
End-to-end test stage	Under 15 minutes
Individual critical test	Under 5 seconds

These are illustrative values. Real thresholds should be based on the application's current performance, CI infrastructure, and delivery requirements.

A useful policy can alert the team when:

Total test duration increases beyond an agreed percentage.
A test becomes significantly slower than its historical baseline.
The number of slow tests increases.
A pipeline stage exceeds its performance budget.
A test repeatedly times out.

Avoid failing every build because of a small timing fluctuation. A warning threshold can identify trends, while a stricter quality gate can be reserved for sustained or significant regressions.

Common Mistakes When Optimizing Slow Tests

Mistake	Why it creates problems	Better approach
Optimizing the total pipeline without identifying slow tests	Time may be spent on only a few expensive tests	Rank tests by duration first
Removing slow tests	Reduces coverage of important scenarios	Optimize or move them to an appropriate stage
Increasing parallel workers blindly	Can create resource contention and flaky failures	Measure performance at different worker counts
Using fixed sleep statements	Adds unnecessary waiting	Wait for specific conditions
Ignoring setup and teardown time	Misses expensive fixtures and cleanup operations	Measure every test phase
Retrying every failure	Hides instability and increases pipeline time	Investigate flaky behavior
Comparing different CI environments	Produces misleading performance trends	Compare equivalent runners and configurations

Conclusion

Slow tests should be identified before they become a regular source of CI delays. A pipeline that reports only pass or fail results provides limited information about the cost of individual tests.

Start by collecting test durations, ranking the slowest tests, and examining setup, execution, and teardown separately. Retain historical results so that performance regressions can be identified before they affect the entire development workflow.

Then optimize the actual bottleneck, whether it is repeated database setup, unnecessary waiting, external service calls, browser initialization, or resource contention.

The goal is not to make every test execute in milliseconds. It is to give developers reliable feedback as quickly as possible while preserving the test coverage needed to protect the application.