A flaky test is one of the most frustrating problems in a software delivery pipeline.
The test passes locally. It passes in CI five times. Then it fails once without an obvious code change.
Developers rerun the pipeline.
It passes.
The team moves on.
Over time, this creates a dangerous pattern:
Test fails
|
v
Rerun
|
v
Test passes
|
v
Ignore failure
|
v
Flaky test becomes normal
Eventually, developers stop trusting the test suite.
That is much more serious than a single unreliable test. Automated tests are supposed to provide confidence. If failures are routinely dismissed as "probably flaky," genuine regressions can be hidden among the noise.
The solution is not simply to rerun failed tests more often.
A better approach is to collect historical test results and analyze failure patterns over time.
This article explains how to identify flaky .NET tests using historical execution data, calculate useful metrics, build a failure history, and turn recurring patterns into actionable engineering work.
What Is a Flaky Test?
A flaky test is a test whose result is nondeterministic even when the relevant code and environment have not intentionally changed.
For example:
Run 1 -> Pass
Run 2 -> Pass
Run 3 -> Fail
Run 4 -> Pass
Run 5 -> Pass
The test is not consistently broken.
Its result depends on something that is not fully controlled.
Common causes include:
The important distinction is:
Consistently failing test
!=
Flaky test
A consistently failing test is usually easier to diagnose.
A flaky test requires historical evidence because individual executions can look perfectly normal.
Why Historical Analysis Matters
Suppose a test has failed twice.
That alone does not tell you much.
Now consider its history:
Last 100 executions
Pass: 94
Fail: 6
The failure rate is:
6 / 100 = 6%
That is meaningful.
Now compare another test:
Last 100 executions
Pass: 99
Fail: 1
The second test may still deserve investigation, but the operational priority is different.
Historical analysis lets teams prioritize based on evidence rather than anecdotes.
Start With a Test Result Dataset
A useful historical record should contain more than pass/fail.
For each execution, capture information such as:
| Field | Example |
|---|
| Test name | OrderServiceTests.CreateOrder_ShouldPersist |
| Project | OrderService.Tests |
| Result | Passed |
| Duration | 1.42 s |
| Timestamp | 2026-08-17T08:15:00Z |
| Commit | a82f31c |
| Branch | main |
| Build ID | 18452 |
| Agent | windows-runner-03 |
| Framework | net10.0 |
| Failure message | Timeout |
| Retry number | 0 |
The more context you retain, the easier it becomes to identify patterns.
For example, a test might appear flaky globally but actually fail only on:
Linux
+
Parallel execution
+
Specific test runner
That is a much more actionable finding.
Calculate a Basic Flake Rate
The simplest metric is the failure rate over a defined number of executions.
Flake Rate =
Number of unexpected failures
/
Total executions
For example:
Executions: 500
Failures: 15
Flake rate = 15 / 500
= 3%
Do not interpret every failure as a flaky failure automatically.
A test that consistently fails after a known code change is more likely to be a genuine regression.
Historical analysis should classify failures before assigning them to the flake-rate metric.
Track Failure Streaks
Failure rate alone can hide useful information.
Consider:
P P P F P P P P F P
versus:
P P P F F F P P P P
Both contain three failures if the window is larger, but the second pattern may indicate a persistent environmental or code problem.
Track:
Consecutive failures
Consecutive passes
Maximum failure streak
Maximum pass streak
A sudden failure streak often deserves more immediate investigation than isolated historical failures.
Track Failure Recency
Old failures and recent failures should not necessarily receive equal weight.
Consider:
100 executions
Failures:
- 90 runs ago
- 80 runs ago
- 75 runs ago
versus:
100 executions
Failures:
- 3 runs ago
- 5 runs ago
- 8 runs ago
The second pattern is much more relevant to the current state of the system.
A practical dashboard should therefore expose:
Last failure
Failures in last 10 runs
Failures in last 50 runs
Failures in last 100 runs
This provides both long-term and recent context.
Analyze Test Duration
Flakiness is not always represented by failure alone.
A test that normally takes:
200 ms
but occasionally takes:
20 seconds
may have a timing or resource problem even if it eventually passes.
Track duration statistics:
Average duration
Median duration
p95 duration
Maximum duration
Duration variance
For example:
| Metric | Value |
|---|
| Median | 210 ms |
| p95 | 340 ms |
| Maximum | 18.4 s |
That is a useful investigation signal.
The test may be sensitive to a slow database, thread scheduling, contention, or an external dependency.
Correlate Failures With Execution Environment
One of the strongest benefits of historical analysis is finding environmental correlations.
Suppose the history contains:
| Environment | Runs | Failures |
|---|
| Windows | 500 | 2 |
| Linux | 500 | 18 |
The overall failure rate hides the important detail.
Linux is clearly worth investigating.
Continue breaking the data down by:
Operating system
.NET version
CPU architecture
Runner
Database version
Container runtime
Browser version
Test category
Parallelism
The relevant dimensions depend on the test.
Analyze Parallel Execution
A common source of flaky tests is shared state.
For example:
public class OrderTests
{
[Fact]
public async Task CreatesOrder()
{
await database.ClearAsync();
// Test...
}
}
If another test is simultaneously modifying the same database, the result may depend on execution order.
The test may pass in isolation:
Test A
|
v
Pass
but fail when the complete suite runs:
Test A ----+
|
Test B ----+----> Shared database
|
Test C ----+
Historical data can reveal whether failures increase when test parallelism is enabled.
Capture Failure Categories
Do not store only the complete exception message.
Classify failures where possible.
For example:
Timeout
Assertion
Connection
File system
HTTP
Database
Concurrency
Serialization
Environment
Unknown
A simple classification makes trends easier to identify.
Suppose a test fails 20 times:
Timeout: 15
Database: 3
Assertion: 1
Unknown: 1
That strongly suggests investigating timeout and database behavior first.
Normalize Failure Messages
Raw exception messages often contain dynamic values.
For example:
Expected 42 but received 41
and:
Expected 43 but received 42
may represent the same underlying failure category.
Likewise:
Timeout after 5000 ms connecting to 10.0.1.20
and:
Timeout after 5000 ms connecting to 10.0.1.21
may represent the same network problem.
Normalize dynamic values before grouping failures.
The goal is to identify recurring failure signatures.
Example Historical Storage Model
A simple relational table could contain:
CREATE TABLE TestExecutionHistory
(
Id BIGINT PRIMARY KEY,
TestName NVARCHAR(500) NOT NULL,
ProjectName NVARCHAR(200) NOT NULL,
Result NVARCHAR(20) NOT NULL,
DurationMs BIGINT NOT NULL,
ExecutedAtUtc DATETIME2 NOT NULL,
CommitSha NVARCHAR(64) NULL,
BranchName NVARCHAR(200) NULL,
EnvironmentName NVARCHAR(100) NULL,
Framework NVARCHAR(50) NULL,
FailureCategory NVARCHAR(100) NULL,
FailureSignature NVARCHAR(500) NULL
);
This is intentionally simple.
A production implementation can add:
Build identifiers
Test suite
Agent information
Retry count
Pull request ID
Container image
Database version
Runtime version
The schema should reflect the questions the engineering team actually needs to answer.
Query Tests With High Failure Rates
Once historical data exists, start with simple queries.
For example:
SELECT
TestName,
COUNT(*) AS TotalRuns,
SUM(
CASE
WHEN Result = 'Failed'
THEN 1
ELSE 0
END
) AS Failures
FROM TestExecutionHistory
GROUP BY TestName
HAVING COUNT(*) >= 30;
You can calculate the failure rate in application code or in SQL depending on the database.
The important point is to avoid ranking tests based on only one or two executions.
A test with:
1 failure / 2 runs
should not automatically outrank:
20 failures / 1000 runs
The second test has much stronger historical evidence.
Add a Minimum Sample Size
A useful rule is to require a minimum number of executions before classifying a test as flaky.
For example:
Minimum history = 30 executions
The exact threshold is a policy decision.
The purpose is statistical sanity.
Without a minimum sample size, a newly added test can immediately appear highly flaky after one failure.
Build a Flakiness Score
A single failure rate is useful but incomplete.
A more practical score can combine:
Failure frequency
+
Recent failures
+
Failure streak
+
Execution volume
+
Duration anomalies
For example:
Flakiness Score =
Weighted Failure Rate
+ Recent Failure Weight
+ Duration Anomaly Weight
The exact formula should be validated against actual engineering outcomes.
Do not pretend that a score is mathematically objective if its weights are arbitrary.
The score is primarily a prioritization mechanism.
Detect Tests That Pass Only After Retry
Retries are particularly useful historical signals.
Consider:
Attempt 1 -> Fail
Attempt 2 -> Pass
If this happens frequently, the test deserves investigation.
Track:
Initial failures
Recovered on retry
Failed after all retries
For example:
| Test | Runs | First Attempt Failures | Retry Recoveries |
|---|
| Test A | 500 | 18 | 16 |
| Test B | 500 | 4 | 1 |
Test A is a much stronger flakiness candidate.
Retries should not be used to hide this information.
Do Not Use Retries as the Fix
A retry can be useful operationally:
Test fails
|
v
Retry
|
v
Pipeline continues
But this does not make the test reliable.
If the test fails 5% of the time and retries make the pipeline green 99% of the time, the underlying reliability problem still exists.
The correct interpretation is:
Retry = mitigation
Not = root-cause fix
Track the initial failure even when the retry passes.
Identify Time-Based Failures
Some flaky tests correlate with specific times.
For example:
Failures:
09:00-10:00 -> 1
10:00-11:00 -> 2
12:00-13:00 -> 17
That may indicate:
Time-based analysis is especially useful for tests that interact with shared environments.
Identify Commit Correlations
A test may be stable for months and become flaky after a particular change.
For example:
Commit A
100 runs -> 0 failures
Commit B
100 runs -> 12 failures
That is a valuable signal.
The test may have been affected by:
Historical test data should therefore be correlated with commit history.
Detect Test Order Dependencies
Order-dependent tests are particularly difficult to diagnose.
Imagine:
Test A -> Pass
Test B -> Pass
Test C -> Pass
but:
Test C
Test A
Test B
causes Test B to fail.
Run-history data should capture enough information to reconstruct execution order when parallelism is relevant.
If possible, record:
Test group
Worker
Execution sequence
Parallel worker ID
This can turn a seemingly random failure into a reproducible ordering problem.
Separate Product Failures From Test Failures
Not every intermittent failure is a flaky test.
Suppose an integration test calls an external service.
If the service returns:
HTTP 503
the test may be correctly reporting a real dependency outage.
Historical analysis should therefore distinguish:
Application defect
Test defect
Environment defect
External dependency
Infrastructure failure
This classification prevents teams from rewriting good tests to accommodate unreliable infrastructure.
Build a Flaky Test Dashboard
A useful dashboard can show:
Top Flaky Tests
Test Name Flake Rate
------------------------------------------------
OrderServiceTests.CreateOrder 5.2%
PaymentTests.TimeoutHandling 3.8%
CacheTests.ConcurrentRefresh 2.9%
Recent Flaky Tests
Test Failures / Last 20
--------------------------------------------
CreateOrder 4
RefreshCache 3
ExportReport 2
Environment Correlation
Environment Failure Rate
--------------------------------
Windows 0.4%
Linux 3.7%
This provides a much more useful view than a simple "build passed" indicator.
A Practical CI Workflow
A robust workflow can look like:
Run test suite
|
v
Collect results
|
v
Store execution history
|
v
Classify failures
|
v
Calculate flakiness metrics
|
v
Identify recurring patterns
|
v
Create engineering work
The important part is the last step.
Historical analysis should result in action.
Examples include:
Fix race condition
Isolate test data
Remove external dependency
Improve synchronization
Fix time handling
Correct test cleanup
Replace unstable environment
Common Mistakes
Looking Only at Current CI Results
One build rarely provides enough evidence.
Treating Every Failure as Flaky
A genuine regression can be mislabeled as flaky if failures are not investigated.
Ignoring Retry Results
A retry that passes is often one of the strongest flakiness signals.
Using Only Average Failure Rate
Recent failures, streaks, environment, and duration patterns matter.
Keeping Only Test Names
Without commit, environment, framework, and timing context, historical analysis becomes much less useful.
Automatically Quarantining Every Flaky Test
Quarantine can reduce pipeline noise, but it can also allow important failures to disappear from normal development.
Use it deliberately.
Deleting Historical Data Too Quickly
A short history makes it difficult to determine whether reliability is improving or getting worse.
Troubleshooting a Suspected Flaky Test
When a test is suspected of being flaky, start with its recent history.
Check:
1. Last 100 executions
2. Failure rate
3. Last failure
4. Retry recovery rate
5. Failure signatures
6. Execution duration
7. Environment
8. .NET version
9. Parallelism
10. Related commits
Then reproduce it deliberately.
For example:
dotnet test \
--filter "FullyQualifiedName~CreateOrder" \
--no-restore
Run the test repeatedly under the same conditions.
If it only fails under parallel execution, test the parallelism hypothesis.
If it only fails on one operating system, investigate environment-specific behavior.
Historical data should narrow the search space.
It should not replace actual debugging.
A Practical Flaky-Test Classification
After analysis, classify tests into categories such as:
| Classification | Meaning | Action |
|---|
| Stable | No meaningful intermittent failures | Keep |
| Suspected flaky | Limited evidence | Monitor |
| Flaky | Repeated intermittent failures | Investigate |
| Environment-dependent | Failures correlate with environment | Fix environment/test |
| External dependency | Failure depends on external service | Isolate or control dependency |
| Consistently failing | Reproducible failure | Treat as defect |
This prevents the word "flaky" from becoming a generic label for every difficult test.
Frequently Asked Questions
How many runs are needed to identify a flaky test?
There is no universal number. A reasonable minimum history should be large enough to distinguish a recurring pattern from a one-off failure. Many teams can start with dozens of executions and increase the window for low-frequency tests.
Should failed tests automatically be retried?
Retries can reduce pipeline disruption, but they should never erase the original failure from historical analysis. A test that fails and then passes on retry is an important flakiness signal.
Should flaky tests be removed from CI?
Usually not as a first response. Removing a test eliminates its signal rather than fixing the underlying reliability problem. Temporary quarantine can be appropriate when a test seriously blocks delivery, but it should have ownership and follow-up.
Can historical analysis find race conditions?
It can provide strong clues. Correlations with parallel execution, test ordering, timing, CPU load, or specific environments can point toward concurrency problems, but the race condition still needs to be reproduced and diagnosed.
Should test duration be tracked?
Yes. Intermittent latency spikes can reveal resource contention, timeouts, synchronization issues, or unstable dependencies even when the test ultimately passes.
Is a low flake rate acceptable?
It depends on the test's importance and the cost of failure. A 1% failure rate can still be unacceptable for a test executed thousands of times in CI if it creates significant developer disruption.
Conclusion
Flaky tests become manageable when teams stop treating them as isolated CI annoyances and start treating test execution as historical data. A single failure rarely explains why a test is unstable, but a history of hundreds of executions can reveal failure frequency, retry recovery, timing anomalies, environment correlations, execution-order dependencies, and relationships with recent code changes.
For .NET teams, the practical goal is not to create a complicated analytics platform. Start by storing test results with enough context to answer basic questions: when did the test fail, how often does it fail, does it pass on retry, where does it fail, and what changed around the failure? From there, use historical evidence to prioritize investigations and fix the underlying reliability problem. A trustworthy test suite is not one that never fails; it is one where failures have a predictable meaning and intermittent failures are actively identified and removed.