A failing test does not always mean that the application is broken.
Sometimes the test is exposing a real regression. Sometimes the test depends on timing, network availability, shared state, ordering, or another unstable condition.
The difficult part is telling these two situations apart.
A regression usually looks like this:
Code change
↓
Behavior changes
↓
Test fails consistently
A flaky test looks different:
Same code
↓
Same test
↓
Pass
Pass
Fail
Pass
Fail
That distinction matters because the engineering response is different.
A regression should block delivery and trigger investigation of the application change.
A flaky test should trigger investigation of the test or its environment.
Microsoft.Testing.Platform (MTP) provides capabilities that make it easier to build a more deterministic testing workflow, including test filtering, execution controls, diagnostics, retries through test-framework capabilities, and structured test execution. The important engineering task is to combine these features with repeatable experiments rather than simply rerunning a failing test until it passes.
What Is a Flaky Test?
A flaky test is a test whose result changes even though the tested code and test inputs have not meaningfully changed.
For example:
Run 1 → Pass
Run 2 → Pass
Run 3 → Fail
Run 4 → Pass
Run 5 → Fail
Common causes include:
Race conditions
Timing assumptions
Shared mutable state
Test ordering
Randomized input
External services
Network dependencies
Database state
File-system timing
Resource contention
Time-zone assumptions
Incorrect cleanup
A test that fails every time after a code change is much more likely to represent a regression.
The key word is repeatability.
Regression vs Flakiness
A useful first classification looks like this:
| Behavior | Likely interpretation |
|---|
| Always passes before change, always fails after change | Regression |
| Passes and fails without code changes | Flaky test |
| Fails only on one environment | Environment-specific issue |
| Fails only under parallel execution | Concurrency/shared-state issue |
| Fails only under load | Timing/resource issue |
| Fails after external service changes | Dependency issue |
This is not a proof.
It is an investigation starting point.
Build a Reproducible Experiment
Suppose a test fails in CI:
Test: CreateOrder_WhenInventoryIsAvailable
Result: Failed
Do not immediately modify the application.
First establish a baseline.
Run the test repeatedly:
Run 1 → Pass
Run 2 → Pass
Run 3 → Fail
Run 4 → Pass
Run 5 → Pass
Run 6 → Fail
Now you have evidence that the test may be nondeterministic.
The next question becomes:
What changes between the passing and failing executions?
Repeat the Same Test
A simple repeated execution strategy can expose flaky behavior.
For example:
dotnet test \
--filter "FullyQualifiedName~CreateOrder_WhenInventoryIsAvailable"
Run the same test repeatedly without changing the source code.
For a more systematic experiment, create a small shell or PowerShell loop:
1..20 | ForEach-Object {
Write-Host "Run $_"
dotnet test `
--filter "FullyQualifiedName~CreateOrder_WhenInventoryIsAvailable" `
--no-restore
if ($LASTEXITCODE -ne 0) {
Write-Host "Test failed on run $_"
}
}
The objective is to collect evidence rather than simply obtain a passing result.
Why One Successful Retry Is Not Enough
Suppose CI reports:
Failed
You rerun the pipeline:
Passed
That does not prove the original failure was a flake.
It could be:
First run → genuine regression
Second run → different code path
or:
First run → transient dependency failure
Second run → dependency recovered
or:
First run → race condition
Second run → race did not occur
A useful investigation needs multiple executions and additional evidence.
Add Failure Metadata
When a test fails intermittently, record information that can distinguish execution environments.
Useful fields include:
Test name
Commit SHA
Branch
OS
Architecture
.NET version
Test framework version
MTP version
Execution duration
Retry count
Environment
Timestamp
Random seed
External dependency status
For example:
Test:
CreateOrder_WhenInventoryIsAvailable
Commit:
a17c92e
Environment:
Linux x64
.NET:
Target runtime version
Duration:
2.8 s
Result:
Failed
This makes patterns easier to identify.
Look for Timing Correlation
Flaky tests frequently have timing signatures.
Imagine the following results:
| Run | Result | Duration |
|---|
| 1 | Pass | 120 ms |
| 2 | Pass | 118 ms |
| 3 | Fail | 2,004 ms |
| 4 | Pass | 121 ms |
| 5 | Fail | 2,011 ms |
The failures occurring near two seconds are a clue.
The test may be hitting:
Timeout
Retry delay
Polling interval
Lock contention
Network delay
This is why execution duration should be captured alongside the result.
Parallel Execution Can Expose Flakes
A test may pass individually:
dotnet test --filter "FullyQualifiedName~MyTest"
but fail when the entire suite executes.
That often points toward shared state.
For example:
Test A
↓
Writes shared database row
Test B
↓
Assumes row does not exist
When executed sequentially:
A → B
the behavior might be predictable.
Under parallel execution:
A ────────┐
├── Shared state
B ────────┘
the ordering can change.
This is a strong signal that the test suite needs isolation rather than retries.
Randomness Is Another Common Cause
Tests sometimes use random data:
var number = Random.Shared.Next();
That makes failures difficult to reproduce.
A better testing pattern is to make the seed explicit:
var random = new Random(12345);
Now a failure can be reproduced using the same input.
For property-based or randomized testing, capture the seed whenever a test fails.
The principle is:
Randomized execution
↓
Capture seed
↓
Failure
↓
Replay exact input
This converts a difficult intermittent failure into a deterministic test case.
External Dependencies Create Flaky Boundaries
Consider:
await httpClient.GetAsync(
"https://service.example/api/orders");
The test result now depends on something outside the test process.
Possible failure causes include:
DNS
Network
TLS
Remote service
Authentication
Rate limiting
Service deployment
Firewall
If the objective is to test your application logic, use a controlled dependency where appropriate.
For example:
Application
↓
Mock/Fake/Test Server
↓
Deterministic response
Integration tests should still validate real integrations, but those tests should be identified separately from deterministic unit tests.
Database State Is a Major Source of Flakiness
Tests that share database state can behave differently depending on execution order.
For example:
await db.Users.AddAsync(user);
await db.SaveChangesAsync();
If the test assumes that [email protected] does not already exist, another test can cause it to fail.
Better isolation strategies include:
The correct strategy depends on the test type and database architecture.
Use Test Filtering During Investigation
MTP supports filtering test execution so that developers can isolate a failing test or test group.
For example:
dotnet test \
--filter "FullyQualifiedName~PaymentTests"
This is useful when a large test suite contains thousands of tests.
A practical investigation sequence is:
Entire suite
↓
Failing test class
↓
Failing test
↓
Repeated execution
↓
Minimal reproduction
The smaller the reproduction, the easier it becomes to identify the cause.
Separate Detection From Mitigation
A retry can be useful, but it should not become the primary solution.
Consider:
Test fails
↓
Retry
↓
Pass
The pipeline may become green, but the underlying instability remains.
A better workflow is:
Test fails
↓
Record failure
↓
Retry for classification
↓
Determine whether failure is reproducible
↓
Investigate root cause
↓
Fix test or application
Retries are a diagnostic tool and temporary mitigation, not a substitute for fixing nondeterminism.
Use Retries Carefully
A retry policy can be useful for identifying intermittent failures.
For example:
Attempt 1 → Fail
Attempt 2 → Pass
Attempt 3 → Pass
This is evidence worth recording.
However, a retry should not silently transform:
Fail → Pass
into:
Success
without preserving the original failure information.
Otherwise, the test suite's reported health can become misleading.
Test for Order Dependence
One useful experiment is changing test execution order.
Suppose:
A → B → C → D
passes, but:
D → C → B → A
fails.
That strongly suggests hidden state or test coupling.
The goal of a well-designed test suite is to make tests independently executable whenever practical.
A test should ideally establish its own required state:
Setup
↓
Execute
↓
Assert
↓
Cleanup
rather than depending on:
Previous test
↓
Hidden state
↓
Current test
Use Diagnostics When the Failure Is Hard to Reproduce
Microsoft.Testing.Platform provides diagnostic options that can help investigate test execution behavior.
For example:
dotnet test -- --diagnostic
Additional diagnostic verbosity can also be configured when deeper execution information is required.
This becomes useful when a failure occurs only in CI:
Developer machine → Pass
CI runner → Fail
The goal is to compare the execution environment rather than repeatedly rerunning the same pipeline without collecting additional information.
Build a Flakiness Detection Pipeline
A useful CI workflow can separate normal test execution from stability analysis.
Pull Request
↓
Normal Test Suite
↓
Failures?
/ \
No Yes
| |
Pass Classify
↓
Re-run evidence
↓
Regression or Flake
For critical repositories, a separate scheduled stability job can repeatedly execute historically problematic tests.
For example:
Nightly
↓
Known unstable test set
↓
20 repeated executions
↓
Collect results
↓
Calculate failure rate
↓
Create investigation signal
This helps identify tests that are not consistently broken but are becoming less reliable.
Measure Flakiness Quantitatively
Instead of saying:
"This test is flaky."
record a measurable failure rate.
For example:
Executions: 100
Failures: 7
Failure rate: 7%
Now teams can track whether a fix actually improved reliability.
A simple metric is:
Flake Rate =
Intermittent Failures / Total Executions × 100
Track this over time:
| Period | Executions | Failures | Flake Rate |
|---|
| Week 1 | 500 | 18 | 3.6% |
| Week 2 | 500 | 11 | 2.2% |
| Week 3 | 500 | 3 | 0.6% |
This turns test reliability into an engineering metric.
A Practical Classification Model
A useful classification process is:
Step 1: Reproduce
Run the same test multiple times.
Step 2: Compare
Compare environment, duration, inputs, and logs.
Step 3: Isolate
Run the test independently from the full suite.
Step 4: Change Execution Conditions
Test sequential and parallel execution where relevant.
Step 5: Remove External Variables
Replace unnecessary external services with deterministic test doubles.
Step 6: Identify the Boundary
Determine whether the instability originates from:
Application
Test
Environment
Infrastructure
External dependency
Step 7: Fix the Cause
Do not stop at the first successful retry.
Common Mistakes
Treating Every Intermittent Failure as a Flake
A dependency failure or infrastructure outage can also produce intermittent results.
Investigate the evidence.
Increasing Timeouts Indefinitely
A longer timeout can hide a real performance regression.
Adding Retries Everywhere
Retries reduce visibility into failures if they are not reported correctly.
Ignoring Test Isolation
Shared state is one of the most common causes of nondeterministic behavior.
Using Uncontrolled Randomness
Always capture enough information to reproduce randomized failures.
Depending on Live External Services
Use deterministic dependencies when the purpose of the test is to validate application logic.
Best Practices
Treat flaky-test detection as an evidence-gathering process.
Repeat failed tests before classifying them.
Record commit, environment, runtime, duration, and inputs.
Capture random seeds for randomized tests.
Investigate test ordering and parallel execution.
Isolate database and file-system state.
Minimize unnecessary external dependencies.
Use retries for diagnosis, not as a permanent substitute for fixing the test.
Preserve the original failure even when a retry passes.
Measure flake rates instead of relying on anecdotal reports.
Use Microsoft.Testing.Platform filtering and diagnostics to narrow investigations.
Track recurring flaky tests as engineering debt until the underlying cause is fixed.
Frequently Asked Questions
How can I tell whether a failing test is flaky?
Run the same test repeatedly without changing the code or test inputs. If the result alternates between pass and fail, investigate nondeterminism. A consistent failure after a code change is more indicative of a regression.
Should flaky tests be retried automatically?
Retries can reduce the impact of transient failures, but they should not hide the original failure. Preserve retry information and investigate recurring instability.
Can parallel test execution cause flaky tests?
Yes. Tests that share databases, files, static state, ports, or other resources can interfere with each other when executed concurrently.
Should I disable parallel testing?
Not automatically. Disabling parallelism can hide shared-state problems and increase test duration. First determine whether the tests can be isolated correctly.
How can I reduce flaky integration tests?
Control external dependencies, isolate test data, clean up resources, avoid timing assumptions, use deterministic test environments, and make asynchronous operations cancellation-aware.
Conclusion
Flaky tests create a difficult problem because they reduce confidence without necessarily indicating an application regression.
The solution is not simply to rerun the test until it passes.
A stronger approach uses Microsoft.Testing.Platform's execution and diagnostic capabilities together with repeatable experiments:
Failure
↓
Repeat
↓
Collect evidence
↓
Isolate
↓
Classify
↓
Fix
↓
Verify
The distinction between a regression and a flaky test ultimately comes from evidence.
A regression should lead developers toward the changed application behavior.
A flaky test should lead them toward nondeterministic test logic, shared state, timing, dependencies, or infrastructure.
When teams measure flakiness, preserve failure evidence, and systematically isolate the cause, the test suite becomes more than a pass/fail gate. It becomes a reliable signal for software quality.