A test that passes on one run and fails on another without any relevant code change is one of the most frustrating problems in software development. These tests are commonly called flaky tests.
Flaky tests are especially harmful in CI/CD because they reduce confidence in automated validation. Developers may start rerunning failed pipelines instead of investigating the actual problem.
Microsoft.Testing.Platform (MTP) provides a modern execution platform for .NET tests and supports features that can help teams build more reliable test execution workflows. With the right configuration and CI practices, teams can identify repeated failures, collect useful diagnostics, and distinguish genuine defects from unstable tests.
For .NET teams, the important goal is not simply to rerun a failed test. It is to detect, classify, and eventually eliminate the underlying source of flakiness.
What Is a Flaky Test?
A flaky test is a test whose result is nondeterministic under apparently equivalent conditions.
For example:
Run 1 → PASS
Run 2 → PASS
Run 3 → FAIL
Run 4 → PASS
Run 5 → FAIL
The application code may not have changed between these executions.
Common causes include:
Race conditions
Timing assumptions
Shared test state
External services
Network dependencies
Database state
Parallel execution
File-system dependencies
Time-zone assumptions
Random data
Insufficient cleanup
A flaky test is different from a consistently failing test.
Consistent failure:
PASS → FAIL after code change → FAIL → FAIL
Flaky failure:
PASS → FAIL → PASS → PASS → FAIL
The second pattern requires a different investigation strategy.
What Is Microsoft.Testing.Platform?
Microsoft.Testing.Platform is a lightweight test execution platform designed for .NET testing scenarios.
It provides a common foundation for running tests and supports extensibility through test platform capabilities.
A typical .NET test project might look like:
MyApplication.Tests
|
+-- UnitTests
+-- IntegrationTests
+-- Test configuration
+-- Test project file
The test platform executes the tests and produces results that can be consumed by local development tools and CI/CD systems.
For teams migrating from older test execution approaches, the important consideration is that MTP is a platform rather than a testing framework itself. Your test framework, such as MSTest, NUnit, or xUnit, still determines how tests are written.
Why Flaky Tests Are Dangerous in CI/CD
Consider a pipeline:
Commit
↓
Build
↓
Unit Tests
↓
Integration Tests
↓
Deploy
If an integration test fails intermittently, developers may repeatedly execute:
dotnet test
until it passes.
This creates a dangerous habit.
The team may eventually stop treating failures as meaningful signals.
A better workflow is:
Test Failure
↓
Determine whether failure is deterministic
↓
Collect diagnostics
↓
Classify flaky behavior
↓
Fix root cause
↓
Track stability
The goal is to preserve trust in the pipeline.
Running Tests with .NET
A standard test command remains:
dotnet test
For CI environments, you can use:
dotnet test --configuration Release
If you need additional diagnostics:
dotnet test \
--configuration Release \
--logger "console;verbosity=detailed"
The exact options available depend on the test SDK, test framework, and Microsoft.Testing.Platform configuration used by the project.
The important point is to configure diagnostic output intentionally rather than generating excessively large logs for every successful build.
Detecting Intermittent Failures
A single failure does not prove that a test is flaky.
A practical approach is to record test outcomes over multiple runs.
For example:
| Test | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Classification |
|---|
| CreateOrder_ShouldSucceed | Pass | Pass | Pass | Pass | Pass | Stable |
| UpdateOrder_ShouldPersist | Pass | Fail | Pass | Pass | Fail | Suspected flaky |
| DeleteOrder_ShouldFailForMissingId | Fail | Fail | Fail | Fail | Fail | Consistent failure |
The exact number of repetitions should depend on the cost of the test suite.
The important thing is to identify patterns rather than automatically labeling every failed test as flaky.
Using Retry Carefully
Retries can help confirm intermittent failures, but they should not become a permanent solution.
For example:
Test fails
↓
Retry
↓
Pass
The result should be treated as:
Potential flaky test
not:
Test is healthy
A retry can reduce false pipeline failures, but excessive retries can hide real defects.
If a test requires repeated retries to become green, the team should investigate why.
Example of a Timing-Dependent Test
Consider this test:
[Fact]
public async Task Order_ShouldBecomeAvailable()
{
await service.CreateOrderAsync();
await Task.Delay(100);
var order =
await service.GetOrderAsync();
Assert.NotNull(order);
}
This test assumes that 100 milliseconds is enough for the system to process the operation.
That assumption may work on a developer machine but fail under CI load.
A better approach is to wait for the actual condition.
For example:
[Fact]
public async Task Order_ShouldBecomeAvailable()
{
await service.CreateOrderAsync();
var order = await WaitUntilAsync(
async () => await service.GetOrderAsync(),
result => result is not null,
TimeSpan.FromSeconds(5));
Assert.NotNull(order);
}
The implementation of WaitUntilAsync should include a bounded timeout and appropriate polling interval.
The principle is more important than the helper itself:
wait for a state, not an arbitrary amount of time.
Database-Related Flakiness
Integration tests frequently become flaky because they depend on shared database state.
For example:
[Fact]
public async Task ShouldCreateCustomer()
{
await repository.CreateAsync(
new Customer
{
Email = "[email protected]"
});
var customer =
await repository.GetByEmailAsync(
"[email protected]");
Assert.NotNull(customer);
}
If another test uses the same email, the result can depend on execution order.
A better design isolates test data:
var email =
$"test-{Guid.NewGuid():N}@example.com";
Alternatively, tests can use transactional isolation, dedicated databases, containers, or controlled fixtures depending on the test architecture.
Parallel Execution Problems
Parallel execution can expose hidden dependencies.
Suppose two tests modify a shared file:
Test A → writes config.json
Test B → deletes config.json
Running independently may succeed:
Test A → PASS
Test B → PASS
Running simultaneously can produce:
Test A + Test B
↓
Race condition
↓
Intermittent failure
The correct fix is usually to remove the shared mutable state rather than simply disabling parallel execution for the entire test suite.
Time and Date Flakiness
Tests that depend on the current system time are another common source of instability.
Avoid logic like:
var expected =
DateTime.UtcNow.AddMinutes(5);
when the test compares exact timestamps.
Instead, inject a clock abstraction:
public interface IClock
{
DateTimeOffset UtcNow { get; }
}
Production code can use the real clock while tests use a deterministic implementation.
This makes time-dependent behavior reproducible.
External Service Dependencies
A test that calls a real third-party service can fail because of:
For unit tests, prefer deterministic mocks or fakes where appropriate.
For integration tests that intentionally verify external integrations, isolate them from the main fast feedback pipeline when the dependency cannot provide deterministic availability.
Capturing Useful Diagnostics
When a flaky test fails, the diagnostic information surrounding the failure is often more valuable than the test result itself.
Useful information includes:
A CI pipeline should preserve failed-test artifacts when practical.
For example:
- name: Run tests
run: dotnet test --configuration Release
- name: Upload test artifacts
if: failure()
uses: actions/upload-artifact@v4
with:
name: test-results
path: TestResults/
The exact CI configuration depends on the platform. The principle is to retain enough information to investigate the failure without immediately reproducing the entire pipeline.
Building a Flaky-Test Detection Workflow
A practical workflow can be:
Step 1: Record Failures
Store test results from every CI execution.
Step 2: Identify Repeated Failures
Look for tests that alternate between pass and fail without corresponding code changes.
Step 3: Classify the Cause
Group suspected failures into categories:
Timing
Concurrency
External dependency
Shared state
Environment
Randomness
Resource exhaustion
Step 4: Reproduce
Run the test repeatedly in an environment similar to CI.
Step 5: Fix the Root Cause
Remove timing assumptions, isolate state, control dependencies, or correct synchronization.
Step 6: Monitor After the Fix
Continue tracking the test after the change.
A test that has been stable for several runs is encouraging, but long-term monitoring is more useful than declaring it fixed after one successful execution.
Common Mistakes
Blindly Retrying Failed Tests
Retries can hide real defects.
Using Thread.Sleep
Fixed delays often create unreliable tests.
Prefer condition-based waiting with a timeout.
Sharing Mutable Test Data
Tests should avoid depending on data created or modified by another test.
Disabling Parallel Tests Everywhere
This can mask concurrency problems and significantly increase execution time.
Ignoring Test Duration
Tests that become unusually slow may be showing resource contention or environmental problems even when they eventually pass.
Labeling Every Failure as Flaky
A consistently failing test is usually a defect, not a flaky test.
Troubleshooting Checklist
When a test fails intermittently, check:
Does it depend on current time?
Does it use Task.Delay or fixed sleeps?
Does it share files, databases, ports, or global state?
Does it depend on test execution order?
Does it run in parallel with another test?
Does it access a network service?
Does it use random data?
Does the CI environment have different CPU or memory characteristics?
Does the failure correlate with a particular machine or runner?
Are logs and test artifacts available?
This checklist often narrows the problem faster than simply rerunning the entire pipeline.
Best Practices
Keep unit tests deterministic and isolated.
Avoid fixed delays when testing asynchronous behavior.
Inject clocks for time-dependent code.
Generate isolated test data.
Control external dependencies.
Keep integration tests separate from unit tests.
Collect diagnostic artifacts for failed runs.
Use retries only as a temporary safety mechanism.
Track suspected flaky tests over time.
Fix the root cause instead of normalizing intermittent failures.
Advantages and Disadvantages
Advantages
Provides a modern test execution foundation for .NET.
Supports integration with automated development workflows.
Helps teams standardize test execution.
Can be combined with CI diagnostics and test-result collection.
Makes it possible to build systematic flaky-test detection workflows.
Disadvantages
MTP does not automatically eliminate flaky tests.
Diagnosing nondeterministic failures can still be difficult.
CI retry mechanisms can hide underlying problems.
Integration tests remain sensitive to external dependencies.
Test stability still depends heavily on test design.
Conclusion
Microsoft.Testing.Platform provides a modern foundation for executing .NET tests, but flaky-test detection is ultimately a testing and engineering discipline rather than a feature that can simply be switched on.
The most reliable approach is to combine test execution with historical results, diagnostics, controlled retries, and root-cause analysis.
When a test behaves like:
PASS → FAIL → PASS → FAIL
do not treat the next successful run as proof that everything is fine.
Investigate the timing, state, dependencies, concurrency, and environment surrounding the failure.
A trustworthy CI/CD pipeline depends on tests that developers believe. Microsoft.Testing.Platform can support that workflow, but the final responsibility for deterministic and maintainable tests remains with the development team.
Join the conversation! Your thoughts help the community grow.