Handling Transient Failures in .NET 8 With Polly

In this article, we can learn how to implement a retry mechanism when interacting with a service and the service is experiencing some Transient Faults.

Transient faults relate to fault occurrences that exist for shorter periods.

For example. A network connection is down due to a router reboot. A service is down due to deployment starting up or a Refusing connection due to exhaustion of resources.

In the case of Transient failures, the failure is for a short period, and the service will come back again. So, to make sure of accepting failure, it's better to retry for n number of times and then accept failure.

We can Handle Transient Failures by a retry Policy, which means trying the request again and again to see if it is successful this time.

Retry Policy can be configured using the options below.

Number of Retires
Time Interval between retries.

We shall cover three retry policies in this article

Policy 1. Retry Immediately 5 times

As per this policy, the request retries 5 times until it gets a successful Response. After the 5th request, if it still gets a failure, it accepts the failure.

Request retries

Policy 2. Retry 5 times and wait 3s

As per this policy, the request Service waits for 3s before it makes another request to the response service.

Response service

Policy 3. Retry 5 times with Exponential Back Off

As per this policy, the request service retires 5 times with exponential wait time between the requests like 1s, 3s, 5s, 8s.

Exponential Back Off

The retry mechanism can be achieved by using Polly, and we implement the retry mechanism with a class-based configuration. Lets code

Let us create a new dotnet web api application and name it Response Service.

Map the Controllers and add Controllers within the Program.cs

builder.Services.AddSwaggerGen();
builder.Services.AddControllers();
var app = builder.Build();
app.MapControllers();

Let's create a ResponseController.cs file and add an action method.

[Route("api/[Controller]")]
[ApiController]
public class ResponseController : ControllerBase
{
    [HttpGet]
    [Route("{id:int}")]
    public ActionResult GetAResponse(int id)
    {
        Random rnd = new Random();
        var rndInteger = rnd.Next(1, 101);
        
        if (rndInteger >= id)
        {
            Console.WriteLine("Failure - Generate an Internal Error");
            return StatusCode(StatusCodes.Status500InternalServerError);
        }

        Console.WriteLine("Failure - Generated a Success");
        return Ok("Success");
    }
}

As you can see in the code, we implemented the transient failure within our service by using a Random function. If the randomly generated integer is less than the input ID, there is a chance of returning an internal server error.

Let's run the code and check it via Postman. Based on the random integer generated, the response from the response service varies as 200 or 500 status codes.

GET

We are now ready with the Response Service. Let's Create a Request Service.

Create a new dotnet web API application and name it ResponseService. As in the ResponseService Program.cs similarly adds Controllers to the pipeline.

Let us create a RequestController.cs, which will have basic logic to call the api using HttpClient. Add the code below.

namespace RequestService.Controllers
{
    [ApiController]    
    [Route("api/[Controller]")]
    public class RequestController: ControllerBase
    {
        public RequestController()
        {
        }
        [HttpGet]    
        public async Task<ActionResult> MakeRequest()
        {
           var client = new HttpClient();        
            var response = await client.GetAsync("http://localhost:5202/api/response/25");
             var response = await _clientPolicy.LinearHttpRetry.ExecuteAsync( () => 
             client.GetAsync("http://localhost:5202/api/response/25")
             );
            if(response.IsSuccessStatusCode)
            {
               Console.WriteLine("--> Response Service Retuned Success");
               return Ok();
            }
            Console.WriteLine("--> Response Service Retuned Failure");
            return StatusCode(StatusCodes.Status500InternalServerError); 
        }     
    }
}

We can now run the request service and validate it via Postman. We can see that we will get a failure message from the Response Service, but we haven't implemented the retry mechanism yet. Let's Implement it

Now, use the dotnet cli run command to add the Polly nuget package to the Request Service.

dotnet add package Microsoft.Extensions.Http.Polly

Create a folder named Policies, add the ClientPolicy class file, and code it as below. We have referenced the Polly in the ClientPolicy.cs file.

using Polly;
using Polly.Retry;
namespace RequestService.Policies
{
    public class ClientPolicy 
    {
        public AsyncRetryPolicy<HttpResponseMessage> ImmediateHttpRetry {get;}
        public AsyncRetryPolicy<HttpResponseMessage> LinearHttpRetry  {get;}
        public AsyncRetryPolicy<HttpResponseMessage> ExponentialHttpRetry  {get;}
        public ClientPolicy()
        {
            ImmediateHttpRetry = Policy.HandleResult<HttpResponseMessage>
            ( res => !res.IsSuccessStatusCode).RetryAsync(5);
            LinearHttpRetry = Policy.HandleResult<HttpResponseMessage>
            ( res => !res.IsSuccessStatusCode).
            WaitAndRetryAsync(5, retryAttempt => TimeSpan.FromSeconds(3));
            ExponentialHttpRetry = Policy.HandleResult<HttpResponseMessage>
            ( res => !res.IsSuccessStatusCode).
            WaitAndRetryAsync(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2,retryAttempt)));
        }
    }
}

As you can see in the code, we have implemented three retry policies in the same file. Within the Constructor of this class, we have initialized the different retry policies.

LinearHttpRetry = Policy.HandleResult<HttpResponseMessage>
       ( res => !res.IsSuccessStatusCode).
       WaitAndRetryAsync(5, retryAttempt => TimeSpan.FromSeconds(3));

If we see the LinearHttpRetry, We have used Policy.HandleResult, and then if it is not a SuccessStatusCode, we mentioned the WaitAndRetryAsync method 5 times and with a time span of 3 seconds.

Similarly, for ExponentialHttpRetry, we are using two power of retry attempts as timespan between the requests.

Let's instantiate the clientPolicy within our Requestcontroller by dependency injection and configuring it in the Program.cs

builder.Services.AddSingleton<ClientPolicy> (new ClientPolicy());

Let's Change the RequestController to use ClientPolicy. Change your HttpClient Code as below. As you can see within ClientPolicy.LinearHttpRetry.ExecuteAsync method, we have added our HTTP client logic.

private readonly ClientPolicy _clientPolicy;
    private readonly IHttpClientFactory _clientFactory;
    public RequestController(ClientPolicy clientPolicy,IHttpClientFactory clientFactory)
   {
    _clientPolicy = clientPolicy;
    _clientFactory = clientFactory;
} 
[HttpGet]    
public async Task<ActionResult> MakeRequestNormalHttpClient() {
    var client = new HttpClient();
    //var response = await client.GetAsync("http://localhost:5202/api/response/25");
    var response = await _clientPolicy.LinearHttpRetry.ExecuteAsync( () => 
    client.GetAsync("http://localhost:5202/api/response/25")
    );
    if(response.IsSuccessStatusCode)
    {
        Console.WriteLine("--> Response Service Retuned Success");
        return Ok();
    }
    Console.WriteLine("--> Response Service Retuned Failure");
        return StatusCode(StatusCodes.Status500InternalServerError); 
}

Instead of injecting the ClientPolicy within the Controller, let us add the policy when we create a named http client within the Program.cs Let us change the code below.

builder.Services.AddHttpClient("Test")
    .AddPolicyHandler(request => 
        request.Method == HttpMethod.Get ? 
        new ClientPolicy().LinearHttpRetry : 
        new ClientPolicy().LinearHttpRetry
    );
public async Task<ActionResult> MakeRequest()
{
    var client = _clientFactory.CreateClient("Test");  
    var response = await client.GetAsync("http://localhost:5202/api/response/25");    
    if (response.IsSuccessStatusCode)
    {
        Console.WriteLine("--> Response Service Returned Success");
        return Ok();
    }   
    Console.WriteLine("--> Response Service Returned Failure");
    return StatusCode(StatusCodes.Status500InternalServerError);
}

Since we have configured policy at Program.cs for a Named Http Client, we can directly CreateClient using IHttpClientFactory and the policy is enabled on it. Let's run the code and test the LinearHttpRetry in Postman.

We succeeded within Postman with a linear wait.

Postman

As we can see in the response service debugging, we can see there were four failures before we got a successful response.

Service debug

We just implemented the Retry Policy with Polly in this article. We also have other patterns like Circuit Breaker with Polly.

That's it from this article. Please comment with any questions.