Back to: Microservices using ASP.NET Core Web API Tutorials
Implementing Circuit Breaker in ASP.NET Core Web API Microservices
In this Post, I will discuss Implementing Circuit Breaker in ASP.NET Core Web API Microservices. A circuit breaker opens (trips) when it determines that the downstream service is unhealthy. The goal is simple: stop wasting time on calls that are likely to fail/timeout, and protect your service (threads, connections, memory). It usually does not open because of a single small error—it opens because it detects a pattern of failures over a short period.
Real-time Scenario in our ECommerce Microservices
Implementing Circuit Breaker in OrderService (REST + gRPC) Using .NET 8. In a microservices system, OrderService depends on other services to complete a single feature:
- OrderService calls PaymentService (REST) to initiate payment.
- OrderService calls NotificationService (REST) to send a notification.
- OrderService calls UserService (gRPC) to fetch user details.
- OrderService calls ProductService (gRPC) to validate products and stock.
Now imagine this real situation:
- PaymentService is down or slow.
- If OrderService keeps calling PaymentService, requests will keep waiting (with timeouts), threads will block, and soon OrderService will become slow too.
Circuit Breaker prevents this cascading failure by doing this:
- It watches recent calls.
- If too many calls fail or time out in a short window, it opens the circuit.
- When open, it stops calling the dependency for a short time and fails fast.
Step 1: Add the Package in OrderService.Infrastructure
First, we need to install Microsoft’s official resilience package Microsoft.Extensions.Http.Resilience for HttpClient. So, please execute the following command in Package Manager Console, and also please select OrderService.Infrastructure layer project while executing the command.
- Install-Package Microsoft.Extensions.Http.Resilience -Version 8.0.0
Step 2: Creating Resilience Folder
First, create a folder named Resilience at the root directory of the OrderService.Infrastructure layer project. This folder serves as a single place to keep all Resilience/Circuit-Breaker-related code. Instead of scattering circuit breaker logic across multiple files, we keep it centralized. This makes our codebase clean, maintainable, and easy to modify later.
- Keeps all resilience logic in one place
- Prevents InfrastructureServiceRegistration from becoming messy
- Helps us tune settings and change policies without touching business code
So, this folder will contain all:
- Circuit Breaker Options
- REST Resilience Registration
- gRPC Resilience Registration
- gRPC Interceptor
Step 3: Add Resilience Settings to appsettings.json (OrderService API project)
The following settings hold our circuit breaker and timeout values. In real applications, these values are Tuning Parameters, meaning we might change them based on production traffic and behaviour. So, it’s best to keep them in configuration rather than hardcoding them.
In the OrderService API project (not Infrastructure), open appsettings.json and add the following settings. This will be read at runtime by Othe ptions Pattern.
"Resilience": {
"Rest": {
"Payment": {
"TimeoutSeconds": 2,
"SamplingDurationSeconds": 10,
"FailureRatio": 0.5,
"MinimumThroughput": 10,
"BreakDurationSeconds": 15
},
"Notification": {
"TimeoutSeconds": 1,
"SamplingDurationSeconds": 10,
"FailureRatio": 0.7,
"MinimumThroughput": 10,
"BreakDurationSeconds": 10
}
},
"Grpc": {
"Default": {
"TimeoutSeconds": 2,
"SamplingDurationSeconds": 10,
"FailureRatio": 0.5,
"MinimumThroughput": 10,
"BreakDurationSeconds": 15
}
}
}
Step 4: Create Options Classes
This file contains the Strongly-Typed Classes that match our appsettings.json resilience section. The main job of this file is to help ASP.NET Core read configuration values and store them in C# objects, so our code can access them safely and cleanly.
Create a class file named ResilienceOptions.cs within the OrderService.Infrastructure/Resilience folder, and copy-paste the following code.
namespace OrderService.Infrastructure.Resilience
{
// Root options class for ALL resilience settings.
// This class binds to the "Resilience" section in appsettings.json.
//
// Example:
// "Resilience": {
// "Rest": { ... },
// "Grpc": { ... }
// }
public sealed class ResilienceOptions
{
// Settings for REST (HttpClient) based services like Payment and Notification
public RestResilienceOptions Rest { get; set; } = new();
// Settings for gRPC based services like ProductService and UserService
public GrpcResilienceOptions Grpc { get; set; } = new();
}
// Represents: "Resilience:Rest" section in appsettings.json
// Example:
// "Resilience": {
// "Rest": {
// "Payment": { ... },
// "Notification": { ... }
// }
// }
//
// Why separate RestResilienceOptions?
// - REST calls typically fail through HTTP status codes (500, 408, 429)
// - We apply HttpClient resilience policies using these values
public sealed class RestResilienceOptions
{
// Policy settings used for PaymentService REST calls
// Example path: Resilience:Rest:Payment
public CircuitBreakerPolicyOptions Payment { get; set; } = new();
// Policy settings used for NotificationService REST calls
// Example path: Resilience:Rest:Notification
public CircuitBreakerPolicyOptions Notification { get; set; } = new();
}
// Represents: "Resilience:Grpc" section in appsettings.json
// Example:
// "Resilience": {
// "Grpc": {
// "Default": { ... }
// }
// }
//
// Why separate GrpcResilienceOptions?
// - gRPC failures usually happen as RpcException (Unavailable, DeadlineExceeded, etc.)
// - We apply resilience using a Polly pipeline + interceptor using these settings
public sealed class GrpcResilienceOptions
{
// Default policy for all gRPC clients (UserService, ProductService, etc.)
// Example path: Resilience:Grpc:Default
//
// Note:
// - If later you want different rules per service,
// you can add properties like:
// public CircuitBreakerPolicyOptions UserService { get; set; }
// public CircuitBreakerPolicyOptions ProductService { get; set; }
public CircuitBreakerPolicyOptions Default { get; set; } = new();
}
// Common circuit breaker + timeout settings shared by both REST and gRPC
// This class is reused in multiple places:
// - REST (HttpClient) circuit breaker + timeout policies
// - gRPC (Polly pipeline) circuit breaker + timeout policies
public sealed class CircuitBreakerPolicyOptions
{
// Timeout (in seconds) for ONE call attempt.
// REST example:
// - If PaymentService takes more than 2 seconds, we stop waiting (timeout).
// gRPC example:
// - If UserService gRPC call takes more than 2 seconds, we cancel the call.
public int TimeoutSeconds { get; set; } = 2;
// Sampling window (in seconds) used by circuit breaker to measure failure rate.
// Example:
// - If SamplingDurationSeconds = 10,
// circuit breaker checks failures for the last 10 seconds.
public int SamplingDurationSeconds { get; set; } = 10;
// Failure ratio threshold (0.0 to 1.0)
// Example:
// - FailureRatio = 0.5 means: if 50% or more calls fail within sampling window,
// open the circuit.
public double FailureRatio { get; set; } = 0.5;
// Minimum number of requests needed before circuit breaker starts deciding.
// Why needed?
// - Prevents false trips when traffic is low.
// Example:
// - If MinimumThroughput = 10,
// circuit breaker will only calculate failure ratio after at least 10 calls.
public int MinimumThroughput { get; set; } = 10;
// Break duration (cooldown time) in seconds once the circuit is OPEN.
// Example:
// - If BreakDurationSeconds = 15,
// circuit breaker blocks all calls for 15 seconds.
// After that:
// - Circuit moves to Half-Open
// - Allows a few trial calls to check recovery
public int BreakDurationSeconds { get; set; } = 15;
}
}
Step 5: Register Options using the Options Pattern
This file registers our options using the ASP.NET Core Options Pattern. It binds the “Resilience” section of appsettings.json into ResilienceOptions and also validates the values at startup.
Create a class file named ResilienceOptionsRegistration.cs within the OrderService.Infrastructure/Resilience folder, and copy-paste the following code. Now, values from appsettings.json will be bound to ResilienceOptions.
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
namespace OrderService.Infrastructure.Resilience
{
// This class is responsible for registering ResilienceOptions into DI.
// In simple words:
// - appsettings.json has "Resilience" settings
// - we want to read those settings as strongly-typed C# objects
// - and use them in our REST + gRPC circuit breaker configuration
// This follows the ASP.NET Core Options Pattern.
public static class ResilienceOptionsRegistration
{
// Extension method to register resilience options
// Usage: services.AddResilienceOptions(configuration);
public static IServiceCollection AddResilienceOptions(
this IServiceCollection services,
IConfiguration configuration)
{
// Register ResilienceOptions using Options Pattern
// - AddOptions<ResilienceOptions>() tells DI:
// "ResilienceOptions will be available to inject using IOptions / IOptionsMonitor"
// - Bind(configuration.GetSection("Resilience")) binds:
// appsettings.json "Resilience" section -> ResilienceOptions object
services.AddOptions<ResilienceOptions>()
.Bind(configuration.GetSection("Resilience"))
// Validation rules (Fail Fast)
// These validations ensure wrong configuration does NOT silently run.
// If config is invalid, the application will fail at startup with a clear error.
// Why important?
// - In production, config mistakes happen.
// - Validations prevent runtime failures and unpredictable behavior.
// PaymentService REST timeout must be positive
.Validate(o => o.Rest.Payment.TimeoutSeconds > 0,
"Resilience:Rest:Payment:TimeoutSeconds must be > 0")
// Minimum requests must be positive
// Circuit breaker should not calculate failure ratio with 0 requests.
.Validate(o => o.Rest.Payment.MinimumThroughput > 0,
"Resilience:Rest:Payment:MinimumThroughput must be > 0")
// gRPC timeout must be positive
.Validate(o => o.Grpc.Default.TimeoutSeconds > 0,
"Resilience:Grpc:Default:TimeoutSeconds must be > 0")
// gRPC MinimumThroughput must be positive
.Validate(o => o.Grpc.Default.MinimumThroughput > 0,
"Resilience:Grpc:Default:MinimumThroughput must be > 0")
// 3) ValidateOnStart (Very important)
// Without ValidateOnStart():
// - Options are validated only when they are first used.
//
// With ValidateOnStart():
// - Options are validated during app startup itself.
// - If config is wrong, app fails immediately (fail-fast).
.ValidateOnStart();
// Return IServiceCollection to support method chaining
return services;
}
}
}
Step 6: REST Circuit Breaker (Payment + Notification)
This file defines circuit breaker and timeout settings for REST calls (HttpClient), specifically PaymentService and NotificationService. It attaches resilience policies to HttpClients using AddResilienceHandler(…). This means every call made using those HttpClients automatically gets timeout + circuit breaker protection.
- Applies Timeout + Circuit Breaker for REST HttpClient calls
- Reads policy values from options (IOptionsMonitor<ResilienceOptions>)
- Protects OrderService from slow/down Payment/Notification services
- Uses ShouldHandle to treat only Real Transient Failures as failures (timeouts, 500+, 408, 429)
Create a class file named RestResilienceRegistration.cs within the OrderService.Infrastructure/Resilience folder, and copy-paste the following code. Now, Payment & Notification REST calls are protected by Circuit Breaker.
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Polly;
using Polly.Timeout;
using System.Net;
namespace OrderService.Infrastructure.Resilience
{
// This class contains extension methods that attach resilience policies
// (Timeout + Circuit Breaker) to REST HttpClients.
// Why this class exists?
// - To keep InfrastructureServiceRegistration clean.
// - So we can reuse the same resilience configuration for multiple REST clients.
// - To centralize timeout and circuit breaker rules for REST calls.
public static class RestResilienceRegistration
{
// Adds resilience policy for PaymentService HttpClient
// This will protect all calls made through the named HttpClient "PaymentServiceClient".
// In real-time:
// - If PaymentService becomes slow or starts failing (500s),
// circuit breaker will OPEN and OrderService will stop calling it temporarily.
public static IHttpClientBuilder AddPaymentResilience(this IHttpClientBuilder httpClientBuilder)
{
// AddResilienceHandler(...) creates a resilience pipeline for HttpClient.
// "payment-pipeline" is just a logical name for identifying this pipeline.
httpClientBuilder.AddResilienceHandler("payment-pipeline", (builder, context) =>
{
// Fetch current resilience settings from appsettings.json (Options Pattern)
// We use IOptionsMonitor so config changes can be picked up without restart.
var options = context.ServiceProvider
.GetRequiredService<IOptionsMonitor<ResilienceOptions>>()
.CurrentValue;
// Payment policy settings (TimeoutSeconds, FailureRatio, etc.)
var policy = options.Rest.Payment;
// Timeout Strategy
// Purpose:
// - Do not wait too long for PaymentService.
// - If PaymentService is slow, fail fast instead of blocking threads.
// Example:
// - TimeoutSeconds = 2 means: "Wait max 2 seconds for a response"
builder.AddTimeout(TimeSpan.FromSeconds(policy.TimeoutSeconds));
// Circuit Breaker Strategy
// Purpose:
// - If too many calls fail in a short time, OPEN the circuit.
// - When OPEN: stop calling PaymentService for some time (cooldown).
// - This prevents cascading failure (OrderService also becoming slow).
builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
// Observation window:
// Circuit breaker checks failures inside this time window
// Example: last 10 seconds
SamplingDuration = TimeSpan.FromSeconds(policy.SamplingDurationSeconds),
// Failure threshold:
// Example: 0.5 means "if 50% calls fail, open circuit"
FailureRatio = policy.FailureRatio,
// Minimum calls required before decision:
// Prevents false trips when traffic is low
MinimumThroughput = policy.MinimumThroughput,
// How long circuit stays OPEN:
// During this time, calls are blocked and fail fast.
BreakDuration = TimeSpan.FromSeconds(policy.BreakDurationSeconds),
// What should be counted as a "failure"?
// We count only transient failures:
// - Network Exception (HttpRequestException)
// - Timeout (TimeoutRejectedException)
// - Status Codes: 408, 429, and 500+
ShouldHandle = static args =>
{
// Exceptions (network failures / timeout)
if (args.Outcome.Exception is HttpRequestException or TimeoutRejectedException)
return ValueTask.FromResult(true);
// HTTP response codes (if response exists)
return ValueTask.FromResult(
args.Outcome.Result is HttpResponseMessage r &&
(
r.StatusCode is HttpStatusCode.RequestTimeout or // 408
HttpStatusCode.TooManyRequests || // 429
(int)r.StatusCode >= 500 // 500+
)
);
},
OnOpened = args =>
{
var logger = context.ServiceProvider
.GetRequiredService<ILoggerFactory>()
.CreateLogger("Resilience.Payment");
logger.LogWarning("PAYMENT circuit OPENED for {BreakDuration}. Reason: {Reason}",
args.BreakDuration,
args.Outcome.Exception?.Message ?? args.Outcome.Result?.StatusCode.ToString());
return ValueTask.CompletedTask;
},
OnHalfOpened = args =>
{
var logger = context.ServiceProvider
.GetRequiredService<ILoggerFactory>()
.CreateLogger("Resilience.Payment");
logger.LogWarning("PAYMENT circuit is HALF-OPEN (testing).");
return ValueTask.CompletedTask;
},
OnClosed = args =>
{
var logger = context.ServiceProvider
.GetRequiredService<ILoggerFactory>()
.CreateLogger("Resilience.Payment");
logger.LogInformation("PAYMENT circuit CLOSED (service recovered).");
return ValueTask.CompletedTask;
}
});
});
// Return builder so caller can continue chaining
return httpClientBuilder;
}
// Adds resilience policy for NotificationService HttpClient
// Similar to Payment, but typically notification is less critical.
// If notification fails, order/payment should still succeed.
public static IHttpClientBuilder AddNotificationResilience(this IHttpClientBuilder httpClientBuilder)
{
httpClientBuilder.AddResilienceHandler("notification-pipeline", (builder, context) =>
{
// Load configuration from appsettings.json using Options Pattern
var options = context.ServiceProvider
.GetRequiredService<IOptionsMonitor<ResilienceOptions>>()
.CurrentValue;
// Notification policy settings
var policy = options.Rest.Notification;
// 1) Timeout Strategy
// Notification should be fast. If it's slow, don't wait too long.
builder.AddTimeout(TimeSpan.FromSeconds(policy.TimeoutSeconds));
// 2) Circuit Breaker Strategy
// If NotificationService is unhealthy, open circuit to prevent repeated calls.
builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
SamplingDuration = TimeSpan.FromSeconds(policy.SamplingDurationSeconds),
FailureRatio = policy.FailureRatio,
MinimumThroughput = policy.MinimumThroughput,
BreakDuration = TimeSpan.FromSeconds(policy.BreakDurationSeconds),
// What counts as a failure for NotificationService?
// Same transient conditions:
// - network/timeout exceptions
// - 408, 429, and 500+
ShouldHandle = static args =>
{
if (args.Outcome.Exception is HttpRequestException or TimeoutRejectedException)
return ValueTask.FromResult(true);
return ValueTask.FromResult(
args.Outcome.Result is HttpResponseMessage r &&
(
r.StatusCode is HttpStatusCode.RequestTimeout or
HttpStatusCode.TooManyRequests ||
(int)r.StatusCode >= 500
)
);
}
});
});
return httpClientBuilder;
}
}
}
Step 7: gRPC Circuit Breaker (Pipeline + Interceptor)
gRPC failures usually come as RpcException, not HTTP status codes. We use a Polly pipeline and apply it via an interceptor.
Create a gRPC Pipeline Registration
This file registers a Polly Resilience Pipeline for gRPC. gRPC calls usually fail with RpcException, not HTTP status codes. We define a named pipeline that handles gRPC failure signals (Unavailable, DeadlineExceeded, etc.) and applies Timeout and Circuit Breaker rules.
- Creates a named pipeline (example: “grpc-pipeline”)
- Adds Timeout + Circuit Breaker for gRPC
- Detects failures using RpcException.StatusCode
- Keeps gRPC resilience logic centralized and reusable
Create a class file named GrpcResilienceRegistration.cs within the OrderService.Infrastructure/Resilience folder, and copy-paste the following code.
using Grpc.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Polly;
using Polly.CircuitBreaker;
namespace OrderService.Infrastructure.Resilience
{
// This class runs at application STARTUP.
// It does NOT execute for every request.
// Its job:
// 1) Create a resilience pipeline (Timeout + Circuit Breaker).
// 2) Register that pipeline in Dependency Injection.
// 3) Register the gRPC interceptor.
public static class GrpcResilienceRegistration
{
// This is just the name of our pipeline.
// We will use this same name later to fetch the pipeline.
public const string GrpcPipelineName = "grpc-pipeline";
// Extension method used in Startup / Program.cs:
// services.AddGrpcResilience();
public static IServiceCollection AddGrpcResilience(this IServiceCollection services)
{
// Here we CREATE and REGISTER a resilience pipeline.
// A pipeline is like a wrapper that applies rules
// around execution (similar to middleware).
services.AddResiliencePipeline(GrpcPipelineName, (builder, context) =>
{
// Read configuration from appsettings.json
// Example: Resilience:Grpc:Default
// We use IOptionsMonitor so values are strongly typed and centralized.
var options = context.ServiceProvider
.GetRequiredService<IOptionsMonitor<ResilienceOptions>>()
.CurrentValue;
// Pick gRPC default policy
var policy = options.Grpc.Default;
// TIMEOUT STRATEGY
// If the gRPC call takes longer than TimeoutSeconds,
// we stop waiting and treat it as failure.
// Why?
// So OrderService does not wait forever.
builder.AddTimeout(TimeSpan.FromSeconds(policy.TimeoutSeconds));
// CIRCUIT BREAKER STRATEGY
// Circuit breaker watches recent calls.
// If:
// - Too many failures
// - Within a short time
// Then:
// - It OPENs the circuit
// - Future calls fail immediately (no waiting)
builder.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
// Look at failures in the last X seconds
SamplingDuration = TimeSpan.FromSeconds(policy.SamplingDurationSeconds),
// Example: 0.5 = 50% failure threshold
FailureRatio = policy.FailureRatio,
// Minimum calls before breaker starts calculating failure ratio
// prevents false trips during low traffic
MinimumThroughput = policy.MinimumThroughput,
// Cooldown period
// How long circuit stays OPEN
BreakDuration = TimeSpan.FromSeconds(policy.BreakDurationSeconds),
// What counts as a failure for gRPC?
// In REST we use HTTP status codes (500, 408, etc.)
// In gRPC, failures usually come as RpcException with StatusCode.
//
// We treat only transient/unhealthy cases as failures:
// - Unavailable -> service is down / network issue
// - DeadlineExceeded -> service is too slow
// - ResourceExhausted -> service overloaded (rate limit)
// - Internal -> server side error
ShouldHandle = args =>
{
if (args.Outcome.Exception is RpcException ex)
{
// These are considered "service unhealthy"
return ValueTask.FromResult(
ex.StatusCode is StatusCode.Unavailable
or StatusCode.DeadlineExceeded
or StatusCode.ResourceExhausted
or StatusCode.Internal);
}
return ValueTask.FromResult(false);
},
OnOpened = args =>
{
var logger = context.ServiceProvider.GetRequiredService<ILoggerFactory>()
.CreateLogger("Resilience.gRPC");
logger.LogWarning("gRPC circuit OPENED for {BreakDuration}. Reason: {Reason}",
args.BreakDuration,
args.Outcome.Exception?.Message);
return ValueTask.CompletedTask;
},
OnHalfOpened = args =>
{
var logger = context.ServiceProvider.GetRequiredService<ILoggerFactory>()
.CreateLogger("Resilience.gRPC");
logger.LogWarning("gRPC circuit is HALF-OPEN (testing).");
return ValueTask.CompletedTask;
},
OnClosed = args =>
{
var logger = context.ServiceProvider.GetRequiredService<ILoggerFactory>()
.CreateLogger("Resilience.gRPC");
logger.LogInformation("gRPC circuit CLOSED (service recovered).");
return ValueTask.CompletedTask;
}
});
});
// Register the interceptor.
// Without this, the pipeline exists,
// but gRPC calls will NOT use it.
services.AddSingleton<GrpcResilienceInterceptor>();
return services;
}
}
}
Create the gRPC Interceptor
This file is the bridge that actually applies the gRPC pipeline to real gRPC calls. A pipeline is just a definition—this interceptor ensures every outgoing gRPC call passes through the pipeline, so circuit breaker and timeout really get enforced.
- Intercepts outgoing gRPC calls from OrderService to other services
- Executes the call through the resilience pipeline
- Ensures timeout/circuit breaker works for gRPC same as REST
- Keeps your gRPC client code clean (no resilience code inside each client method)
Create a class file named GrpcResilienceInterceptor.cs within the OrderService.Infrastructure/Resilience folder, and copy-paste the following code
using Grpc.Core;
using Grpc.Core.Interceptors;
using Polly.Registry;
namespace OrderService.Infrastructure.Resilience
{
// This interceptor runs EVERY TIME a gRPC call is made.
// Its job:
// - Take the real gRPC call
// - Wrap it inside the resilience pipeline
// - Take the real gRPC call and run it THROUGH the pipeline rules
// (timeout + circuit breaker).
public sealed class GrpcResilienceInterceptor : Interceptor
{
// Pipeline provider helps us "fetch the pipeline by name"
// Example: GetPipeline("grpc-pipeline")
private readonly ResiliencePipelineProvider<string> _pipelineProvider;
public GrpcResilienceInterceptor(ResiliencePipelineProvider<string> pipelineProvider)
{
_pipelineProvider = pipelineProvider;
}
// This method intercepts outgoing gRPC calls.
public override AsyncUnaryCall<TResponse> AsyncUnaryCall<TRequest, TResponse>(
TRequest request,
ClientInterceptorContext<TRequest, TResponse> context,
AsyncUnaryCallContinuation<TRequest, TResponse> continuation)
{
// STEP 1: Get the pipeline rules
// These rules were registered at startup in GrpcResilienceRegistration.
var pipeline = _pipelineProvider.GetPipeline(GrpcResilienceRegistration.GrpcPipelineName);
// Make the real gRPC call.
// continuation(...) makes the actual call to remote service.
var originalCall = continuation(request, context);
// STEP 3: Wrap only the response part with pipeline.ExecuteAsync(...)
// This means:
// - Timeout will apply
// - Circuit breaker will apply
Task<TResponse> responseAsync =
pipeline.ExecuteAsync(
ct => new ValueTask<TResponse>(originalCall.ResponseAsync),
context.Options.CancellationToken // stop quickly if request disconnects
).AsTask();
// Return a new AsyncUnaryCall object
// We replace ONLY ResponseAsync. Everything else remains original.
// Why new object?
// - Because we replaced only ResponseAsync with a resilient response.
// - But gRPC needs other things too (headers/status/trailers/dispose).
// - We reuse those from originalCall so behavior stays correct.
return new AsyncUnaryCall<TResponse>(
responseAsync, // Protected response or Resilient Response
originalCall.ResponseHeadersAsync, // Keep Original headers
originalCall.GetStatus, // Keep Original status
originalCall.GetTrailers, // Keep Original trailers
originalCall.Dispose // Keep Original cleanup
);
}
}
}
Step 8: Modify InfrastructureServiceRegistration
This file is the wiring/DI setup of your infrastructure layer. It doesn’t implement circuit breaker logic itself. Instead, it calls your clean registration methods, registers options, registers pipelines, and attaches interceptors/handlers. Basically, it’s the “assembly point” where everything gets connected properly.
- Registers options: services.AddResilienceOptions(configuration)
- Registers gRPC resilience: services.AddGrpcResilience()
- Adds interceptor to gRPC clients (AddInterceptor<GrpcResilienceInterceptor>())
- Adds resilience to REST HttpClients (AddPaymentResilience, AddNotificationResilience)
- Keeps configuration and plumbing separate from business logic
Please open the InfrastructureServiceRegistration.cs class file and paste the following code.
using ECommerce.GrpcContracts.Products;
using ECommerce.GrpcContracts.Users;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using OrderService.Contracts.ExternalServices;
using OrderService.Domain.Repositories;
using OrderService.Infrastructure.ExternalServices;
using OrderService.Infrastructure.GrpcClients;
using OrderService.Infrastructure.Repositories;
using OrderService.Infrastructure.Resilience;
namespace OrderService.Infrastructure.DependencyInjection
{
// Infrastructure Service Registration
// This class is the "wiring place" for OrderService.Infrastructure.
// - All external service clients (gRPC + REST) are registered here.
// - All repositories are registered here.
// - Resilience (timeout + circuit breaker) is also attached here.
//
// Why we do this?
// - Keeps Program.cs clean.
// - Keeps dependency registrations in one place.
public static class InfrastructureServiceRegistration
{
// Extension method called from Program.cs:
// builder.Services.AddInfrastructureServices(builder.Configuration);
public static IServiceCollection AddInfrastructureServices(
this IServiceCollection services,
IConfiguration configuration)
{
// STEP 1: Register Resilience Options (Options Pattern)
// - Reads "Resilience" section from appsettings.json
// - Binds to ResilienceOptions class
// - Validates values at startup (fail fast if config is wrong)
services.AddResilienceOptions(configuration);
// STEP 2: Register gRPC Resilience (Pipeline + Interceptor)
// What happens inside AddGrpcResilience()?
// - Registers a named pipeline: "grpc-pipeline"
// -> contains Timeout + Circuit Breaker rules
// - Registers GrpcResilienceInterceptor in DI
//
// Important:
// - Registering the pipeline alone is not enough.
// - Interceptor is required to APPLY these rules to actual gRPC calls.
services.AddGrpcResilience();
// STEP 3: Register gRPC Clients (UserService + ProductService)
// These clients are auto-generated from .proto files.
//
// We attach:
// .AddInterceptor<GrpcResilienceInterceptor>()
// so EVERY outgoing gRPC call goes through:
// Timeout + Circuit Breaker pipeline.
services.AddGrpcClient<UserGrpc.UserGrpcClient>(o =>
{
// Base URL of UserService read from configuration
// Example: "https://localhost:5001"
o.Address = new Uri(configuration["ExternalServices:UserServiceUrl"]
?? throw new ArgumentNullException("UserServiceUrl not configured"));
})
.AddInterceptor<GrpcResilienceInterceptor>(); // resilience applied to all UserService gRPC calls
services.AddGrpcClient<ProductGrpc.ProductGrpcClient>(o =>
{
// Base URL of ProductService read from configuration
o.Address = new Uri(configuration["ExternalServices:ProductServiceUrl"]
?? throw new ArgumentNullException("ProductServiceUrl not configured"));
})
.AddInterceptor<GrpcResilienceInterceptor>(); // resilience applied to all ProductService gRPC calls
// STEP 4: Register REST HttpClients (Payment + Notification)
// For REST services, we use HttpClientFactory.
// We attach resilience using our extension methods:
// - AddPaymentResilience()
// - AddNotificationResilience()
//
// These methods apply:
// - Timeout strategy
// - Circuit breaker strategy
services.AddHttpClient("PaymentServiceClient", client =>
{
// Base URL of PaymentService read from configuration
client.BaseAddress = new Uri(configuration["ExternalServices:PaymentServiceUrl"]
?? throw new ArgumentNullException("PaymentServiceUrl not configured"));
})
.AddPaymentResilience(); // resilience applied to all Payment REST calls
services.AddHttpClient("NotificationServiceClient", client =>
{
// Base URL of NotificationService read from configuration
client.BaseAddress = new Uri(configuration["ExternalServices:NotificationServiceUrl"]
?? throw new ArgumentNullException("NotificationServiceUrl not configured"));
})
.AddNotificationResilience(); // resilience applied to all Notification REST calls
// STEP 5: Register Wrapper Clients
services.AddScoped<IUserServiceClient, UserServiceGrpcClient>();
services.AddScoped<IProductServiceClient, ProductServiceGrpcClient>();
services.AddScoped<IPaymentServiceClient, PaymentServiceClient>();
services.AddScoped<INotificationServiceClient, NotificationServiceClient>();
// STEP 6: Register Repositories (DB Layer Implementation)
services.AddScoped<ICancellationRepository, CancellationRepository>();
services.AddScoped<ICartRepository, CartRepository>();
services.AddScoped<IMasterDataRepository, MasterDataRepository>();
services.AddScoped<IOrderRepository, OrderRepository>();
services.AddScoped<IRefundRepository, RefundRepository>();
services.AddScoped<IReturnRepository, ReturnRepository>();
services.AddScoped<IShipmentRepository, ShipmentRepository>();
return services;
}
}
}
Step 9: Update PaymentMethodEnum (OrderService.Contracts.Enums)
Please open the PaymentMethodEnum file and then replace it with the following code. We need to keep this in sync with Payment.
namespace OrderService.Contracts.Enums
{
public enum PaymentMethodEnum
{
CreditCard = 1,
DebitCard = 2,
UPI = 3,
Wallet = 4,
COD = 5,
NetBanking = 6
}
}
Step 10: Update CreatePaymentResponseDTO (OrderService.Contracts.DTOs)
Please open the CreatePaymentResponseDTO file and then replace it with the following code. We need to keep this in sync with Payment.
namespace OrderService.Contracts.DTOs
{
public class CreatePaymentResponseDTO
{
public Guid PaymentId { get; set; }
public string? Status { get; set; }
public string? PaymentUrl { get; set; } // For online payments
public string? ErrorMessage { get; set; }
public int? SuggestedHttpStatusCode { get; set; }
}
}
Step 11: Update PaymentServiceClient (OrderService.Infrastructure.ExternalServices)
Please open OrderService.Infrastructure.ExternalServices file and then replace it with the following code. We need to keep this in sync with Payment.
using OrderService.Contracts.DTOs;
using OrderService.Contracts.Enums;
using OrderService.Contracts.ExternalServices;
using Polly.CircuitBreaker;
using Polly.Timeout;
using System.Net;
using System.Net.Http.Headers;
using System.Net.Http.Json;
namespace OrderService.Infrastructure.ExternalServices
{
public class PaymentServiceClient : IPaymentServiceClient
{
private readonly HttpClient _httpClient;
public PaymentServiceClient(IHttpClientFactory httpClientFactory)
{
_httpClient = httpClientFactory.CreateClient("PaymentServiceClient");
}
public async Task<CreatePaymentResponseDTO> InitiatePaymentAsync(CreatePaymentRequestDTO request, string accessToken)
{
try
{
var paymentApiRequest = new
{
request.UserId,
request.OrderId,
request.Amount,
PaymentMethodTypeId = request.PaymentMethod,
Currency = "INR"
};
using var httpRequest = new HttpRequestMessage(HttpMethod.Post, "api/payments/initiate")
{
Content = JsonContent.Create(paymentApiRequest)
};
httpRequest.Headers.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
using var response = await _httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead);
// Auth errors are NOT resilience failures. They are token/permission issues.
if (response.StatusCode is HttpStatusCode.Unauthorized)
{
return Fail("Unauthorized to initiate payment.",
(int)HttpStatusCode.Unauthorized);
}
if (response.StatusCode is HttpStatusCode.Forbidden)
{
return Fail("Forbidden to initiate payment.",
(int)HttpStatusCode.Forbidden);
}
// If Payment service replied with overload / unavailable, treat as 503
if (response.StatusCode is HttpStatusCode.TooManyRequests
or HttpStatusCode.ServiceUnavailable)
{
var body = await response.Content.ReadAsStringAsync();
return Fail($"Payment service is overloaded/unavailable. Please try again. Body={body}",
(int)HttpStatusCode.ServiceUnavailable);
}
// Any other non-success from Payment service:
// We DID reach Payment service, but it returned an error -> 502 from our side.
if (!response.IsSuccessStatusCode)
{
var body = await response.Content.ReadAsStringAsync();
return Fail($"Payment initiation failed. Payment returned {(int)response.StatusCode}. Body={body}",
(int)HttpStatusCode.BadGateway);
}
// Parse response JSON
var apiResponse = await response.Content.ReadFromJsonAsync<ApiResponse<CreatePaymentResponseDTO>>();
if (apiResponse?.Success != true || apiResponse.Data == null)
{
// Payment responded but payload not as expected => 502
return Fail(apiResponse?.Message ?? "Payment returned an invalid/empty response.",
(int)HttpStatusCode.BadGateway);
}
return new CreatePaymentResponseDTO
{
PaymentId = apiResponse.Data.PaymentId,
Status = apiResponse.Data.Status,
PaymentUrl = apiResponse.Data.PaymentUrl,
ErrorMessage = null,
SuggestedHttpStatusCode = null
};
}
// Circuit breaker OPEN -> fail fast => 503
catch (BrokenCircuitException)
{
return Fail("Payment service temporarily unavailable (circuit open). Please try again.",
(int)HttpStatusCode.ServiceUnavailable);
}
// Timeout -> 504
catch (TimeoutRejectedException)
{
return Fail("Payment service timed out. Please try again.",
(int)HttpStatusCode.GatewayTimeout);
}
// Network issue / Payment is DOWN -> 503
catch (HttpRequestException ex)
{
return Fail($"Payment service is unreachable. Details: {ex.Message}",
(int)HttpStatusCode.ServiceUnavailable);
}
// Any unexpected client-side failure -> 502
catch (Exception ex)
{
return Fail($"Payment initiation failed due to an unexpected error. Details: {ex.Message}",
(int)HttpStatusCode.BadGateway);
}
static CreatePaymentResponseDTO Fail(string message, int statusCode) => new()
{
PaymentId = Guid.Empty,
Status = PaymentStatusEnum.Failed.ToString(),
PaymentUrl = null,
ErrorMessage = message,
SuggestedHttpStatusCode = statusCode
};
}
public async Task<PaymentInfoResponseDTO?> GetPaymentInfoAsync(PaymentInfoRequestDTO request, string accessToken)
{
// Simulate async delay
await Task.Delay(100);
// Return dummy payment info
return new PaymentInfoResponseDTO
{
OrderId = Guid.NewGuid(),
PaymentId = Guid.NewGuid(),
PaymentStatus = PaymentStatusEnum.Completed,
PaymentMethod = PaymentMethodEnum.CreditCard,
PaidAmount = 1000.00m,
PaymentDate = DateTime.UtcNow.AddMinutes(-10),
TransactionReference = "TXN123456789",
FailureReason = null
};
}
public async Task<RefundResponseDTO> InitiateRefundAsync(RefundRequestDTO request, string accessToken)
{
// Simulate async delay
await Task.Delay(100);
// Return dummy refund response
return new RefundResponseDTO
{
Success = true,
ErrorMessage = null
};
}
}
}
Updating CreateOrderCommandHandler:
Please update the Payment Service Client Call Logic within the Handle method of the CreateOrderCommandHandler class. This class is available in OrderService.Application.Orders.Handlers.Commands folder.
var paymentResponse = await _paymentServiceClient.InitiatePaymentAsync(paymentRequest, accessToken);
if (paymentResponse == null || paymentResponse.Status == "Failed")
{
var msg = paymentResponse?.ErrorMessage ?? "Payment initiation failed.";
var statusCode = paymentResponse?.SuggestedHttpStatusCode ?? 503;
throw new HttpRequestException(
message: msg,
inner: null,
statusCode: (System.Net.HttpStatusCode)statusCode
);
}
Updating OrderController
Please update the CreateOrder method within the OrderController as follows.
[Authorize]
[HttpPost]
public async Task<ActionResult<ApiResponse<OrderResponseDTO>>> CreateOrder(
[FromBody] CreateOrderRequestDTO request)
{
_logger.LogInformation("CreateOrder request received for UserId: {UserId}", request.UserId);
try
{
// Extract bearer token for downstream microservice calls
var accessToken = Request.Headers["Authorization"]
.ToString()
.Replace("Bearer ", "");
// Send Command to MediatR → CommandHandler will execute logic
// MediatR routes it to CreateOrderCommandHandler
var result = await _mediator.Send(new CreateOrderCommand(request, accessToken));
_logger.LogInformation("Order created successfully for UserId: {UserId}", request.UserId);
return Ok(ApiResponse<OrderResponseDTO>.SuccessResponse(
result, "Order placed successfully."));
}
// IMPORTANT CATCH: Downstream service failures (Payment/User/Product/Notification)
// We throw HttpRequestException from the handler when:
// - Payment service is DOWN -> 503
// - Circuit is OPEN -> 503
// - Payment timed out -> 504
// - Payment returned invalid/failed response -> 502
catch (HttpRequestException ex)
{
_logger.LogError(ex,
"Downstream dependency failure while creating order for UserId: {UserId}",
request.UserId);
// If status code was provided, use it.
// Otherwise default to 503 (service unavailable).
var statusCode = ex.StatusCode.HasValue
? (int)ex.StatusCode.Value
: StatusCodes.Status503ServiceUnavailable;
return StatusCode(statusCode, ApiResponse<OrderResponseDTO>.FailResponse(
"Order creation failed.",
new List<string> { ex.Message }));
}
catch (Exception ex)
{
_logger.LogError(ex, "Error occurred while creating order for UserId: {UserId}", request.UserId);
return BadRequest(ApiResponse<OrderResponseDTO>.FailResponse(
"Order creation failed.",
new List<string> { ex.Message }));
}
}
Run Consul in Development Mode
Please run the following command in Command Prompt.
consul agent -dev -client=0.0.0.0 -ui -node=consul-dev
Note: Keep this command window open while working with your microservices. If you close it, Consul stops, and services cannot register or be discovered.
Open Consul UI
Once Consul is running, we can monitor everything from the browser. Open your browser and navigate to: http://localhost:8500
Now test the Order Creation functionality; it should work as expected.
In this article, I explain how to implement Circuit Breaker in ASP.NET Core Web API Microservices. I hope you enjoy this article on Circuit Breaker in ASP.NET Core Web API Microservices.