In the world of cloud-native development, “stability” is often synonymous with how well your application handles its own demise. When running on Google Kubernetes Engine (GKE), pods are constantly being moved, scaled, or updated. If your application doesn’t handle these transitions politely, your users will see the dreaded 503 Service Unavailable errors.
In this post, we’ll dive into the SIGTERM signal, why it matters for .NET architects, and how to implement a graceful shutdown strategy.
What is a SIGTERM Signal?
When GKE decides to terminate a pod—whether due to a rolling update, a scale-down event, or a node maintenance—it doesn’t just “pull the plug.” Instead, it follows a coordinated sequence:
The Signal: Kubernetes sends a SIGTERM (Signal Terminate) to the process (PID 1) inside your container.
The Grace Period: Kubernetes waits for a defined period (default is 30 seconds).
The Kill: If the process is still running after the grace period, Kubernetes sends a SIGKILL, which force-terminates the app immediately.
The SIGTERM is your application’s “final boarding call.” It is your chance to stop accepting new requests, finish existing work, and flush buffers.
Why “Wait and See” Isn’t a Strategy
If you ignore the SIGTERM signal:
Dropped Requests: Users in the middle of a request will see a connection reset.
Data Corruption: File writes or database transactions might be cut off midstream.
Zombie State: In systems like Kafka, if a consumer dies without committing its offset, the next consumer will re-process the same messages, leading to duplicates.
Implementing Graceful Shutdown in .NET
Modern .NET (Core 3.1 through .NET 8+) makes handling shutdown intuitive via IHostApplicationLifetime and CancellationToken.
1. The Power of CancellationToken
Every asynchronous operation in your API should respect a CancellationToken. When GKE sends a SIGTERM, .NET automatically triggers the StoppingToken in your background services.
public class DataProcessorWorker : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// The framework triggers stoppingToken when SIGTERM is received
while (!stoppingToken.IsCancellationRequested)
{
await ProcessDataAsync(stoppingToken);
}
// Perform cleanup logic here
await CloseConnectionsAsync();
}
}
2. Hooking into Application Lifetime
If you need to perform global cleanup (like flushing distributed logs or closing a singleton Redis connection), inject IHostApplicationLifetime.
public void Configure(IApplicationBuilder app, IHostApplicationLifetime lifetime)
{
lifetime.ApplicationStopping.Register(() =>
{
// This code executes immediately upon receiving SIGTERM
Log.Information("API is shutting down. Finalizing telemetry...");
});
}
The GKE “Gotcha”: The Race Condition
There is a brief lag between GKE sending a SIGTERM and the Load Balancer removing the pod from its rotation. If your app stops immediately upon receiving the signal, the Load Balancer might still send it one last request, resulting in a 502 Bad Gateway.
The Solution: Add a small delay in your Kubernetes preStop hook to allow the Load Balancer to catch up.
# In your Kubernetes Deployment manifest
spec:
containers:
- name: my-dotnet-api
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 10"]
Summary for Architects
To build truly resilient APIs on GKE, your checklist should include:
Use Exec Form in Dockerfiles: ENTRYPOINT [“dotnet”, “App.dll”] ensures your app receives signals directly.
Propagate Tokens: Always pass CancellationToken through your service layers.
Configure Termination Grace Period: If your app needs 60 seconds to flush a Kafka buffer, set terminationGracePeriodSeconds: 60 in your YAML.
Test It: Use kubectl delete pod [name] and watch your logs to see if your cleanup logic actually fires.
By mastering the SIGTERM, we move from building apps that simply “run” to building systems that are truly “cloud-native.”
Happy Coding !!!