Background processing is a common requirement in modern applications. An API may need to generate reports, process uploaded files, synchronize customer records, or handle messages from a queue.
These tasks often take longer than a normal HTTP request. Running them directly inside an API endpoint can increase response times, consume request-handling resources, and make failures harder to recover from.
Google Cloud Run provides several ways to run background workloads without managing virtual machines or Kubernetes clusters. Cloud Run services can process requests and events, Cloud Run jobs can execute tasks to completion, and Cloud Run worker pools support continuously running background processes.
The challenge is choosing the right execution model. A message consumer that must continuously poll a queue has different requirements from a scheduled report generator or a one-time data migration.
This article explains how to choose between Cloud Run services, jobs, and worker pools, with practical deployment examples, scaling considerations, and production recommendations.
Choosing the Right Cloud Run Execution Model
Cloud Run supports different execution models because background workloads do not all behave the same way.
Cloud Run resource | Best suited for | Execution behavior |
|---|
Services | APIs, webhooks, HTTP-triggered processing | Responds to requests and can scale based on traffic |
Jobs | Reports, migrations, batch processing | Runs tasks to completion |
Worker pools | Queue consumers and continuous background processing | Runs containers without requiring incoming HTTP requests |
A Cloud Run service is suitable when work begins with an HTTP request or an event delivered to an HTTP endpoint.
A Cloud Run job is more appropriate when the application needs to execute a defined task and then exit.
A worker pool is designed for processes that continuously perform background work, such as consuming messages from Pub/Sub, Kafka, or RabbitMQ.
The important distinction is that worker pools do not expose an HTTP endpoint and do not require a container to listen on a port. Unlike request-driven Cloud Run services, they are not automatically scaled based on incoming HTTP traffic.
When to Use Cloud Run Worker Pools
Consider an e-commerce application that accepts customer orders.
The API receives an order and stores it in a database. After the order is created, the application must generate an invoice, update inventory, and send a confirmation notification.
Performing all these operations inside the original HTTP request can increase response times and make error handling more complicated.
A better architecture separates request handling from background processing.
Customer places an order
|
v
Cloud Run API Service
|
v
Store order and publish message
|
v
Pub/Sub
|
v
Cloud Run Worker Pool
|
v
Process order
|
v
Update database and send notification
The API can return a response after the order is accepted and its durable work has been recorded.
The worker processes the message independently.
This separation provides several benefits. The API does not need to wait for invoice generation, background failures can be retried, and worker capacity can be managed independently of HTTP traffic.
Worker pools are especially useful when a background process needs to remain active and continuously pull work from a queue.
However, if Pub/Sub push delivery can invoke a Cloud Run service directly, a request-driven service may be simpler and more cost-effective. A continuously running worker is not automatically the best solution for every queue-based application.
Creating a Background Worker Application
A worker pool runs a container, so the application can be written in any language that can execute inside a supported container.
For this example, we will create a .NET background worker that continuously checks for work.
Start by creating a worker project:
dotnet new worker -n OrderProcessingWorker
cd OrderProcessingWorker
The Worker Service template provides a hosted background service that can run independently of an HTTP server.
Open Worker.cs and replace its contents with:
namespace OrderProcessingWorker;
public class Worker : BackgroundService
{
private readonly ILogger<Worker> _logger;
public Worker(ILogger<Worker> logger)
{
_logger = logger;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
_logger.LogInformation(
"Order processing worker started.");
while (!stoppingToken.IsCancellationRequested)
{
_logger.LogInformation(
"Checking for pending orders at {Time}",
DateTimeOffset.UtcNow);
await Task.Delay(
TimeSpan.FromSeconds(10),
stoppingToken);
}
}
}
This example demonstrates the basic structure of a continuously running worker.
The ExecuteAsync method runs until the application receives a shutdown signal.
The cancellation token allows the worker to stop when Cloud Run terminates the container.
The example does not process real orders yet. It simply logs a message every ten seconds.
This is useful for verifying the worker's lifecycle before adding queue integration.
Understanding the Worker Lifecycle
A background worker must handle startup and shutdown correctly.
Cloud Run manages the container infrastructure, but the application remains responsible for processing work safely.
When a worker instance starts, the application initializes its dependencies and begins processing messages.
When Cloud Run stops the instance, the application receives a termination signal and has a limited period to shut down gracefully.
The worker should respond by stopping the retrieval of new messages and completing or safely abandoning any work already in progress.
A simplified lifecycle looks like this:
Container starts
|
v
Initialize dependencies
|
v
Connect to message source
|
v
Receive message
|
v
Process message
|
v
Acknowledge successful work
|
v
Receive next message
|
v
Shutdown signal received
|
v
Stop receiving new work
|
v
Complete or release pending work
|
v
Container exits
A worker should never assume that it will run forever.
Instances may restart because of deployment changes, infrastructure maintenance, application failures, or scaling operations.
For this reason, important processing state should not exist only in the container's memory.
Adding Pub/Sub Message Processing
Google Cloud Pub/Sub is a managed messaging service that can separate an API from its background consumers.
A publisher sends a message to a topic, and a subscriber receives messages through a subscription.
For a continuously running worker pool, a pull subscription is a natural integration.
First, add the Pub/Sub client library:
dotnet add package Google.Cloud.PubSub.V1
Create a Pub/Sub topic and subscription using the Google Cloud CLI:
gcloud pubsub topics create orders
gcloud pubsub subscriptions create orders-worker \
--topic=orders
Next, implement a background service that consumes messages.
using Google.Cloud.PubSub.V1;
namespace OrderProcessingWorker;
public class Worker : BackgroundService
{
private readonly ILogger<Worker> _logger;
public Worker(ILogger<Worker> logger)
{
_logger = logger;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
var projectId = Environment.GetEnvironmentVariable(
"GOOGLE_CLOUD_PROJECT");
if (string.IsNullOrWhiteSpace(projectId))
{
throw new InvalidOperationException(
"GOOGLE_CLOUD_PROJECT is required.");
}
var subscriptionId = SubscriptionName.FromProjectSubscription(
projectId,
"orders-worker");
var subscriber = await SubscriberClient.CreateAsync(
subscriptionId);
_logger.LogInformation(
"Starting Pub/Sub subscriber.");
var subscriberTask = subscriber.StartAsync(
async (message, cancellationToken) =>
{
var orderData = message.Data.ToStringUtf8();
try
{
_logger.LogInformation(
"Processing order: {Order}",
orderData);
await ProcessOrderAsync(
orderData,
cancellationToken);
return SubscriberClient.Reply.Ack;
}
catch (Exception ex)
{
_logger.LogError(
ex,
"Failed to process order.");
return SubscriberClient.Reply.Nack;
}
});
try
{
await Task.Delay(
Timeout.Infinite,
stoppingToken);
}
catch (OperationCanceledException)
when (stoppingToken.IsCancellationRequested)
{
_logger.LogInformation(
"Stopping Pub/Sub subscriber.");
}
finally
{
await subscriber.StopAsync(
TimeSpan.FromSeconds(5));
await subscriberTask;
}
}
private static Task ProcessOrderAsync(
string orderData,
CancellationToken cancellationToken)
{
// Replace with actual order-processing logic.
return Task.CompletedTask;
}
}
This example demonstrates the basic Pub/Sub consumption pattern.
The subscriber receives messages and invokes the callback.
When processing succeeds, the callback returns Ack. When processing fails, it returns Nack, allowing the message to become eligible for redelivery.
The shutdown path requests the subscriber to stop and waits for its processing task to finish.
For a production application, configure subscriber concurrency, flow control, retry behavior, and shutdown timing according to the workload. A message-processing callback should also avoid logging sensitive message contents.
Handling Duplicate Messages Safely
Message-driven applications must account for duplicate delivery.
Pub/Sub subscriptions commonly use at-least-once delivery semantics. This means the same message may be delivered more than once.
For example, a worker might successfully update an order in the database but fail before acknowledging the message.
Pub/Sub can then redeliver the message.
If the worker processes the same operation again without checking its previous state, the application may generate duplicate invoices or send multiple notifications.
The solution is idempotent processing.
An idempotent operation can be executed multiple times without producing unintended additional effects.
For example, the application can store a unique message ID or business operation ID in a database.
public async Task ProcessOrderAsync(
string messageId,
string orderId,
CancellationToken cancellationToken)
{
var alreadyProcessed = await _db.ProcessedMessages
.AnyAsync(
x => x.MessageId == messageId,
cancellationToken);
if (alreadyProcessed)
{
return;
}
// Execute the order-processing operation.
_db.ProcessedMessages.Add(new ProcessedMessage
{
MessageId = messageId
});
await _db.SaveChangesAsync(cancellationToken);
}
This illustrates the idea, but a simple check followed by an insert is not enough to prevent races between concurrent workers.
In production, use a unique database constraint on the message or operation ID, and place the business changes and processing record in an appropriate transaction.
For operations involving external services, such as payment providers, use the external service's idempotency mechanisms where available.
The goal is to make message redelivery safe rather than assuming that each message will arrive exactly once.
Containerizing the Background Worker
Cloud Run worker pools deploy container images.
Create a Dockerfile in the project directory:
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish \
-c Release \
-o /app/publish \
--no-restore
FROM mcr.microsoft.com/dotnet/runtime:10.0
WORKDIR /app
COPY --from=build /app/publish ./
ENTRYPOINT ["dotnet", "OrderProcessingWorker.dll"]
This Dockerfile uses a multi-stage build.
The first stage restores dependencies and publishes the application.
The second stage contains the smaller runtime image and the published worker.
Because a worker pool does not require an HTTP listener, the application does not need to expose a port or start a web server.
Build the image locally:
docker build -t order-processing-worker .
Before deploying, run the container locally with appropriate Google Cloud credentials and configuration.
For the initial logging-only worker, no Pub/Sub access is required. Once Pub/Sub integration is added, the container needs an authenticated Google Cloud identity with subscription access.
Deploying the Worker to Cloud Run
Before deployment, enable the required Google Cloud APIs and create an Artifact Registry repository.
For example:
gcloud services enable \
run.googleapis.com \
artifactregistry.googleapis.com \
pubsub.googleapis.com
Create a Docker repository:
gcloud artifacts repositories create worker-images \
--repository-format=docker \
--location=us-central1
Configure Docker authentication:
gcloud auth configure-docker \
us-central1-docker.pkg.dev
Tag the image:
docker tag order-processing-worker \
us-central1-docker.pkg.dev/PROJECT_ID/worker-images/order-processing-worker:1.0
Replace PROJECT_ID with your Google Cloud project ID.
Push the image:
docker push \
us-central1-docker.pkg.dev/PROJECT_ID/worker-images/order-processing-worker:1.0
Deploy the worker pool:
gcloud run worker-pools deploy order-processing-worker \
--image=us-central1-docker.pkg.dev/PROJECT_ID/worker-images/order-processing-worker:1.0 \
--region=us-central1 \
--instances=1
This creates a worker pool with one running instance.
The worker starts executing its background process after deployment.
Unlike a Cloud Run service, the worker pool does not provide a public HTTP URL. Its purpose is to run background code rather than respond to incoming web requests.
Configuring Service Account Permissions
A worker that reads Pub/Sub messages needs permission to access the subscription.
The recommended approach is to assign a dedicated service account to the worker pool.
Create a service account:
gcloud iam service-accounts create order-worker \
--display-name="Order Processing Worker"
Grant the required Pub/Sub subscriber role:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:order-worker@PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/pubsub.subscriber"
Then configure the worker pool to use that service account:
gcloud run worker-pools update order-processing-worker \
--region=us-central1 \
--service-account=order-worker@PROJECT_ID.iam.gserviceaccount.com
The worker can use Application Default Credentials through the Google Cloud client library.
There is no need to embed a service account key file inside the container image.
For stronger isolation, grant the Pub/Sub subscriber permissions on the specific subscription rather than at the project level.
The service account should receive only the permissions required for its background operations.
Scaling Worker Pools
Scaling is an important difference between Cloud Run services and worker pools.
Cloud Run services can automatically scale based on incoming requests.
Worker pools are designed for background workloads that do not receive requests through a load-balanced HTTP endpoint.
Worker pools support manually configured instance counts. Google Cloud also documents external autoscaling approaches based on metrics such as Pub/Sub backlog or Kafka queue depth.
For example, a worker pool may start with one instance:
gcloud run worker-pools deploy order-processing-worker \
--image=IMAGE_URL \
--region=us-central1 \
--instances=1
The number of instances can be changed according to processing demand.
However, simply increasing the instance count does not guarantee better throughput.
The application must support concurrent processing, and its dependencies must be able to handle the additional load.
For example, increasing the number of workers may increase database connections, external API requests, or pressure on downstream services.
Monitor queue depth, processing latency, CPU usage, memory consumption, and error rates before changing capacity.
Also consider cost. A continuously running worker pool consumes resources even when there are no messages to process. If the workload is intermittent, a request-driven Cloud Run service or a Cloud Run job may be more appropriate.
When Cloud Run Jobs Are a Better Choice
Not every background task needs a continuously running worker.
Consider a reporting application that generates a financial report once every night.
The task starts, reads data, generates a report, uploads the result to storage, and exits.
This is a good fit for a Cloud Run job.
A job runs one or more tasks to completion and can be started manually, through Cloud Scheduler, or through an orchestration workflow.
For example, deploy a job:
gcloud run jobs deploy daily-report \
--image=IMAGE_URL \
--region=us-central1
Execute the job:
gcloud run jobs execute daily-report \
--region=us-central1
The container starts, performs its work, and exits.
Unlike a continuously running worker pool, the job does not need to remain active waiting for new messages.
Cloud Run jobs also support parallel tasks, retries, and configurable execution timeouts.
This makes them useful for data imports, database maintenance, scheduled reports, and batch processing.
Monitoring Background Workers
A background worker can fail without affecting the availability of the application's HTTP API.
This makes monitoring essential.
Google Cloud provides Cloud Logging and Cloud Monitoring for observing worker behavior.
At a minimum, track:
Number of messages processed.
Number of failed processing attempts.
Message processing duration.
Queue backlog and oldest unacknowledged message age.
Worker restarts and unexpected exits.
CPU and memory usage.
Structured logs make it easier to investigate individual operations.
For example:
_logger.LogInformation(
"Order {OrderId} processed successfully in {DurationMs} ms",
orderId,
durationMs);
Use consistent operation identifiers so that logs can be correlated across the API, queue, and worker.
Also configure alerts for sustained queue growth or repeated processing failures.
A worker can remain technically healthy while processing messages too slowly. Queue backlog is often a more useful signal than process availability alone.
Production Considerations
A reliable background processing system needs more than a working container.
Design for retries
Temporary database failures, network interruptions, and external service errors are normal.
Use bounded retries and exponential backoff where appropriate. Configure dead-letter topics or queues for messages that repeatedly fail.
Handle shutdown correctly
Stop accepting new work when the container receives a termination signal.
Allow in-progress operations to finish when possible, and make sure incomplete operations can be safely retried.
Keep workers stateless
Do not depend on local container storage for durable processing state.
Store important state in a database, object storage, or another durable system.
Protect downstream services
A large number of workers can overwhelm a database or external API.
Use concurrency limits, connection pooling, and rate limiting where appropriate.
Separate worker and API deployments
Deploy the API and background worker independently when they have different scaling, availability, or resource requirements.
This allows each component to be updated without unnecessarily affecting the other.
Summary
Cloud Run provides several ways to run background workloads without managing servers.
Cloud Run services are useful for HTTP-triggered and event-driven processing. Cloud Run jobs are designed for tasks that run to completion. Cloud Run worker pools are intended for continuously running background processes, including message consumers.
Worker pools allow developers to deploy a container that processes work without exposing an HTTP endpoint. They can integrate with Pub/Sub, databases, and other Google Cloud services through a dedicated service account.
However, the application must still handle message redelivery, idempotency, shutdown, retries, concurrency, and monitoring.
The best architecture depends on how work arrives and how long it needs to run. For continuous queue consumption, worker pools provide a managed execution environment. For scheduled or finite tasks, Cloud Run jobs are often simpler. For events that can be delivered through HTTP, a Cloud Run service may provide a better scaling model.
Choosing the correct execution model makes background processing easier to operate while avoiding unnecessary infrastructure management.
Join the conversation! Your thoughts help the community grow.