Running GPU Workloads on Google Cloud Run

Saurav Kumar

Google Cloud Run is commonly used to deploy containerized applications without managing servers or Kubernetes clusters. It handles infrastructure provisioning, scaling, and request routing, making it useful for APIs, background services, and event-driven applications.

However, applications that perform AI inference, image processing, video analysis, or other compute-intensive operations may benefit from GPU acceleration.

Cloud Run supports GPU workloads, allowing developers to run GPU-enabled containers without managing a traditional virtual machine environment.

This article explains how to prepare a GPU-enabled container, deploy it to Google Cloud Run, configure the required resources, and handle common deployment problems.

When Does a Cloud Run Application Need a GPU?

A CPU is sufficient for most web applications, database operations, and lightweight background tasks.

A GPU becomes useful when an application performs large amounts of parallel computation.

Common examples include:

Running machine learning inference.
Processing images and videos.
Generating embeddings.
Running computer vision models.
Performing audio transcription.
Executing GPU-accelerated scientific workloads.

Consider an image processing API that receives uploaded product images and uses a machine learning model to identify objects.

A CPU-based implementation may take several seconds to process each image. A compatible GPU can reduce processing time, depending on the model, batch size, memory requirements, and workload.

However, GPU acceleration does not automatically make every application faster. The model and its dependencies must support GPU execution, and the workload must be large enough to benefit from it.

Understand How GPU Workloads Run on Cloud Run

Cloud Run runs container images and manages the underlying infrastructure.

When deploying a GPU-enabled service, you configure the GPU resource alongside the normal container settings.

The general architecture looks like this:

Client Application
       |
       v
Cloud Run Service
       |
       v
GPU-Enabled Container
       |
       v
Machine Learning Model
       |
       v
GPU-Accelerated Processing
       |
       v
API Response

The application still uses the normal Cloud Run deployment model.

You do not need to create and manage a separate virtual machine just to run the GPU workload.

However, GPU availability depends on the Cloud Run configuration, supported region, quota, GPU type, and current service limitations. Check the supported configuration before designing the deployment.

Prepare the Application

For this example, create a Python application that loads a machine learning model and exposes an HTTP endpoint.

The application will use FastAPI and PyTorch.

Create a project structure:

gpu-image-service/
│
├── app.py
├── requirements.txt
└── Dockerfile

Add the dependencies:

fastapi
uvicorn
torch
torchvision
pillow

For production, pin compatible dependency versions and use the PyTorch installation instructions appropriate for the selected CUDA environment. Do not assume that installing any PyTorch package automatically enables GPU execution.

Create the application:

from fastapi import FastAPI
import torch

app = FastAPI()

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

@app.get("/health")
async def health():
    return {
        "status": "healthy",
        "device": str(device)
    }

@app.post("/process")
async def process():
    tensor = torch.randn(
        1000,
        1000,
        device=device
    )

    result = torch.matmul(tensor, tensor)

    return {
        "device": str(device),
        "result_shape": list(result.shape)
    }

This example performs a matrix multiplication to demonstrate device selection.

A real application would load a trained model and process an image, audio file, or other input.

The health endpoint reports whether PyTorch detected a CUDA-compatible GPU.

Do not rely only on successful container startup to confirm GPU availability. Verify that the actual model and inference operation execute on the expected device.

Create the Dockerfile

The container image must include the application dependencies and a compatible GPU runtime environment.

A basic Dockerfile can look like this:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

This is a basic application container, not a complete GPU-ready image.

For a production GPU workload, use a compatible CUDA-enabled base image or a suitable framework image. The CUDA runtime, PyTorch build, Python version, and GPU driver environment must be compatible.

Avoid installing unnecessary development packages in the final image. A smaller container is easier to build, scan, transfer, and deploy.

You should also test the container locally with an available compatible GPU before deploying it.

Configure Google Cloud

Before deployment, configure the Google Cloud project.

Install the Google Cloud CLI if it is not already available, then authenticate:

gcloud auth login

Select the project:

gcloud config set project YOUR_PROJECT_ID

Enable the required services:

gcloud services enable \
  run.googleapis.com \
  artifactregistry.googleapis.com \
  cloudbuild.googleapis.com

Replace YOUR_PROJECT_ID with your actual project ID.

Create an Artifact Registry repository:

gcloud artifacts repositories create gpu-apps \
  --repository-format=docker \
  --location=us-central1

The region is an example. Select a region that supports the required Cloud Run GPU configuration and is appropriate for your users and data residency requirements.

Configure Docker authentication:

gcloud auth configure-docker us-central1-docker.pkg.dev

This allows Docker to push images to the Artifact Registry repository.

Build and Push the Container Image

Build the image:

docker build -t gpu-image-service .

Tag the image for Artifact Registry:

docker tag gpu-image-service \
  us-central1-docker.pkg.dev/YOUR_PROJECT_ID/gpu-apps/gpu-image-service:v1

Push the image:

docker push \
  us-central1-docker.pkg.dev/YOUR_PROJECT_ID/gpu-apps/gpu-image-service:v1

The image is now available for Cloud Run deployment.

For repeatable deployments, use a CI/CD pipeline that builds, scans, tags, and publishes the image rather than relying on manual commands.

Deploy the GPU Service to Cloud Run

Cloud Run supports GPU-enabled service deployments through its GPU configuration.

For a supported configuration, a deployment command follows this pattern:

gcloud run deploy gpu-image-service \
  --image us-central1-docker.pkg.dev/YOUR_PROJECT_ID/gpu-apps/gpu-image-service:v1 \
  --region us-central1 \
  --gpu 1 \
  --no-allow-unauthenticated

The command requests one GPU for the service.

The selected region and GPU configuration must be supported by Cloud Run. You must also have sufficient quota and the necessary permissions.

Cloud Run may require additional configuration depending on the selected GPU type and service settings. Check the current deployment requirements before using this command in production.

The --no-allow-unauthenticated option prevents public access. Configure IAM permissions for the users or services that should call the endpoint.

After deployment, retrieve the service URL:

gcloud run services describe gpu-image-service \
  --region us-central1 \
  --format="value(status.url)"

Use the returned URL to access the application.

Verify GPU Availability

After deployment, call the health endpoint.

curl https://YOUR_CLOUD_RUN_URL/health \
  -H "Authorization: Bearer YOUR_ID_TOKEN"

The response should identify the device used by PyTorch.

For example:

{
  "status": "healthy",
  "device": "cuda"
}

If the application reports cpu, investigate the container's CUDA and framework configuration.

Check the following:

The container uses a compatible GPU-enabled framework build.
The CUDA runtime dependencies are available.
The deployed service actually has a GPU assigned.
The application is using the correct device.
The selected GPU configuration is supported in the deployment region.

Also run an actual inference operation. Device detection alone does not prove that the complete model can execute successfully.

Configure Concurrency and Request Handling

GPU workloads have different resource requirements from ordinary web APIs.

A normal API might handle many lightweight requests simultaneously. A large machine learning model may consume significant GPU memory for each concurrent inference operation.

If too many requests execute simultaneously, the application may run out of GPU memory.

Consider limiting request concurrency based on the model's memory requirements.

For example, a service that performs large image transformations may work better with a lower concurrency setting than a simple JSON API.

You should also consider:

Model loading time.
GPU memory consumption.
Request duration.
Container startup time.
Maximum request size.
CPU and memory allocation.
Number of service instances.

Do not choose concurrency based only on CPU utilization. Monitor GPU memory and inference latency under realistic workloads.

Handle Cold Starts and Model Loading

GPU applications often have larger container images and heavier model initialization requirements.

When Cloud Run starts a new instance, the application may need to:

Initialize the Python runtime.
Load machine learning dependencies.
Download or access model files.
Load model weights into memory.
Initialize the GPU execution environment.

These operations can increase startup time.

Avoid downloading large model files every time an instance starts. Store model artifacts in a suitable storage service and use controlled caching where appropriate.

If the application requires predictable response times, evaluate minimum instance settings and their cost implications.

Also, avoid loading a model separately for every incoming request. Initialize it once per application instance and reuse it safely.

Monitor GPU Workloads

Monitoring is important because GPU performance problems are not always visible through ordinary API metrics.

Track the following measurements:

Metric	Why it matters
Request latency	Identifies slow inference
Requests per second	Measures throughput
Container memory	Detects memory pressure
CPU utilization	Identifies CPU bottlenecks
GPU utilization	Shows GPU activity
GPU memory usage	Detects memory exhaustion
Container startup time	Identifies cold-start delays
Error rate	Tracks failed requests

Use Cloud Logging and Cloud Monitoring to inspect application behavior and service performance.

For GPU-specific measurements, expose or collect the metrics supported by the selected runtime and monitoring setup. Do not assume every GPU metric is automatically available through standard Cloud Run dashboards.

Common Mistakes to Avoid

One common mistake is using a CPU-only machine learning package inside a GPU-enabled container. Assigning a GPU to Cloud Run does not automatically make the application GPU-compatible.

Another mistake is using an unsupported region or GPU configuration. Verify availability, quota, and deployment requirements before building the deployment pipeline.

Developers also sometimes increase concurrency without considering GPU memory consumption. This can cause inference failures under load.

Finally, avoid loading model files on every request. Initialize the model once per instance and reuse it where the framework and model support safe reuse.

Summary

Google Cloud Run provides a managed way to deploy containerized applications that require GPU acceleration.

It can be useful for machine learning inference, image processing, and other workloads that benefit from parallel computation without requiring teams to manage GPU virtual machines directly.

A successful deployment requires more than adding a GPU option to the deployment command. The container image, framework, CUDA runtime, model, region, quota, and resource settings must all work together.

Start with a small GPU-enabled service, verify actual inference execution, and measure performance under realistic workloads before moving it into production.

Google Cloud RunGPU workloadsGoogle CloudPyTorchFastAPICUDADockermachine learning inference

Comments

Join the conversation! Your thoughts help the community grow.