Uploading a file to cloud storage is usually treated as a simple operation:
Application
|
v
Upload File
|
v
Cloud StorageBut there is another question that production systems need to answer:
How do you know the bytes that arrived at Cloud Storage are the same bytes you intended to upload?
A network transfer can involve application code, memory, operating-system buffers, network hardware, routers, proxies, and storage infrastructure. Software bugs, hardware problems, memory errors, or transmission issues can result in corrupted data.
Google Cloud Storage provides checksums to detect this type of corruption. It supports CRC32C and MD5 checksums, with CRC32C being the recommended validation method. Cloud Storage can validate uploaded data and clients can validate downloaded data as well.
For most developers, the important concept is simple:
Source Bytes
|
| Calculate checksum
v
Upload
|
v
Cloud Storage
|
| Calculate / compare checksum
v
Valid ObjectThis article explains how Cloud Storage checksums work, when they matter, how uploads and downloads are validated, and how to build integrity checks into production applications.
What Is a Checksum?
A checksum is a value calculated from a block of data.
For example:
File
|
+-- hello.txt
|
+-- "Hello Cloud Storage"
|
v
Checksum
|
+-- CRC32C = calculated valueIf even part of the file changes, the calculated checksum will generally change.
That gives the application and storage service a way to detect accidental corruption.
A checksum is not encryption.
It does not hide the contents of the file.
It also should not be treated as a security mechanism against an attacker deliberately modifying data. Google recommends TLS/HTTPS for protection against modification by an attacker in transit.
Why Data Can Become Corrupted
Data corruption can happen at different points in a data pipeline.
Consider:
Application
|
v
Memory
|
v
Operating System
|
v
Network
|
v
Cloud StoragePotential causes include:
Software bugs
Hardware errors
Memory errors
Network problems
Router or transmission errors
Problems during long-running uploads
Changes to source data while an upload is in progress
Google Cloud Storage specifically recommends checksums to detect corruption during transfers and to validate object data.
The probability of a problem may be low, but the impact can be high.
A corrupted image may be inconvenient.
A corrupted backup can be much more serious.
A corrupted financial export or machine-learning dataset can create incorrect downstream results.
CRC32C vs MD5
Cloud Storage supports two main checksum types:
Checksum | Primary Use | Cloud Storage Recommendation |
|---|---|---|
CRC32C | Data integrity validation | Recommended |
MD5 | Data integrity validation | Supported in applicable cases |
CRC32C is the recommended validation method for Cloud Storage integrity checks. MD5 is supported for single-file uploads, but it is not supported for certain object types such as composite objects and XML API multipart uploads.
There is also an important terminology detail.
CRC32C is not the same algorithm as generic CRC32.
Cloud Storage uses the CRC32C algorithm, and its object metadata represents the CRC32C value in Base64 encoding.
What Happens During an Upload?
A checksum can be calculated before the upload begins.
Suppose your application has:
report.pdfThe application calculates:
CRC32C(report.pdf)and sends the checksum with the upload request.
Cloud Storage receives the file and calculates its own checksum.
Conceptually:
Client Cloud Storage
File
|
+-- Calculate CRC32C
|
+-- Upload ------------------------>
| |
| +-- Calculate CRC32C
| |
| +-- Compare
| |
| +----+----+
| | |
| Match Mismatch
| | |
| v v
| Store RejectWhen a checksum is supplied with an object write, Cloud Storage compares the supplied checksum with the checksum it calculates. If they do not match, the write request is rejected.
This is stronger than uploading first and discovering the problem much later.
Server-Side Validation
Server-side validation is useful because the storage service validates the upload as part of the write operation.
For example, an HTTP request can provide a CRC32C checksum:
X-Goog-Hash: crc32c=<base64-checksum>The checksum represents the complete object for the relevant upload operation.
If the values do not match, Cloud Storage rejects the request.
Conceptually:
Expected checksum
|
v
Cloud Storage
|
+-- Calculated checksum
|
v
CompareThe application does not need to independently inspect the stored object after every upload just to determine whether the server received the expected bytes.
Example With the Google Cloud CLI
For command-line workflows, Cloud Storage tools already perform integrity validation during object transfers.
A simple upload might look like:
gcloud storage cp report.pdf gs://my-data-bucket/reports/A download might look like:
gcloud storage cp gs://my-data-bucket/reports/report.pdf ./report.pdfThe Google Cloud CLI automatically validates copies to and from Cloud Storage using available checksums. If the source and destination checksums do not match, the invalid copy can be removed and the transfer can be retried.
That means developers using the CLI do not necessarily need to implement their own checksum comparison for ordinary copy operations.
What Happens During a Download?
Integrity validation is not only an upload problem.
Suppose the object is already stored:
Cloud Storage
|
v
report.pdf
|
v
ApplicationThe server can provide the stored checksum along with the object data.
The client calculates the checksum of the bytes it receives and compares it with the server-provided value.
Stored checksum
|
v
Client
^
|
Downloaded bytes
|
v
Calculated checksum
|
v
CompareCloud Storage client libraries support checksum validation for reads and downloads.
If the checksums do not match, the application should treat the downloaded data as invalid and follow an appropriate retry or recovery strategy.
.NET Example
For a .NET application, the Google Cloud Storage client library provides download options that include checksum validation behavior.
A simplified download might look like:
using Google.Cloud.Storage.V1;
var client = await StorageClient.CreateAsync();
await client.DownloadObjectAsync(
"my-data-bucket",
"reports/report.pdf",
"report.pdf");The client library can perform checksum validation as part of the download path.
If you need to explicitly control checksum validation, the download options expose a checksum validation setting:
using Google.Cloud.Storage.V1;
var client = await StorageClient.CreateAsync();
var options = new DownloadObjectOptions
{
ChecksumValidation = DownloadValidationMode.Always
};
await client.DownloadObjectAsync(
"my-data-bucket",
"reports/report.pdf",
"report.pdf",
options);The exact behavior depends on the client library version and transport being used, so production applications should use a current supported Google Cloud Storage .NET client version. Google's documentation currently lists checksum support in the .NET client library and recommends a current release for server-side checksum validation.
Verifying an Object After Upload
There are cases where the application does not know the checksum before uploading.
For example:
Generate File
|
v
Upload
|
v
Object CreatedThe application can retrieve object metadata after the upload and compare the returned checksum with a checksum calculated from the original data.
Conceptually:
var expectedChecksum = CalculateCrc32C(filePath);
await UploadAsync(filePath);
var metadata = await GetObjectMetadataAsync();
if (metadata.Crc32c != expectedChecksum)
{
await DeleteObjectAsync();
throw new InvalidOperationException(
"Object integrity validation failed.");
}This is a client-side validation pattern.
Join the conversation! Your thoughts help the community grow.