Stop Sending Large Files Through Your ASP.NET Core API: A Better File Upload Architecture

Direct-to-Storage vs. API-Proxy Uploads — Which Approach Scales Better?

Introduction

Imagine you're building an ASP.NET Core Web API for a school management platform.

Parents need to upload student documents such as birth certificates, identification documents, and academic records.

Each parent can upload up to 10 files, with a maximum size of 10 MB per file.

Everything works perfectly during development.

But what happens when hundreds of users start uploading documents simultaneously?

Your API suddenly needs to handle hundreds of active file transfers.

Even if you use streaming to reduce memory consumption, the file data still travels through your API infrastructure.

This raises an important architectural question:

Should your backend handle the actual file bytes, or should it only authorize the upload and let cloud storage handle the transfer?

Let's compare two approaches.

1. Two Common File Upload Architectures

There are two widely used approaches to uploading files in modern web applications.

Approach A: Direct-to-Storage Upload

The client uploads the file directly to cloud storage using a temporary signed URL.

               Client / Browser
                 |          |
                 |          |
     1. Auth     |          | 3. Upload file
     request     |          | directly
                 v          v
          ASP.NET Core     Amazon S3
             API              ^
                 |            |
                 +------------+
                   2. Generate
                   signed URL

The backend handles authentication, authorization, and upload metadata.

The file bytes go directly from the client to Amazon S3.

Approach B: API-Proxy Upload

The client sends the file to your ASP.NET Core API, which forwards it to storage.

Client / Browser
       |
       | 1. Upload file
       v
ASP.NET Core API
       |
       | 2. Forward file
       v
Amazon S3

The backend participates in the entire file transfer.

Both approaches are valid, but they have different performance, security, and operational trade-offs.

2. Why Streaming Doesn't Completely Solve the Problem

A common assumption is:

"If I use streaming in ASP.NET Core, large file uploads won't affect my API."

Streaming certainly helps, but we need to understand what it actually solves.

Consider:

[HttpPost("upload")]
public async Task<IActionResult> Upload(
    IFormFile file,
    CancellationToken cancellationToken)
{
    await using var stream = file.OpenReadStream();

    await _s3Client.UploadObjectFromStreamAsync(
        stream,
        file.FileName,
        cancellationToken);

    return Ok();
}

This is a conceptual example; UploadObjectFromStreamAsync represents an application-defined storage abstraction rather than a built-in AWS SDK method.

There is another important detail: IFormFile is typically used with buffered model binding. Calling OpenReadStream() doesn't, by itself, make the entire request pipeline an unbuffered streaming implementation.

For genuinely streaming large uploads in ASP.NET Core, you can process the request body or multipart sections directly, subject to appropriate request limits and validation.

But even with true streaming, the file still passes through the API.

The backend continues to consume:

Incoming network bandwidth
Outgoing network bandwidth to storage
An active request and associated resources
Connection capacity and infrastructure resources

Streaming reduces memory pressure, but it doesn't remove the backend from the data path.

That's the fundamental difference.

3. Why Direct-to-Storage Uploads Are Often Better for Large Files

In a direct-upload architecture, the backend doesn't transfer the file itself.

Instead, it grants temporary permission to upload a specific object.

Let's understand the complete workflow.

Step 1: Authenticate and Authorize

The user sends an authenticated request:

POST /api/documents/upload-request
Authorization: Bearer <JWT_TOKEN>

Request body:

{
  "studentId": 101,
  "fileName": "certificate.pdf",
  "contentType": "application/pdf",
  "fileSize": 5242880
}

The backend validates the user's permissions and the requested upload metadata.

For example, it verifies that the parent is authorized to upload documents for student 101.

Step 2: Create a Pending Upload Record

Before generating the signed URL, the backend creates a record in SQL Server.

Example:

Id	StudentId	Status	ObjectKey
501	101	Pending	students/101/abc123

This record represents an authorized upload that hasn't yet been verified as completed.

Step 3: Generate a Pre-Signed URL

The backend generates a short-lived S3 pre-signed URL for the specific object key.

Example:

var request = new GetPreSignedUrlRequest
{
    BucketName = bucketName,
    Key = objectKey,
    Verb = HttpVerb.PUT,
    Expires = DateTime.UtcNow.AddMinutes(5),
    ContentType = "application/pdf"
};

string uploadUrl = s3Client.GetPreSignedURL(request);

The client receives the URL and the upload identifier.

The URL grants temporary permission for the intended S3 operation.

It shouldn't grant broad bucket access.

Step 4: Upload Directly to S3

The browser uploads the file:

const response = await fetch(uploadUrl, {
    method: "PUT",
    headers: {
        "Content-Type": file.type
    },
    body: file
});

if (!response.ok) {
    throw new Error("Upload failed");
}

Notice that the file doesn't pass through the ASP.NET Core API.

Amazon S3 receives the file directly.

Step 5: Verify the Upload

After the browser reports success, the client calls:

POST /api/documents/501/complete

The backend retrieves the pending upload record and checks the corresponding S3 object.

It verifies information such as:

Expected bucket and object key
Actual uploaded size
Expected metadata
Upload ownership and authorization
Required content inspection or malware-scanning status

Only after the required checks pass should the application mark the file as ready.

Step 6: Mark the Document as Ready

Pending
   |
   v
Uploaded
   |
   v
Validation / Scanning
   |
   v
Ready

If verification fails, the application can reject the upload and clean up the object.

This is an important distinction:

A successful S3 upload doesn't automatically mean the file is safe or accepted by your application.

4. Direct-to-Storage vs. API-Proxy: Side-by-Side Comparison

Feature	Direct-to-Storage	API-Proxy
File passes through API	No	Yes
API bandwidth usage	Lower	Higher
Backend memory usage	Low for upload coordination	Can be low with streaming
File inspection	Usually after upload	Can inspect during transfer
Authentication	API authorizes URL issuance	API authorizes upload request
Upload validation	Before and after transfer	During and/or after transfer
Implementation complexity	Requires signed URLs and completion workflow	Simpler for basic uploads
Large-file scalability	Often better	Depends on infrastructure capacity
Best use case	Large or frequent uploads	Small files or inline inspection

Direct-to-storage is often preferable when file transfers are large, frequent, or expensive for the API infrastructure.

However, it's not automatically the right choice for every application.

5. Security: A Signed URL Is a Temporary Credential

One of the most important architectural considerations is security.

A pre-signed URL should be treated like a temporary bearer credential.

Anyone who obtains a valid URL may be able to perform the authorized operation until the URL expires or becomes otherwise invalid.

Therefore, follow these principles:

Use short expiration periods. Give clients enough time to start the upload without unnecessarily extending the URL's lifetime.

Generate object keys on the server. Don't trust arbitrary client-provided storage paths.

Verify ownership before signing. Confirm that the authenticated user is authorized to upload the requested document.

Restrict the operation. Generate a URL for the intended bucket, object key, HTTP method, and applicable signed headers.

Enforce file restrictions. For strict upload size limits, consider S3 pre-signed POST policies with content-length-range, rather than trusting a client-declared file size.

Keep the bucket private. Use appropriate IAM permissions, bucket policies, and S3 Block Public Access.

Verify uploaded content. File extensions and declared MIME types are not proof of actual content. Perform content validation and malware scanning where required.

One subtle issue is that a pre-signed PUT URL isn't inherently single-use. Reusing it may overwrite the same object while the URL remains valid.

For stronger workflows, consider conditional requests, object versioning, and application-side state controls.

6. What Happens If the Upload Succeeds but the Database Update Fails?

This is a common distributed-systems problem.

Consider:

The backend creates a pending upload record.
The browser successfully uploads the file to S3.
The client calls the completion endpoint.
The database becomes temporarily unavailable.
The application cannot mark the document as ready.

Now the S3 object exists, but the database record remains pending.

How Do We Handle This?

Design the completion endpoint to be safely retryable.

The backend should be able to check whether the object exists and finish the pending record later.

A scheduled cleanup or reconciliation process can inspect abandoned uploads and resolve inconsistencies.

For example:

Pending Uploads
       |
       v
Reconciliation Worker
       |
       +-- Object exists and valid
       |       |
       |       v
       |   Complete record
       |
       +-- Missing or invalid
               |
               v
         Reject / Clean up

This is a practical example of handling eventual consistency between your application database and external object storage.

7. When Should You Still Upload Through Your API?

Direct-to-storage uploads are useful, but API-proxy uploads still have valid use cases.

For example:

Small files with modest traffic
Applications that must inspect or transform bytes during upload
Systems requiring tightly controlled ingestion gateways
Workflows where clients cannot communicate directly with cloud storage
Applications where a simple upload endpoint is sufficient

An API proxy also provides a centralized place for applying custom validation, transformations, or content filtering before forwarding data.

The correct choice depends on your requirements, not simply the file size.

8. Real-World Example: School Document Management

Consider a school management platform with multiple schools, students, and parents.

Each parent can upload documents for their children.

A practical architecture might use:

ASP.NET Core Web API for JWT authentication, school-based authorization, upload policy validation, and signed URL generation.

SQL Server for storing document metadata, ownership, upload status, and audit information.

Amazon S3 for storing private document objects.

Background workers for cleanup, reconciliation, and optional malware scanning.

Pre-signed GET URLs for short-lived, authorized downloads.

The backend must verify both the student's school and the requesting user's relationship to that student before issuing an upload or download URL.

This design separates business authorization from file transfer and storage responsibilities.

9. Senior .NET Developer Interview Questions

Q1. Why is direct-to-S3 upload more scalable than sending files through the API?

It removes file-byte transfer from the backend data path, reducing API bandwidth consumption and the resources needed to maintain long-running upload requests.

Q2. Does streaming eliminate the overhead of API-proxy uploads?

No. Streaming can reduce memory buffering, but the API still receives and forwards the file data.

Q3. Why create a pending upload record before generating the signed URL?

It allows the application to track authorized upload intent, associate the object with its owner, and verify completion before making the file available.

Q4. Can we trust the file size sent by the browser?

No. Client-supplied metadata must be treated as untrusted.

Enforce restrictions using appropriate storage-side policies and verify the uploaded object.

Q5. What happens if a user shares a pre-signed URL?

Someone with the URL may be able to use it while it remains valid. That's why signed URLs should be short-lived, narrowly scoped, and protected from unnecessary exposure.

Q6. How would you handle failed uploads?

Use retryable operations, pending upload records, expiration policies, and background cleanup or reconciliation.

Q7. How do you prevent users from accessing another student's documents?

Verify the authenticated user's authorization against trusted student and document records before issuing signed URLs.

Conclusion

When designing file uploads in ASP.NET Core, one important question is often overlooked:

Does the backend really need to handle the file bytes?

For small files or workflows requiring inline inspection, API-proxy uploads can be a reasonable choice.

But for large or frequent uploads, direct-to-storage architecture can reduce backend bandwidth usage and improve scalability.

The backend remains responsible for authentication, authorization, upload intent, and verification.

Cloud storage handles the actual transfer.

Key Takeaway

Let your API decide who can upload. Let cloud storage handle the file transfer. And never trust an upload until you've verified what actually arrived.

Good architecture isn't just about making the system faster. It's about assigning responsibilities to the right components.

Official References

AWS S3 — Uploading objects with pre-signed URLs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html
AWS S3 — Browser-based POST uploads: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html
Microsoft Learn — File uploads in ASP.NET Core: https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads
AWS S3 — Multipart upload overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

#DotNet #ASPNETCore #AWS #AmazonS3 #SystemDesign #FileUpload #BackendDevelopment #CloudArchitecture #CSharp #SoftwareEngineering #S3Bucket #AWSS3