Large file uploads are common in business applications. Users upload CSV files containing thousands of customer records, Excel sheets with financial data, JSON exports, or documents that need validation before being imported into a database.
A common mistake is to upload the file and process all its contents inside the same ASP.NET Core HTTP request. This approach may work during development with small files, but it can create serious problems when multiple users upload large files at the same time.
The better approach is to separate file uploading from file processing. The API should accept the file, store it safely, create a processing job, and return a job ID. A background worker can then process the file without keeping the HTTP request open.
In this article, we will build this approach using ASP.NET Core, background processing, durable storage, and a job status system.
Why Large File Processing Should Not Run Inside an HTTP Request
Consider an application where users upload CSV files containing customer information. After receiving a file, the API reads every row, validates the data, checks for duplicate records, and saves valid records to the database.
If the file contains a few hundred rows, the process may finish quickly. However, a file containing several hundred thousand rows can take much longer.
Processing everything inside the request creates several problems.
Long-running HTTP requests
HTTP requests are not designed to remain open indefinitely. A reverse proxy, load balancer, hosting platform, or client may terminate a request after its configured timeout.
The server might still be processing the file when the connection closes. The user may not know whether the operation completed successfully.
High memory consumption
Loading an entire file into memory creates unnecessary pressure on the application.
For example:
var fileBytes = await file.OpenReadStream()
.ReadAllBytesAsync();
This approach is unsuitable for large files because the application must hold the file contents in memory.
If several users upload large files simultaneously, memory consumption can increase quickly.
Request resource exhaustion
Long-running operations occupy server resources. Database connections, memory, file handles, and other resources may remain in use while processing continues.
Under heavy traffic, this can affect unrelated API requests.
Duplicate processing
Suppose the server finishes processing a file, but the response never reaches the client. The user may upload the same file again because they believe the first attempt failed.
Without duplicate detection or idempotent processing, the application may import the same records twice.
Use an Asynchronous File Processing Architecture
A more reliable design separates the upload process from the actual processing work.
The API handles the upload and creates a job. A background worker handles the processing.
The workflow looks like this:
Client
|
| Upload file
v
ASP.NET Core API
|
| Validate upload
| Store file
| Create job record
| Publish job message
v
Object Storage + Message Queue
|
| Background worker receives job
v
File Processing Worker
|
| Read file in chunks
| Validate records
| Save results
| Update job status
v
Database
Client --> Job Status API --> Processing Status
Each component has a separate responsibility.
Component | Responsibility |
|---|
ASP.NET Core API | Accept and validate file uploads |
Object storage | Store the uploaded file |
Database | Maintain job status and progress |
Message queue | Notify workers about pending jobs |
Background worker | Process files asynchronously |
Status API | Allow clients to check job progress |
For cloud applications, use durable object storage such as Azure Blob Storage or Amazon S3 instead of relying on a local application directory. Local storage can become unavailable when an application instance restarts or another instance receives the background job.
Create a File Processing Job Model
Before implementing the upload endpoint, define a model that tracks the processing operation.
public enum FileJobStatus
{
Pending,
Processing,
Completed,
Failed,
Cancelled
}
public class FileProcessingJob
{
public Guid Id { get; set; }
public string FileName { get; set; } = string.Empty;
public string StorageKey { get; set; } = string.Empty;
public FileJobStatus Status { get; set; }
public int ProcessedRecords { get; set; }
public int FailedRecords { get; set; }
public string? ErrorMessage { get; set; }
public DateTime CreatedAtUtc { get; set; }
public DateTime? StartedAtUtc { get; set; }
public DateTime? CompletedAtUtc { get; set; }
}
The job record stores information about the operation, not the actual file contents.
The StorageKey identifies the file in object storage. The processing status helps the API report whether the job is waiting, running, completed, or failed.
For production applications, consider adding tenant information, retry count, a lease expiration time, and a row version for concurrency control.
Upload Files Without Processing Them
The upload endpoint should perform only the work required to accept the file safely.
The following example uses IFormFile and a storage abstraction. The storage implementation is responsible for writing the file to durable object storage.
[ApiController]
[Route("api/files")]
public class FileUploadController : ControllerBase
{
private readonly IFileStorageService _storage;
private readonly IFileJobService _jobService;
public FileUploadController(
IFileStorageService storage,
IFileJobService jobService)
{
_storage = storage;
_jobService = jobService;
}
[HttpPost("upload")]
[RequestSizeLimit(100_000_000)]
public async Task<IActionResult> Upload(
IFormFile file,
CancellationToken cancellationToken)
{
if (file == null || file.Length == 0)
{
return BadRequest("Please select a valid file.");
}
if (file.Length > 100_000_000)
{
return BadRequest("File size exceeds the allowed limit.");
}
var extension = Path.GetExtension(file.FileName);
if (!string.Equals(
extension,
".csv",
StringComparison.OrdinalIgnoreCase))
{
return BadRequest("Only CSV files are supported.");
}
var jobId = Guid.NewGuid();
var storageKey = $"imports/{jobId:N}.csv";
await using var fileStream = file.OpenReadStream();
await _storage.UploadAsync(
storageKey,
fileStream,
cancellationToken);
var job = new FileProcessingJob
{
Id = jobId,
FileName = Path.GetFileName(file.FileName),
StorageKey = storageKey,
Status = FileJobStatus.Pending,
CreatedAtUtc = DateTime.UtcNow
};
await _jobService.CreateJobAsync(
job,
cancellationToken);
await _jobService.EnqueueAsync(
jobId,
cancellationToken);
return AcceptedAtAction(
nameof(GetJobStatus),
new { jobId },
new
{
JobId = jobId,
Status = "Pending",
Message = "File uploaded and queued for processing."
});
}
[HttpGet("jobs/{jobId:guid}")]
public async Task<IActionResult> GetJobStatus(
Guid jobId,
CancellationToken cancellationToken)
{
var job = await _jobService.GetJobAsync(
jobId,
cancellationToken);
if (job == null)
{
return NotFound();
}
return Ok(job);
}
}
The important part is that the endpoint does not read or process every CSV record. It uploads the file, creates a job, and returns an HTTP 202 Accepted response.
The client receives a job ID and can use it to check the processing status.
Important considerations for large uploads
This example uses IFormFile for clarity, but it is not automatically a fully streaming upload implementation. ASP.NET Core multipart form handling can buffer uploaded content in memory or temporary files, depending on configuration and file size.
For genuinely large uploads, use a streaming multipart implementation with MultipartReader, or allow clients to upload directly to private object storage using a short-lived, restricted upload URL.
Configure request size limits, multipart limits, proxy limits, and storage limits consistently. Do not rely only on checking file.Length after the request has already been accepted.
Also, never use the original filename as a storage path. Generate a unique storage key and treat the original name as display information only.
Use a Background Worker to Process Files
Once the upload is complete and the job is queued, a background worker can process the file.
For a simple application, ASP.NET Core provides BackgroundService. It is useful for hosted background work, but an in-memory queue alone is not durable. Pending work can be lost when the application restarts.
For production workloads, use a durable message broker such as RabbitMQ or Azure Service Bus. The worker can consume messages from the queue and retrieve the corresponding file from object storage.
A simplified worker might look like this:
public class FileProcessingWorker : BackgroundService
{
private readonly IFileJobQueue _queue;
private readonly IFileJobService _jobService;
private readonly IFileStorageService _storage;
private readonly IServiceScopeFactory _scopeFactory;
public FileProcessingWorker(
IFileJobQueue queue,
IFileJobService jobService,
IFileStorageService storage,
IServiceScopeFactory scopeFactory)
{
_queue = queue;
_jobService = jobService;
_storage = storage;
_scopeFactory = scopeFactory;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var jobId = await _queue.DequeueAsync(
stoppingToken);
if (jobId == null)
{
continue;
}
await ProcessJobAsync(
jobId.Value,
stoppingToken);
}
}
private async Task ProcessJobAsync(
Guid jobId,
CancellationToken cancellationToken)
{
var job = await _jobService.GetJobAsync(
jobId,
cancellationToken);
if (job == null)
{
return;
}
try
{
await _jobService.MarkProcessingAsync(
jobId,
cancellationToken);
await using var stream =
await _storage.OpenReadAsync(
job.StorageKey,
cancellationToken);
await using var reader =
new StreamReader(stream);
var processed = 0;
while (await reader.ReadLineAsync(
cancellationToken) is { } line)
{
if (string.IsNullOrWhiteSpace(line))
{
continue;
}
await ProcessRecordAsync(
line,
cancellationToken);
processed++;
if (processed % 500 == 0)
{
await _jobService.UpdateProgressAsync(
jobId,
processed,
cancellationToken);
}
}
await _jobService.MarkCompletedAsync(
jobId,
processed,
cancellationToken);
}
catch (OperationCanceledException)
when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
await _jobService.MarkFailedAsync(
jobId,
ex.Message,
CancellationToken.None);
}
}
private async Task ProcessRecordAsync(
string record,
CancellationToken cancellationToken)
{
// Parse, validate, and save the record.
await Task.CompletedTask;
}
}
This example demonstrates the processing flow. The queue, storage service, and job service are abstractions that need production implementations.
The worker reads the file line by line instead of loading the complete file into memory. It also updates progress periodically rather than writing to the database after every record.
For real CSV processing, use a CSV parser that correctly handles quoted fields, escaped delimiters, and embedded line breaks. Splitting a CSV row using string.Split(',') is not reliable for general CSV data.
Process Records in Batches
Reading a file line by line reduces memory consumption, but saving every record individually can create unnecessary database overhead.
A better approach is to collect a limited number of validated records and save them in batches.
var batch = new List<CustomerImportRecord>();
while (await reader.ReadLineAsync(
cancellationToken) is { } line)
{
var record = ParseCustomerRecord(line);
if (record == null)
{
continue;
}
batch.Add(record);
if (batch.Count >= 500)
{
await repository.SaveBatchAsync(
batch,
cancellationToken);
batch.Clear();
}
}
if (batch.Count > 0)
{
await repository.SaveBatchAsync(
batch,
cancellationToken);
}
The batch size should be based on database performance, record size, transaction limits, and available memory.
A batch size of 500 is only an example. Measure the application under realistic workloads before choosing a production value.
For applications where partial imports are allowed, record invalid rows separately and continue processing valid records. If the import must be all-or-nothing, use a staging table and a controlled final transaction or promotion step.
Make Job Creation Reliable
There is an important problem in the upload example.
Suppose the application saves the job record successfully but fails before publishing the queue message. The database contains a pending job, but no worker receives it.
The opposite situation can also happen if a message is published before the job transaction is committed.
The transactional outbox pattern helps solve this problem.
In a single database transaction, save both the job record and an outbox message. A separate publisher reads unpublished outbox records and sends them to the message broker. After successful publication, it marks the outbox message as published.
This makes job creation and message publication recoverable without requiring a distributed transaction.
Workers must also be idempotent. Message brokers can deliver messages more than once, especially when a worker crashes after completing work but before acknowledging the message.
Before processing, the worker should acquire the job using an atomic state transition or lease. A completed job should not be processed again simply because an old message was delivered.
For database imports, use a stable business key or an import-specific idempotency key to prevent duplicate records.
Track Processing Progress
Users should not have to guess whether their uploaded file is still being processed.
A status endpoint can return information such as:
{
"jobId": "9d0c4d7e-0e2a-4c4a-bc4f-0d2e8c1f9a31",
"status": "Processing",
"processedRecords": 12500,
"failedRecords": 18,
"createdAtUtc": "2026-11-02T08:30:00Z",
"startedAtUtc": "2026-11-02T08:30:04Z"
}
The frontend can periodically call the status endpoint to display progress.
For applications that need immediate updates, SignalR can notify connected clients when the job status changes.
Do not update progress for every row. Frequent database writes can become a performance problem. Update after a fixed number of records or after a reasonable time interval.
If the total number of records is unknown, show the number processed rather than displaying an inaccurate percentage.
Handle Failures and Retries
Background processing does not mean failures disappear. Storage services, databases, message brokers, and file parsers can all fail.
A production implementation should consider the following cases:
Failure | Recommended handling |
|---|
Temporary database outage | Retry with exponential backoff |
Object storage unavailable | Retry the storage operation |
Invalid file structure | Mark the job as failed |
Invalid individual record | Record the error and continue if partial imports are allowed |
Worker restart | Allow the durable queue to redeliver unacknowledged messages |
Repeated processing failure | Move the message to a dead-letter queue |
Duplicate queue message | Use job state checks and idempotent operations |
Do not retry every exception indefinitely. A malformed file will not become valid after another attempt.
Separate transient errors from permanent errors. Configure a maximum retry count and preserve enough diagnostic information to investigate failures.
Avoid exposing internal exception messages, database details, or storage paths through the public status API. Store detailed diagnostics in protected logs and return a safe error message to the client.
Security Considerations for File Processing
File uploads create a security boundary. A file extension alone does not prove that a file contains valid data.
Apply these controls:
Validate file size, extension, expected format, and actual content.
Generate storage keys instead of trusting user-provided filenames.
Store uploaded files in private storage.
Verify that the requesting user owns the job before returning its status.
Scan files for malware when the application handles untrusted uploads.
Restrict archive extraction size and file count to protect against archive bombs.
Apply parser limits to avoid excessively large fields, deeply nested JSON, or malformed documents.
Delete temporary files and expired uploads according to a defined retention policy.
Avoid logging sensitive file contents or personal information.
If the application processes files for multiple customers, include tenant ownership in job queries and storage access checks. A user should never be able to retrieve another tenant's file by changing a job ID.
Common Mistakes to Avoid
Loading the complete file into memory
Avoid ReadAllBytes, File.ReadAllText, or copying an entire large file into a MemoryStream when streaming is possible.
Using an in-memory queue for critical jobs
An in-memory queue is convenient for demonstrations, but pending messages may disappear when the application restarts. Use durable messaging for important processing tasks.
Running too many jobs simultaneously
Unrestricted concurrency can overload the database, storage service, or application instance. Configure worker concurrency based on available resources and downstream service capacity.
Treating every failure as retryable
Invalid file formats and business validation failures should usually be handled as permanent failures. Retrying them wastes resources.
Keeping uploaded files forever
Define a retention policy. Delete files after successful processing when they are no longer needed, or retain them for a documented audit period.
Ignoring job ownership
Every status, cancellation, and download endpoint should verify the caller's authorization to access the job.
How to Test Large File Processing
Testing with a small sample file is not enough. Test the complete workflow under realistic conditions.
Include these scenarios:
Upload a file close to the configured size limit.
Upload multiple files concurrently.
Include malformed records and unexpected data types.
Disconnect the client during an upload.
Restart a worker while a job is running.
Deliver the same queue message more than once.
Simulate temporary database and object storage failures.
Verify that a failed job can be investigated and safely retried.
Measure memory usage while processing files of different sizes.
Monitor upload duration, queue waiting time, processing duration, records processed per second, failure rates, worker concurrency, memory consumption, and temporary storage usage.
These measurements help identify whether the bottleneck is file reading, validation, database writes, or downstream services.
Summary
Large file processing should not depend on an HTTP request remaining open until every record has been processed.
A better ASP.NET Core design accepts the upload, stores the file in durable object storage, creates a processing job, and returns HTTP 202 Accepted with a job ID. A background worker then reads the file in manageable chunks, validates the data, saves records in batches, and updates the job status.
For production applications, combine durable messaging, transactional outbox publishing, idempotent processing, controlled concurrency, retry policies, and proper file security.
This design keeps API requests responsive, limits memory consumption, and makes long-running file imports easier to monitor, recover, and scale.
Join the conversation! Your thoughts help the community grow.