🚀 Batch Processing in C# using async and await — The Smart Way to Handle Workloads

Ck Nitin
20h
1.2k
0
1

Article

In modern applications, especially in high-performance systems like e-commerce platforms, video streaming, or large-scale data processing, we often need to handle multiple tasks simultaneously — without blocking the main thread.

That’s where asynchronous programming in C# using async and await comes into play. One common real-world use case is batch processing — executing multiple operations in parallel and waiting for all of them to complete efficiently.

💡 What is Batch Processing?

Batch processing means executing a collection (batch) of operations together rather than one by one.

For example:

Sending bulk emails or SMS to multiple users
Processing multiple API calls
Importing a large CSV file with 10,000 records in chunks of 100
Running multiple image conversions or video compressions

Doing these tasks sequentially would be slow, so instead we execute them asynchronously in parallel batches.

⚙️ Why Use `async` and `await`?

In traditional synchronous processing:

foreach (var item in data)
{
    ProcessItem(item); // Waits for one to complete before starting next
}

This blocks the main thread until each operation finishes — very inefficient for I/O or network-based operations.

With async and await, we can:

Run multiple tasks concurrently
Avoid blocking the main thread
Use system resources efficiently
Improve throughput and scalability

🧠 Real-World Example: Sending Notifications in Batches

Let’s imagine you are building a notification service that sends welcome messages to 1,000 users after registration.
Instead of sending all 1,000 requests sequentially (which can take minutes), you can process them in batches of 100 using async tasks.

✅ Step 1: Sample Data

var userIds = Enumerable.Range(1, 1000).ToList(); // 1000 users

✅ Step 2: Define an Async Method for Each Operation

public async Task SendNotificationAsync(int userId)
{
    // Simulate API call or database write
    await Task.Delay(100); // mimic network delay
    Console.WriteLine($"✅ Notification sent to user {userId}");
}

✅ Step 3: Process in Batches using Async and Await

public async Task ProcessInBatchesAsync(List<int> userIds, int batchSize = 100)
{
    for (int i = 0; i < userIds.Count; i += batchSize)
    {
        var batch = userIds.Skip(i).Take(batchSize);

        // Create list of tasks for this batch
        var tasks = batch.Select(id => SendNotificationAsync(id)).ToList();

        // Run all tasks concurrently
        await Task.WhenAll(tasks);

        Console.WriteLine($"🚀 Batch {i / batchSize + 1} completed!");
    }

    Console.WriteLine("🎉 All notifications sent successfully!");
}

✅ Step 4: Execute It

public static async Task Main(string[] args)
{
    var processor = new NotificationProcessor();
    var userIds = Enumerable.Range(1, 1000).ToList();

    await processor.ProcessInBatchesAsync(userIds, 100);
}

🧩 Explanation

SendNotificationAsync – simulates an I/O-bound operation (e.g., API call, database write).
Task.WhenAll(tasks) – runs all async tasks in a batch concurrently and waits until all are complete.
Batching (Skip and Take) – ensures we don’t overload the system by running too many tasks at once.
await ensures each batch completes before starting the next.

⚡ Advantages

✅ Better performance than sequential processing
✅ Prevents system overload with controlled concurrency
✅ Easy to implement and maintain
✅ Scalable for high-volume workloads

🧰 Advanced Optimization (Optional)

You can further optimize by:

Using SemaphoreSlim for throttling (control max parallel tasks)
Adding retry logic for failed operations
Using Parallel.ForEachAsync (in .NET 6+) for even cleaner syntax

Example with Parallel.ForEachAsync:

await Parallel.ForEachAsync(userIds, async (id, token) =>
{
    await SendNotificationAsync(id);
});

🔍 Real-World Scenarios

Scenario	Description
📩 Email Campaign	Sending thousands of promotional emails in batches
📷 Media Processing	Processing user-uploaded images or videos asynchronously
📊 Data Import	Reading and inserting millions of records in chunks
🔍 API Aggregation	Fetching data from multiple APIs simultaneously

🏁 Conclusion

Batch processing with async and await in C# is a powerful pattern for scaling applications efficiently.
It allows you to:

Run multiple operations in parallel
Control concurrency
Keep applications responsive
Save time and resources

Next time you face a large workload, think in batches + async! 🚀