Imagine you are building a .NET Web API that checks whether a customer email already exists in SQL Server. If thousands of requests arrive every minute, repeatedly querying the database can increase its workload.

A Bloom Filter can help reduce unnecessary database queries.

What Is a Bloom Filter?

A Bloom Filter is a memory-efficient data structure that quickly checks whether an item might exist in a collection.

It provides two possible results:

  • Definitely not present: The item has not been added to the filter.
  • Possibly present: The item might exist, so we need to verify it against the database.

A Bloom Filter can produce false positives, but it does not produce false negatives for items that have been successfully added, provided the filter is functioning correctly.

How Does It Work?

A Bloom Filter uses a bit array and multiple hash functions.

Initially, all bits are set to 0.

When we add [email protected], the hash functions calculate positions in the bit array. Suppose the positions are 5, 12, and 20. The filter changes those bits to 1.

When we check another email, the filter calculates its positions again.

If any position contains 0, the email is definitely absent from the filter. If all positions contain 1, the email might exist.

Different emails can share the same positions. Therefore, the filter can sometimes report that an email exists when it does not.

Example Using a .NET Web API

Source code can be downloaded from Git

Suppose SQL Server contains customer emails. We can create a Bloom Filter and populate it with existing emails.

var filter = new BloomFilter(
    size: 100_000,
    hashCount: 7);

filter.Add("[email protected]");
filter.Add("[email protected]");
filter.Add("[email protected]");
Here, BloomFilter represents our implementation of the bit array and hash functions.
Now, our API can check an email before querying SQL Server.
app.MapGet("/customers/check/{email}",
    async (string email,
           BloomFilter filter,
           AppDbContext db) =>
{
    if (!filter.MightContain(email))
        return Results.NotFound();

    var customer = await db.Customers
        .FirstOrDefaultAsync(c => c.Email == email);

    return customer is null
        ? Results.NotFound()
        : Results.Ok(customer);
});

If the filter says the email is definitely absent, the API skips the database query. If it might exist, the API queries SQL Server to confirm.

How Do We Keep the Filter Synchronized?

This is an important part of a production application.

Suppose a new customer, [email protected], is inserted into SQL Server, but the filter is not updated. The filter might incorrectly report that Peter's email is absent.

We can avoid this problem by following a reliable update process.

  1. Insert the customer into SQL Server.
  2. Save an event in an Outbox table within the same database transaction.
  3. A background worker reads the event and adds the email to the Bloom Filter.
  4. If processing fails, retry the event.
  5. Rebuild the filter from SQL Server when necessary, such as after an application restart.

The Transactional Outbox pattern helps ensure that a successful database transaction does not lose its corresponding update event.

If multiple API instances run independently, each instance must also receive the updates or use an appropriate shared solution.

When Should We Use a Bloom Filter?

Bloom Filters are useful when applications frequently check membership in large datasets. Examples include checking whether a URL has been processed, reducing unnecessary cache lookups, and screening customer records.

However, a Bloom Filter is not a replacement for SQL Server indexes. If indexed database lookups are already fast, adding a Bloom Filter may introduce unnecessary complexity.

For duplicate customer registration, always enforce uniqueness with a unique database index. The Bloom Filter is only an optimization.

Conclusion

A Bloom Filter helps reduce unnecessary database queries by quickly identifying items that are definitely absent from its filter.

The key principle is simple: a Bloom Filter can tell you when an item is definitely absent, but it cannot prove that an item exists.

For production applications, combine it with reliable synchronization, background processing, and database validation to get the performance benefits without compromising data correctness.

Happy Coding!