Performance Benchmarking With BenchmarkDotNet

Introduction

Benchmarking is the process of measuring and baselining performance of your code. It is helpful in identifying bottlenecks and in comparing performance of different algorithms or approaches that target same set of problem(s) and choosing the one that has optimal time and memory consumption. BenchmarkDotNet is a popular library that makes it simple and elegant to write and run benchmarks and export and share them. In this article, we will explore the basics of how to use BenchmarkDotNet to measure the performance of .NET console application code.

Setup

We will measure the performance of various ways to sort an array of strings in descending order. Create a .NET console application and install BenchmarkDotNet nuget package.

Add a class called Sorter and a method that can sort the array in descending order using the bubble sort algorithm as follows,

public class Sorter {
    private string[] input;
    public Sorter(string[] array) {
        input = array;
    }
    public void BubbleSort() {
        for (int i = 0; i < input.Length - 1; i++) {
            for (int j = i + 1; j < input.Length; j++) {
                if (input[i].CompareTo(input[j]) < 0) {
                    (input[j], input[i]) = (input[i], input[j]);
                }
            }
        }
    }
}

Create a class called SortBenchmark as follows,

[MemoryDiagnoser]
public class SortBenchmark
{
    private static string[] stringArr = new[] { "Bar", "Baz", "Foo", "Qux", "Abc", "Zxy", "Try", "Cab" };
    public static readonly Sorter arraySorter = new Sorter(stringArr);

    [Benchmark]
    public void BubbleSort() => arraySorter.BubbleSort();
}

MemoryDiagnoser attribute captures the GC and memory consumption information. Benchmark attribute marks the method for benchmarking measurements.

Inside the Program class write the following code to run the benchmarks when the project is executed:-

BenchmarkRunner.Run<SortBenchmark>();

Benchmarking

Before running the benchmarks, build the project in release mode. Never use debug mode for benchmarking as debug mode is not optimized and hence can produce misleading results.

dotnet build -c Release

Run the benchmarks using dotnet CLI command. Replace <path_to_release_dll> with the path of the project DLL file (including the DLL file name) inside bin/Release folder.

dotnet "<path_to_release_dll>"

If you are running this inside Visual Studio, make sure to change the build configuration from Debug to Release.

After benchmarking is complete, you can see the time and memory consumption information for all the benchmarks in your code. The bubble sort method takes a mean time of 1.218 microseconds to perform sorting and does not allocate any additional memory.

Let’s add few more ways to sort the array inside Sorter class.

public class Sorter {
    private string[] input;
    public Sorter(string[] array) {
        input = array;
    }
    public void BubbleSort() {
        for (int i = 0; i < input.Length - 1; i++) {
            for (int j = i + 1; j < input.Length; j++) {
                if (input[i].CompareTo(input[j]) < 0) {
                    (input[j], input[i]) = (input[i], input[j]);
                }
            }
        }
    }
    public void BuiltInSortWithComparison() {
        Array.Sort(input, (a, b) => b.CompareTo(a));
    }
    public void SortWithLINQ() {
        input = input.OrderByDescending(s => s).ToArray();
    }
}

Add the corresponding benchmarks to SortBenchmark class. You can also choose one of the benchmark methods as baseline. Doing so will show you the mean value of ratio distribution against the baseline in the output.

[MemoryDiagnoser]
[Orderer(SummaryOrderPolicy.FastestToSlowest)]
[RankColumn]
public class SortBenchmark
{
    private static string[] stringArr = new[] { "Bar", "Baz", "Foo", "Qux", "Abc", "Zxy", "Try", "Cab" };
    public static readonly Sorter arraySorter = new Sorter(stringArr);

    [Benchmark]
    public void BubbleSort() => arraySorter.BubbleSort();

    [Benchmark]
    public void BuiltInSortWithComparison() => arraySorter.BuiltInSortWithComparison();

    [Benchmark(Baseline = true)]
    public void SortWithLINQ() => arraySorter.SortWithLINQ();
}

Order attribute orders the output from fastest to slowest performing benchmarks. RankColumn attribute adds a Rank column to the output.

Build and execute the code again to see the benchmarking output inside the terminal.

Clearly, built-in sort algorithms are faster than the bubble sort algorithm. Sorting with LINQ uses additional memory as it does not perform an in memory sorting like other algorithms.

BenchmarkDotNet produces log files and benchmark output in different formats inside BechmarkDotNet.Artifacts folder inside your project. The results subfolder contains latest benchmark output in CSV, HTML, and markdown formats.

Summary

BenchmarkDotNet is a powerful library that enables you to measure the performance of your code by writing simple benchmark methods in a way that is very similar to writing unit tests. The library has various configuration options in order to collect additional statistics and run benchmarks based on different characteristics like platform, toolchain, etc. Check out their documentation for more information.

References

https://benchmarkdotnet.org/index.html
https://www.youtube.com/watch?v=EWmufbVF2A4