How I Benchmark Code

Benchmarking is the process of measuring and baselining the performance of your code. It helps identify bottlenecks in comparing the performance of different algorithms or approaches that target the same set of problems and choosing the one that has optimal time and memory consumption. There are many ways to code the same thing in .NET, so how do you know which one is more performant? There can be big differences that not only affect performance but memory too. Sadly, I have never worked for a company that did benchmarking, unless I wrote the code on my own. Benchmarking the code that you put in the cloud is critically important since, for many of the services, you are charged for the length of time that the code is executing.

For many years for the work for my code performance book and OSS assemblies, I have been using BenchmarkDotNet. I highly recommend using this NuGet package. Microsoft even uses it to benchmark .NET! In previously released versions of Spargine, I have had an assembly to make it easier to use BenchmarkDotNet, but for my work moving Spargine to .NET 6, I have completely rewritten these classes to make it even easier to benchmark your code. The information in this article is from the dotNetTips.Spargine.Benchmarking assembly and NuGet package. The code and NuGet packages can be found below.

GitHub: https://bit.ly/SpargineCode
NuGet: http://bit.ly/dotNetDaveNuGet

To view the benchmark reports for Spargine, go here: https://bit.ly/Spargine6BenchmarkReports

Overview

The dotNetTips.Spargine.Benchmarking assembly not only makes setting up BenchmarkDotNet easier but features classes and methods that pre-generates data that you can use in your benchmark tests. First, let me show you what a benchmark test method looks like.

[Benchmark(Description = "EasyLogger.LogCritical")]
[BenchmarkCategory(Categories.Logging)]
public void LogEasyCriticalBenchmark() 
{
    var testException = new ArgumentInvalidException("TEST EX MESSAGE");
    for (var index = 0; index < this.Count; index++) 
    {
        EasyLogger.LogCritical(this._logger, "CRITICAL ENTRY", testException);
    }
}

The required attribute for a benchmark test is [Benchmark]. I always add a description since it will be used in the reports. In much of my code, I also use [BenchmarkCategory] which is also used in the reports and allows you to run a suite of tests for a given category. The output of this test will look something like this,

How I Benchmark Code

All of the columns above are configured in the dotNetTips.Spargine.Benchmarking.Benchmark class. Below are the classes that make up this assembly.

How I Benchmark Code

Benchmark.cs

The Benchmark abstract class is the core type to help run benchmark tests. This is where the information in the reports is set up along with many helper methods constants and properties. All the data is either constants or loaded when the class is created so that does not interfere with the benchmark test timing.

Constants

Below are the constants defined that I use for many of my benchmark tests.

LongTestString	LowerCaseString	ProperCaseString
TestEmailLowerCase	TestEmailMixedCase	UpperCaseString

Properties

Here are the properties that are preloaded with data.

Base64String	Coordinate01	Coordinate02
CoordinateProper01	CoordinateProper02	JsonTestDataPersonProper
JsonTestDataPersonRecord	PersonProper01	PersonProper02
PersonRecord01	PersonRecord02	String10Characters01
String10Characters02	String15Characters01	String15Characters02
StringEmpty	StringNull	StringToTrim
XmlTestDataPersonProper	XmlTestDataPersonRecord	TestGuid
LaunchDebugger: Set this to true if you want to launch the BenchmarkDotNet debugger.

Methods

Here is a list of the methods for Benchmark and descriptions.

Cleanup (virtual)	Code to clean up the data.
GetByteArray	Returns a byte array in the size requested (in Kb).
GetStringArray	Returns a string array of words using a specified count and the minimum and maximum length of the words can be set.
GlobalCleanup	This is called when the benchmark tests are completed.
GlobalSetup	This is called before the benchmark tests are run. It preloads data for the properties.
Setup (virtual)	This method preloads the data for Benchmark. Call this method from your benchmark classes before any setup that you do.

Benchmark is the core type used throughout this assembly and I use it to benchmark much of the code in Spargine as shown below.

public class TypeHelperBenchmark: Benchmark

CollectionsBenchmark.cs

The CollectionsBenchmark class holds and loads a few different collection types that can be used for benchmarking. The collection types use the models from the dotNetTips.Spargine.6.Tester assembly. These models are listed below,

Coordinate	A typical structure that uses X and Y as coordinates.
CoordinateProper	Is a structure that uses X and Y as coordinates and implements important methods: CompareTo, Equals, GetHashCode. It also overrides ToString and implements default operators.
Person	Typical Person class that has common properties such as FirstName, LastName Email, etc.
PersonProper	Is a Person class that has common properties such as FirstName, LastName, Email, etc., and implements important methods: CompareTo, Equals, and GetHashCode. It also overrides ToString and implements default operators.
PersonRecord	Is a Person record class that has common properties such as FirstName, LastName, Email, etc. This class uses a collection of Address to allow a person to have more than one address.

Constructor

The constructor requires a maxCount that is used when loading the arrays. This number is also used for the MaxCount property.

Methods

Below are the methods and their descriptions.

GetPeopleToInsert	Returns a cloned PersonProper[] that can be used when adding people to a collection.
GetCoordinateArray *	Returns a Coordinate[].
GetCoordinateProperArray *	Returns a CoordinateProper[].
GetPersonRefArray *	Returns reference type Person[].
GetPersonValArray *	Returns a value type Person[].
GetPersonProperArray *	Returns reference type PersonProper[].
GetPersonProperDictionary *	Returns a Dictionary<string, PersonProper>.
GetPersonRecordArray *	Returns a PersonRecord[].

* All these methods have the following parameters.

Tristate clone	If set to Tristate.True or Tristate.UseDefault, will return a cloned copy of the array.
CollectionSize collectionSize	If set to CollectionSize.Full will return the full array or when set to CollectionSize.Half will return half of the array.

LargeCollectionBenchmark.cs

This is the main type (inherits CollectionsBenchmark) that I use whenever I need to benchmark code using collections. It will run benchmarks with the following collection counts: 10, 25, 50, 100, 250, 500, 1000, and 2500. I use this class with many of the benchmark tests for Spargine as shown below.

public class ListExtensionsCollectionBenchmark: LargeCollectionBenchmark

SmallCounterBenchmark.cs

This class inherits Benchmark and sets up running a benchmark test with the following collection counts: 2, 5, 10, 20, 25, 50, 75, 100, and 250. I use it for some of the tests for Spargine as shown below.

public class StringBuilderHelperCounterBenchmark: SmallCounterBenchmark

LargeCounterBenchmark.cs

This class inherits Benchmark and sets up running a benchmark test with the following collection counts: 10, 25, 50, 100, 250, 500, 1000, and 2500. I use it for some of the tests for Spargine as shown below.

public class ListExtensionsCollectionBenchmark: LargeCollectionBenchmark

Setup of the Reports

There are many ways to set up the reports that BenchmarkDotNet will create. This is done in the Spargine Benchmark class. Benchmark will set up the output of most of the common reports that include HTML, GitHub markdown, Json, RP plots, and more. Furthermore, it will set up columns for the reports like the Max, Minimum, Namespace, and more. Here is a list of all the attributes used to set this up in Benchmark.

[AllStatisticsColumn]	[AsciiDocExporter]	[Atlassian]
[BaselineColumn]	[CategoriesColumn]	[ConfidenceIntervalErrorColumn]
[CsvExporter]	[CsvMeasurementsExporter]	[Default]
[DisassemblyDiagnoser]	[EvaluateOverhead]	[Full]
[GcServer(true)]	[GitHub]	[HtmlExporter]
[IterationsColumn]	[JsonExporter]	[KurtosisColumn]
[LogicalGroupColumn]	[MarkdownExporter]	[MaxColumn]
[MemoryDiagnoser]	[MinColumn]	[MValueColumn]
[NamespaceColumn]	[Orderer(SummaryOrderPolicy.Method)]	[PlainExporter]
[RankColumn]	[RPlotExporter]	[SkewnessColumn]
[StackOverflow]	[StatisticalTestColumn]

The reports I use the most are HTML and GitHub markdown. For more information on these attributes, check out the documentation here: https://benchmarkdotnet.org/articles/overview.html

Examples

Here are just a few examples of how I use this assembly when benchmarking code for Spargine and my code performance book.

In most of the benchmarking tests that I write I use the Consumer class in BenchmarkDotNet that will consume the type that you are testing, giving a more realistic result as shown below.

[Benchmark(Description = nameof(XmlSerialization.Deserialize) + ": XML=PersonProper")]
[BenchmarkCategory(Categories.XML)]
public void Deserialize01() 
{
    var result = XmlSerialization.Deserialize < PersonProper > (Resources.PersonProperXml);
    base.Consumer.Consume(result);
}

I have exposed the BenchmarkDotNet Consumer in the Benchmark type.

[Benchmark(Description = "for()", Baseline = true)]
[BenchmarkCategory(Categories.GenericCollections)]
public async Task TestLooping01() 
{
    var channel = new ChannelQueue < PersonProper > ();
    var collection = PersonProperList;
    for (var count = 0; count < collection.Count; count++) 
    {
        await channel.WriteAsync(collection[count]);
    }
    base.Consumer.Consume(channel.Count);
}

[Benchmark(Description = "Combine: From Array with += in a loop")]
public void TestCombineStrings100() 
{
    var result = string.Empty;
    for (var count = 0; count < _stringArray.Length; count++) 
    {
        result += _stringArray[count];
    }
    base.Consumer.Consume(result);
}

[Benchmark(Description = "foreach()")]
[BenchmarkCategory(Categories.GenericCollections)]
public async Task TestLooping02() 
{
    var channel = new ChannelQueue < PersonProper > ();
    var collection = PersonProperList;
    foreach(var person in collection) 
    {
        await channel.WriteAsync(person);
    }
    base.Consumer.Consume(channel.Count);
}

Summary

If you use BenchmarkDotNet, I hope you will find this assembly useful. If you would like anything added, please do a pull request, or submit an issue. If you have any comments or suggestions, please make them below. Happy benchmarking!