Your benchmarks are so wrong!

TL;DR – If you are measuring a relative execution time of 2 pieces of code, make sure to run under the Visual Studio debugger.

Disclaimer: this post is relevant to Visual Studio debugger. I did not check it on other platforms or debuggers.

How many times have you tested and bench-marked some operation in your code? How many

Yesterday i read a post on the internet that attempted to benchmark 2 pieces of code that perform the same task in C#, and compare the 2 solutions for the same problem (comparing execution time complexity) to decide which is faster. The methodology was to create 2 ‘for’ loops and run them a million times each, one after the other in the same program, when each of the loops measuring it’s own execution time beginning to end.

The outcome was unequivocal, the second loop completed twice as faster than the first one! Amazing! “Not so fast buddy” was my initial thought.

I was skeptic, on the face of it the 2 loops looked like they should perform more or less the same (time-wise). I decided to run the same test on my machine.

Results:

Execution time of the 2 loops on my machine was almost the same (with very minor differences).

The findings:

First, i noticed he ran his code without the debugger. He simply pressed “Ctrl+F5” and ran the program outside of the debugger. Well, i ran mine in debug mode. Interesting. Could that be the answer? Do you see the problem already?

The technical explanation:

When you run your code from Visual Studio when the studio’s debugger is attached, the process is optimized to run at a constant priority which is pretty high. Here are the 2 loops on my machine, notice how the processor is stable in the amount of attention it dedicates to my process. (the drop before the end is the end of the program)

WithDebugger

Now take a look how the same graph looks like when i run the same program with the debugger detached. It seems as if the process starts slow and then the system increases the attention it dedicates to the process and the execution speed almost doubles.

NoDebugger

Amazing, isn’t it?

The conclusion:

Obviously, if the process priority gets higher when executing the second loop, it will seem as if the second loop performed much faster. This will lead to a complete false benchmark result.

Solution:

The best way to compare the 2 loops is by running them under the debugger of Visual Studio. Another way is to insert a dumb loop at the beginning of your program to “burn” the unstable priority time and continue when the CPU gets up to speed and starts executing at constant priority.

If you are angry, mad or just happy about this post and want to share it with me – leave a comment.

Code on,
Shonn Lyga.

About Shonn Lyga

Obsessed with anything and everything in Software Engineering, Technology and Science
This entry was posted in .NET and tagged , , , , . Bookmark the permalink.

2 Responses to Your benchmarks are so wrong!

  1. nerushdennis says:

    I’m not sure that a debug based benchmark will be correct since your code won’t run in debug mode in production, furthermore when using a debugger Visual Studio doesn’t just run the code it also collects data about the local variables, background events (and many more if your intellitrace is running). Keep in mind that in order to debug Visual Studio decompiles the target DLLs and translates the PDB files so that you could see the source code in C#.
    All these will take time and increase the time you think that your code runs which will make your benchmark also falsy.

    I think that a true benchmark should be when the code runs in release mode in an environment as similar to the production environment as possible.

    Like

  2. Shonn Lyga says:

    @nerushdennis – you are correct, but i was referring to relative benchmarking of 2 pieces of code in the same program. The relative performance differences will be wrong when using release mode without attaching the debugger.

    Regarding absolute performance benchmarking – your comment is correct, release mode with appropriate environment would be the best way.

    (I fixed the TL;DR to clarify that the post is about relative performance of 2 pieces of code)

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s