[ The PC Guide | Articles and Editorials | Benchmark This! ]

The Origins of Benchmarking

Computer benchmarking was originally the realm of computer scientists and systems engineers. By running tests designed specifically to test the operation of a particular component, researchers could measure how well a new design compared to previous designs in terms of speed or throughput. The values were given as numbers due to the fact that digital computations were being measured. By understanding the component and benchmark design, intelligent conclusions could be made about the effects of various features and design changes. Later, marketing departments discovered that benchmarks could be used to "prove" that their product was "faster" than their competitors', usually by choosing the benchmark tests and results that were the most favorable for their purposes.

In an attempt to provide standardization, several organizations were formed that developed their own benchmark tests. Some of these were formed as non-profit organizations that received funding from manufacturers for the purpose of developing these benchmark programs. Most of these programs were offered as source code to be compiled by each manufacturer, making them more platform- and operating-system-independent. It didn't take very long before manufacturers realized that they could optimize their product or drivers for these benchmarks, and even use special compiler flags to take advantage of their particular hardware features.

In order to combat the "benchmarketing" performed by manufacturers, new versions of these benchmarks are released on a periodic basis, though this does not completely eliminate the criticisms. Many of these programs are offered for "free" (though sometimes with an "administrative" fee), while some are run only by the organization itself to ensure the results are not skewed. Since the inception of the World Wide Web on the Internet, the free benchmark programs have become available to anyone, causing a virtual explosion of benchmark activity by end users.

Next: The (Invalid) Benchmarking Phenomenon

Home  -  Search  -  Topics  -  Up