Benchmark (computing): Difference between revisions

Revision as of 08:51, 6 May 2025

Latest revision as of 14:20, 31 July 2025

(4 intermediate revisions by 3 users not shown)

Line 20:

Prior to 2000, computer and microprocessor architects used [[SPEC]] to do this, although SPEC's Unix-based benchmarks were quite lengthy and thus unwieldy to use intact.

Computer manufacturerscompanies are known to configure their systems to give unrealistically high performance on benchmark tests that are not replicated in real usage. For instance, during the 1980s some compilers could detect a specific mathematical operation used in a well-known floating-point benchmark and replace the operation with a faster mathematically equivalent operation. However, such a transformation was rarely useful outside the benchmark until the mid-1990s, when [[RISC]] and [[VLIW]] architectures emphasized the importance of [[compiler]] technology as it related to performance. Benchmarks are now regularly used by [[compiler]] companies to improve not only their own benchmark scores, but real application performance.

CPUs that have many execution units — such as a [[superscalar]] CPU, a [[VLIW]] CPU, or a [[reconfigurable computing]] CPU — typically have slower clock rates than a sequential CPU with one or two execution units when built from transistors that are just as fast. Nevertheless, CPUs with many execution units often complete real-world and benchmark tasks in less time than the supposedly faster high-clock-rate CPU.

Given the large number of benchmarks available, a manufacturervendor can usually find at least one benchmark that shows its system will outperform another system; the other systems can be shown to excel with a different benchmark.

ManufacturersSoftware vendors also use benchmarks in their marketing, such as the "benchmark wars" between rival [[relational database]] makers in the 1980s and 1990s. Companies commonly report only those benchmarks (or aspects of benchmarks) that show their products in the best light. They also have been known to mis-represent the significance of benchmarks, again to show their products in the best possible light.<ref Takenname="rdbmsinformix20070612">{{Cite together,interview these|interviewer=Luanne practicesJohnson are|title=RDBMS calledWorkshop: ''benchInformix |url=https://archive.computerhistory.org/resources/access/text/2013/05/102702566-marketing05-01-acc.''pdf |access-date=2025-05-30 |publisher=Computer History Museum |date=2007-06-12}}</ref><ref name="rdbmsingressybase20070613">{{Cite interview |interviewer=Doug Jerger |title=RDBMS Workshop: Ingres and Sybase |url=https://archive.computerhistory.org/resources/access/text/2013/05/102702565-05-01-acc.pdf |access-date=2025-05-30 |publisher=Computer History Museum |date=2007-06-13}}</ref>

Ideally benchmarks should only substitute for real applications if the application is unavailable, or too difficult or costly to port to a specific processor or computer system. If performance is critical, the only benchmark that matters is the target environment's application suite.

Line 97:

=== Industry standard (audited and verifiable) ===

* [[BAPCo consortium|Business Applications Performance Corporation (BAPCo)]]

* [[EEMBC|Embedded Microprocessor Benchmark Consortium (EEMBC)]]

* [[Standard Performance Evaluation Corporation]] (SPEC), in particular their [[SPECint]] and [[SPECfp]]

Line 119 ⟶ 118:

* [[NAS benchmarks|NAS parallel benchmarks]]

* [[NBench]] – synthetic benchmark suite measuring performance of integer arithmetic, memory operations, and floating-point arithmetic

* [[PAL (software)|PAL]] – a benchmark for realtime physics engines

* [[PerfKitBenchmarker]] – A set of benchmarks to measure and compare cloud offerings.

* [[Phoronix Test Suite]] – open-source cross-platform benchmarking suite for Linux, OpenSolaris, FreeBSD, OSX and Windows. It includes a number of other benchmarks included on this page to simplify execution.

Line 130 ⟶ 129:

=== Microsoft Windows benchmarks ===

* [[BAPCo consortium|BAPCo]]: MobileMark, SYSmark, WebMark

* [[CrystalDiskMark]]

* [[Futuremark|Underwriters Laboratories (UL)]]: [[3DMark]], [[PCMark]]

Line 141 ⟶ 139:

* [[Windows System Assessment Tool]], included with Windows Vista and later releases, providing an index for consumers to rate their systems easily

* [[Worldbench]] (discontinued)

===Unusual benchmark===

* [[Will Smith Eating Spaghetti test]] - an informal test to determine the capabilities of [[text-to-video]] models.

=== Others ===

* [[AnTuTu]] – commonly used on phones and ARM-based devices.