Benchmark (computing): Difference between revisions

Revision as of 05:16, 29 November 2024

Revision as of 05:17, 29 November 2024

Line 1:

{{Short description|Standardized performance evaluation}}

{{About|the use of '''[https://modefiles.com/understanding-benchmark-in-android benchmarks]''' in computing||Benchmark (disambiguation){{!}}Benchmark}}

{{Multiple issues|{{more citations needed|date=July 2015}}

{{Expert needed|Computer science|talk=Outdated sources|reason=Outdated or deprecated sources|date=October 2022}}}}

Line 9:

Benchmarking is usually associated with assessing performance characteristics of [[computer hardware]], for example, the [[floating point operation]] performance of a [[Central processing unit|CPU]], but there are circumstances when the technique is also applicable to [[software]]. Software benchmarks are, for example, run against [[compiler]]s or [[database management system]]s (DBMS).

'''[https://modefiles.com/understanding-benchmark-in-android Benchmarks]''' provide a method of comparing the performance of various subsystems across different chip/system [[Computer architecture|architectures]]. Benchmarking as a part of [[continuous integration]] is called Continuous Benchmarking.<ref>{{cite conference|doi=10.1109/IC2E.2019.00039|chapter-url=https://www.researchgate.net/publication/333918034|chapter=Continuous Benchmarking: Using System Benchmarking in Build Pipelines|year=2019|access-date=2023-12-03|first1=Martin|last1=Grambow|first2=Fabian|last2=Lehmann|first3=David|last3=Bermbach|title=2019 IEEE International Conference on Cloud Engineering (IC2E) |pages=241–246 |isbn=978-1-7281-0218-4 }}</ref>

== Purpose ==

As [[computer architecture]] advanced, it became more difficult to compare the performance of various computer systems simply by looking at their specifications. Therefore, tests were developed that allowed comparison of different architectures. For example, [[Pentium 4]] processors generally operated at a higher clock frequency than [[Athlon XP]] or [[PowerPC]] processors, which did not necessarily translate to more computational power; a processor with a slower clock frequency might perform as well as or even better than a processor operating at a higher frequency. See [[BogoMips]] and the [[megahertz myth]].

'''[https://modefiles.com/understanding-benchmark-in-android Benchmarks]''' are designed to mimic a particular type of workload on a component or system. Synthetic benchmarks do this by specially created programs that impose the workload on the component. Application benchmarks run real-world programs on the system. While application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks are useful for testing individual components, like a [[hard disk]] or networking device.

Benchmarks are particularly important in [[CPU design]], giving processor architects the ability to measure and make tradeoffs in [[microarchitecture|microarchitectural]] decisions. For example, if a benchmark extracts the key [[algorithms]] of an application, it will contain the performance-sensitive aspects of that application. Running this much smaller snippet on a cycle-accurate simulator can give clues on how to improve performance.

Line 31:

== Functionality ==

Features of benchmarking software may include recording/[[data export|exporting]] the course of performance to a [[spreadsheet]] file, visualization such as drawing [[line graph]]s or [[color-coded]] tiles, and pausing the process to be able to resume without having to start over. Software can have additional features specific to its purpose, for example, disk '''[https://modefiles.com/understanding-benchmark-in-android benchmarking software]''' may be able to optionally start measuring the disk speed within a specified range of the disk rather than the full disk, measure [[random access]] reading speed and [[latency (engineering)|latency]], have a "quick scan" feature which measures the speed through samples of specified intervals and sizes, and allow specifying a [[Block (data storage)|data block]] size, meaning the number of requested bytes per read request.<ref>Software: HDDScan, GNOME Disks</ref>

== Challenges ==

Line 52:

== Benchmarking principles ==

There are seven vital characteristics for '''[https://modefiles.com/understanding-benchmark-in-android benchmarks]'''.<ref>{{cite conference|first1=Wei |last1=Dai |first2=Daniel |last2=Berleant |title=Benchmarking Contemporary Deep Learning Hardware and Frameworks: a Survey of Qualitative Metrics |date=December 12–14, 2019 |___location=Los Angeles, CA, USA |book-title=2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI)|publisher=IEEE |doi=10.1109/CogMI48466.2019.00029 |pages=148–155|url=https://dberleant.github.io/papers/BenchmarkingContemporaryDeepLearningHardwareAndFrameworks.pdf |arxiv=1907.03626 }}</ref> These key properties are:

# Relevance: Benchmarks should measure relatively vital features.

# Representativeness: Benchmark performance metrics should be broadly accepted by industry and academia.