Content deleted Content added
Add description of Continuous Benchmarking cite conference paper |
|||
Line 51:
* Benchmarking institutions often disregard or do not follow basic scientific method. This includes, but is not limited to: small sample size, lack of variable control, and the limited repeatability of results.<ref>{{cite web|url=http://donutey.com/hardwaretesting.php|title=Hardware Testing and Benchmarking Methodology|year=2006|access-date=2008-02-24|first=Kevin|last=Castor|url-status=dead|archive-url=https://web.archive.org/web/20080205031133/http://www.donutey.com/hardwaretesting.php|archive-date=2008-02-05}}</ref>
== Benchmarking
There are seven vital characteristics for benchmarks.<ref>{{cite conference|first1=Wei |last1=Dai |first2=Daniel |last2=Berleant |title=Benchmarking Contemporary Deep Learning Hardware and Frameworks: a Survey of Qualitative Metrics |date=December 12–14, 2019 |___location=Los Angeles, CA, USA |book-title=2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI)|publisher=IEEE |doi=10.1109/CogMI48466.2019.00029 |pages=148–155|url=https://dberleant.github.io/papers/BenchmarkingContemporaryDeepLearningHardwareAndFrameworks.pdf |arxiv=1907.03626 }}</ref> These key properties are:
# Relevance: Benchmarks should measure relatively vital features.
| |||