The Computer Language Benchmarks Game: Difference between revisions

Revision as of 11:07, 13 December 2017

Revision as of 13:51, 13 December 2017

Line 8:

* A website to facilitate the interactive comparison of the results

== Supported languages ==

Due to resource constraints, only a small subset of common programming languages are supported, up to the discretion of the game's operator.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/|title=The Computer Language Benchmarks Game|website=benchmarksgameBenchmarksgame.alioth.debian.org|accessdate=13 December 2017}}</ref>

Due to resource constraints, only a small subset of common programming languages are supported, up to the discretion of the game's operator.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/|title=The Computer Language Benchmarks Game|website=benchmarksgame.alioth.debian.org}}</ref>

{{Collapsible list

|title=List of supported languages

Line 46 ⟶ 44:

}}

== Metrics ==

The following aspects of each given implementation are measured:<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/how-programs-are-measured.html|title=details - The Computer Language Benchmarks Game|website=benchmarksgameBenchmarksgame.alioth.debian.org|accessdate=13 December 2017}}</ref>

The following aspects of each given implementation are measured:<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/how-programs-are-measured.html|title=details - The Computer Language Benchmarks Game|website=benchmarksgame.alioth.debian.org}}</ref>

* overall user runtime

* peak memory allocation

Line 58 ⟶ 54:

It is common to see multiple solutions in the same programming language for the same problem. This highlights that within the bounds of a given language, a solution can be given which is either of high abstraction, is memory efficiency, fast, or parallelizes better.

== Benchmark programs ==

It was a design choice from the start to only include very simple toy problems, each providing a different kind of programming challenge.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/why-measure-toy-benchmark-programs.html|title=Toy benchmark programs - The Computer Language Benchmarks Game|website=Benchmarksgame.alioth.debian.org|accessdate=13 December 2017}}</ref>

ItThis wasprovides ausers designof choicethe fromBenchmark Game the startopportunity to onlyscrutinize include very simple toy problems, each providing a different kind ofthe programmingvarious challengeimplementations.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/whyu64q/nbody-measure-toy-benchmark-programsdescription.html#nbody|title=Toyn-body benchmark programsdescription (64-bit TheUbuntu quad core) - Computer Language Benchmarks Game|website=benchmarksgameBenchmarksgame.alioth.debian.org|accessdate=13 December 2017}}</ref>

This provides users of the Benchmark Game the opportunity to scrutinize the various implementations.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/u64q/nbody-description.html#nbody|title=n-body description (64-bit Ubuntu quad core) - Computer Language Benchmarks Game|website=benchmarksgame.alioth.debian.org}}</ref>

* [[Memory management#Dynamic memory allocation|binary-trees]]

* [[Synchronization (computer science)#Thread or process synchronization|chameneos-redux]]

Line 77 ⟶ 71:

* [[Context switch#Multitasking|thread-ring]]

== History ==

ThisThe providesproject userswas ofknown theas Benchmark''The GameGreat theComputer opportunity to scrutinizeLanguage theShootout'' variousuntil implementations2007.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/u64q/nbodysometimes-people-descriptionjust-make-up-stuff.html#nbody|title=n-bodystories description (64-bit Ubuntu quad core) -The Computer Language Benchmarks Game|website=benchmarksgameBenchmarksgame.alioth.debian.org|accessdate=13 December 2017}}</ref>

TheA projectport wasfor knownWindows aswas ''Themaintained Greatseparately Computerbetween Language Shootout''2002 untiland 20072003.<ref>{{cite web|url=httpshttp://benchmarksgamedada.aliothperl.debian.orgit/shootout/sometimes-people-just-make-up-stuff.html|title=storiesThe -Great TheWin32 Computer Language Benchmarks GameShootout|website=benchmarksgameDada.aliothperl.debian.orgit|accessdate=13 December 2017}}</ref>

Information about the project's history and lineage can be found at WikiWikiWeb.<ref>[http://wiki.c2.com/?GreatComputerLanguageShootout] {{dead link|date=December 2017}}</ref><ref>[http://wiki.c2.com/?ComputerLanguageBenchmarksGame] {{dead link|date=December 2017}}</ref>

A port for Windows was maintained separately between 2002 and 2003.<ref>{{cite web|url=http://dada.perl.it/shootout/|title=The Great Win32 Computer Language Shootout|website=dada.perl.it}}</ref>

The sources are kept in CVS, but it also has multiple forks on GitHub.<ref>{{cite web|url=https://github.com/Byron/benchmarksgame-cvs-mirror|title=benchmarksgame-cvs-mirror: A git mirror of the benchmarksgame cvs repository|first=Sebastian|last=Thiel|date=24 October 2017|publisher=[[GitHub]]|viaaccessdate=GitHub13 December 2017}}</ref>

Information about the project's history and lineage can be found at WikiWikiWeb.<ref>http://wiki.c2.com/?GreatComputerLanguageShootout</ref><ref>http://wiki.c2.com/?ComputerLanguageBenchmarksGame</ref>

The project is continuously evolving. The list of supported programming languages is updated approximately once per annum, following market trends. Users can also submit improved solutions to any of the problems or suggest testing methodology refinement.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/play.html|title=Contribute your programs - The Computer Language Benchmarks Game|website=benchmarksgameBenchmarksgame.alioth.debian.org|accessdate=13 December 2017}}</ref>

The sources are kept in CVS, but it also has multiple forks on GitHub.<ref>{{cite web|url=https://github.com/Byron/benchmarksgame-cvs-mirror|title=benchmarksgame-cvs-mirror: A git mirror of the benchmarksgame cvs repository|first=Sebastian|last=Thiel|date=24 October 2017|publisher=|via=GitHub}}</ref>

The project is continuously evolving. The list of supported programming languages is updated approximately once per annum, following market trends. Users can also submit improved solutions to any of the problems or suggest testing methodology refinement.<ref>{{cite web|url=https://benchmarksgame.alioth.debian.org/play.html|title=Contribute your programs - The Computer Language Benchmarks Game|website=benchmarksgame.alioth.debian.org}}</ref>

== Caveats ==

== Caveats ==

The developers themselves highlight the fact that those doing research should exercise caution when using such microbenchmarks:

{{quotation|[...] the JavaScript benchmarks are fleetingly small, and behave in ways that are significantly different than the real applications. We have documented numerous differences in behavior, and we conclude from these measured differences that results based on the benchmarks may mislead JavaScript engine implementers. Furthermore, we observe interesting behaviors in real JavaScript applications that the benchmarks fail to exhibit, suggesting that previously unexplored optimization strategies may be productive in practice.|https://benchmarksgame.alioth.debian.org/for-programming-language-researchers.html}}

== Impact ==

== Impact ==

The benchmark results have uncovered various compilers issues. Sometimes a given compiler failed to process unusual, but otherwise grammatically valid constructs. At other times, runtime performance was shown to be below expectations, which prompted compiler developers to revise their optimization capabilities.

Line 121 ⟶ 112:

<ref>{{cite conference|author1=Bradford L. Chamberlain|author2=Ben Albrecht|author3=Lydia Duncan|author4=Ben Harshbarger|title=Entering the Fray: Chapel’s Computer Language Benchmark Game Entry|date=2017|url=http://chapel.cray.com/CHIUW/2017/chamberlain-abstract.pdf|accessdate=25 March 2017}}</ref>

== See also ==

* [[Benchmark (computing)]]

* [[Comparison of programming languages]]

== References ==

== External links ==

{{Reflist|30em}}

== External links ==

* [https://benchmarksgame.alioth.debian.org/ official website]

== References ==

{{Reflist|30em}}

[[Category:Programming language comparisons| ]]

{{linkrot|date=December 2017}}