Latest revision as of 16:59, 10 March 2011

Haskell Benchmark Suite

Recent discussions have illustrated the need for an up-to-date Haskell benchmark suite. The goal would be to package a number of representative benchmarks, written in Haskell, which users can run on their machine. Gathering (and publishing) performance numbers for these benchmarks on a number of different machines will hopefully help in guiding future development of both Haskell compilers and programs.

News

July 29, 2010: The effort has been rebooted, now with the goal of obtaining a happy ending :-) All will be done in the open, see the repo on github. See also the mail thread on the Haskell cafe list.

Contributors

Kenneth Hoste - boegel on #haskell@FreeNode (ideas/advice/Wiki page maintainer)
Donald Bruce Stewart - dons on #haskell@FreeNode (darcs repo maintainer (probably))
Andy Georges - Itkovian on #haskell@FreeNode (measurements/analysis/advice)
John Meacam - JohnMeacam on #haskell@FreeNode (advice/JHC support)
Arthur van Leeuwen - earthy on #haskell@FreeNode (Computer Language Shootout man-on-the-insde)
Neil Mitchell - ndm on #haskell@FreeNode (suggestions/Yhc support/user)
Tom Davie - beelsebob on #haskell@FreeNode (suggestions/debugging support/user)
Jeremy Shaw - stepcut on #haskell@FreeNode (suggestions)
< Want to contribute? Add your name here! >

Naming

 * HaBench (intented to be the Haskell Benchmark Suite)
 * nofib/nofib 2.0 (follow-up of well-known nofib)

Ideas

start from the (out-of-date?) nofib benchmark suite (GHC-only, includes ghcisms, but very usefull programs (gzip in pure Haskell, a whole strictness analysis pass of a compiler, quantum mechanics simulator and other _real_ stuff (JohnMeacam)) (website)
include Computer Language Shootout submissions (website)
YHC regressions suite (URL)
microbenchmarks for comparison with C (for example bytecopying) (dons)
both a Haskell98 and Haskell'-only (sub)suite (JohnMeacam)
multiple Haskell compilers (GHC, JHC, ...)
maintain benchmark according to Hackage (Hackage list)
Rename this project as nofib, otherwise it is like that people will have to run both this and nofib, because people will ask for the nofib results as well (ndm)
Libraries and tools in Haskell
Make sure code is pure Haskell as accepted by standard Haskell 98 (with addendums) or Haskell'
For each program include a correct version along with several buggy versions (perhaps even bugs that the programmer introduced while writing the benchmark) so that the programs can be used to test debuggers
For each program, include a clean, simple idiomatic version, and a hand-optimized fast version. This should hopefully give clues as to which programs have the most room for improvement, as well as what types of optimizations are helpful for a given program.
Divide benchmarks into imaginary, spectral and real (just like nofib did). (SimonPJ)

Notes

Random notes are below.

(JohnMeacham) Insert non-formatted text here pretty much every compiler has a mode similar to 'ghc --make', so just let the user specify an arbitrary command line for the compiler to use, like 'jhc -v -flint $< -o $@' or 'ghc --make $< -o $@'
(beelsebob) read Olaf's paper, he has evidence that speed problems in Haskell are usually because people are doing things too strictly

More info?

Comments

Before I knew about the HaBench project, I (David Peixotto) created the Fibon benchmark suite. Fibon is an effort to create a new Haskell benchmark suite and benchmarking tools. I'm all for merging efforts, but just wanted to put a pointer to the benchmark suite here for anyone arriving via web search. It currently contains 35 benchmarks taken from Hackge, the Shootout, DPH, and Repa libraries. The source code for the tools and benchmarks are available on github.

Comments? Feel free to add them! Suggestions? Mail Kenneth at kenneth [dot] hoste [at] elis [dot] ugent [dot] be .

@@ Line 1: / Line 1: @@
+[[Category:Proposals]]
 = Haskell Benchmark Suite =
 Recent discussions have illustrated the need for an up-to-date Haskell benchmark suite. The goal would be to package a number of representative benchmarks, written in Haskell, which users can run on their machine. Gathering (and publishing) performance numbers for these benchmarks on a number of different machines will hopefully help in guiding future development of both Haskell compilers and programs.
+== News ==
+July 29, 2010: The effort has been rebooted, now with the goal of obtaining a happy ending :-) All will be done in the open, see the repo on [http://github.com/itkovian/HaBench github]. See also the [http://www.haskell.org/pipermail/haskell-cafe/2010-June/079352.html mail thread] on the Haskell cafe list.
 == Contributors ==
@@ Line 11: / Line 17: @@
 * '''Arthur van Leeuwen''' - ''earthy on #haskell@FreeNode'' (Computer Language Shootout man-on-the-insde)
 * '''Neil Mitchell''' - ''ndm on #haskell@FreeNode'' (suggestions/Yhc support/user)
+* '''Tom Davie''' - ''beelsebob on #haskell@FreeNode'' (suggestions/debugging support/user)
-* <add '''your name''' here if you want to contribute!>
+* '''Jeremy Shaw''' - ''stepcut on #haskell@FreeNode'' (suggestions)
+* < Want to contribute? Add '''your name''' here! >
 == Naming ==
@@ Line 29: / Line 37: @@
 * Rename this project as nofib, otherwise it is like that people will have to run both this ''and'' nofib, because people will ask for the nofib results as well (ndm)
 * [http://haskell.org/haskellwiki/Libraries_and_tools Libraries and tools in Haskell]
+* Make sure code is pure Haskell as accepted by standard Haskell 98 (with addendums) or Haskell'
+* For each program include a correct version along with several buggy versions (perhaps even bugs that the programmer introduced while writing the benchmark) so that the programs can be used to test debuggers
+* For each program, include a clean, simple idiomatic version, and a hand-optimized fast version. This should hopefully give clues as to which programs have the most room for improvement, as well as what types of optimizations are helpful for a given program.
+* Divide benchmarks into imaginary, spectral and real (just like nofib did). (SimonPJ)
 == Notes ==
@@ Line 35: / Line 47: @@
 * (''JohnMeacham'') <nowiki>Insert non-formatted text here pretty much every compiler has a mode similar to 'ghc --make', so just let the user specify an arbitrary command line for the compiler to use, like 'jhc -v -flint $< -o $@'  or 'ghc --make $< -o $@'</nowiki>
+* (beelsebob)	<nowiki>read Olaf's paper, he has evidence that speed problems in Haskell are usually because people are doing things too strictly</nowiki>
 == More info? ==
@@ Line 43: / Line 56: @@
 == Comments ==
+* Before I knew about the HaBench project, I (David Peixotto) created the Fibon benchmark suite. Fibon is an effort to create a new Haskell benchmark suite and benchmarking tools.  I'm all for merging efforts, but just wanted to put a pointer to the benchmark suite here for anyone arriving via web search. It currently contains 35 benchmarks taken from Hackge, the Shootout, DPH, and Repa libraries. The source code for the tools and benchmarks are available on [https://github.com/dmpots/fibon github].
 Comments? Feel free to add them! Suggestions? Mail Kenneth at kenneth [dot] hoste [at] elis [dot] ugent [dot] be .

Difference between revisions of "HaBench"