HaBench

Haskell Benchmark Suite

Recent discussions have illustrated the need for an up-to-date Haskell benchmark suite. The goal would be to package a number of representative benchmarks, written in Haskell, which users can run on their machine. Gathering (and publishing) performance numbers for these benchmarks on a number of different machines will hopefully help in guiding future development of both Haskell compilers and programs.

News

July 29, 2010: The effort has been rebooted, now with the goal of obtaining a happy ending :-) All will be done in the open, see the repo on github. See also the mail thread on the Haskell cafe list.

Contributors

• Kenneth Hoste - boegel on #haskell@FreeNode (ideas/advice/Wiki page maintainer)
• Donald Bruce Stewart - dons on #haskell@FreeNode (darcs repo maintainer (probably))
• Andy Georges - Itkovian on #haskell@FreeNode (measurements/analysis/advice)
• John Meacam - JohnMeacam on #haskell@FreeNode (advice/JHC support)
• Arthur van Leeuwen - earthy on #haskell@FreeNode (Computer Language Shootout man-on-the-insde)
• Neil Mitchell - ndm on #haskell@FreeNode (suggestions/Yhc support/user)
• Tom Davie - beelsebob on #haskell@FreeNode (suggestions/debugging support/user)
• Jeremy Shaw - stepcut on #haskell@FreeNode (suggestions)
• < Want to contribute? Add your name here! >

Naming

 * HaBench (intented to be the Haskell Benchmark Suite)
 * nofib/nofib 2.0 (follow-up of well-known nofib)

Ideas

• start from the (out-of-date?) nofib benchmark suite (GHC-only, includes ghcisms, but very usefull programs (gzip in pure Haskell, a whole strictness analysis pass of a compiler, quantum mechanics simulator and other _real_ stuff (JohnMeacam)) (website)
• include Computer Language Shootout submissions (website)
• YHC regressions suite (URL)
• microbenchmarks for comparison with C (for example bytecopying) (dons)
• both a Haskell98 and Haskell'-only (sub)suite (JohnMeacam)
• multiple Haskell compilers (GHC, JHC, ...)
• maintain benchmark according to Hackage (Hackage list)
• Rename this project as nofib, otherwise it is like that people will have to run both this and nofib, because people will ask for the nofib results as well (ndm)
• Libraries and tools in Haskell
• Make sure code is pure Haskell as accepted by standard Haskell 98 (with addendums) or Haskell'
• For each program include a correct version along with several buggy versions (perhaps even bugs that the programmer introduced while writing the benchmark) so that the programs can be used to test debuggers
• For each program, include a clean, simple idiomatic version, and a hand-optimized fast version. This should hopefully give clues as to which programs have the most room for improvement, as well as what types of optimizations are helpful for a given program.
• Divide benchmarks into imaginary, spectral and real (just like nofib did). (SimonPJ)

Notes

Random notes are below.

• (JohnMeacham) Insert non-formatted text here pretty much every compiler has a mode similar to 'ghc --make', so just let the user specify an arbitrary command line for the compiler to use, like 'jhc -v -flint $< -o $@' or 'ghc --make $< -o $@'
• (beelsebob) read Olaf's paper, he has evidence that speed problems in Haskell are usually because people are doing things too strictly

More info?

• #haskell IRC channel at Freenode
• initial thread @ Haskell mailing list
• follow-up thread @ Haskell-Cafe mailing list

Comments

Comments? Feel free to add them! Suggestions? Mail Kenneth at kenneth [dot] hoste [at] elis [dot] ugent [dot] be .