<div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Hi all,</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Problem</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif">=======</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">After having used Makefiles to build my benchmark suites for quite some time I'm convinced we need support for building benchmark using Cabal. Make doesn't solve the problem of building a benchmarks particularly well. There are several problems:</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">* Building your library using Cabal and your benchmarks using Make sometimes ends up with different package versions being used for the two separate builds, resulting in compilation errors.</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif">* Lots of functionality that's already in Cabal is duplicated in Makefiles (e.g. preprocessor rules).</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">* It's hard to get proper incremental compilation; I always use `make -C benchmarks clean all` to rebuild the benchmarks, just to make sure that everything is up-to-date.</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Proposal</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">========</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Step 1. Add basic support for building simple benchmark suites e.g. modelled after test suite sections:</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><font class="Apple-style-span" face="'courier new', monospace">Name: foo</font><div><font class="Apple-style-span" face="'courier new', monospace">Version: 1.0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">License: BSD3</font></div><div><font class="Apple-style-span" face="'courier new', monospace">Cabal-Version: >= 1.9.2</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">Build-Type: Simple</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace">Benchmark bench-foo</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> type: stdio-1.0</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> main-is: bench-foo.hs</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> build-depends: base</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">This doesn't add much on top of normal executable sections, except that it tells Cabal that this isn't an executable it should build by default or install.</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Step 2. Add a detailed benchmark interface that allows the benchmark to associate a real valued *measure* (optionally with upper and lower bounds) with a benchmark name. Benchmark naming would work like test naming (e.g. group/subgroup/name). This would allow us to track performance over time.</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Step 2 needs to be fleshed out some more, but I think we can get quite a lot of bang-for-the-buck with step 1.</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Cheers,</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Johan</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div>