nofib benchmarks for measuring the effects of compiler optimizations

[Simon Marlow]

On 09/10/2009 17:25, David Peixotto wrote:
> Thanks for the input. I'll take a look at the nobench suite. I might be
> interested in working to improve nobench. Is there some obvious
> improvements that people have in mind (besides using Criterion), or it
> just generally needs some updated benchmarks? I suppose one question to
> consider is whether it should include both parallel and sequential
> benchmarks.
>
> Simon M, can you say what your reliance on nobench-analyse is (output
> format, specific collected stats, etc.)?

Ok, here are the ways I use it:

  - to compare the results from two nofib runs, on various metrics:
    runtime, GC time, mutator time, allocations, binary sizes, sizes
    of modules etc.  Pretty much all the stats that nofib-analyse
    compares are useful to us.  I normally use the text output option,
    rather that HTML.

  - to generate LaTeX output for papers.  This is a bit more ad-hoc,
    typically I add more functionality as we need it.

  - to generate CSV files for importing into Excel to generate graphs,
    for papers and talks.  Again, quite ad-hoc and funcionality added
    on demand.

The tool itself is a bit ugly and could do with a rewrite.  I wouldn't 
object to having to add e.g. LaTeX/CSV support myself to something else, 
as long as the replacement is suitably clean and extensible.

Cheers,
	Simon


> Thanks,
> David
>
> On Oct 9, 2009, at 6:13 AM, Simon Marlow wrote:
>
>> On 09/10/2009 09:43, Malcolm Wallace wrote:
>>>> Would you say that the nofib benchmarks are the best available for
>>>> measuring the effectiveness of compiler optimizations, or is there a
>>>> better benchmarking suite for that purpose?
>>>
>>> I don't know about "better", but the nobench suite (which is largely
>>> based on nofib, but goes further) automates the comparison of two or
>>> more compiler builds, and produces nice HTML tables of the results. It
>>> is currently languishing a bit - could do with some love and attention:
>>>
>>> http://code.haskell.org/nobench/
>>
>> We do need a better benchmark suite. In GHC we still use nofib, but
>> it's becoming less useful as time goes by. I have some reliance on the
>> nofib-analyse tool that comes with nofib, which would need updating
>> (or replacing) to work with nobench.
>>
>> Also we should be looking at using Criterion for our stats.
>>
>> Working on nobench would be a really useful project for someone... any
>> takers?
>>
>> Cheers,
>> Simon
>>
>> _______________________________________________
>> Cvs-ghc mailing list
>> Cvs-ghc at haskell.org
>> http://www.haskell.org/mailman/listinfo/cvs-ghc
>>
>