[Haskell-cafe] Shootout favouring C

[Sebastian Sylvan]

On 1/16/06, Daniel Fischer <daniel.is.fischer at web.de> wrote:
> Hi,
>
> Is it only my machime, or can you confirm that for the Ackermann benchmark,
> it's very good for C that they chose 9 and not a larger value? For 10, we are
> significantly faster and for 11,12,13, we can run rings around the
> C-programme:
> dafis at linux:~/Documents/haskell/shootout> time ./cacker3 10; time ./acker 10;
> time ./cacker3 11; time ./acker 11; time ./cacker3 12; time ./acker 12
> Ack(3,10): 8189
>
> real    0m0.664s
> user    0m0.660s
> sys     0m0.010s
> Ack(3,10): 8189
>
> real    0m0.405s
> user    0m0.400s
> sys     0m0.000s
> Ack(3,11): 16381
>
> real    0m6.255s
> user    0m6.220s
> sys     0m0.000s
> Ack(3,11): 16381
>
> real    0m1.731s
> user    0m1.710s
> sys     0m0.000s
> Ack(3,12): 32765
>
> real    0m35.270s
> user    0m35.050s
> sys     0m0.000s
> Ack(3,12): 32765
>
> real    0m10.673s
> user    0m10.590s
> sys     0m0.000s
> dafis at linux:~/Documents/haskell/shootout> time ./acker 13; time ./cacker3 13
> Ack(3,13): 65533
>
> real    1m4.476s
> user    1m4.050s
> sys     0m0.010s
> Ack(3,13): 65533
>
> real    2m50.645s
> user    2m47.020s
> sys     0m0.020s
>

This is interesting. Hopefully it's not intentional, but it's quite
obvious that for benchmarks where the fastest time is only a few
fractions of a second, languages with more complex runtime systems
will be unfairly slow due to the startup cost. I suspect that e.g.
Java suffers quite a bit from this in the shootout.
There is already a startup benchmark in there (although for some
lightweight langauges I suspect that the cost of writing "hello world"
is larger than the startup, so it's perhaps not optimal) so it would
be good to remove the startup factor in benchmarks which are not meant
to test that.

In other words I'd prefer if all benchmarks are reconfigured to target
an execution time of at least a few seconds for the fastest
benchmarks.

/S
--
Sebastian Sylvan
+46(0)736-818655
UIN: 44640862