[Haskell-cafe] Speed of Error handling with Continuations vs. Eithers

[Andrea Vezzosi]

On Thu, May 13, 2010 at 8:13 PM, wren ng thornton <wren at freegeek.org> wrote:
> Andrea Vezzosi wrote:
>>
>> On Thu, May 13, 2010 at 10:51 AM, wren ng thornton <wren at freegeek.org>
>> wrote:
>>>
>>> Andrea Vezzosi wrote:
>>>>
>>>> wren ng thornton  wrote:
>>>>>
>>>>> With this change [1] I can't notice any difference for your
>>>>> benchmark[2].
>>>>> Then again, all the runTest calls take 0 msec and I've had no luck
>>>>> making
>>>>> the computation take much time; perhaps your computer can detect a
>>>>> difference.
>>>>
>>>> On my machine, with ghc-6.12.1, yours and the original ErrCPS give
>>>> quite similar results, both ~2x slower than Either.
>>>> However it's important to note that these results are highly dependent
>>>> on the monadic expressions being evaluated, with a different benchmark
>>>> you can get an huge speedup with the CPS versions.
>>>
>>> That's very curious. After installing Criterion, my machine (OSX 10.5.8
>>> 2.8GHz Intel Core2Duo, GHC 6.12.1 with -O2) shows only 1% difference
>>> between
>>> my ErrCPS and Either on this benchmark. Alas, I can't print kernel
>>> density
>>> graphs since Crieterion charts are broken on 6.12. It seems buggy that
>>> your
>>> platform would behave so much differently...
>>
>> I got the measurements from the original code, could you share the
>> code that uses criterion instead?
>
> The 1% number was buggy because I hadn't factored the generation of random
> lists out of the benchmark. But, having fixed that, I still can't replicate
> your numbers: I get 12us for Either, vs 17us for EitherCPS.
>
> http://community.haskell.org/~wren/wren-extras/test/Control/Monad/ErrCPS/CriterionBenchmark.hs
>
>
>
> Yet another version of the same benchmark, this time using Microbench:
>
> http://community.haskell.org/~wren/wren-extras/test/Control/Monad/ErrCPS/MicrobenchBenchmark.hs
>
> Microbench seems to replicate your numbers better: 2551.930ns vs 4466.832ns
> (or 391.86 vs 223.87 calls per second)--- though this is getting into the
> range where there might be Int overflow issues corrupting the results (a
> similar problem showed up when benchmarking Data.Trie vs Data.Map), so it
> may warrant further investigation.
>

That might be the case, i'm on 64bit:

saizan at astarte:~$ uname -a
Linux astarte 2.6.32-ARCH #1 SMP PREEMPT Tue Feb 23 19:43:46 CET 2010
x86_64 Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz GenuineIntel
GNU/Linux

saizan at astarte:~$ ./CriterionBenchmark
warming up
estimating clock resolution...
mean is 6.834442 us (80001 iterations)
found 1240 outliers among 79998 samples (1.6%)
  1131 (1.4%) high severe
estimating cost of a clock call...
mean is 107.2316 ns (41 iterations)

benchmarking Either
collecting 100 samples, 1039 iterations each, in estimated 683.8220 ms
bootstrapping with 100000 resamples
mean: 6.563462 us, lb 6.553649 us, ub 6.570454 us, ci 0.950
std dev: 41.74602 ns, lb 23.76971 ns, ub 67.67842 ns, ci 0.950
found 8 outliers among 100 samples (8.0%)
  2 (2.0%) low severe
  4 (4.0%) high mild
  2 (2.0%) high severe
variance introduced by outliers: 0.990%
variance is unaffected by outliers

benchmarking ErrCPS
collecting 100 samples, 1 iterations each, in estimated 1.334000 s
bootstrapping with 100000 resamples
mean: 13.14468 ms, lb 13.10442 ms, ub 13.18208 ms, ci 0.950
std dev: 198.3150 us, lb 182.0600 us, ub 220.7957 us, ci 0.950
variance introduced by outliers: 0.993%
variance is unaffected by outliers

If i'm reading it correctly this gives even worse results: 6us vs. 13ms

> --
> Live well,
> ~wren
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>