[Haskell-cafe] Difficult memory leak in array processing

[isto]

Hi Niko,

to, 2006-11-23 kello 12:11 +0200, Niko Korhonen kirjoitti:
> I've tried applying seq and some other strictness tricks (such as x ==
> x) pretty much everywhere on the code with no results. Could you
> please help me understand what is going on here? Have I misunderstood
> something critical in how Haskell works? Here is the relevant portion
> of the code: 

> main = do
>     -- This should allocate a 40 MB array
>     buf <- newArray_ (0, 10000000) :: IO Buffer
>     -- Fill the array with dither 
>     genSeries buf tpdf (2, 12)


main = do
    -- This should allocate a 40 MB array
    buf <- newArray_ (0, 100000000) :: IO Buffer
    -- Fill the array with dither 
    genSeries buf tpdf (2, 12)
    a <- readArray buf  100000000
    putStrLn $ "a is " ++ (show a)

By adding -O3 -optc-O3  -funfolding-use-threshold=16 
compile flags the above code with 100'000'000 elements
worked.  And by still adding -ddump-simpl > core.txt
flag and looking the generated core, the worker-loop 
seemed to use primitives.

I cannot say, if this was the helping part here.  Have you
tried profiling: -prof -auto-all  and running with +RTS -p -RTS?
Or running with  +RTS -sstderr
gives

14,257,786,344 bytes allocated in the heap
  4,282,040 bytes copied during GC (scavenged)
  1,646,936 bytes copied during GC (not scavenged)
 80,733,232 bytes maximum residency (2 sample(s))

      27045 collections in generation 0 (  0.31s)
          2 collections in generation 1 (  0.00s)

         78 Mb total memory in use

  INIT  time    0.00s  (  0.00s elapsed)
  MUT   time   22.61s  ( 24.07s elapsed)
  GC    time    0.31s  (  0.32s elapsed)
  EXIT  time    0.00s  (  0.00s elapsed)
  Total time   22.92s  ( 24.39s elapsed)

  %GC time       1.3%  (1.3% elapsed)

  Alloc rate    630,612,876 bytes per MUT second

  Productivity  98.6% of total user, 92.7% of total elapsed


It seems that garbage collector has not used very much time here.
There is more information on haskell wiki:
http://www.haskell.org/haskellwiki/Performance
http://www.haskell.org/haskellwiki/Performance/GHC

This GHC specific part does not mention -O3 -optc-O3
-funfolding-use-threshold=nn flags.  They were hinted here
on this list; I have found them very helpful a couple of weeks 
ago - thanks again :)
btw, Could the GHC specific wiki page be updated to contain and
explain these flags?

Hopefully this helped you a bit!  And hopefully someone who knows
how these things go have time to give you a detailed answer!

br, Isto