[Haskell-cafe] ST not strict enough?

[Roman Cheplyaka]

* Jason Dusek <jason.dusek at gmail.com> [2011-11-15 20:08:48+0000]
> I'm having some trouble with memory usage in rebuilding a
> ByteString with some sequences escaped. I thought I'd try
> vectors. However, it seems that even a relatively simple
> function, one that places all the bytes of a ByteString in to a
> vector, uses a great deal of memory.
> 
> Should I be annotating my functions with strictness, for the
> vector reference, for example? Should I be using STUArrays,
> instead?

Hi Jason,

I think what's happening here is ByteString's "strictness" makes things
actually lazy on your side.

Namely, unpack function produces its result "strictly", whole list at
once. As a result, the resulting list cannot be consumed one-by-one,
so it takes memory. You see ST thunks because

  mapM_ f as =  sequence_ (map f as)

and that map probably gets fused with unpack.

I guess the proper solution here is to use lazy bytestring and make sure
the chunks are not very big.

-- 
Roman I. Cheplyaka :: http://ro-che.info/