[Haskell-cafe] Debugging ByteString and Data.Binary.Get memory usage

[Johan Tibell]

A good starting point is to estimate how much space you think the data
should take using e.g.


http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html

If you do that, is the actual space usage close to what you expected?


On Thu, Aug 29, 2013 at 5:35 PM, Kyle Hanson <hanooter at gmail.com> wrote:

> OK
>
> I have a bunch of BSON documents that I convert to ByteStrings, put in a
> Map, and write to a socket based on the response. I noticed some high
> memory usage (in the GBs) so I decided to investigate. I simplified my
> problem into a small program that demonstrates clearer what is happening.
>
> I wrote two versions, one with a Lazy Map and Lazy ByteStrings and one
> with a Strict Map and Strict ByteStrings. Both share the same memory
> behavior (except the lazy BS one is faster)
>
> Here is the strict version:
>
> http://lpaste.net/92298
>
> And here is the lazy version:
>
> http://lpaste.net/92299
>
> I wrote this and compared the memory and speed behavior of ByteStrings
> generated by converting it from a BSON document and ByteStrings generated
> more purely.
>
> The length of the ByteString from a BSON document is 68k and the length of
> the "pure" BS is 70k.
>
> This is my weird memory behavior, both BSON and "pure" methods use the
> same amount of memory after inserting 10k of them (90mb)
>
> However when I go to lookup a value, the BSON Map explodes the memory to
> over 250mb. Even if I lookup just 1 value. Looking up any number of values
> in the "pure BS" keeps the memory usage stable (90mb).
>
> I am hoping someone can help me understand this. I have read some posts
> about Temporary ByteStrings causing memory issues but I don't know how to
> get started debugging.
>
> --
> Kyle Hanson
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20130829/b4b13262/attachment.htm>