[Haskell-cafe] Debugging ByteString and Data.Binary.Get memory usage

[Kyle Hanson]

OK

I have a bunch of BSON documents that I convert to ByteStrings, put in a
Map, and write to a socket based on the response. I noticed some high
memory usage (in the GBs) so I decided to investigate. I simplified my
problem into a small program that demonstrates clearer what is happening.

I wrote two versions, one with a Lazy Map and Lazy ByteStrings and one with
a Strict Map and Strict ByteStrings. Both share the same memory behavior
(except the lazy BS one is faster)

Here is the strict version:

http://lpaste.net/92298

And here is the lazy version:

http://lpaste.net/92299

I wrote this and compared the memory and speed behavior of ByteStrings
generated by converting it from a BSON document and ByteStrings generated
more purely.

The length of the ByteString from a BSON document is 68k and the length of
the "pure" BS is 70k.

This is my weird memory behavior, both BSON and "pure" methods use the same
amount of memory after inserting 10k of them (90mb)

However when I go to lookup a value, the BSON Map explodes the memory to
over 250mb. Even if I lookup just 1 value. Looking up any number of values
in the "pure BS" keeps the memory usage stable (90mb).

I am hoping someone can help me understand this. I have read some posts
about Temporary ByteStrings causing memory issues but I don't know how to
get started debugging.

--
Kyle Hanson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20130829/282bf5d2/attachment.htm>