<div dir="ltr">A good starting point is to estimate how much space you think the data should take using e.g.<div><br></div><div> <a href="http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html">http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html</a></div>
<div><br></div><div>If you do that, is the actual space usage close to what you expected?</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 29, 2013 at 5:35 PM, Kyle Hanson <span dir="ltr"><<a href="mailto:hanooter@gmail.com" target="_blank">hanooter@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">OK<div><br></div><div>I have a bunch of BSON documents that I convert to ByteStrings, put in a Map, and write to a socket based on the response. I noticed some high memory usage (in the GBs) so I decided to investigate. I simplified my problem into a small program that demonstrates clearer what is happening. </div>
<div><br></div><div>I wrote two versions, one with a Lazy Map and Lazy ByteStrings and one with a Strict Map and Strict ByteStrings. Both share the same memory behavior (except the lazy BS one is faster)</div><div><br></div>
<div>Here is the strict version:</div><div><br></div><div><a href="http://lpaste.net/92298" target="_blank">http://lpaste.net/92298</a><br></div><div><br></div><div>And here is the lazy version:</div><div><br></div><div>
<a href="http://lpaste.net/92299" target="_blank">http://lpaste.net/92299</a><br>
</div><div><br></div><div>I wrote this and compared the memory and speed behavior of ByteStrings generated by converting it from a BSON document and ByteStrings generated more purely.</div><div><br></div><div>The length of the ByteString from a BSON document is 68k and the length of the "pure" BS is 70k. </div>
<div><br></div><div>This is my weird memory behavior, both BSON and "pure" methods use the same amount of memory after inserting 10k of them (90mb)</div><div><br></div><div>However when I go to lookup a value, the BSON Map explodes the memory to over 250mb. Even if I lookup just 1 value. Looking up any number of values in the "pure BS" keeps the memory usage stable (90mb).</div>
<div><br></div><div>I am hoping someone can help me understand this. I have read some posts about Temporary ByteStrings causing memory issues but I don't know how to get started debugging.</div><div><br></div><div>--</div>
<div>Kyle Hanson</div></div>
<br>_______________________________________________<br>
Haskell-Cafe mailing list<br>
<a href="mailto:Haskell-Cafe@haskell.org">Haskell-Cafe@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/haskell-cafe" target="_blank">http://www.haskell.org/mailman/listinfo/haskell-cafe</a><br>
<br></blockquote></div><br></div>