[Haskell-cafe] Compressed Data.Map for more efficient RAM usage?

[Antoine Latter]

On Thu, Feb 16, 2012 at 4:29 PM, Johan Tibell <johan.tibell at gmail.com> wrote:
> On Thu, Feb 16, 2012 at 2:03 PM, Antoine Latter <aslatter at gmail.com> wrote:
>> You could have a re-implemented HashMap which would un-pack the
>> payload's ByteString constructor into the leaves of the HashMap type
>> itself.
>>
>> Then you would save on both the keys and the values.

Hah hah - forget what I said. HashMap may be smaller in terms of the
structure of the tree it builds, but it can't be smaller in the size
of its keys since the hash is lossy :-/

>
> Note that ByteString has a high per-value overhead due to the internal
> use of ForeignPtrs and indicies that track offset/size:
> http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.html
>

You could store just the raw GHC ByteArray, which drops the indices.
You could then promote to a ByteString to call into standard
deserialization libraries. I don't know at what point we descend too
far into voodoo-magic.

> -- Johan