[Haskell-cafe] High memory usage with 1.4 Million records?

[Thomas Schilling]

On 8 June 2012 01:39, Andrew Myers <asm198 at gmail.com> wrote:
> Hi Cafe,
> I'm working on inspecting some data that I'm trying to represent as records
> in Haskell and seeing about twice the memory footprint than I was
> expecting.

That is to be expected in a garbage-collected language. If your
program requires X bytes of memory then allocators will usually
trigger garbage collection once the heap reaches a size of 2X bytes.
If it didn't do this then every allocation would require a GC.  You
can change this factor with +RTS -F option.  E.g., +RTS -F1.5 should
reduce this to only 50% overhead, but will trigger more frequent
garbace collections. To find the actual residency (live data) see the
output of +RTS -s

There may still be room for improvement.  For example, you could try
turning on the compacting GC -- which trades GC performance for lower
memory usage.  You can enable it with +RTS -c

The reason that -hc runs slowly is that it performs a GC every 1s (I
think).  You can change this using the -i option.  E.g.  -i60 only
examines the heap every 60s.  It will touch almost all your live data,
so it is an inherently RAM-speed bound operation.

HTH,
 / Thomas