[Haskell-cafe] OS swapping and haskell data structures

[Alex Jacobson]

Ok, so for low throughput applications, you actually need a disk 
strategy.  Got it.

Ok, is there a standard interface to BerkleyDB or some other disk based 
store?

-Alex-




Duncan Coutts wrote:
> On Wed, 2007-08-01 at 11:31 -0700, Bryan O'Sullivan wrote:
>> Alex Jacobson wrote:
>>> If you create a Data.Map or Data.Set larger than fits in physical 
>>> memory, will OS level swapping enable your app to behave reasonably or 
>>> will things just die catastrophically as you hit a memory limit?
>> Relying on the OS to page portions of your app in and out should always 
>> be the fallback of last resort.  You are fairly guaranteed to get 
>> terrible performance because the VM subsystem can't anticipate your 
>> app's memory access patterns, and catastrophic death of either your app 
>> or other system processes is a strong possibility (Google for "OOM 
>> killer" if you want some horror stories).  In many cases, you can't even 
>> rely on paging being possible.
> 
> Furthermore, as I understand it, GC does not interact well with paging
> since the GC has to traverse the data structures on major GCs it'll
> force it all to be kept in memory.
> 
> Duncan