[Haskell-cafe] 1G strings in Haskell
Donald Bruce Stewart
dons at cse.unsw.edu.au
Thu Apr 20 03:34:53 EDT 2006
> On 4/19/06, Donald Bruce Stewart <dons at cse.unsw.edu.au> wrote:
> > Question:
> > Can I manipulate 1G strings in Haskell?
> > Failed due to memory exhaustion.
> > Almost made it though, just need a tad more ram than I had.
> > filter !
> > unlines !
> > unwords !
> > reverse ! -- copy
> > cons ! -- copy
> > snoc ! -- involves a copy
> > ++ ! -- can't concat two 1G strings on this box
> I would say given the nature of the experiment that you want to feed
> these functions with values that lead to 1 GB results. At least, that
> makes sense in the case of ++, but maybe not the others.
Yes, good point. Previous experiments with 500M strings indicated that
these all worked fine:
Size of test data: 531416k
Still wouldn't work at 0.5G
> Did the machine have any available swap? If so, why was it not used?
Swap was used, 1G more, but ulimits kicked in a few times (and I'm not
an admin on the box). This was a bit frustrating, but the 0.5G results
show what is possible.
> The strings had to be in memory all at once to not skew the benchmark
> by including I/O time?
The strings were all in memory, yes, with some deepSeq tricks.
More information about the Haskell-Cafe