[Haskell-cafe] Slow IO

[Daniel Fischer]

Am Dienstag, 12. September 2006 22:26 schrieben Sie:
> Daniel Fischer wrote:
> > The programme consumed more and more memory (according to top),
> > kswapd started to have a higher CPU-percentage than my programme,
> > programme died, system yelling 'Speicherzugriffsfehler', top displays
> > 'kswapd<defunct>'.
> > I believe that means my programme demanded more memory than I have
> > available (only 256MB RAM + 800MB swap). Is that a segfault or what is
> > the correct term?
> >
> > That is probably due to (apart from the stupidity of my IO-code) the
> > large overhead of Haskell lists.
>
> Most certainly not.  I'm pretty sure this is to a bug in your code.
> Something retains a data structure which is actually unneeded.  Probably

Apparently. And my money is on a load of lines from the file (of which I need 
only the first and last Char).

> a case of "foldl" where "foldl'" should be used or a "try" in Parsec
> code where it should be left out or a lot of "updateWiths" to a Map,
> etc.  Or it could be a bad choice of data structure.  I bet, it's the
> map you're using to represent the graph (which you don't even need to
> represent at all, btw).

No foldl nor parsec around. I represent the graph as a

UArray (Char,Char) Int 

(I've switched to Int for the index type, too, when tuning the code), so that 
shouldn't use much memory (array size is 676).
The array is built via accumArray, I hope that's sufficiently efficient
(though now I use unsafeAccumArrayUArray, that's faster).

How could I solve the problem without representing the graph in some way?
Possibly that could be done more efficiently than I do it, but I can't imagine 
how to do it without representing the graph in some data structure.
>
> > So the chunk of the file which easily fits into my
> > RAM in ByteString form is too large as a list of ordinary Strings.
>
> The chunk of file should never need to fit into RAM.  If that's a
> problem, you also forgot to prime a crucial "foldl".
>

Forgive the stupid question, but where if not RAM would the chunk currently 
processed reside?

>
> Udo.

Cheers,
Daniel

-- 

"In My Egotistical Opinion, most people's C programs should be
indented six feet downward and covered with dirt."
	-- Blair P. Houghton