glynn at gclements.plus.com
Fri Apr 22 16:56:20 EDT 2005
David Brown wrote:
> >> And hGet/PutWord8 are fast enough for most situations.
> > Are you certain? Which interesting applications do you know
> > that read and write one byte at a time? I can't speak for
> > "most" situations, but in those situations where I needed
> > binary I/O this API would have been impracticable.
> I'm going to chime in agreement with the disagreement.
> I have one application which uses single character binary I/O. Because it
> works on small files, it borders on usefuleness. As time goes on, the
> size of these inputs grows, and now the tool has become nearly worthless.
> I seem to run into this kind of thing a lot with Haskell. Dominic's ASN
> library is useful to me, however I'm finding I'll probably just have to
> use it as a template for new code. It works fine for what designed for,
> parsing keys and such, but I'm looking to use it to represent gigabytes of
> data. Processing data like that one Word8 at a time isn't practical.
Personally, I doubt that Haskell will ever be practical for processing
very large amounts of data (e.g. larger than your system's RAM).
When processing large amounts of data, rule #1 is to do as little as
possible to most of the data; don't even read it into memory if you
can avoid it (e.g. create an index and use lseek() as much as
possible). You certainly don't want to be boxing/unboxing gigabytes.
Glynn Clements <glynn at gclements.plus.com>
More information about the Libraries