[Haskell-cafe] Re: sizeFM type

[Tomasz Zielonka]

On Tue, Apr 27, 2004 at 02:05:06PM -0400, S. Alexander Jacobson wrote:
> On Tue, 27 Apr 2004, Tomasz Zielonka wrote:
> > On Mon, Apr 26, 2004 at 08:43:44PM -0400, ajb at spamcop.net wrote:
> >
> > It seems I qualify as almost nobody, cause I've had a database with more
> > than 2^34 records on a single IA32 machine. However, I wouldn't ever
> > think of loading the entire database to RAM.
> 
> Why not?  Its only 17gb.

About 17000000000 records, yes, but not 17GB. It was more like 300 GB
compressed data.

> If you are doing any substantive volume of transactions on that
> database* you may be better off storing the database in memory than
> buying the multiple CPUs and reliable RAID arrays you need to serve it
> off disk.

I was not a typical relational database, so it's hard to compare. I
managed on a 1 CPU P4 2.6 with 1GB mem and a RAID5 array. The database
engine was optimised for bulk processing, reducing disk seeks and memory
requirements.

> A Dell PowerEdge 4600 w/ 24gb of memory and a
> write-caching controller w/ 6 36GB harddrives
> (RAID0 state,logging,status) costs around $34k.
> <http://www1.us.dell.com/content/products/compare.aspx/tower?c=us&cs=555&l=en&s=biz>
> Using e.g. a Prevayler style in memory logged data
> structures** you may be able to serve >100,000
> queries/second. (see e.g.
> <http://www.prevayler.org/wiki.jsp?topic=ScalabilityTestResults>)

We were able to achieve a throughput of 200,000 records per seconds (not
queries), which was about 3 times faster then necessary.

The data you provided is interesting. However, if I had such a machine,
my work could be much less challenging :)

> * OTOH, if this is just your personal mail archive
> then nevermind :- )

No.

Best regards,
Tom

-- 
.signature: Too many levels of symbolic links