[Hackage] #288: the package indexes are very slow

[Duncan Coutts]

On Wed, 2010-07-14 at 22:37 -0500, Antoine Latter wrote:
> On Wed, Jul 14, 2010 at 4:48 PM, Hackage <cvs-ghc at haskell.org> wrote:
> > Comment(by duncan):
> >
> >  It should be straightforward to extend the size of the types used to cope
> >  with bigger tarballs. The only cost will be a bigger index. The reason for
> >  the limitations in the hackage code is simply to save space by keeping the
> >  indexes very compact.
> >
> 
> Yup. My branch of cabal-install has this done:
> 
> http://community.haskell.org/~aslatter/code/cabal-install/index/
> 
> which requires:
> 
> http://community.haskell.org/~aslatter/code/tarindex/
> 
> This should be considered a prototype, as we would need to more
> carefully consider what dependencies we should be pulling in for this.
> 
> For "cabal list $pkgname" I go from ~1s to ~550ms.
> For "cabal install --dry-run happstack" I go from ~2.5s to ~1.9s.
> 
> According to GHC profiling most of the time spent in "cabal list" is
> in looking up paths in the tar-index offset store (this happens ~9000
> times). I don't have profiling data through package boundaries for
> some reason, so there might yet be low-hanging fruit over there.

Nice, I'll try and find some time to look at this.

Duncan