Hi all,<div><br></div><div>I'm very much looking forward to a future where cabal install exercises all my core's with some heavy duty Haskell work ;-) Thanks Frank for taking this up.</div><div><br></div><div>I personally like progress reports on the individual builds very much. I agree that they are not super important, but nevertheless I think that a progress report significantly improves the user experience. I also have a simple, ad-hoc scheme that should result in an OK progress report for most cases.</div>
<div><br></div><div>Gather patterns of the form</div><div><br></div><div> "["<integer>" of "<integer>"]"</div><div><br></div><div>in the program output and interpret the resulting sequence such that the second to last "measurement" is a conservative estimate of the real progress; i.e.,</div>
<div><br></div><div>progress :: [(Int,Int)] -> Maybe Double</div><div>progress xs = case reverse xs of</div><div> (_:(i,n):_) -> return (fromIntegral i / fromIntegral n)</div><div> _ -> mzero</div><div>
<br></div><div>Probably some more filtering of this sequence is required to cater for repeated calls to GHC. I guess that, as long as progress never goes from 100% to something below, the user will be happy about the progress estimate. Moreover, the chance that such a pattern occurs where it doesn't indicate some interesting progress is reasonably low.</div>
<div><br></div><div>best regards,</div><div>Simon</div><div><br></div><div><br><div class="gmail_quote">2011/3/30 Johan Tibell <span dir="ltr"><<a href="mailto:johan.tibell@gmail.com">johan.tibell@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Frank,<br>
<br>
Thanks for reaching out and gathering input.<br>
<div class="im"><br>
On Tue, Mar 29, 2011 at 9:54 PM, Frank Murphy <<a href="mailto:anirishduck@gmail.com">anirishduck@gmail.com</a>> wrote:<br>
> - Parallelize executeInstallPlan. When given a target load average as a flag it<br>
> will determine whether it should spawn a worker (if below the target load<br>
> average) or wait. If waiting, it will listen to all worker status channels<br>
> and print out their current build status and the load average. Once a worker<br>
> exits, it will again check the load average and spawn a new thread if<br>
> necessary.<br>
<br>
</div>I think the most important setting is the number of worker threads<br>
(e.g. -jX). Load average sounds like a cool idea but I don't know how<br>
well it'll work in practice. Gentoo's Portage uses it so you might<br>
snoop around there for more info.<br>
<div class="im"><br>
> - Rewrite install.*Package and their callees to use the CHP (Communicating<br>
> Haskell Process) monad where possible. Use channels to communicate build<br>
> status back to the main thread.<br>
<br>
</div>CHP might be a bit overkill, an MVar and a Chan or two should be<br>
enough. At least start simple.<br>
<div class="im"><br>
> - It might be necessary to parse the output of external builds in some way so<br>
> that meaningful status can be communicated back to the user.<br>
<br>
</div>I'm not sure this is worth it and even possible in the general case. See below.<br>
<div class="im"><br>
> - Add a default parallel build log path template. Allow the user to specify one<br>
> on the command line to override the default.<br>
<br>
</div>I'm not quite sure what you mean here. Do you mean that we'd write<br>
"cabal install" logs to e.g. .cabal/logs or something along those<br>
lines?<br>
<div class="im"><br>
> - On single-threaded (sequential) builds, revert to the old output style.<br>
<br>
</div>Sounds good. One possible policy would be: If you run "cabal build",<br>
you get the old output format (and a single threaded build). If you<br>
run "cabal install", you get the new output format, regardless of if<br>
the build runs in parallel or not.<br>
<br>
What do people think? Is it worth displaying all the build output for<br>
"cabal install" in the single threaded case? Does the user care to see<br>
it? Perhaps it's good for debugging to let single threaded "cabal<br>
install" show the old output (i.e. if a parellel build fails, run the<br>
single threaded one to get more output).<br>
<div class="im"><br>
> On multi-threaded builds, display the current status of all running builds, load<br>
> averages and nothing else. Possible output:<br>
><br>
> Resolving dependencies...<br>
> Building derive-2.3.0.2... [17 of 58]<br>
> Building regex-base-0.93.1... [1 of 4]<br>
> Building dyre-0.8.6... [5 of 7]<br>
> Configuring xdg-basedir-0.2... [in progress]<br>
><br>
> Dependencies Built: [0 of 9]<br>
> Load Average: [3.4/4.0]<br>
> Running 4 Jobs.<br>
<br>
</div>Cabal allows packages to use any build system they want (e.g. make),<br>
which means that we can't know the progress of a single build in the<br>
general case. Today, Cabal simply shows the stdout of the build<br>
process, whatever it is. This means that we cannot show progress of<br>
individual packages. I suggest something like (take from Gentoo's<br>
Portage):<br>
<br>
Building (1 of 9) derive-2.3.0.2...<br>
Building (2 of 9) regex-base-0.93.1...<br>
Building (3 of 9) dyre-0.8.6...<br>
Building (4 of 9) aeson-0.3.2.1...<br>
Building (5 of 9) binary-0.5.0.2...<br>
Installing derive-2.3.0.2<br>
Installing regex-base-0.93.1<br>
Building (6 of 9) text-0.11.0.6...<br>
Installing dyre-0.8.6<br>
Jobs: 3 of 9 complete, 3 running Load avg: 3.44, 1.46, 0.69<br>
<br>
We could perhaps make a special case for the Simple build type and<br>
parse the GHC output and show progress on individual builds. I don't<br>
think it's worth it, at least not initially.<br>
<div class="im"><br>
> A possible error message might look like:<br>
><br>
> derive-2.3.0.2 failed during the building phase.<br>
> Log stored in /home/frank/cabal/logs/build/derive-2.3.0.2.log<br>
<br>
</div>For build failures I think we should output the content of the log<br>
file to stdout (as one chunk, using a lock to avoid interleaving).<br>
This will make it quicker for users to get to the build failure. For<br>
successful builds I don't think we need to output more than in the<br>
example above.<br>
<br>
Cheers,<br>
<font color="#888888">Johan<br>
</font><div><div></div><div class="h5"><br>
_______________________________________________<br>
cabal-devel mailing list<br>
<a href="mailto:cabal-devel@haskell.org">cabal-devel@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/cabal-devel" target="_blank">http://www.haskell.org/mailman/listinfo/cabal-devel</a><br>
</div></div></blockquote></div><br></div>