[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Thomas Tuegel]

On Tue, Apr 6, 2010 at 7:43 PM, Rogan Creswick <creswick at gmail.com> wrote:
> test-framework and test-runner both address the second problem, and
> those solutions can be kept separate, at least for now.  Figuring out
> the best way to specify test commands, dependencies, build/execution
> order, etc. is going to take some substantial effort, and I think that
> should be the first goal of the project.

Ok, this is the bottom-line that I didn't understand after our first
exchange, but I think now I do: I should entirely scrap the second
aspect of my proposal and focus exclusively on making Cabal build and
run test programs.

> Cabal allows for multiple executable sections -- are multiple test
> sections allowed? If so, how are they handled when 'cabal test' is
> invoked?  If not, will there be any support for running multiple test
> suites? (more on this below).
>
> While the test executable could be configured to run different sets of
> tests (at runtime? hm.. we may need more flags to 'cabal test'), there
> are some situations it's necessary to build multiple test suites
> because of odd library dependencies.  (for example, testing certain
> combinations of libraries--imagine supporting multiple versions of
> ghc.)

I had intended to allow multiple 'Executable' sections; that's what I
meant by "this is really an 'Executable' stanza by another name".  I
think that takes care of your concern about building multiple test
suites with different dependencies, also.

My thinking is that 'cabal test' should run all the tests that were
built, and 'cabal test foo' should run only the test named 'foo'.

> The existing Executable sections may serve the need fine, if we could
> specify how to run the tests in a different way.  Perhaps a list of
> test commands could be specified instead, eg:
>
>> TestCommands: foo-test-ghc6.6,
>>                            foo-test-ghc6.8,
>>                            foo-props --all
>
> Anyhow, just food for thought.

One of the reasons I prefer implementing a dedicated 'Test' stanza is
that it makes it easier to tell which executables to install.  There
may be situations where we want to install test programs, but there
will _always_ be situations where we _don't_ want to.  (Maybe
'--{en,dis}able-tests' passed to 'cabal install' should control this?)
 Could we take a 'TestCommands' list and parse out the options to get
the executable names?  Yes, but relying on that to work makes me
uneasy.  I think the more robust way to specify executable-specific
options instead would be to add a field to the 'Test' section
('run-options' seems to be consistent with the naming scheme) that
doesn't exist in 'Executable'.

(I've snipped some of the comments here; if forget about test
frameworks for this proposal, it's all tangential.)

> See my comments about running multiple test suites, and parameterized
> test suites above.  I think richer parameters are necessary.
> (possibly just a --pass-through flag that hands all the subsequent
> parameters off to the test executable(s))

That is an excellent suggestion that I will definitely adopt in my
eventual proposal.

Thanks!
-- 
Thomas Tuegel