Checkins and test passing
simonmarhaskell at gmail.com
Mon Oct 23 11:24:33 EDT 2006
Malcolm Wallace wrote:
> Simon Marlow <simonmarhaskell at gmail.com> wrote:
>>Malcolm, you're misunderstanding the use of the term "fail" in this
>>context. I stated it in that message, but you cut off the quote:
>> fail = "exhibits incorrect behaviour"
>>This makes perfect sense in the context of the result of a particular
>>*test*. You want to know whether the test demonstrated incorrect
>>behaviour of the thing you were testing; whether the thing returned a
>>non-zero exit code or not is beside the point.
> Ah, yes I'm sorry, I was indeed misunderstanding. The difference
> between "program failure" and "test failure" is pretty subtle. Although
> in most cases they are the same thing,
I wouldn't say in most cases they're the same: for example, out of GHC's
typechecker tests, 168 are tests where the compiler should emit an error
(carefully avoiding using the word "fail" here :-) and 216 where the compiler
should compile the module successfully. It's just as important to test what the
compiler refuses to compile as it is to test what it does compile, and to test
that the error messages remain relevant and accurate.
In fact, for a long time I've wanted to merge the 'should_fail' and
'should_compile' directories in the test suite. They're all just tests, not to
mention the fact that this terminology is obviously confusing (should_fail does
not contain "expected failures"!).
More information about the Cvs-ghc