Checkins and test passing

[Simon Marlow]

Malcolm Wallace wrote:
> Simon Marlow <simonmarhaskell at gmail.com> wrote:
> 
> 
>>Malcolm, you're misunderstanding the use of the term "fail" in this
>>context.  I  stated it in that message, but you cut off the quote:
>>
>>    fail     = "exhibits incorrect behaviour"
>>
>>This makes perfect sense in the context of the result of a particular
>>*test*.  You want to know whether the test demonstrated incorrect
>>behaviour of the thing  you were testing; whether the thing returned a
>>non-zero exit code or not is  beside the point.
> 
> 
> Ah, yes I'm sorry, I was indeed misunderstanding.  The difference
> between "program failure" and "test failure" is pretty subtle.  Although
> in most cases they are the same thing,

I wouldn't say in most cases they're the same: for example, out of GHC's 
typechecker tests, 168 are tests where the compiler should emit an error 
(carefully avoiding using the word "fail" here :-) and 216 where the compiler 
should compile the module successfully.  It's just as important to test what the 
compiler refuses to compile as it is to test what it does compile, and to test 
that the error messages remain relevant and accurate.

In fact, for a long time I've wanted to merge the 'should_fail' and 
'should_compile' directories in the test suite.  They're all just tests, not to 
mention the fact that this terminology is obviously confusing (should_fail does 
not contain "expected failures"!).

Cheers,
	Simon