regarding testsuite

[Claus Reinke]

>> - why do i need to install python to run haskell tests? isn't
>>    haskell good enough (dogfood and all that..)?
> 
> A couple of reasons come to mind:
> 
> * The testsuite driver itself is actually a pretty good test of low
>  level stuff: forking, threads for timing out tests, etc. Both old GHC
>  (used to build the HEAD) and the HEAD itself may not work correctly on
>  all platforms, and it's easier to track down bugs in something
>  smaller than the testsuite driver.

yes, and i recall a time when our HaRe testsuite would run beautifully
on win98, because ghc's running external programs would always 
return ExitSuccess there, no matter what!-)

but we're talking about testing ghc here, so we want to expose bugs.
it sounds like a bootstrap problem: write a small program that exercises
the low-level stuff needed for the main testsuite, and run that first: if it 
works start the main suite of tests, otherwise, you've got a small 
program to track down bugs in.

> * Python, being interpreted, makes a nicer language in which to make an
>  EDSL for this.

i'm not sure i buy that, but if there was truth in it, that ought to 
change!-) nothing against python itself, but i thought the aim was 
to get rid of dependencies, such as perl, rather than add them.

and seeing ghc recommending python instead of haskell for
writing testsuites seems like setting a bad example!-)
 
>> - the usage documentation page is only online. it should be
>>    copied into the download, for offline users.
> But then we have the problem of keeping it in sync.

that is exactly the problem your users run into, only aggravated
by not having the page in the first place. can't the trak wiki push
the plain text version of the wiki page into the repo after each edit?

>>    (and can it be that 'make
>>    distclean' deleted my test-orig.log file that i kept for
>>    comparison with after-patch results?).
> 
> If it did then that sounds like a bug to me.

nevertheless, '*.log' is part of MOSTLY_CLEAN_FILES
(mk/paths.mk), so make distclean will try to delete files that 
weren't created by make or configure. interestingly, there's 
also '*_stub.[ch]' in there, but there are usually some of 
those left over in the libraries after distclean.

so, make distclean's coverage is neither sufficient nor 
necessary. currently, the former hurts me more often than 
the latter, but neither is nice, really.

ignoring boring files when recording is one thing, but deleting
files not re-creatable is quite another. perhaps an approach
similar to cabal's dist/ directory might work, directing all
output into special directories, which can be removed as one?
 
>>    -utf8_005.hs:2:0: lexical error (UTF-8 decoding error)
>>    +utf8_005.hs:2:0: lexical error at end of input ]
> 
> This is caused by using an old version of alex.

thanks, will try to update.
 
>> - it takes a lot more that 5 minutes to run 
>>    'make -C testsuite/tests/ghc-regress/ fast stage=2'
> 
> This is at least partly because you've compiled the extralibs, which
> means you're running more tests.

but i thought that 'fast' was meant to select a specific subset
of tests, namely the same subset used for validate? how can
that vary with the libs i've got?

is that also the reason why there are unexpected failures/
successes left in there?
 
>>    running the tests, from within that directory, i was
>>    surprised to see tests outside that directory being
>>    run (such as libraries/network/tests)
> 
> That's meant to happen.

that is a rather confusing and very non-standard property,
isn't it? not just because testsuite/tests has separate 
directories for h98 and libraries, which would suggest
otherwise, but also because, generally, make makes in
subdirectories, not parents or children of siblings of parents.

it is also inconsistent, because the docs tell me to go to
subdirectories of the testsuite, such as ghci, to run subsets
of tests.
 
>> - libraries/network/net00[12] ran into my firewall, and
>>    since i had not expected them, nor could find them
>>    in the test directory (see above), i blocked them.
>>    not that that stopped the testsuite in any way, so
>>    it took me a while to reassure myself that these
>>    were run as part of the testsuite, but it is strange
>>    that the blocking does not affect them..
> 
> net001 certainly ought to fail if it is blocked. Are you sure 
> your firewall really blocked it?

it told me so, and added net001 to the list of disabled
exceptions, but no, it doesn't seem to be blocked (not
that it should, being local). another case of me vs the
ms firewall - me losing, of course..

thanks,
claus