Personal tools

Library tests

From HaskellWiki

Jump to: navigation, search


1 Goals

  1. Help library developers ensure that their changes work.
  2. Enable something like BuildBot to ensure that libraries keep working across all the Implementations.
  3. Enable test-driven development if a developer wants to use it.
  4. Provide a series of testcases for implementation developers.

2 Status

According to Ian Lynagh, SimonM has been moving tests a directory at a time from the testsuite package that comes with ghc to libraries/foo/tests. It looks like he's gotten to unix and network. The tests "assume the package is part of a GHC build tree with the testsuite installed in ../../../testsuite."

Neil Mitchell has a program at to automatically extract tests from Haddock comments like the ones in uses a single Tests.hs file and some Template Haskell. It's integrated with both Cabal and darcs.

3 Design

I think the tests should be run with runhaskell Setup.hs test. Cabal's going to need some work to make this work portably. JeffreyYasskin 06:39, 21 March 2007 (UTC)

3.1 Abbreviated QuickCheck in Haddock comments

Need to recognise parameters. GenTests.hs currently recognises a small range of special variables. What about declaring the parameter names and associated types in a header comment? For instance

-- > :type xs, ys :: [Int]

this would declare that xs and ys are free variables in any following tests and of the type specified. They can then be recognised at the lexical level.

Alternatively, what about special "forall" syntax such as:

forall (xs, ys :: [Int]) . reverse xs ++ reverse ys == reverse (ys ++ xs)
The problem is that xs and ys are actually of type
forall a. [a]
, and we don't want to mislead people who are reading the documentation. We could automatically substitute
for type variables in forall expressions (which are easy to recognize because they're lower case and between the "::" and the "."). So:
-- > forall (xs, ys) :: (MyTyCon a b, MyTyCon a b) . myprop xs ys
means QuickCheck will see
prop_N = \xs ys -> myprop xs ys where types = (xs, ys) :: (MyTyCon Int Int, MyTyCon Int Int)
It's likely we'd also eventually need similar defaulting for particular typeclasses, which might force us to integrate more of Haskell than we'd like... JeffreyYasskin 07:06, 21 March 2007 (UTC)
Oops, I didn't make myself clear. I was thinking of the QuickCheck "forAll" function rather than the "forall" type keyword. What I wanted to say was that we need a way to represent the QuickCheck idiom of
 prop_myTest = forAll myGenerator $ \(x, y) -> myProperty x y
There's tension here between the actual property that a function satisfies and the code QuickCheck needs in order to check that property. I'm worried that the documentation will be worse if we encourage people to include implementation details in it. This should be the 80% solution—if they need QuickCheck's full power, they should fall back to one of the next two methods. Of course, maybe
'is' needed for most tests, in which case, I'd agree. JeffreyYasskin 19:01, 24 March 2007 (UTC)
In the ranged sets library I needed to ensure that properties held for sets of Ints and Reals because they were handled differently (due to the fact that
x > 4
is equivalent to
x >= 5
x :: Int
but not for
x :: Real
. The library expresses this distinction with
class DiscreteOrdered
. Is this the kind of thing you had in mind when you said "defaulting for particular typeclasses"? If so then optionally wrapping the :type lines in an outer structure might work. PaulJohnson 20:14, 21 March 2007 (UTC)
Yes. Although I'd rather avoid :type lines entirely, since they don't seem to belong in documentation. JeffreyYasskin 19:01, 24 March 2007 (UTC)

GenTests currently has flag syntax for Posix and Windows tests (necessary for FilePath). Should this be in a general solution? If so, are other flags required?

I think we need something with the same functionality. Posix- and Windows-specific tests are relatively easy. You just write the property directly in terms of qualified names. The needed imports could be specified in a recognized
-- > import Foo as F
comment toward the top of the file. Doing generic tests that should hold over several modules (or several typeclass instances?) is harder. If overloading
works, maybe:
-- > forall [x <- ["P", "W"]] . x.isPathSeparator x.pathSeparator
JeffreyYasskin 07:06, 21 March 2007 (UTC)

ed tests in code could easily be moved to this form if we replace the
{- ... Testing ... -}
comments with
#ifdef TESTING .... #endif
. We'll need to extend Haddock to be able to copy function bodies into the generated documentation.

This poses some problems for arranging the compiled output. The #define flags generally don't show up in the name of the output file, but we want to make sure that even if you've run the tests, runhaskell Setup.hs install installs the non-testing libraries.

We could probably re-build all of the libraries in a separate dist/test/ tree with -DTESTING on. If we have A.hs and B.hs where each exports some implementation functions behind
#ifdef TESTING
, this gives B's tests access to A's private functions. Is that a problem? JeffreyYasskin 06:39, 21 March 2007 (UTC)

3.3 Separate test programs

Should be in the style of, but with a driver that doesn't depend on living inside a ghc source tree.

I personally dislike the "golden file" approach that these test use, but the existing driver also supports a run_command_ignore_output test-fn, which we could combine with the exit_code(0) opt-fn to use any testing framework we want. JeffreyYasskin 06:39, 21 March 2007 (UTC)