Any remaining test patches?

[Thomas Tuegel]

On Sat, May 21, 2011 at 9:20 AM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:
> data TestInstance
>   = TestInstance {
>       run            :: IO TestResult
>       name           :: String,
>
>       concurrentSafe :: Bool,
>       expectedFail   :: Bool,
>
>       options        :: [OptionDescr]
>       setOption      :: String -> String -> Either String TestInstance
>     }

> Note that I don't have an IO action to get any default options. In
> this design it must always be possible to run the test with no options
> supplied. Internally, if a test does need some options specified then
> it just has to fill them in with its internal defaults.

> data OptionDescr
>   = OptionDescr {
>       optionName        :: String,
>       optionDescription :: String,
>       optionType        :: OptionType,
>       optionDefault    :: Maybe String
>     }

The reason I used an IO action to get at the default options was to
allow extraction of the random seed that was going to be used, e.g.,
by QuickCheck tests, so the test would be reproducible. Since this
design doesn't use IO anywhere, is it still possible for the test
agent to record the random seed used? That is, if optionType ==
OptionRngSeed, can optionDefault be anything other than Nothing?

> data OptionType
>   = OptionFile {
>       optionFileMustExist  :: Bool,
>       optionFileIsDir      :: Bool,
>       optionFileExtensions :: [String]
>     }
>   | OptionString {
>       optionStringMultiline :: Bool
>     }
>   | OptionNumber {
>       optionNumberIsInt   :: Bool,
>       optionNumberBounds  :: (Maybe String, Maybe String)
>     }
>   | OptionBool
>   | OptionEnum  [String]
>   | OptionSet   [String]
>   | OptionRngSeed
>
> You can imagine how each of these could be used as hints to make a
> more helpful web form or GUI interface. They could also let a command
> line ui do some up front validation too.
>
> The most general case is OptionString for when no more specific
> alternative is appropriate.

Perhaps we could add a field to the OptionString constructor to take a
human-readable description of what is expected? I wouldn't expect a
test framework to necessarily accept just any string, for example; I
can see this option type being used for custom types that don't really
fit any of the predefined types.

> Honestly, I don't know why Max did it this way, seems to me a much
> more direct approach is:
>
> data TestProgress
>    = Finished TestResult
>    | Progress String (IO TestProgress)
>
> So it's not a lazy list, it's a list with explicit IO to get the tail.
> The test-framework approach could be lifted into this fairly easily I
> think.

I really like this approach. When I wrote my interface, I wanted to
allow tests to return partial progress, but I couldn't devise a way to
do it without Max's Concurrent-based approach (and I wasn't sure that
was appropriate for Cabal).

-- 
Thomas Tuegel