Library tests

Goals

1. Help library developers ensure that their changes work.
2. Enable something like BuildBot to ensure that libraries keep working across all the Implementations.
3. Enable test-driven development if a developer wants to use it.
4. Provide a series of testcases for implementation developers.

Status

According to Ian Lynagh, SimonM has been moving tests a directory at a time from the testsuite package that comes with ghc to libraries/foo/tests. It looks like he's gotten to unix and network. The tests "assume the package is part of a GHC build tree with the testsuite installed in ../../../testsuite."

Neil Mitchell has a program at http://www.cs.york.ac.uk/fp/darcs/filepath/GenTests.hs to automatically extract tests from Haddock comments like the ones in http://www.cs.york.ac.uk/fp/darcs/filepath/System/FilePath/Version_0_11.hs.

Design

I think the tests should be run with runhaskell Setup.hs test. Cabal's going to need some work to make this work portably. JeffreyYasskin 06:39, 21 March 2007 (UTC)

Abbreviated QuickCheck in Haddock comments

Need to recognise parameters. GenTests.hs currently recognises a small range of special variables. What about declaring the parameter names and associated types in a header comment? For instance

-- > :type xs, ys :: [Int]

this would declare that xs and ys are free variables in any following tests and of the type specified. They can then be recognised at the lexical level.

Alternatively, what about special "forall" syntax such as:

forall (xs, ys :: [Int]) . reverse xs ++ reverse ys == reverse (ys ++ xs)

The problem is that xs and ys are actually of type forall a. [a], and we don't want to mislead people who are reading the documentation. We could automatically substitute Int for type variables in forall expressions (which are easy to recognize because they're lower case and between the "::" and the "."). So:

-- > forall (xs, ys) :: (MyTyCon a b, MyTyCon a b) . myprop xs ys

means QuickCheck will see

prop_N = \xs ys -> myprop xs ys where types = (xs, ys) :: (MyTyCon Int Int, MyTyCon Int Int)

It's likely we'd also eventually need similar defaulting for particular typeclasses, which might force us to integrate more of Haskell than we'd like... JeffreyYasskin 07:06, 21 March 2007 (UTC)

Oops, I didn't make myself clear. I was thinking of the QuickCheck "forAll" function rather than the "forall" type keyword. What I wanted to say was that we need a way to represent the QuickCheck idiom of

 prop_myTest = forAll myGenerator $ \(x, y) -> myProperty x y

. PaulJohnson 19:48, 21 March 2007 (UTC)

GenTests currently has flag syntax for Posix and Windows tests (necessary for FilePath). Should this be in a general solution? If so, are other flags required?

I think we need something with the same functionality. Posix- and Windows-specific tests are relatively easy. You just write the property directly in terms of qualified names. The needed imports could be specified in a recognized

-- > import Foo as F

comment toward the top of the file. Doing generic tests that should hold over several modules (or several typeclass instances?) is harder. If overloading forall works, maybe:

-- > forall [x <- ["P", "W"]] . x.isPathSeparator x.pathSeparator

JeffreyYasskin 07:06, 21 March 2007 (UTC)

#ifdefed tests in code

http://darcs.haskell.org/packages/base/Data/Map.hs could easily be moved to this form if we replace the {- ... Testing ... -} comments with #ifdef TESTING .... #endif. We'll need to extend Haddock to be able to copy function bodies into the generated documentation.

This poses some problems for arranging the compiled output. The #define flags generally don't show up in the name of the output file, but we want to make sure that even if you've run the tests, runhaskell Setup.hs install installs the non-testing libraries.

We could probably re-build all of the libraries in a separate dist/test/ tree with -DTESTING on. If we have A.hs and B.hs where each exports some implementation functions behind #ifdef TESTING, this gives B's tests access to A's private functions. Is that a problem? JeffreyYasskin 06:39, 21 March 2007 (UTC)

Separate test programs

Should be in the style of http://darcs.haskell.org/packages/network/tests/, but with a driver that doesn't depend on living inside a ghc source tree.

I personally dislike the "golden file" approach that these test use, but the existing driver also supports a run_command_ignore_output test-fn, which we could combine with the exit_code(0) opt-fn to use any testing framework we want. JeffreyYasskin 06:39, 21 March 2007 (UTC)