(Talk a bit about testing with typeclasses.)
Revision as of 20:14, 21 March 2007
- Help library developers ensure that their changes work.
- Enable something like BuildBot to ensure that libraries keep working across all the Implementations.
- Enable test-driven development if a developer wants to use it.
- Provide a series of testcases for implementation developers.
According to Ian Lynagh, SimonM has been moving tests a directory at a time from the testsuite package that comes with ghc to libraries/foo/tests. It looks like he's gotten to unix and network. The tests "assume the package is part of a GHC build tree with the testsuite installed in ../../../testsuite."
Neil Mitchell has a program at http://www.cs.york.ac.uk/fp/darcs/filepath/GenTests.hs to automatically extract tests from Haddock comments like the ones in http://www.cs.york.ac.uk/fp/darcs/filepath/System/FilePath/Version_0_11.hs.
I think the tests should be run with
runhaskell Setup.hs test. Cabal's going to need some work to make this work portably. JeffreyYasskin 06:39, 21 March 2007 (UTC)
3.1 Abbreviated QuickCheck in Haddock comments
Need to recognise parameters. GenTests.hs currently recognises a small range of special variables. What about declaring the parameter names and associated types in a header comment? For instance
-- > :type xs, ys :: [Int]
this would declare that xs and ys are free variables in any following tests and of the type specified. They can then be recognised at the lexical level.
Alternatively, what about special "forall" syntax such as:
forall (xs, ys :: [Int]) . reverse xs ++ reverse ys == reverse (ys ++ xs)
- The problem is that xs and ys are actually of type , and we don't want to mislead people who are reading the documentation. We could automatically substituteforall a. [a]for type variables in forall expressions (which are easy to recognize because they're lower case and between the "::" and the "."). So:Intmeans QuickCheck will see
-- > forall (xs, ys) :: (MyTyCon a b, MyTyCon a b) . myprop xs ys
prop_N = \xs ys -> myprop xs ys where types = (xs, ys) :: (MyTyCon Int Int, MyTyCon Int Int)
- It's likely we'd also eventually need similar defaulting for particular typeclasses, which might force us to integrate more of Haskell than we'd like... JeffreyYasskin 07:06, 21 March 2007 (UTC)
- Oops, I didn't make myself clear. I was thinking of the QuickCheck "forAll" function rather than the "forall" type keyword. What I wanted to say was that we need a way to represent the QuickCheck idiom of .
prop_myTest = forAll myGenerator $ \(x, y) -> myProperty x y
- In the ranged sets library I needed to ensure that properties held for sets of Ints and Reals because they were handled differently (due to the fact that is equivalent tox > 4forx >= 5but not forx :: Int. The library expresses this distinction withx :: Real. Is this the kind of thing you had in mind when you said "defaulting for particular typeclasses"? If so then optionally wrapping the :type lines in an outer structure might work. PaulJohnson 20:14, 21 March 2007 (UTC)class DiscreteOrdered
GenTests currently has flag syntax for Posix and Windows tests (necessary for FilePath). Should this be in a general solution? If so, are other flags required?
- I think we need something with the same functionality. Posix- and Windows-specific tests are relatively easy. You just write the property directly in terms of qualified names. The needed imports could be specified in a recognized comment toward the top of the file. Doing generic tests that should hold over several modules (or several typeclass instances?) is harder. If overloading
-- > import Foo as Fworks, maybe:forallJeffreyYasskin 07:06, 21 March 2007 (UTC)
-- > forall [x <- ["P", "W"]] . x.isPathSeparator x.pathSeparator
3.2 #ifdefed tests in code
http://darcs.haskell.org/packages/base/Data/Map.hs could easily be moved to this form if we replace the
This poses some problems for arranging the compiled output. The #define flags generally don't show up in the name of the output file, but we want to make sure that even if you've run the tests,
runhaskell Setup.hs install installs the non-testing libraries.
dist/test/tree with -DTESTING on. If we have A.hs and B.hs where each exports some implementation functions behind
3.3 Separate test programs
Should be in the style of http://darcs.haskell.org/packages/network/tests/, but with a driver that doesn't depend on living inside a ghc source tree.
I personally dislike the "golden file" approach that these test use, but the existing driver also supports a
run_command_ignore_output test-fn, which we could combine with the
exit_code(0) opt-fn to use any testing framework we want. JeffreyYasskin 06:39, 21 March 2007 (UTC)