https://wiki.haskell.org/api.php?action=feedcontributions&user=Steve+C&feedformat=atomHaskellWiki - User contributions [en]2021-03-07T19:05:56ZUser contributionsMediaWiki 1.27.4https://wiki.haskell.org/index.php?title=Tail_recursion&diff=34800Tail recursion2010-05-23T09:13:04Z<p>Steve C: Make 'foldl' the object of the link, and move below quote from Brent.</p>
<hr />
<div>A recursive function is tail recursive if the final result of the<br />
recursive call is the final result of the function itself. If the<br />
result of the recursive call must be further processed (say, by adding<br />
1 to it, or consing another element onto the beginning of it), it is<br />
not tail recursive.<br />
<br />
With that said, tail recursion is not that useful of a concept in a<br />
lazy language like Haskell. The important concept to know in Haskell<br />
is [[guarded recursion]], where any recursive calls occur within a data<br />
constructor (such as <hask>foldr</hask>, where the recursive call to foldr occurs<br />
as an argument to <hask>(:)</hask>). This allows the result of the function to be<br />
consumed lazily, since it can be evaluated up to the data constructor<br />
and the recursive call delayed until needed.<br />
<br />
Note that [[Fold|foldl]] is always tail recursive.<br />
<br />
== Source ==<br />
<br />
* Brent Yorgey in Haskell-Cafe on [http://www.haskell.org/pipermail/haskell-cafe/2009-March/058607.html Definition of "tail recursive" wrt Folds]<br />
<br />
[[Category:Glossary]]<br />
[[Category:Idioms]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=32791Prime numbers2009-12-31T11:56:27Z<p>Steve C: Add "Using ST Array" section</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a <i>prime number</i> (or a <i>prime</i>) is a natural number which has exactly two distinct natural number divisors: 1 and itself. The smallest prime is thus 2.<br />
<br />
[http://en.wikipedia.org/wiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Finding Primes ==<br />
<br />
Any natural number is representable as a product of powers of its prime factors, and so a prime has no prime divisors other than itself. That means that starting with 2, <i>for each</i> newly found <i>prime</i> we can <i>eliminate</i> from the rest of the numbers <i>all such numbers</i> that have this prime as their divisor, giving us the <i>next available</i> number as next prime. This is known as <i>sieving</i> the natural numbers, so that in the end what we are left with are just primes. <br />
<br />
=== The Classic Simple Primes Sieve ===<br />
<br />
Attributed to David Turner <i>(SASL Language Manual, 1975)</i>, the following is a direct translation of that idea, generating a list of all prime numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where<br />
sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0] <br />
-- or: filter ((/=0).(`mod`p)) xs<br />
</haskell><br />
<br />
This should only be considered a <i>specification</i>, not a <i>code</i>. When run as is, it is <i>extremely inefficient</i> because it starts up the filters prematurely, immediately after each prime, instead of only after the prime's square has been reached. To be admitted as prime, <i>each number</i> will be <i>tested for divisibility</i> here by all its preceding primes, while just those not greater than its square root would suffice. This means that e.g. to find the <b>1001</b>st prime (<code>7927</code>), <b>1000</b> filters are used, when in fact just the first <b>24</b> are needed (upto <code>89</code>'s filter only).<br />
<br />
So this in effect creates a cascade of nested filters in front of the infinite numbers supply, and in <i>extremely premature</i> fashion at that. One way of fixing that would be to <i>postpone</i> the creation of filters until the right moment, by decoupling the primes supply from the numbers supply, as in<br />
<br />
=== Postponed Filters Sieve ===<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..]<br />
where <br />
sieve (p:ps) xs = h ++ sieve ps [x | x<-t, x `rem` p /= 0] <br />
-- or: filter ((/=0).(`rem`p)) t<br />
where (h,~(_:t)) = span (< p*p) xs<br />
</haskell><br />
<br />
This can be seen as essential framework for all the code to come. It only tests odd numbers, and only by the primes that are needed, for each <i>numbers span</i> between successive squares of primes. To find the <b>1001</b>st prime, the divisibility test is performed by only <b>24</b> nested filters corresponding to the first <b>24</b> odd primes.<br />
<br />
Whereas the first version exhibits near O(<math>{n^2}</math>) behavior, this one exhibits near O(<math>{n^{1.5}}</math>) behavior, with an orders-of-magnitude speedup. <br />
<br />
There is another way for the composites to be found - by generating all the multiples of successive primes, in advance. Any number thus generated will of course be divisible by the corresponding prime.<br />
<br />
==== Postponed Multiples Removal i.e. Euler's Sieve ====<br />
Instead of testing <i>each number</i> for divisibility by a prime we can just <i>remove</i> the prime's <i>multiples</i> in advance. We gain in speed because we now get the primes <i>for free</i>, after all the multiples are removed on a particular span, <i>without</i> performing any divisibility tests <i>at all</i>:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..] <br />
where<br />
sieve (p:ps) xs = h ++ sieve ps (t `minus` tail [q,q+2*p..])<br />
where (h,~(_:t)) = span (< q) xs <br />
q = p*p<br />
<br />
<br />
minus :: (Ord a) => [a] -> [a] -> [a]<br />
minus a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: xs `minus` b<br />
EQ -> xs `minus` ys<br />
GT -> a `minus` ys<br />
minus a b = a<br />
</haskell> <br />
<br />
This is, in fact, Euler's sieve.<br />
<br />
==== Merged Multiples Removal Sieve ====<br />
<code>(...((s-a)-b)-...)</code> is the same as <code>(s-(a+b+...))</code>, and so we can just remove the ''merged'' infinite primes multiples, each starting at its prime's square, from the ''initial'' numbers supply. This way we needn't explicitly jump to a prime's square because it's where its multiples start anyway:<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3] ++ [5,7..] `minus` foldr merge' [] mults<br />
mults = map (\p->let q=p*p in (q,tail [q,q+2*p..])) $ primes'<br />
merge' (q,qs) xs = q : merge qs xs<br />
<br />
<br />
<br />
merge :: (Ord a) => [a] -> [a] -> [a] <br />
merge a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: merge xs b <br />
EQ -> x: merge xs ys<br />
GT -> y: merge a ys<br />
merge a b = if null a then b else a<br />
</haskell><br />
<br />
This code is yet faster. Its main deficiency still is that it creates a linear nested merging structure, instead of a tree-like structure. Each multiple produced by a prime has to percolate to the top eventually, so it's better to make its path shorter. It'll have to go through fewer merge nodes this way.<br />
<br />
The linearity is imposed by type asymmetry of our <code>(merge' :: a -> b -> b)</code> function, forcing us into the <code>1+(1+(1+(1+...)))</code> pattern, <code>+</code> standing for <code>merge'</code> (which was defined that way to prevent the run-ahead when folding over the infinite list of lists of multiples).<br />
<br />
We need to turn it into an associative operation of uniform type <code>(:: a -> a -> a)</code> to be able to freely rearrange the combinations into arbitrary tree-like patterns, as in e.g. ((1+1)+(2+2))+(...) etc. The type uniformity is what makes compositionality possible.<br />
<br />
The code in the "Implicit Heap" section below improves on that, and is essentially equivalent to using a treefold instead of a standard linear <code>foldr</code>, as in:<br />
<br />
==== Treefold Merged Multiples Removal ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3,5] ++ drop 2 [3,5..] `minus` comps<br />
mults = map (\p-> let q=p*p in ([q],tail [q,q+2*p..])) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
<br />
pairwise f (x:y:ys) = f x y : pairwise f ys<br />
<br />
tfold f (a: ~(b: ~(c:xs)))<br />
= (a `f` (b `f` c)) `f` tfold f (pairwise f xs)<br />
<br />
mergeSP (a,b) ~(c,d) = let (bc,b') = spMerge b c<br />
in (a ++ bc, merge b' d)<br />
where <br />
spMerge :: (Ord a) => [a] -> [a] -> ([a],[a]) <br />
spMerge a@(x:xs) b@(y:ys) = case compare x y of<br />
LT -> (x:c,d) where (c,d) = spMerge xs b<br />
EQ -> (x:c,d) where (c,d) = spMerge xs ys<br />
GT -> (y:c,d) where (c,d) = spMerge a ys<br />
spMerge a [] = ([] ,a)<br />
spMerge [] b = ([] ,b)<br />
</haskell><br />
The fold used here creates a <code>(2+(2+2))+( (4+(4+4)) + ( (8+(8+8)) + ... ))</code> structure, better adjusted for primes multiples production than <code>1+(2+(4+(8+...)))</code>, used by the "Implicit Heap" code, giving it additional 10% speedup.<br />
<br />
<code> mergeSP </code> is an associative operation, preserving of the invariant such that for a list of multiples <code>[(a,b),(c,d), ...]</code>, it's always the case that <code> last a < head b && last a < head c</code>. These "split pairs" represent ordered list as a pair of its known and (currently) finite prefix, and the rest of it. Such pairs under <code>mergeSP</code> operation form a <i>monoid</i>, and if we were to declare a <hask>newtype SplitPair a = SP ([a],[a])</hask> a <hask>Monoid</hask> instance, with <code>mergeSP</code> its <hask>mappend</hask> (and <code>tfold</code> its <hask>mconcat</hask>), the above code for <code>comps</code> would just become <hask>SP (comps,_) = mconcat mults</hask>.<br />
<br />
This code exhibits approximately O(<math>{n^{1.20}}</math>)..O(<math>{n^{1.15}}</math>) local asymptotic behavior (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced). When compared with Melissa O'Neill's PQ code from the ZIP package which was modified to work on odds only as well, it is 3.2x times faster, with used memory reported about 2.5x times smaller.<br />
<br />
It can be further improved with the wheel optimization (as described below at the Prime Wheels section):<br />
<br />
==== Treefold Merged Multiples, with Wheel ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:5:7:primes' <br />
where<br />
primes' = [11,13] ++ drop 2 (rollFrom 11) `minus` comps<br />
mults = map (\p-> fromList $ map (p*) $ rollFrom p) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
fromList (x:xs) = ([x],xs)<br />
rollFrom n = let x = (n-11) `mod` 210<br />
(y,_) = span (< x) wheelNums<br />
in roll n $ drop (length y) wheel<br />
<br />
wheelNums = roll 0 wheel<br />
roll = scanl (+)<br />
wheel = 2:4:2:4:6:2:6:4:2:4:6:6:2:6:4:2:6:4:6:8:4:2:4:2:<br />
4:8:6:4:6:2:4:6:2:6:6:4:2:4:6:2:6:4:2:4:2:10:2:10:wheel<br />
</haskell><br />
<br />
This runs about 2.5x times faster than the Priority Queue-based code as present in Melissa O'Neill's ZIP package, with similar local asymptotic behavior of about O(<math>{n^{1.25}}</math>)..O(<math>{n^{1.17}}</math>) (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced), with used memory reported about twice less as well.<br />
<br />
=== More Filtering Sieves ===<br />
<br />
The primes list creation with divisibility testing can be reformulated in a few more ways, using the list of primes <i>as it is being built</i> (a la "circular programming").<br />
<br />
==== Odd numbers, by Trial Division ====<br />
This is also good for generating a few 100,000s primes (when GHC-compiled as well):<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: filter isPrime [5,7..]<br />
where<br />
isPrime n = all (notDivs n) $ takeWhile (\p-> p*p <= n) (tail primes)<br />
notDivs n p = n `mod` p /= 0 <br />
</haskell><br />
<br />
Instead of relying on nested filters, it tests each odd number by an explicit list of all the needed prime factors. But for each number tested it re-fetches this list <i>anew</i> which will be <i>the same</i> for the increasing spans of numbers between the successive squares of primes.<br />
<br />
==== Generated Spans, by Nested Filters ====<br />
The other way to go instead of concentrating on the numbers supply, is to directly work on the successive spans between the primes squares.<br />
<br />
This version is a bit faster still, creating <i>158,000 primes</i> (again, GHC-compiled) in the same time as the postponed filters does 100,000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
notDivsBy d n = n `mod` d /= 0<br />
sieve ds (p:ps) x = foldr (filter . notDivsBy) [x, x+2..p*p-2] ds<br />
++ sieve (p:ds) ps (p*p+2)<br />
</haskell><br />
<br />
This one explicitly maintains the list of primes needed for testing each odds span between successive primes squares, which it also explicitly generates. But it tests with nested <code>filter</code>s, which it repeatedly recreates.<br />
<br />
==== Generated Spans, by List of Primes ====<br />
The list of primes needed to test each range of odds is actually just the prefix of the primes list itself, of known length, and need not be specifically generated at all. Combined with one-call testing by the explicit list of primes, and direct generation of odds between the successive primes squares, this leads to:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve 0 (tail primes) 5<br />
sieve k (p:ps) x = [x | x<-[x,x+2..p*p-2], and [x`rem`p/=0 | p<-fs]]<br />
-- or: all ((>0).(x`rem`)) fs<br />
++ sieve (k+1) ps (p*p+2) <br />
where fs = take k (tail primes) <br />
</haskell><br />
<br />
It produces about <i>222,000 primes</i> in the same amount of time, and is good for creating about a million first primes, compiled.<br />
<br />
The reason to have <code>sieve</code> function available separately too is that it can also be used to produce primes above a given number, as in<br />
<br />
<haskell><br />
primesFrom m = sieve (length h) ps $ m`div`2*2+1 <br />
where <br />
(h,(_:ps)) = span (<= (floor.sqrt.fromIntegral) m) primes<br />
</haskell><br />
<br />
It can thus produce a few primes e.g. above <code>239812076741689</code>, which is a square of the millionth odd prime, without having to compute all the preceding primes (which would number in trillions).<br />
<br />
==== Multiples Removal on Generated Spans, or Sieve of Eratosthenes ====<br />
<br />
The divisibility testing too should be considered a specification (as in "no multiples of p"), and not a code per se, because although testing composites is cheap (as most of them will have small factors, so the test is soon aborted), testing prime numbers is costly, and is to be avoided.<br />
<br />
All the filtering versions thus far try to <i>keep the primes</i> among all numbers by testing <i>each number</i> in isolation. Instead, all the relevant primes' multiples can be removed from the corresponding segments of odds, and what's left after that will be just primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
sieve fs (p:ps) x = [x,x+2..q-2] `minus` foldl merge [] mults<br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
</haskell> <br />
<br />
This modifies the preceding sieve to "mark" the odd composites in a given range (instead of testing their divisibility) by generating them - just as a person performing the original sieve of Eratosthenes would do, marking one by one the multiples of the relevant primes.<br />
<br />
Compared with the previous version, interpreted under both GHCi and WinHugs, it runs <i>faster</i>, takes <i>less</i> memory, and has better asymptotic behavior, its performance approximately the same as in the Merged Multiples Removal sieve. The advantage in working with spans explicitly is that this code is easily amendable to using arrays for the composites marking and removal on each <i>finite</i> span. <br />
<br />
=== Using Immutable Arrays ===<br />
<br />
==== Generating Segments of Primes ====<br />
<br />
The removal of multiples on each segment of odds in the sieve of Eratosthenes can be done by actually marking them in an array, instead of manipulating lists with "minus" and "merge":<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5 <br />
where <br />
sieve fs (p:ps) x = [i | i<- [x,x+2..q-2], a!i] <br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
a = accumArray (\a b->False) True (x,q-2) <br />
[(i,()) | ms<- mults, i<- ms] <br />
</haskell><br />
Apparently, arrays are ''fast'' too. This code, compared with Treefold Merged with Wheel version (itself 2.5x times faster than Melissa O'Neill's PQ version), runs at about the same time and memory usage, but improving slightly with slightly better local asymptotics.<br />
<br />
==== Calculating Primes Upto a Given Value ====<br />
<br />
<haskell><br />
primesToN n = 2: [i | i<-[3,5..n], ar!i]<br />
where<br />
ar = f 5 $ accumArray (\a b->False) True (3,n) <br />
[(i,()) | i<- [9,15..n]]<br />
f p a | q>=n = a<br />
| True = if null x then a' else f (head x) a'<br />
where q = p*p<br />
a'= a//[(i,False)|i<-[q,q+2*p..n]]<br />
x = [i | i<-[p+2,p+4..n], a' !i]<br />
</haskell><br />
<br />
==== Calculating Primes in a Given Range ====<br />
<br />
<haskell><br />
primesFromTo a b = (if a<3 then [2] else []) <br />
++ [i | i<-[o,o+2..b], ar!i]<br />
where <br />
o = max (if even a then a+1 else a) 3<br />
r = (1+).floor.sqrt.fromInteger $ b<br />
ar = accumArray (\a b->False) True (o,b) <br />
[(i,()) | p <- [3,5..r]<br />
, let q = p*p <br />
s = 2*p <br />
(n,x) = quotRem (o-q) s <br />
q' = if o <= q then q<br />
else if x==0 then q+n*s<br />
else q+(n+1)*s<br />
, i<- [q',q'+s..b] ]<br />
</haskell><br />
<br />
Although testing by odds instead of by primes, the array generation is so fast that it is very much feasible and even preferable for quick generation of some short spans of relatively big primes.<br />
<br />
=== Using Mutable Arrays ===<br />
<br />
Using mutable arrays is the fastest and most memory efficient way to calculate prime numbers in Haskell.<br />
<br />
==== Using ST Array ====<br />
<br />
This method implements the Sieve of Eratosthenes, similar to how you might do it<br />
in C.<br />
<br />
<haskell><br />
import Control.Monad<br />
import Control.Monad.ST<br />
import Data.Array.IArray<br />
import Data.Array.MArray<br />
import Data.Array.ST<br />
import Data.Array.Unboxed<br />
<br />
primesToNA :: Int -> UArray Int Bool<br />
primesToNA n = runSTUArray sieve where<br />
sieve = do<br />
a <- newArray (2,n) True :: ST s (STUArray s Int Bool)<br />
let sr = floor . (sqrt::Double->Double) . fromIntegral $ n+1<br />
forM_ [4,6..n] $ \j -> writeArray a j False<br />
forM_ [3,5..sr] $ \i -> do<br />
si <- readArray a i<br />
when si $<br />
forM_ [i*i,i*i+i+i..n] $ \j -> writeArray a j False<br />
return a<br />
<br />
primesToN :: Int -> [Int]<br />
primesToN n = [i | (i,e) <- assocs . primesToNA $n, e]<br />
</haskell><br />
<br />
==== Bitwise prime sieve with Template Haskell ====<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
=== Implicit Heap ===<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd (x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = x `f` foldTree f (pairs xs)<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*p,p*p+2*p..]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
=== Prime Wheels ===<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers first that avoid a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
<br />
=== Using IntSet for a traditional sieve ===<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== Testing Primality ==<br />
<br />
=== Primality Test and Integer Factorization ===<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
=== Miller-Rabin Primality Test ===<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]<br />
[[Category:Mathematics]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=32790Prime numbers2009-12-31T11:19:06Z<p>Steve C: Rename "Arrays" section to "Immutable Arrays", add "Mutable Arrays" section</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a <i>prime number</i> (or a <i>prime</i>) is a natural number which has exactly two distinct natural number divisors: 1 and itself. The smallest prime is thus 2.<br />
<br />
[http://en.wikipedia.org/wiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Finding Primes ==<br />
<br />
Any natural number is representable as a product of powers of its prime factors, and so a prime has no prime divisors other than itself. That means that starting with 2, <i>for each</i> newly found <i>prime</i> we can <i>eliminate</i> from the rest of the numbers <i>all such numbers</i> that have this prime as their divisor, giving us the <i>next available</i> number as next prime. This is known as <i>sieving</i> the natural numbers, so that in the end what we are left with are just primes. <br />
<br />
=== The Classic Simple Primes Sieve ===<br />
<br />
Attributed to David Turner <i>(SASL Language Manual, 1975)</i>, the following is a direct translation of that idea, generating a list of all prime numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where<br />
sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0] <br />
-- or: filter ((/=0).(`mod`p)) xs<br />
</haskell><br />
<br />
This should only be considered a <i>specification</i>, not a <i>code</i>. When run as is, it is <i>extremely inefficient</i> because it starts up the filters prematurely, immediately after each prime, instead of only after the prime's square has been reached. To be admitted as prime, <i>each number</i> will be <i>tested for divisibility</i> here by all its preceding primes, while just those not greater than its square root would suffice. This means that e.g. to find the <b>1001</b>st prime (<code>7927</code>), <b>1000</b> filters are used, when in fact just the first <b>24</b> are needed (upto <code>89</code>'s filter only).<br />
<br />
So this in effect creates a cascade of nested filters in front of the infinite numbers supply, and in <i>extremely premature</i> fashion at that. One way of fixing that would be to <i>postpone</i> the creation of filters until the right moment, by decoupling the primes supply from the numbers supply, as in<br />
<br />
=== Postponed Filters Sieve ===<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..]<br />
where <br />
sieve (p:ps) xs = h ++ sieve ps [x | x<-t, x `rem` p /= 0] <br />
-- or: filter ((/=0).(`rem`p)) t<br />
where (h,~(_:t)) = span (< p*p) xs<br />
</haskell><br />
<br />
This can be seen as essential framework for all the code to come. It only tests odd numbers, and only by the primes that are needed, for each <i>numbers span</i> between successive squares of primes. To find the <b>1001</b>st prime, the divisibility test is performed by only <b>24</b> nested filters corresponding to the first <b>24</b> odd primes.<br />
<br />
Whereas the first version exhibits near O(<math>{n^2}</math>) behavior, this one exhibits near O(<math>{n^{1.5}}</math>) behavior, with an orders-of-magnitude speedup. <br />
<br />
There is another way for the composites to be found - by generating all the multiples of successive primes, in advance. Any number thus generated will of course be divisible by the corresponding prime.<br />
<br />
==== Postponed Multiples Removal i.e. Euler's Sieve ====<br />
Instead of testing <i>each number</i> for divisibility by a prime we can just <i>remove</i> the prime's <i>multiples</i> in advance. We gain in speed because we now get the primes <i>for free</i>, after all the multiples are removed on a particular span, <i>without</i> performing any divisibility tests <i>at all</i>:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..] <br />
where<br />
sieve (p:ps) xs = h ++ sieve ps (t `minus` tail [q,q+2*p..])<br />
where (h,~(_:t)) = span (< q) xs <br />
q = p*p<br />
<br />
<br />
minus :: (Ord a) => [a] -> [a] -> [a]<br />
minus a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: xs `minus` b<br />
EQ -> xs `minus` ys<br />
GT -> a `minus` ys<br />
minus a b = a<br />
</haskell> <br />
<br />
This is, in fact, Euler's sieve.<br />
<br />
==== Merged Multiples Removal Sieve ====<br />
<code>(...((s-a)-b)-...)</code> is the same as <code>(s-(a+b+...))</code>, and so we can just remove the ''merged'' infinite primes multiples, each starting at its prime's square, from the ''initial'' numbers supply. This way we needn't explicitly jump to a prime's square because it's where its multiples start anyway:<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3] ++ [5,7..] `minus` foldr merge' [] mults<br />
mults = map (\p->let q=p*p in (q,tail [q,q+2*p..])) $ primes'<br />
merge' (q,qs) xs = q : merge qs xs<br />
<br />
<br />
<br />
merge :: (Ord a) => [a] -> [a] -> [a] <br />
merge a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: merge xs b <br />
EQ -> x: merge xs ys<br />
GT -> y: merge a ys<br />
merge a b = if null a then b else a<br />
</haskell><br />
<br />
This code is yet faster. Its main deficiency still is that it creates a linear nested merging structure, instead of a tree-like structure. Each multiple produced by a prime has to percolate to the top eventually, so it's better to make its path shorter. It'll have to go through fewer merge nodes this way.<br />
<br />
The linearity is imposed by type asymmetry of our <code>(merge' :: a -> b -> b)</code> function, forcing us into the <code>1+(1+(1+(1+...)))</code> pattern, <code>+</code> standing for <code>merge'</code> (which was defined that way to prevent the run-ahead when folding over the infinite list of lists of multiples).<br />
<br />
We need to turn it into an associative operation of uniform type <code>(:: a -> a -> a)</code> to be able to freely rearrange the combinations into arbitrary tree-like patterns, as in e.g. ((1+1)+(2+2))+(...) etc. The type uniformity is what makes compositionality possible.<br />
<br />
The code in the "Implicit Heap" section below improves on that, and is essentially equivalent to using a treefold instead of a standard linear <code>foldr</code>, as in:<br />
<br />
==== Treefold Merged Multiples Removal ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3,5] ++ drop 2 [3,5..] `minus` comps<br />
mults = map (\p-> let q=p*p in ([q],tail [q,q+2*p..])) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
<br />
pairwise f (x:y:ys) = f x y : pairwise f ys<br />
<br />
tfold f (a: ~(b: ~(c:xs)))<br />
= (a `f` (b `f` c)) `f` tfold f (pairwise f xs)<br />
<br />
mergeSP (a,b) ~(c,d) = let (bc,b') = spMerge b c<br />
in (a ++ bc, merge b' d)<br />
where <br />
spMerge :: (Ord a) => [a] -> [a] -> ([a],[a]) <br />
spMerge a@(x:xs) b@(y:ys) = case compare x y of<br />
LT -> (x:c,d) where (c,d) = spMerge xs b<br />
EQ -> (x:c,d) where (c,d) = spMerge xs ys<br />
GT -> (y:c,d) where (c,d) = spMerge a ys<br />
spMerge a [] = ([] ,a)<br />
spMerge [] b = ([] ,b)<br />
</haskell><br />
The fold used here creates a <code>(2+(2+2))+( (4+(4+4)) + ( (8+(8+8)) + ... ))</code> structure, better adjusted for primes multiples production than <code>1+(2+(4+(8+...)))</code>, used by the "Implicit Heap" code, giving it additional 10% speedup.<br />
<br />
<code> mergeSP </code> is an associative operation, preserving of the invariant such that for a list of multiples <code>[(a,b),(c,d), ...]</code>, it's always the case that <code> last a < head b && last a < head c</code>. These "split pairs" represent ordered list as a pair of its known and (currently) finite prefix, and the rest of it. Such pairs under <code>mergeSP</code> operation form a <i>monoid</i>, and if we were to declare a <hask>newtype SplitPair a = SP ([a],[a])</hask> a <hask>Monoid</hask> instance, with <code>mergeSP</code> its <hask>mappend</hask> (and <code>tfold</code> its <hask>mconcat</hask>), the above code for <code>comps</code> would just become <hask>SP (comps,_) = mconcat mults</hask>.<br />
<br />
This code exhibits approximately O(<math>{n^{1.20}}</math>)..O(<math>{n^{1.15}}</math>) local asymptotic behavior (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced). When compared with Melissa O'Neill's PQ code from the ZIP package which was modified to work on odds only as well, it is 3.2x times faster, with used memory reported about 2.5x times smaller.<br />
<br />
It can be further improved with the wheel optimization (as described below at the Prime Wheels section):<br />
<br />
==== Treefold Merged Multiples, with Wheel ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:5:7:primes' <br />
where<br />
primes' = [11,13] ++ drop 2 (rollFrom 11) `minus` comps<br />
mults = map (\p-> fromList $ map (p*) $ rollFrom p) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
fromList (x:xs) = ([x],xs)<br />
rollFrom n = let x = (n-11) `mod` 210<br />
(y,_) = span (< x) wheelNums<br />
in roll n $ drop (length y) wheel<br />
<br />
wheelNums = roll 0 wheel<br />
roll = scanl (+)<br />
wheel = 2:4:2:4:6:2:6:4:2:4:6:6:2:6:4:2:6:4:6:8:4:2:4:2:<br />
4:8:6:4:6:2:4:6:2:6:6:4:2:4:6:2:6:4:2:4:2:10:2:10:wheel<br />
</haskell><br />
<br />
This runs about 2.5x times faster than the Priority Queue-based code as present in Melissa O'Neill's ZIP package, with similar local asymptotic behavior of about O(<math>{n^{1.25}}</math>)..O(<math>{n^{1.17}}</math>) (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced), with used memory reported about twice less as well.<br />
<br />
=== More Filtering Sieves ===<br />
<br />
The primes list creation with divisibility testing can be reformulated in a few more ways, using the list of primes <i>as it is being built</i> (a la "circular programming").<br />
<br />
==== Odd numbers, by Trial Division ====<br />
This is also good for generating a few 100,000s primes (when GHC-compiled as well):<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: filter isPrime [5,7..]<br />
where<br />
isPrime n = all (notDivs n) $ takeWhile (\p-> p*p <= n) (tail primes)<br />
notDivs n p = n `mod` p /= 0 <br />
</haskell><br />
<br />
Instead of relying on nested filters, it tests each odd number by an explicit list of all the needed prime factors. But for each number tested it re-fetches this list <i>anew</i> which will be <i>the same</i> for the increasing spans of numbers between the successive squares of primes.<br />
<br />
==== Generated Spans, by Nested Filters ====<br />
The other way to go instead of concentrating on the numbers supply, is to directly work on the successive spans between the primes squares.<br />
<br />
This version is a bit faster still, creating <i>158,000 primes</i> (again, GHC-compiled) in the same time as the postponed filters does 100,000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
notDivsBy d n = n `mod` d /= 0<br />
sieve ds (p:ps) x = foldr (filter . notDivsBy) [x, x+2..p*p-2] ds<br />
++ sieve (p:ds) ps (p*p+2)<br />
</haskell><br />
<br />
This one explicitly maintains the list of primes needed for testing each odds span between successive primes squares, which it also explicitly generates. But it tests with nested <code>filter</code>s, which it repeatedly recreates.<br />
<br />
==== Generated Spans, by List of Primes ====<br />
The list of primes needed to test each range of odds is actually just the prefix of the primes list itself, of known length, and need not be specifically generated at all. Combined with one-call testing by the explicit list of primes, and direct generation of odds between the successive primes squares, this leads to:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve 0 (tail primes) 5<br />
sieve k (p:ps) x = [x | x<-[x,x+2..p*p-2], and [x`rem`p/=0 | p<-fs]]<br />
-- or: all ((>0).(x`rem`)) fs<br />
++ sieve (k+1) ps (p*p+2) <br />
where fs = take k (tail primes) <br />
</haskell><br />
<br />
It produces about <i>222,000 primes</i> in the same amount of time, and is good for creating about a million first primes, compiled.<br />
<br />
The reason to have <code>sieve</code> function available separately too is that it can also be used to produce primes above a given number, as in<br />
<br />
<haskell><br />
primesFrom m = sieve (length h) ps $ m`div`2*2+1 <br />
where <br />
(h,(_:ps)) = span (<= (floor.sqrt.fromIntegral) m) primes<br />
</haskell><br />
<br />
It can thus produce a few primes e.g. above <code>239812076741689</code>, which is a square of the millionth odd prime, without having to compute all the preceding primes (which would number in trillions).<br />
<br />
==== Multiples Removal on Generated Spans, or Sieve of Eratosthenes ====<br />
<br />
The divisibility testing too should be considered a specification (as in "no multiples of p"), and not a code per se, because although testing composites is cheap (as most of them will have small factors, so the test is soon aborted), testing prime numbers is costly, and is to be avoided.<br />
<br />
All the filtering versions thus far try to <i>keep the primes</i> among all numbers by testing <i>each number</i> in isolation. Instead, all the relevant primes' multiples can be removed from the corresponding segments of odds, and what's left after that will be just primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
sieve fs (p:ps) x = [x,x+2..q-2] `minus` foldl merge [] mults<br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
</haskell> <br />
<br />
This modifies the preceding sieve to "mark" the odd composites in a given range (instead of testing their divisibility) by generating them - just as a person performing the original sieve of Eratosthenes would do, marking one by one the multiples of the relevant primes.<br />
<br />
Compared with the previous version, interpreted under both GHCi and WinHugs, it runs <i>faster</i>, takes <i>less</i> memory, and has better asymptotic behavior, its performance approximately the same as in the Merged Multiples Removal sieve. The advantage in working with spans explicitly is that this code is easily amendable to using arrays for the composites marking and removal on each <i>finite</i> span. <br />
<br />
=== Using Immutable Arrays ===<br />
<br />
==== Generating Segments of Primes ====<br />
<br />
The removal of multiples on each segment of odds in the sieve of Eratosthenes can be done by actually marking them in an array, instead of manipulating lists with "minus" and "merge":<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5 <br />
where <br />
sieve fs (p:ps) x = [i | i<- [x,x+2..q-2], a!i] <br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
a = accumArray (\a b->False) True (x,q-2) <br />
[(i,()) | ms<- mults, i<- ms] <br />
</haskell><br />
Apparently, arrays are ''fast'' too. This code, compared with Treefold Merged with Wheel version (itself 2.5x times faster than Melissa O'Neill's PQ version), runs at about the same time and memory usage, but improving slightly with slightly better local asymptotics.<br />
<br />
==== Calculating Primes Upto a Given Value ====<br />
<br />
<haskell><br />
primesToN n = 2: [i | i<-[3,5..n], ar!i]<br />
where<br />
ar = f 5 $ accumArray (\a b->False) True (3,n) <br />
[(i,()) | i<- [9,15..n]]<br />
f p a | q>=n = a<br />
| True = if null x then a' else f (head x) a'<br />
where q = p*p<br />
a'= a//[(i,False)|i<-[q,q+2*p..n]]<br />
x = [i | i<-[p+2,p+4..n], a' !i]<br />
</haskell><br />
<br />
==== Calculating Primes in a Given Range ====<br />
<br />
<haskell><br />
primesFromTo a b = (if a<3 then [2] else []) <br />
++ [i | i<-[o,o+2..b], ar!i]<br />
where <br />
o = max (if even a then a+1 else a) 3<br />
r = (1+).floor.sqrt.fromInteger $ b<br />
ar = accumArray (\a b->False) True (o,b) <br />
[(i,()) | p <- [3,5..r]<br />
, let q = p*p <br />
s = 2*p <br />
(n,x) = quotRem (o-q) s <br />
q' = if o <= q then q<br />
else if x==0 then q+n*s<br />
else q+(n+1)*s<br />
, i<- [q',q'+s..b] ]<br />
</haskell><br />
<br />
Although testing by odds instead of by primes, the array generation is so fast that it is very much feasible and even preferable for quick generation of some short spans of relatively big primes.<br />
<br />
=== Using Mutable Arrays ===<br />
<br />
==== Bitwise prime sieve ====<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
=== Implicit Heap ===<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd (x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = x `f` foldTree f (pairs xs)<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*p,p*p+2*p..]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
=== Prime Wheels ===<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers first that avoid a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
<br />
=== Using IntSet for a traditional sieve ===<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== Testing Primality ==<br />
<br />
=== Primality Test and Integer Factorization ===<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
=== Miller-Rabin Primality Test ===<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]<br />
[[Category:Mathematics]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=32789Prime numbers2009-12-31T09:47:27Z<p>Steve C: Fix typo</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a <i>prime number</i> (or a <i>prime</i>) is a natural number which has exactly two distinct natural number divisors: 1 and itself. The smallest prime is thus 2.<br />
<br />
[http://en.wikipedia.org/wiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Finding Primes ==<br />
<br />
Any natural number is representable as a product of powers of its prime factors, and so a prime has no prime divisors other than itself. That means that starting with 2, <i>for each</i> newly found <i>prime</i> we can <i>eliminate</i> from the rest of the numbers <i>all such numbers</i> that have this prime as their divisor, giving us the <i>next available</i> number as next prime. This is known as <i>sieving</i> the natural numbers, so that in the end what we are left with are just primes. <br />
<br />
=== The Classic Simple Primes Sieve ===<br />
<br />
Attributed to David Turner <i>(SASL Language Manual, 1975)</i>, the following is a direct translation of that idea, generating a list of all prime numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where<br />
sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0] <br />
-- or: filter ((/=0).(`mod`p)) xs<br />
</haskell><br />
<br />
This should only be considered a <i>specification</i>, not a <i>code</i>. When run as is, it is <i>extremely inefficient</i> because it starts up the filters prematurely, immediately after each prime, instead of only after the prime's square has been reached. To be admitted as prime, <i>each number</i> will be <i>tested for divisibility</i> here by all its preceding primes, while just those not greater than its square root would suffice. This means that e.g. to find the <b>1001</b>st prime (<code>7927</code>), <b>1000</b> filters are used, when in fact just the first <b>24</b> are needed (upto <code>89</code>'s filter only).<br />
<br />
So this in effect creates a cascade of nested filters in front of the infinite numbers supply, and in <i>extremely premature</i> fashion at that. One way of fixing that would be to <i>postpone</i> the creation of filters until the right moment, by decoupling the primes supply from the numbers supply, as in<br />
<br />
=== Postponed Filters Sieve ===<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..]<br />
where <br />
sieve (p:ps) xs = h ++ sieve ps [x | x<-t, x `rem` p /= 0] <br />
-- or: filter ((/=0).(`rem`p)) t<br />
where (h,~(_:t)) = span (< p*p) xs<br />
</haskell><br />
<br />
This can be seen as essential framework for all the code to come. It only tests odd numbers, and only by the primes that are needed, for each <i>numbers span</i> between successive squares of primes. To find the <b>1001</b>st prime, the divisibility test is performed by only <b>24</b> nested filters corresponding to the first <b>24</b> odd primes.<br />
<br />
Whereas the first version exhibits near O(<math>{n^2}</math>) behavior, this one exhibits near O(<math>{n^{1.5}}</math>) behavior, with an orders-of-magnitude speedup. <br />
<br />
There is another way for the composites to be found - by generating all the multiples of successive primes, in advance. Any number thus generated will of course be divisible by the corresponding prime.<br />
<br />
==== Postponed Multiples Removal i.e. Euler's Sieve ====<br />
Instead of testing <i>each number</i> for divisibility by a prime we can just <i>remove</i> the prime's <i>multiples</i> in advance. We gain in speed because we now get the primes <i>for free</i>, after all the multiples are removed on a particular span, <i>without</i> performing any divisibility tests <i>at all</i>:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..] <br />
where<br />
sieve (p:ps) xs = h ++ sieve ps (t `minus` tail [q,q+2*p..])<br />
where (h,~(_:t)) = span (< q) xs <br />
q = p*p<br />
<br />
<br />
minus :: (Ord a) => [a] -> [a] -> [a]<br />
minus a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: xs `minus` b<br />
EQ -> xs `minus` ys<br />
GT -> a `minus` ys<br />
minus a b = a<br />
</haskell> <br />
<br />
This is, in fact, Euler's sieve.<br />
<br />
==== Merged Multiples Removal Sieve ====<br />
<code>(...((s-a)-b)-...)</code> is the same as <code>(s-(a+b+...))</code>, and so we can just remove the ''merged'' infinite primes multiples, each starting at its prime's square, from the ''initial'' numbers supply. This way we needn't explicitly jump to a prime's square because it's where its multiples start anyway:<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3] ++ [5,7..] `minus` foldr merge' [] mults<br />
mults = map (\p->let q=p*p in (q,tail [q,q+2*p..])) $ primes'<br />
merge' (q,qs) xs = q : merge qs xs<br />
<br />
<br />
<br />
merge :: (Ord a) => [a] -> [a] -> [a] <br />
merge a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: merge xs b <br />
EQ -> x: merge xs ys<br />
GT -> y: merge a ys<br />
merge a b = if null a then b else a<br />
</haskell><br />
<br />
This code is yet faster. Its main deficiency still is that it creates a linear nested merging structure, instead of a tree-like structure. Each multiple produced by a prime has to percolate to the top eventually, so it's better to make its path shorter. It'll have to go through fewer merge nodes this way.<br />
<br />
The linearity is imposed by type asymmetry of our <code>(merge' :: a -> b -> b)</code> function, forcing us into the <code>1+(1+(1+(1+...)))</code> pattern, <code>+</code> standing for <code>merge'</code> (which was defined that way to prevent the run-ahead when folding over the infinite list of lists of multiples).<br />
<br />
We need to turn it into an associative operation of uniform type <code>(:: a -> a -> a)</code> to be able to freely rearrange the combinations into arbitrary tree-like patterns, as in e.g. ((1+1)+(2+2))+(...) etc. The type uniformity is what makes compositionality possible.<br />
<br />
The code in the "Implicit Heap" section below improves on that, and is essentially equivalent to using a treefold instead of a standard linear <code>foldr</code>, as in:<br />
<br />
==== Treefold Merged Multiples Removal ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3,5] ++ drop 2 [3,5..] `minus` comps<br />
mults = map (\p-> let q=p*p in ([q],tail [q,q+2*p..])) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
<br />
pairwise f (x:y:ys) = f x y : pairwise f ys<br />
<br />
tfold f (a: ~(b: ~(c:xs)))<br />
= (a `f` (b `f` c)) `f` tfold f (pairwise f xs)<br />
<br />
mergeSP (a,b) ~(c,d) = let (bc,b') = spMerge b c<br />
in (a ++ bc, merge b' d)<br />
where <br />
spMerge :: (Ord a) => [a] -> [a] -> ([a],[a]) <br />
spMerge a@(x:xs) b@(y:ys) = case compare x y of<br />
LT -> (x:c,d) where (c,d) = spMerge xs b<br />
EQ -> (x:c,d) where (c,d) = spMerge xs ys<br />
GT -> (y:c,d) where (c,d) = spMerge a ys<br />
spMerge a [] = ([] ,a)<br />
spMerge [] b = ([] ,b)<br />
</haskell><br />
The fold used here creates a <code>(2+(2+2))+( (4+(4+4)) + ( (8+(8+8)) + ... ))</code> structure, better adjusted for primes multiples production than <code>1+(2+(4+(8+...)))</code>, used by the "Implicit Heap" code, giving it additional 10% speedup.<br />
<br />
<code> mergeSP </code> is an associative operation, preserving of the invariant such that for a list of multiples <code>[(a,b),(c,d), ...]</code>, it's always the case that <code> last a < head b && last a < head c</code>. These "split pairs" represent ordered list as a pair of its known and (currently) finite prefix, and the rest of it. Such pairs under <code>mergeSP</code> operation form a <i>monoid</i>, and if we were to declare a <hask>newtype SplitPair a = SP ([a],[a])</hask> a <hask>Monoid</hask> instance, with <code>mergeSP</code> its <hask>mappend</hask> (and <code>tfold</code> its <hask>mconcat</hask>), the above code for <code>comps</code> would just become <hask>SP (comps,_) = mconcat mults</hask>.<br />
<br />
This code exhibits approximately O(<math>{n^{1.20}}</math>)..O(<math>{n^{1.15}}</math>) local asymptotic behavior (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced). When compared with Melissa O'Neill's PQ code from the ZIP package which was modified to work on odds only as well, it is 3.2x times faster, with used memory reported about 2.5x times smaller.<br />
<br />
It can be further improved with the wheel optimization (as described below at the Prime Wheels section):<br />
<br />
==== Treefold Merged Multiples, with Wheel ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:5:7:primes' <br />
where<br />
primes' = [11,13] ++ drop 2 (rollFrom 11) `minus` comps<br />
mults = map (\p-> fromList $ map (p*) $ rollFrom p) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
fromList (x:xs) = ([x],xs)<br />
rollFrom n = let x = (n-11) `mod` 210<br />
(y,_) = span (< x) wheelNums<br />
in roll n $ drop (length y) wheel<br />
<br />
wheelNums = roll 0 wheel<br />
roll = scanl (+)<br />
wheel = 2:4:2:4:6:2:6:4:2:4:6:6:2:6:4:2:6:4:6:8:4:2:4:2:<br />
4:8:6:4:6:2:4:6:2:6:6:4:2:4:6:2:6:4:2:4:2:10:2:10:wheel<br />
</haskell><br />
<br />
This runs about 2.5x times faster than the Priority Queue-based code as present in Melissa O'Neill's ZIP package, with similar local asymptotic behavior of about O(<math>{n^{1.25}}</math>)..O(<math>{n^{1.17}}</math>) (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced), with used memory reported about twice less as well.<br />
<br />
=== More Filtering Sieves ===<br />
<br />
The primes list creation with divisibility testing can be reformulated in a few more ways, using the list of primes <i>as it is being built</i> (a la "circular programming").<br />
<br />
==== Odd numbers, by Trial Division ====<br />
This is also good for generating a few 100,000s primes (when GHC-compiled as well):<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: filter isPrime [5,7..]<br />
where<br />
isPrime n = all (notDivs n) $ takeWhile (\p-> p*p <= n) (tail primes)<br />
notDivs n p = n `mod` p /= 0 <br />
</haskell><br />
<br />
Instead of relying on nested filters, it tests each odd number by an explicit list of all the needed prime factors. But for each number tested it re-fetches this list <i>anew</i> which will be <i>the same</i> for the increasing spans of numbers between the successive squares of primes.<br />
<br />
==== Generated Spans, by Nested Filters ====<br />
The other way to go instead of concentrating on the numbers supply, is to directly work on the successive spans between the primes squares.<br />
<br />
This version is a bit faster still, creating <i>158,000 primes</i> (again, GHC-compiled) in the same time as the postponed filters does 100,000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
notDivsBy d n = n `mod` d /= 0<br />
sieve ds (p:ps) x = foldr (filter . notDivsBy) [x, x+2..p*p-2] ds<br />
++ sieve (p:ds) ps (p*p+2)<br />
</haskell><br />
<br />
This one explicitly maintains the list of primes needed for testing each odds span between successive primes squares, which it also explicitly generates. But it tests with nested <code>filter</code>s, which it repeatedly recreates.<br />
<br />
==== Generated Spans, by List of Primes ====<br />
The list of primes needed to test each range of odds is actually just the prefix of the primes list itself, of known length, and need not be specifically generated at all. Combined with one-call testing by the explicit list of primes, and direct generation of odds between the successive primes squares, this leads to:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve 0 (tail primes) 5<br />
sieve k (p:ps) x = [x | x<-[x,x+2..p*p-2], and [x`rem`p/=0 | p<-fs]]<br />
-- or: all ((>0).(x`rem`)) fs<br />
++ sieve (k+1) ps (p*p+2) <br />
where fs = take k (tail primes) <br />
</haskell><br />
<br />
It produces about <i>222,000 primes</i> in the same amount of time, and is good for creating about a million first primes, compiled.<br />
<br />
The reason to have <code>sieve</code> function available separately too is that it can also be used to produce primes above a given number, as in<br />
<br />
<haskell><br />
primesFrom m = sieve (length h) ps $ m`div`2*2+1 <br />
where <br />
(h,(_:ps)) = span (<= (floor.sqrt.fromIntegral) m) primes<br />
</haskell><br />
<br />
It can thus produce a few primes e.g. above <code>239812076741689</code>, which is a square of the millionth odd prime, without having to compute all the preceding primes (which would number in trillions).<br />
<br />
==== Multiples Removal on Generated Spans, or Sieve of Eratosthenes ====<br />
<br />
The divisibility testing too should be considered a specification (as in "no multiples of p"), and not a code per se, because although testing composites is cheap (as most of them will have small factors, so the test is soon aborted), testing prime numbers is costly, and is to be avoided.<br />
<br />
All the filtering versions thus far try to <i>keep the primes</i> among all numbers by testing <i>each number</i> in isolation. Instead, all the relevant primes' multiples can be removed from the corresponding segments of odds, and what's left after that will be just primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
sieve fs (p:ps) x = [x,x+2..q-2] `minus` foldl merge [] mults<br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
</haskell> <br />
<br />
This modifies the preceding sieve to "mark" the odd composites in a given range (instead of testing their divisibility) by generating them - just as a person performing the original sieve of Eratosthenes would do, marking one by one the multiples of the relevant primes.<br />
<br />
Compared with the previous version, interpreted under both GHCi and WinHugs, it runs <i>faster</i>, takes <i>less</i> memory, and has better asymptotic behavior, its performance approximately the same as in the Merged Multiples Removal sieve. The advantage in working with spans explicitly is that this code is easily amendable to using arrays for the composites marking and removal on each <i>finite</i> span. <br />
<br />
=== Using Arrays ===<br />
<br />
==== Generating Segments of Primes ====<br />
<br />
The removal of multiples on each segment of odds in the sieve of Eratosthenes can be done by actually marking them in an array, instead of manipulating lists with "minus" and "merge":<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5 <br />
where <br />
sieve fs (p:ps) x = [i | i<- [x,x+2..q-2], a!i] <br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
a = accumArray (\a b->False) True (x,q-2) <br />
[(i,()) | ms<- mults, i<- ms] <br />
</haskell><br />
Apparently, arrays are ''fast'' too. This code, compared with Treefold Merged with Wheel version (itself 2.5x times faster than Melissa O'Neill's PQ version), runs at about the same time and memory usage, but improving slightly with slightly better local asymptotics.<br />
<br />
==== Calculating Primes Upto a Given Value ====<br />
<br />
<haskell><br />
primesToN n = 2: [i | i<-[3,5..n], ar!i]<br />
where<br />
ar = f 5 $ accumArray (\a b->False) True (3,n) <br />
[(i,()) | i<- [9,15..n]]<br />
f p a | q>=n = a<br />
| True = if null x then a' else f (head x) a'<br />
where q = p*p<br />
a'= a//[(i,False)|i<-[q,q+2*p..n]]<br />
x = [i | i<-[p+2,p+4..n], a' !i]<br />
</haskell><br />
<br />
==== Calculating Primes in a Given Range ====<br />
<br />
<haskell><br />
primesFromTo a b = (if a<3 then [2] else []) <br />
++ [i | i<-[o,o+2..b], ar!i]<br />
where <br />
o = max (if even a then a+1 else a) 3<br />
r = (1+).floor.sqrt.fromInteger $ b<br />
ar = accumArray (\a b->False) True (o,b) <br />
[(i,()) | p <- [3,5..r]<br />
, let q = p*p <br />
s = 2*p <br />
(n,x) = quotRem (o-q) s <br />
q' = if o <= q then q<br />
else if x==0 then q+n*s<br />
else q+(n+1)*s<br />
, i<- [q',q'+s..b] ]<br />
</haskell><br />
<br />
Although testing by odds instead of by primes, the array generation is so fast that it is very much feasible and even preferable for quick generation of some short spans of relatively big primes.<br />
<br />
=== Implicit Heap ===<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd (x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = x `f` foldTree f (pairs xs)<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*p,p*p+2*p..]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
=== Prime Wheels ===<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers first that avoid a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
=== Bitwise prime sieve ===<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
=== Using IntSet for a traditional sieve ===<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== Testing Primality ==<br />
<br />
=== Primality Test and Integer Factorization ===<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
=== Miller-Rabin Primality Test ===<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]<br />
[[Category:Mathematics]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=32788Prime numbers2009-12-31T09:43:29Z<p>Steve C: correct Wikipedia link so it points to Wikipedia not haskellwiki</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a <i>prime number</i> (or a <i>prime</i>) is a natural number which has exactly two distinct natural number divisors: 1 and itself. The smallest prime is thus 2.<br />
<br />
[http://en.wikipedia.org/wiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Finding Primes ==<br />
<br />
Any natural number is representable as a product of powers of its prime factors, and so a prime has no prime divisors other then itself. That means that starting with 2, <i>for each</i> newly found <i>prime</i> we can <i>eliminate</i> from the rest of the numbers <i>all such numbers</i> that have this prime as their divisor, giving us the <i>next available</i> number as next prime. This is known as <i>sieving</i> the natural numbers, so that in the end what we are left with are just primes. <br />
<br />
=== The Classic Simple Primes Sieve ===<br />
<br />
Attributed to David Turner <i>(SASL Language Manual, 1975)</i>, the following is a direct translation of that idea, generating a list of all prime numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where<br />
sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0] <br />
-- or: filter ((/=0).(`mod`p)) xs<br />
</haskell><br />
<br />
This should only be considered a <i>specification</i>, not a <i>code</i>. When run as is, it is <i>extremely inefficient</i> because it starts up the filters prematurely, immediately after each prime, instead of only after the prime's square has been reached. To be admitted as prime, <i>each number</i> will be <i>tested for divisibility</i> here by all its preceding primes, while just those not greater than its square root would suffice. This means that e.g. to find the <b>1001</b>st prime (<code>7927</code>), <b>1000</b> filters are used, when in fact just the first <b>24</b> are needed (upto <code>89</code>'s filter only).<br />
<br />
So this in effect creates a cascade of nested filters in front of the infinite numbers supply, and in <i>extremely premature</i> fashion at that. One way of fixing that would be to <i>postpone</i> the creation of filters until the right moment, by decoupling the primes supply from the numbers supply, as in<br />
<br />
=== Postponed Filters Sieve ===<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..]<br />
where <br />
sieve (p:ps) xs = h ++ sieve ps [x | x<-t, x `rem` p /= 0] <br />
-- or: filter ((/=0).(`rem`p)) t<br />
where (h,~(_:t)) = span (< p*p) xs<br />
</haskell><br />
<br />
This can be seen as essential framework for all the code to come. It only tests odd numbers, and only by the primes that are needed, for each <i>numbers span</i> between successive squares of primes. To find the <b>1001</b>st prime, the divisibility test is performed by only <b>24</b> nested filters corresponding to the first <b>24</b> odd primes.<br />
<br />
Whereas the first version exhibits near O(<math>{n^2}</math>) behavior, this one exhibits near O(<math>{n^{1.5}}</math>) behavior, with an orders-of-magnitude speedup. <br />
<br />
There is another way for the composites to be found - by generating all the multiples of successive primes, in advance. Any number thus generated will of course be divisible by the corresponding prime.<br />
<br />
==== Postponed Multiples Removal i.e. Euler's Sieve ====<br />
Instead of testing <i>each number</i> for divisibility by a prime we can just <i>remove</i> the prime's <i>multiples</i> in advance. We gain in speed because we now get the primes <i>for free</i>, after all the multiples are removed on a particular span, <i>without</i> performing any divisibility tests <i>at all</i>:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..] <br />
where<br />
sieve (p:ps) xs = h ++ sieve ps (t `minus` tail [q,q+2*p..])<br />
where (h,~(_:t)) = span (< q) xs <br />
q = p*p<br />
<br />
<br />
minus :: (Ord a) => [a] -> [a] -> [a]<br />
minus a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: xs `minus` b<br />
EQ -> xs `minus` ys<br />
GT -> a `minus` ys<br />
minus a b = a<br />
</haskell> <br />
<br />
This is, in fact, Euler's sieve.<br />
<br />
==== Merged Multiples Removal Sieve ====<br />
<code>(...((s-a)-b)-...)</code> is the same as <code>(s-(a+b+...))</code>, and so we can just remove the ''merged'' infinite primes multiples, each starting at its prime's square, from the ''initial'' numbers supply. This way we needn't explicitly jump to a prime's square because it's where its multiples start anyway:<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3] ++ [5,7..] `minus` foldr merge' [] mults<br />
mults = map (\p->let q=p*p in (q,tail [q,q+2*p..])) $ primes'<br />
merge' (q,qs) xs = q : merge qs xs<br />
<br />
<br />
<br />
merge :: (Ord a) => [a] -> [a] -> [a] <br />
merge a@(x:xs) b@(y:ys) = case compare x y of <br />
LT -> x: merge xs b <br />
EQ -> x: merge xs ys<br />
GT -> y: merge a ys<br />
merge a b = if null a then b else a<br />
</haskell><br />
<br />
This code is yet faster. Its main deficiency still is that it creates a linear nested merging structure, instead of a tree-like structure. Each multiple produced by a prime has to percolate to the top eventually, so it's better to make its path shorter. It'll have to go through fewer merge nodes this way.<br />
<br />
The linearity is imposed by type asymmetry of our <code>(merge' :: a -> b -> b)</code> function, forcing us into the <code>1+(1+(1+(1+...)))</code> pattern, <code>+</code> standing for <code>merge'</code> (which was defined that way to prevent the run-ahead when folding over the infinite list of lists of multiples).<br />
<br />
We need to turn it into an associative operation of uniform type <code>(:: a -> a -> a)</code> to be able to freely rearrange the combinations into arbitrary tree-like patterns, as in e.g. ((1+1)+(2+2))+(...) etc. The type uniformity is what makes compositionality possible.<br />
<br />
The code in the "Implicit Heap" section below improves on that, and is essentially equivalent to using a treefold instead of a standard linear <code>foldr</code>, as in:<br />
<br />
==== Treefold Merged Multiples Removal ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:primes'<br />
where<br />
primes' = [3,5] ++ drop 2 [3,5..] `minus` comps<br />
mults = map (\p-> let q=p*p in ([q],tail [q,q+2*p..])) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
<br />
pairwise f (x:y:ys) = f x y : pairwise f ys<br />
<br />
tfold f (a: ~(b: ~(c:xs)))<br />
= (a `f` (b `f` c)) `f` tfold f (pairwise f xs)<br />
<br />
mergeSP (a,b) ~(c,d) = let (bc,b') = spMerge b c<br />
in (a ++ bc, merge b' d)<br />
where <br />
spMerge :: (Ord a) => [a] -> [a] -> ([a],[a]) <br />
spMerge a@(x:xs) b@(y:ys) = case compare x y of<br />
LT -> (x:c,d) where (c,d) = spMerge xs b<br />
EQ -> (x:c,d) where (c,d) = spMerge xs ys<br />
GT -> (y:c,d) where (c,d) = spMerge a ys<br />
spMerge a [] = ([] ,a)<br />
spMerge [] b = ([] ,b)<br />
</haskell><br />
The fold used here creates a <code>(2+(2+2))+( (4+(4+4)) + ( (8+(8+8)) + ... ))</code> structure, better adjusted for primes multiples production than <code>1+(2+(4+(8+...)))</code>, used by the "Implicit Heap" code, giving it additional 10% speedup.<br />
<br />
<code> mergeSP </code> is an associative operation, preserving of the invariant such that for a list of multiples <code>[(a,b),(c,d), ...]</code>, it's always the case that <code> last a < head b && last a < head c</code>. These "split pairs" represent ordered list as a pair of its known and (currently) finite prefix, and the rest of it. Such pairs under <code>mergeSP</code> operation form a <i>monoid</i>, and if we were to declare a <hask>newtype SplitPair a = SP ([a],[a])</hask> a <hask>Monoid</hask> instance, with <code>mergeSP</code> its <hask>mappend</hask> (and <code>tfold</code> its <hask>mconcat</hask>), the above code for <code>comps</code> would just become <hask>SP (comps,_) = mconcat mults</hask>.<br />
<br />
This code exhibits approximately O(<math>{n^{1.20}}</math>)..O(<math>{n^{1.15}}</math>) local asymptotic behavior (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced). When compared with Melissa O'Neill's PQ code from the ZIP package which was modified to work on odds only as well, it is 3.2x times faster, with used memory reported about 2.5x times smaller.<br />
<br />
It can be further improved with the wheel optimization (as described below at the Prime Wheels section):<br />
<br />
==== Treefold Merged Multiples, with Wheel ====<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:5:7:primes' <br />
where<br />
primes' = [11,13] ++ drop 2 (rollFrom 11) `minus` comps<br />
mults = map (\p-> fromList $ map (p*) $ rollFrom p) $ primes'<br />
comps = fst $ tfold mergeSP (pairwise mergeSP mults)<br />
fromList (x:xs) = ([x],xs)<br />
rollFrom n = let x = (n-11) `mod` 210<br />
(y,_) = span (< x) wheelNums<br />
in roll n $ drop (length y) wheel<br />
<br />
wheelNums = roll 0 wheel<br />
roll = scanl (+)<br />
wheel = 2:4:2:4:6:2:6:4:2:4:6:6:2:6:4:2:6:4:6:8:4:2:4:2:<br />
4:8:6:4:6:2:4:6:2:6:6:4:2:4:6:2:6:4:2:4:2:10:2:10:wheel<br />
</haskell><br />
<br />
This runs about 2.5x times faster than the Priority Queue-based code as present in Melissa O'Neill's ZIP package, with similar local asymptotic behavior of about O(<math>{n^{1.25}}</math>)..O(<math>{n^{1.17}}</math>) (tested interpreted, in GHCi, for 10,000 to 300,000 primes produced), with used memory reported about twice less as well.<br />
<br />
=== More Filtering Sieves ===<br />
<br />
The primes list creation with divisibility testing can be reformulated in a few more ways, using the list of primes <i>as it is being built</i> (a la "circular programming").<br />
<br />
==== Odd numbers, by Trial Division ====<br />
This is also good for generating a few 100,000s primes (when GHC-compiled as well):<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: filter isPrime [5,7..]<br />
where<br />
isPrime n = all (notDivs n) $ takeWhile (\p-> p*p <= n) (tail primes)<br />
notDivs n p = n `mod` p /= 0 <br />
</haskell><br />
<br />
Instead of relying on nested filters, it tests each odd number by an explicit list of all the needed prime factors. But for each number tested it re-fetches this list <i>anew</i> which will be <i>the same</i> for the increasing spans of numbers between the successive squares of primes.<br />
<br />
==== Generated Spans, by Nested Filters ====<br />
The other way to go instead of concentrating on the numbers supply, is to directly work on the successive spans between the primes squares.<br />
<br />
This version is a bit faster still, creating <i>158,000 primes</i> (again, GHC-compiled) in the same time as the postponed filters does 100,000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
notDivsBy d n = n `mod` d /= 0<br />
sieve ds (p:ps) x = foldr (filter . notDivsBy) [x, x+2..p*p-2] ds<br />
++ sieve (p:ds) ps (p*p+2)<br />
</haskell><br />
<br />
This one explicitly maintains the list of primes needed for testing each odds span between successive primes squares, which it also explicitly generates. But it tests with nested <code>filter</code>s, which it repeatedly recreates.<br />
<br />
==== Generated Spans, by List of Primes ====<br />
The list of primes needed to test each range of odds is actually just the prefix of the primes list itself, of known length, and need not be specifically generated at all. Combined with one-call testing by the explicit list of primes, and direct generation of odds between the successive primes squares, this leads to:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve 0 (tail primes) 5<br />
sieve k (p:ps) x = [x | x<-[x,x+2..p*p-2], and [x`rem`p/=0 | p<-fs]]<br />
-- or: all ((>0).(x`rem`)) fs<br />
++ sieve (k+1) ps (p*p+2) <br />
where fs = take k (tail primes) <br />
</haskell><br />
<br />
It produces about <i>222,000 primes</i> in the same amount of time, and is good for creating about a million first primes, compiled.<br />
<br />
The reason to have <code>sieve</code> function available separately too is that it can also be used to produce primes above a given number, as in<br />
<br />
<haskell><br />
primesFrom m = sieve (length h) ps $ m`div`2*2+1 <br />
where <br />
(h,(_:ps)) = span (<= (floor.sqrt.fromIntegral) m) primes<br />
</haskell><br />
<br />
It can thus produce a few primes e.g. above <code>239812076741689</code>, which is a square of the millionth odd prime, without having to compute all the preceding primes (which would number in trillions).<br />
<br />
==== Multiples Removal on Generated Spans, or Sieve of Eratosthenes ====<br />
<br />
The divisibility testing too should be considered a specification (as in "no multiples of p"), and not a code per se, because although testing composites is cheap (as most of them will have small factors, so the test is soon aborted), testing prime numbers is costly, and is to be avoided.<br />
<br />
All the filtering versions thus far try to <i>keep the primes</i> among all numbers by testing <i>each number</i> in isolation. Instead, all the relevant primes' multiples can be removed from the corresponding segments of odds, and what's left after that will be just primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5<br />
where<br />
sieve fs (p:ps) x = [x,x+2..q-2] `minus` foldl merge [] mults<br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
</haskell> <br />
<br />
This modifies the preceding sieve to "mark" the odd composites in a given range (instead of testing their divisibility) by generating them - just as a person performing the original sieve of Eratosthenes would do, marking one by one the multiples of the relevant primes.<br />
<br />
Compared with the previous version, interpreted under both GHCi and WinHugs, it runs <i>faster</i>, takes <i>less</i> memory, and has better asymptotic behavior, its performance approximately the same as in the Merged Multiples Removal sieve. The advantage in working with spans explicitly is that this code is easily amendable to using arrays for the composites marking and removal on each <i>finite</i> span. <br />
<br />
=== Using Arrays ===<br />
<br />
==== Generating Segments of Primes ====<br />
<br />
The removal of multiples on each segment of odds in the sieve of Eratosthenes can be done by actually marking them in an array, instead of manipulating lists with "minus" and "merge":<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve [] (tail primes) 5 <br />
where <br />
sieve fs (p:ps) x = [i | i<- [x,x+2..q-2], a!i] <br />
++ sieve ((2*p,q):fs') ps (q+2)<br />
where<br />
q = p*p<br />
mults = [ [y+s,y+2*s..q] | (s,y)<- fs]<br />
fs' = [ (s,last ms) | ((s,_),ms)<- zip fs mults]<br />
a = accumArray (\a b->False) True (x,q-2) <br />
[(i,()) | ms<- mults, i<- ms] <br />
</haskell><br />
Apparently, arrays are ''fast'' too. This code, compared with Treefold Merged with Wheel version (itself 2.5x times faster than Melissa O'Neill's PQ version), runs at about the same time and memory usage, but improving slightly with slightly better local asymptotics.<br />
<br />
==== Calculating Primes Upto a Given Value ====<br />
<br />
<haskell><br />
primesToN n = 2: [i | i<-[3,5..n], ar!i]<br />
where<br />
ar = f 5 $ accumArray (\a b->False) True (3,n) <br />
[(i,()) | i<- [9,15..n]]<br />
f p a | q>=n = a<br />
| True = if null x then a' else f (head x) a'<br />
where q = p*p<br />
a'= a//[(i,False)|i<-[q,q+2*p..n]]<br />
x = [i | i<-[p+2,p+4..n], a' !i]<br />
</haskell><br />
<br />
==== Calculating Primes in a Given Range ====<br />
<br />
<haskell><br />
primesFromTo a b = (if a<3 then [2] else []) <br />
++ [i | i<-[o,o+2..b], ar!i]<br />
where <br />
o = max (if even a then a+1 else a) 3<br />
r = (1+).floor.sqrt.fromInteger $ b<br />
ar = accumArray (\a b->False) True (o,b) <br />
[(i,()) | p <- [3,5..r]<br />
, let q = p*p <br />
s = 2*p <br />
(n,x) = quotRem (o-q) s <br />
q' = if o <= q then q<br />
else if x==0 then q+n*s<br />
else q+(n+1)*s<br />
, i<- [q',q'+s..b] ]<br />
</haskell><br />
<br />
Although testing by odds instead of by primes, the array generation is so fast that it is very much feasible and even preferable for quick generation of some short spans of relatively big primes.<br />
<br />
=== Implicit Heap ===<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd (x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = x `f` foldTree f (pairs xs)<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*p,p*p+2*p..]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
=== Prime Wheels ===<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers first that avoid a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
=== Bitwise prime sieve ===<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
=== Using IntSet for a traditional sieve ===<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== Testing Primality ==<br />
<br />
=== Primality Test and Integer Factorization ===<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
=== Miller-Rabin Primality Test ===<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]<br />
[[Category:Mathematics]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Euler_problems/21_to_30&diff=32407Euler problems/21 to 302009-12-11T07:28:55Z<p>Steve C: Remove redundant 'show'</p>
<hr />
<div>== [http://projecteuler.net/index.php?section=problems&id=21 Problem 21] ==<br />
Evaluate the sum of all amicable pairs under 10000.<br />
<br />
Solution: <br />
(http://www.research.att.com/~njas/sequences/A063990)<br />
<br />
This is a little slow because of the naive method used to compute the divisors.<br />
<haskell><br />
problem_21 = sum [m+n | m <- [2..9999], let n = divisorsSum ! m, amicable m n]<br />
where amicable m n = m < n && n < 10000 && divisorsSum ! n == m<br />
divisorsSum = array (1,9999)<br />
[(i, sum (divisors i)) | i <- [1..9999]]<br />
divisors n = [j | j <- [1..n `div` 2], n `mod` j == 0]<br />
</haskell><br />
<br />
Here is an alternative using a faster way of computing the sum of divisors.<br />
<haskell><br />
problem_21_v2 = sum [n | n <- [2..9999], let m = d n,<br />
m > 1, m < 10000, n == d m, d m /= d (d m)]<br />
d n = product [(p * product g - 1) `div` (p - 1) |<br />
g <- group $ primeFactors n, let p = head g<br />
] - n<br />
primeFactors = pf primes<br />
where<br />
pf ps@(p:ps') n<br />
| p * p > n = [n]<br />
| r == 0 = p : pf ps q<br />
| otherwise = pf ps' n<br />
where (q, r) = n `divMod` p<br />
primes = 2 : filter (null . tail . primeFactors) [3,5..]<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=22 Problem 22] ==<br />
What is the total of all the name scores in the file of first names?<br />
<br />
Solution:<br />
<haskell><br />
import Data.List<br />
import Data.Char<br />
problem_22 =<br />
do input <- readFile "names.txt"<br />
let names = sort $ read$"["++ input++"]"<br />
let scores = zipWith score names [1..]<br />
print . sum $ scores<br />
where score w i = (i *) . sum . map (\c -> ord c - ord 'A' + 1) $ w<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=23 Problem 23] ==<br />
Find the sum of all the positive integers which cannot be written as the sum of two abundant numbers.<br />
<br />
Solution:<br />
<haskell><br />
--http://www.research.att.com/~njas/sequences/A048242<br />
import Data.Array <br />
n = 28124<br />
abundant n = eulerTotient n - n > n<br />
abunds_array = listArray (1,n) $ map abundant [1..n]<br />
abunds = filter (abunds_array !) [1..n]<br />
<br />
rests x = map (x-) $ takeWhile (<= x `div` 2) abunds<br />
isSum = any (abunds_array !) . rests<br />
<br />
problem_23 = putStrLn . show . foldl1 (+) . filter (not . isSum) $ [1..n] <br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=24 Problem 24] ==<br />
What is the millionth lexicographic permutation of the digits 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9?<br />
<br />
Solution:<br />
<haskell><br />
import Data.List <br />
<br />
fac 0 = 1<br />
fac n = n * fac (n - 1)<br />
perms [] _= []<br />
perms xs n= x : perms (delete x xs) (mod n m)<br />
where m = fac $ length xs - 1<br />
y = div n m<br />
x = xs!!y<br />
<br />
problem_24 = perms "0123456789" 999999<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=25 Problem 25] ==<br />
What is the first term in the Fibonacci sequence to contain 1000 digits?<br />
<br />
Solution:<br />
<haskell><br />
fibs = 0:1:(zipWith (+) fibs (tail fibs))<br />
t = 10^999<br />
<br />
problem_25 = length w<br />
where<br />
w = takeWhile (< t) fibs<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=26 Problem 26] ==<br />
Find the value of d < 1000 for which 1/d contains the longest recurring cycle.<br />
<br />
Solution:<br />
<haskell><br />
problem_26 = fst $ maximumBy (\a b -> snd a `compare` snd b)<br />
[(n,recurringCycle n) | n <- [1..999]]<br />
where recurringCycle d = remainders d 10 []<br />
remainders d 0 rs = 0<br />
remainders d r rs = let r' = r `mod` d<br />
in case findIndex (== r') rs of<br />
Just i -> i + 1<br />
Nothing -> remainders d (10*r') (r':rs)<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=27 Problem 27] ==<br />
Find a quadratic formula that produces the maximum number of primes for consecutive values of n.<br />
<br />
Solution:<br />
<haskell><br />
problem_27 = -(2*a-1)*(a^2-a+41)<br />
where n = 1000<br />
m = head $ filter (\x->x^2-x+41>n) [1..]<br />
a = m-1<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=28 Problem 28] ==<br />
What is the sum of both diagonals in a 1001 by 1001 spiral?<br />
<br />
Solution:<br />
<haskell><br />
problem_28 = sum (map (\n -> 4*(n-2)^2+10*(n-1)) [3,5..1001]) + 1<br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=29 Problem 29] ==<br />
How many distinct terms are in the sequence generated by a<sup>b</sup> for 2 â‰¤ a â‰¤ 100 and 2 â‰¤ b â‰¤ 100?<br />
<br />
Solution:<br />
<haskell><br />
import Control.Monad<br />
problem_29 = length . group . sort $ liftM2 (^) [2..100] [2..100] <br />
</haskell><br />
<br />
== [http://projecteuler.net/index.php?section=problems&id=30 Problem 30] ==<br />
Find the sum of all the numbers that can be written as the sum of fifth powers of their digits.<br />
<br />
Solution:<br />
<haskell><br />
import Data.Char (ord)<br />
<br />
limit :: Integer<br />
limit = snd $ head $ dropWhile (\(a,b) -> a > b) $ zip (map (9^5*) [1..]) (map (10^) [1..])<br />
<br />
fifth :: Integer -> Integer<br />
fifth n = foldr (\a b -> (toInteger(ord a) - 48)^5 + b) 0 $ show n<br />
<br />
problem_30 :: Integer<br />
problem_30 = sum $ filter (\n -> n == fifth n) [2..limit]<br />
</haskell></div>Steve Chttps://wiki.haskell.org/index.php?title=Playing_by_the_rules&diff=32087Playing by the rules2009-12-03T09:07:13Z<p>Steve C: Remove from Mathematics category.</p>
<hr />
<div>GHC's [[GHC/Using_rules|term rewriting engine]] provides a playground for all sorts of user-defined optimisations and<br />
tweaks to Haskell code.<br />
This page collects examples of the use of rewrite rules.<br />
There is more information available on the GHC [http://hackage.haskell.org/trac/ghc/wiki/RewriteRules rewrite rules] page.<br />
<br />
== reverse . reverse = id ==<br />
<br />
The following question was asked in [[IRC channel|#haskell]]:<br />
<br />
15:40 Excedrin> why doesn't 'head $ reverse $ reverse [1..]' work in Haskell?<br />
<br />
The answer is: it does work, if you tell the compiler that:<br />
<br />
<haskell><br />
reverse . reverse = id<br />
</haskell><br />
<br />
Using the following rewrite rule:<br />
<br />
<haskell><br />
{-# RULES<br />
"reverse.reverse/id" reverse . reverse = id<br />
#-}<br />
<br />
main = print . head . reverse . reverse $ [1..]<br />
</haskell><br />
<br />
We can "optimise" this program, changing a non-terminating program:<br />
<haskell><br />
$ ./a.out<br />
^C<br />
</haskell><br />
<br />
into a much faster one:<br />
<br />
<haskell><br />
$ ghc -fglasgow-exts -ddump-simpl-stats A.hs<br />
5 PostInlineUnconditionally<br />
4 UnfoldingDone<br />
1 RuleFired<br />
1 reverse.reverse/id<br />
7 BetaReduction<br />
3 SimplifierDone<br />
<br />
$ ./a.out<br />
1<br />
</haskell><br />
<br />
As can be seen here, rewrite rules let you teach the compiler about<br />
interesting algebraic properties you code satisfies, that it can't find<br />
on its own.<br />
<br />
'''Note:''' In general it's a bad idea for rules to change the semantics (meaning) of your code in such a way. Consider someone using <tt>reverse . reverse</tt> to demand strictness of an IO action, this code would no longer work with the above rule.<br />
<br />
== Fast ByteString construction ==<br />
<br />
Another technique is to use rewrite rules to remove intermediate lists<br />
when building bytestrings from literal string constants. The problem is<br />
that string constants must be converted from a [Char] to a ByteString,<br />
to use literal strings as ByteStrings. This is highly inefficient, when<br />
GHC already allocates string constants as unboxed byte arrays. We can<br />
bypass the [Char] conversion using rewrite rules.<br />
<br />
Supposing we want to use a ByteString literal:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as C<br />
<br />
mystring = C.pack "rewrite rules are fun!"<br />
main = C.putStrLn mystring<br />
</haskell><br />
<br />
and it is compiled by GHC to:<br />
<br />
<haskell><br />
mystring_rDR = Data.ByteString.Char8.pack (GHC.Base.unpackCString# "rewrite rules are fun!"#)<br />
<br />
Main.main = Data.ByteString.putStrLn mystring_rDR<br />
<br />
:Main.main = GHC.TopHandler.runMainIO @ () Main.main<br />
</haskell><br />
<br />
Now, the string literal has been packed by GHC for us into a Addr#, pointing<br />
to a proper C string. Now, we'd like to remove that intermediate<br />
unpackCString#, and just pack the string literal straight into a ByteString,<br />
avoiding an intermediate list.<br />
<br />
So we add a rewrite rule:<br />
<br />
<haskell><br />
{-# RULES<br />
"pack/packAddress" forall s . pack (unpackCString# s) = B.packAddress s<br />
#-}<br />
</haskell><br />
<br />
Now, when compiled with -O -fglasgow-exts, we get:<br />
<br />
<haskell><br />
1 RuleFired<br />
1 FPS pack/packAddress<br />
</haskell><br />
<br />
and the code is transformed as follows:<br />
<br />
<haskell><br />
mystring = Data.ByteString.Base.packAddress "rewrite rules are fun!"<br />
</haskell><br />
<br />
to<br />
<br />
<haskell><br />
mystring = Data.ByteString.Char8.pack (GHC.Base.unpackCString# "rewrite rules are fun!"#)<br />
</haskell><br />
<br />
and then the rewrite rule kicks in, and we construct a new ByteString directly<br />
from the Addr#, via a call to strlen to get the length:<br />
<br />
<haskell><br />
mystring =<br />
let addr#_a146 :: GHC.Prim.Addr#<br />
addr#_a146 = "rewrite rules are fun!"#<br />
in case Data.ByteString.Base.$wccall addr#_a146 s2#_a1Q8 of<br />
(# ds3_a1Re, ds4_a1Rf #) -><br />
Data.ByteString.Base.PS addr#_a146<br />
(GHC.ForeignPtr.PlainForeignPtr var#_a1Q9)<br />
0<br />
(GHC.Prim.word2Int# ds4_a1Rf)<br />
</haskell><br />
<br />
''exactly'' what we'd like. So at runtime, this string will require only a call<br />
to strlen to build.<br />
<br />
== Locating errors ==<br />
<br />
Rewrite rules can ''almost'' be used to solve the infamous 'fromJust: Nothing'<br />
error message. Couldn't we just use rewrite rules to rewrite ''*transparently''<br />
all uses of fromJust to safeFromJust, tagging the call site with a location?<br />
To work this requires a few things to go right:<br />
<br />
* a rewrite rule<br />
* assertions<br />
* and rewrite rules firing before assertions are expanded<br />
<br />
Let's try this. Consider the program:<br />
<br />
<haskell><br />
1 import qualified Data.Map as M<br />
2 import Data.Maybe<br />
3<br />
4 main = print f<br />
5<br />
6 f = let m = M.fromList<br />
7 [(1,"1")<br />
8 ,(2,"2")<br />
9 ,(3,"3")]<br />
10 s = M.lookup 4 m<br />
11 in fromJust s<br />
</haskell><br />
<br />
When we run it we get the not so useful error:<br />
<br />
$ ./A<br />
A: Maybe.fromJust: Nothing<br />
<br />
Ok, so we have a few tricks for locating this, using<br />
[http://www.cse.unsw.edu.au/~dons/loch.html LocH], we can catch an assertion<br />
failure, but we have to insert the assertion by hand:<br />
<br />
<haskell><br />
1 import Debug.Trace.Location<br />
2 import qualified Data.Map as M<br />
3 import Data.Maybe<br />
4<br />
5 main = do print f<br />
6<br />
7 f = let m = M.fromList<br />
8 [(1,"1")<br />
9 ,(2,"2")<br />
10 ,(3,"3")]<br />
11 s = M.lookup 4 m<br />
12 in safeFromJust assert s<br />
13<br />
14 safeFromJust a = check a . fromJust<br />
</haskell><br />
<br />
Which correctly identifies the call site:<br />
<br />
$ ./A<br />
A: A.hs:12:20-25: Maybe.fromJust: Nothing<br />
<br />
Now, this approach is a little fragile. 'assert' is only respected by GHC if -O<br />
is ''not'' on, so if we happened to try this trick with -O, we'd get:<br />
<br />
$ ./A<br />
A: Debug.Trace.Location.failure<br />
<br />
So lesson one: you have to do the bug hunting with -Onot.<br />
<br />
Currently there's -fignore-asserts for turning off assertions, but no flag for<br />
turning them on with -O, Simon, could this be fixed? Could we get a<br />
-frespect-asserts that works even with -O ?<br />
<br />
Ok, assuming this assert trick is used, can we get the compiler to insert the<br />
asserts for us? If so, this would be a great advantage, you'd just be able to<br />
switch on a flag, or import a debugging module, and your fromJusts would be<br />
transparently rewritten. With rewrite rules we do just this!<br />
<br />
So, to our initial unsafe use of fromJust, we add a rewrite rule:<br />
<br />
<haskell><br />
--<br />
-- rewrite fromJust to a located version, and hope that GHC expands<br />
-- 'assert' after the rule fires..<br />
--<br />
{-# RULES<br />
"located fromJust" fromJust = check assert . myFromJust<br />
#-}<br />
</haskell><br />
<br />
This just tells the compiler to replace every occurence of fromJust with a<br />
assertion-throwing fromJust, should it fail. We have to use myFromJust here, to<br />
avoid rule recursion.<br />
<br />
<haskell><br />
--<br />
-- Inlined to avoid recursion in the rule:<br />
--<br />
myFromJust :: Maybe a -> a<br />
myFromJust Nothing = error "Maybe.fromJust: Nothing" -- yuck<br />
myFromJust (Just x) = x<br />
</haskell><br />
<br />
Ok, so can we get ghc to rewrite fromJust to the safe fromJust magicaly?<br />
<br />
$ ghc --make -Onot A.hs -fglasgow-exts -ddump-simpl-stats<br />
[1 of 1] Compiling Main ( A.hs, A.o )<br />
1 RuleFired<br />
1 located fromJust<br />
Linking A ...<br />
<br />
Yes, the rule fired! GHC *did* rewrite our fromJust to a more useful fromJust.<br />
Running it:<br />
<br />
<haskell><br />
$ ./A<br />
A: A.hs:19:36-41: Maybe.fromJust: Nothing<br />
</haskell><br />
<br />
Looks good! But that is deceiving: the assert was expanded before the rule<br />
fired, and refers to the rewrite rule source line (line 19), not the fromJust<br />
call site (line 12). Now if we could just have the 'assert' token inserted<br />
into the AST before it was expanded, we'd be home and dry. Could this be done<br />
with TH? Or could we arrange for asserts in rewrite rules not to be expanded<br />
till later?<br />
<br />
Note that this is still a useful technique, we can rewrite head/fromJust/... to<br />
some other possibly more useful message. And if we can constrain the rule to fire<br />
in only particular modules, we may be able to narrow down the bug, just by<br />
turning on a rule. For example, adding:<br />
<br />
<haskell><br />
{-# RULES<br />
"located fromJust" fromJust = safeFromJust<br />
#-}<br />
<br />
safeFromJust s = case s of<br />
Nothing -> "safeFromJust: failed with Nothing. Ouch"<br />
Just x -> x<br />
</haskell><br />
<br />
will produce:<br />
<br />
<haskell><br />
$ ./A<br />
"safeFromJust: failed with Nothing. Ouch"<br />
</haskell><br />
<br />
So rewrite rules can be used to transparently alter uses of partial functions<br />
like head and fromJust.<br />
<br />
== [[Fusion]] and deforestation ==<br />
<br />
By far the most elaborate use of compiler rewrite rules is to perform<br />
automatic [[deforestation]] of intermediate data structures. This is<br />
documented in a number of [[Research_papers/Compilation#Fusion_and_deforestation|research papers]].<br />
Note that also for this application field of rewrite rules the above mentioned problem of potentially changing the semantics of programs occurs. Read further under [[Correctness of short cut fusion|correctness of short cut fusion]].<br />
<br />
== Symbolic differentiation ==<br />
<br />
I like to demonstrate to people, that GHC has already built-in a kind of Computer Algebra system.<br />
The optimizer can be abused to differentiate expressions symbolically.<br />
Unfortunately it cannot be forced to do so.<br />
That's why, some, better say: too many, expressions are not derived.<br />
<br />
* http://www.haskell.org/pipermail/haskell-cafe/2005-March/009489.html<br />
* http://darcs.haskell.org/htam/src/ExploreHaskell/SymbolicDifferentiation.hs<br />
<br />
<br />
[[Category:Tutorials]]<br />
[[Category:Performance]]<br />
[[Category:Program transformation]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=32085Prime numbers2009-12-03T08:15:59Z<p>Steve C: Add Category:Mathematics</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a prime number (or a prime) is a natural number which has exactly two distinct natural number divisors: 1 and itself.<br />
<br />
[http://www.haskell.org/haskellwiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Finding Primes ==<br />
=== Simple Prime Sieve ===<br />
The following is an elegant way to generate a list of all the prime numbers in the universe:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..] where<br />
sieve (p:xs) <br />
= p : sieve [x | x<-xs, x `mod` p /= 0] -- or: filter ((/=0).(`mod`p)) xs<br />
</haskell><br />
<br />
While appearing simple, this method is extremely inefficient and not recommended for more than a few 1000s of prime numbers, even when compiled. Here, every number is tested for divisibility by each prime up to its smallest factor. This means that any prime is checked against all its preceding primes, when in fact just those not greater than its square root would suffice.<br />
<br />
It in effect creates a cascade of filters in front of the infinite numbers supply, and does so in extremely premature fashion, leading to a blowup of unnecessary filters. One way of fixing that would be to <i>postpone</i> the creation of filters until the right moment, as in<br />
<br />
==== Postponed Filters Prime Sieve ====<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve (tail primes) [5,7..] where <br />
sieve (p:ps) xs<br />
= h ++ sieve ps [x|x<-t, x `rem` p /= 0] -- or: filter ((/=0).(`rem`p)) t<br />
where (h,~(_:t))=span (< p*p) xs<br />
</haskell><br />
<br />
This only tests odd numbers, and by the least amount of primes for each numbers span between successive squares of primes.<br />
<br />
Whereas the first version exhibits near O(<math>{n^3}</math>) behavior, this one exhibits near O(<math>{n^{1.5}}</math>) behavior, and is good for creating a few 100,000s of primes (compiled). It takes as much time to generate <i>100,000 primes</i> with it as it takes for the first one to generate <i>5500</i>, GHC-compiled.<br />
<br />
==== Primality Test and Integer Factorization ====<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
=== Simple Prime Sieve II ===<br />
The following method is slightly faster, creating <i>114,000 primes</i> in the same period of time. It works well for generating a few 100,000s of primes as well:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:filter isPrime [3,5..]<br />
where<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes<br />
divides n p = n `mod` p == 0 <br />
</haskell><br />
<br />
Instead of relying on nested filters, it tests by an explicit list of all the needed prime factors, but it recomputes this list, which will be the same for the increasing spans of numbers between the successive squares of primes.<br />
<br />
=== Simple Prime Sieve III ===<br />
<br />
This is faster still, creating <i>158,000 primes</i> (again, GHC-compiled) in the same time span:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:go 5 [] (tail primes)<br />
where<br />
divisibleBy d n = n `mod` d == 0<br />
go start ds (p:ps) = let pSq = p*p in<br />
foldr (\d -> filter (not . divisibleBy d)) [start, start + 2..pSq - 2] ds<br />
++ go (pSq + 2) (p:ds) ps<br />
</haskell><br />
<br />
This one explicitly maintains the list of primes needed for testing each odds span between successive primes squares, which it also explicitly generates. But it tests with nested <code>filter</code>s, which it repeatedly recreates.<br />
<br />
=== Simple Prime Sieve IV ===<br />
<br />
The list of primes needed to test each range of odds is actually just the prefix of the primes list itself, of known length, and need not be specifically generated at all. Combined with one-call testing by the explicit list of primes, and direct generation of odds between the successive primes squares, this leads to:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: sieve 0 (tail primes) 5<br />
sieve k (p:ps) x<br />
= let fs = take k (tail primes) in<br />
[x | x<-[x,x+2..p*p-2], all ((>0).(x`rem`)) fs]<br />
++ sieve (k+1) ps (p*p+2)<br />
</haskell><br />
<br />
It produces about <i>222,000 primes</i> in the same time span, and is good for creating about a million first primes, compiled.<br />
<br />
The reason to have <code>sieve</code> function available separately too is that it can also be used to produce primes above a given number, as in<br />
<br />
<haskell><br />
primesFrom m = sieve (length h) ps $ m`div`2*2+1 <br />
where <br />
(h,(_:ps)) = span (<= (floor.sqrt.fromIntegral) m) primes<br />
</haskell><br />
<br />
It can thus produce a few primes e.g. above <code>239812076741689</code>, which is a square of the millionth odd prime, without having to compute all the preceding primes (which would number in trillions).<br />
<br />
=== Simple Sieve of Eratosthenes ===<br />
<br />
All the versions thus far try to <i>keep the primes</i> among all numbers by testing <i>each number</i> in isolation. Testing composites is cheap (as most of them will have small factors, so the test is soon aborted), but testing prime numbers is costly. The original sieve of Eratosthenes tries to <i>get rid of composites</i> by working on <i>spans</i> of numbers <i>at once</i> and thus gets its primes <i>"for free"</i>, as gaps between the generated/marked/crossed-off composites:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2: 3: 5: sieve [] (tail primes) 7<br />
where<br />
sieve fs (p:ps) x = comb x comps<br />
++ sieve ((2*p,q+2*p):fs') ps (q+2)<br />
where<br />
comb x (y:ys) | x==y = comb (x+2) ys <br />
| True = x: comb (x+2) (y:ys)<br />
comb x [] = if x==q then [] else [x] <br />
q = p*p<br />
comps = foldl mrg [] mults<br />
(mults,fs') = unzip (map count fs)<br />
count (s,x) | x<q = (x:xs,f) <br />
| True = ([],(s,x)) where (xs,f) = count (s,x+s) <br />
mrg u@(a:t) w@(b:r) <br />
| a<b = a: t `mrg` w<br />
| b<a = b: u `mrg` r <br />
| True = a: t `mrg` r<br />
mrg u w = if null u then w else u<br />
</haskell> <br />
<br />
This "marks" the odd composites in a given range by generating them - just as a person performing the original sieve would do, marking one by one the multiples of the relevant primes - and then combs through them looking for gaps.<br />
<br />
Compared with the previous version, if compiled with GHC, it is steadily 1.2x slower and takes 3.5x more memory. But interpreted under both GHCi and WinHugs, it runs <i>2x to 4x faster</i>, takes <i>less</i> memory, and has better asymptotic behavior.<br />
<br />
=== Prime Wheels ===<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers first that avoid a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
=== Implicit Heap ===<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd ~(x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = f x . foldTree f . pairs $ xs<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*k | k <- [p,p+2..]]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
=== Bitwise prime sieve ===<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
=== Using IntSet for a traditional sieve ===<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== Testing Primality ==<br />
=== Miller-Rabin Primality Test ===<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]<br />
[[Category:Mathematics]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=31430Prime numbers2009-11-05T09:23:11Z<p>Steve C: Split into 2 sections: "Finding Primes" and "Testing Primality"</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a prime number (or a prime) is a natural number which has exactly two distinct natural number divisors: 1 and itself.<br />
<br />
[http://www.haskell.org/haskellwiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Finding Primes ==<br />
=== Simple Prime Sieve ===<br />
The following is an elegant way to generate a list of all the prime numbers in the universe:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0]<br />
</haskell><br />
<br />
While simple, this method is rather inefficient and not recommended for more than the first 1000 prime numbers. For every number <hask>x</hask>, it will test <hask>x `mod` p /= 0</hask> for all prime numbers <hask>p</hask> that are smaller than the smallest prime factor of <hask>x</hask>.<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
=== Simple Prime Sieve II ===<br />
The following method is slightly faster and works well for the first 5000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:filter isPrime [3,5..]<br />
where<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes<br />
divides n p = n `mod` p == 0 <br />
</haskell><br />
<br />
Compared to the previous sieve, it only tests odd numbers and avoids testing for prime factors of <math>n</math> that are larger than <math>\sqrt{n}</math>.<br />
<br />
=== Simple Prime Sieve III ===<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:go 5 [] (tail primes)<br />
where<br />
divisibleBy d n = n `mod` d == 0<br />
go start ds (p:ps) = let pSq = p*p in<br />
foldr (\d -> filter (not . divisibleBy d)) [start, start + 2..pSq - 2] ds<br />
++ go (pSq + 2) (p:ds) ps<br />
</haskell><br />
<br />
Compared to the previous sieve, this keeps a list of the primes needed instead of repeatedly generating it with takeWhile.<br />
<br />
=== Prime Wheels ===<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers that avoid the first a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
=== Implicit Heap ===<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd ~(x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = f x . foldTree f . pairs $ xs<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*k | k <- [p,p+2..]]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
=== Bitwise prime sieve ===<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
=== Using IntSet for a traditional sieve ===<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== Testing Primality ==<br />
=== Miller-Rabin Primality Test ===<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Lambdabot&diff=30713Lambdabot2009-10-10T07:31:21Z<p>Steve C: Remove link which is redirected back to this page.</p>
<hr />
<div>[[Image:lambdabot.png|Lambdabot]]<br />
<br />
Lambdabot is an IRC bot written over several years by those on the #haskell<br />
[[IRC channel]]. It also operates in an offline mode as a Haskell development tool, <br />
and embedded as an extension to ghci.<br />
<br />
Lambdabot's source is available via a [http://darcs.net darcs]<br />
repository, [http://code.haskell.org/lambdabot here]<br />
<br />
Lambdabot is written in Haskell, and supports plugins for adding new<br />
commands. It has many plugins, including:<br />
<br />
* [[Haskell]] evaluation (with persistent declarations)<br />
* [[Hoogle]] interface<br />
* [[Pointfree]] refactoring<br />
* A theorem prover, [[Libraries and tools/Theorem_provers|Djinn]].<br />
* Two [http://homepages.inf.ed.ac.uk/wadler/papers/free/free.ps free theorems] generators<br />
* A testing rig using [[QuickCheck]]<br />
* Another test rig using [[SmallCheck]]<br />
* A lambda calculus interpreter<br />
* A Unlambda interpreter<br />
* A Brainfuck interpreter<br />
* Haskell type and kind checking<br />
* Dynamic plugin composition<br />
* Haskell [http://darcs.haskell.org/packages/ library source] lookup<br />
* Language translation<br />
* [http://code.haskell.org/lambdabot/State/quote Quotes]<br />
* Todo lists<br />
* Irc functions<br />
* [[Darcs]] patch tracking<br />
* A vixen/eliza personality<br />
* Random dice<br />
* [http://code.haskell.org/lambdabot/State/where Project database]<br />
* Dictionary lookups<br />
* Karma tracking<br />
* User poll and election support<br />
* Search Google, Wikipedia and more<br />
* Spell checking<br />
* [http://tinyurl.com Tiny URL] generation<br />
* Much more...<br />
<br />
=== Installing ===<br />
If you are using GHC 6.8.x, just <tt>[[cabal install]] lambdabot</tt>. 6.10.x support is forthcoming; see [[/Building]] for details on that.<br />
<br />
=== Interacting with the Bot ===<br />
<br />
Almost all commands start with an <TT>@</TT> or <tt>?</tt>. A command is invoked by<br />
writing <TT>@<command-name> [args]</TT>. For example <TT>@type map</TT><br />
will make lambdabot respond with <hask>(a -> b) -> [a] -> [b]</hask>. <br />
<br />
The special command <tt>@run</tt> has a synonym <tt>></tt>, meaning that<br />
you can evaluate Haskell expressions dynamically with:<br />
<br />
<haskell><br />
> map (+1) [1..10]<br />
[2,3,4,5,6,7,8,9,10,11]<br />
</haskell><br />
<br />
Good commands to know are <tt>@help</tt> and <tt>@list</tt>.<br />
The full command list is [http://code.haskell.org/lambdabot/COMMANDS here].<br />
<br />
=== Contributing ===<br />
<br />
Lambdabot is an open-source application. The <tt>@version</tt> command<br />
provides details on where to find the source. Anyone may write a plugin.<br />
To submit a plugin, use ''darcs send'' to submit it to the bot's current<br />
maintainer.<br />
<br />
For more info, see the sources or ask the bot itself. If you have any<br />
questions about Lambdabot, ask on the #haskell irc channel.<br />
<br />
=== How can it securely execute Haskell, on the Web? ===<br />
<br />
See [[Safely running untrusted Haskell code]]<br />
<br />
=== @version ===<br />
<br />
A list of various lambdabots over time.<br />
#haskell lambdabot 4p534, GHC 6.6 (Linux i686 2.66GHz)<br />
#haskell lambdabot 4p255, GHC 6.5 (OpenBSD i386)<br />
#haskell lambdabot 4p172, GHC 6.5 (OpenBSD i386)<br />
#haskell lambdabot 3p396, GHC 6.4.1 (Linux i686 3.20GHz)<br />
#haskell lambdabot 2p200, GHC 6.4 (OpenBSD i386)<br />
#haskell lambdabot ghc-6.2.2 (Mar 30 20:48:23 EST 2005) (OpenBSD i386)<br />
#maya lambdabot 3.1p48, GHC 6.5 (OpenBSD i386)<br />
#dutchhack (irc.cyberarmy.net) lambdabot 3p399, GHC 6.4.1 (Linux sparc64)<br />
<br />
=== Lambdabot 2 ===<br />
<br />
Sketch of design for a new lambdabot, smaller, more maintainable, more<br />
reliable. Features:<br />
<br />
Core:<br />
* highly concurrent, as for lambdabot 1<br />
* multiple servers<br />
* configuration scripts<br />
* reliable, robust state checkpointing<br />
* command line, irc, and ide interface<br />
* simple plugin api<br />
* single cabalised bundle<br />
<br />
New plugins:<br />
* beta reduction/term rewriting<br />
* hackage search<br />
* rss subscription<br />
<br />
Improved plugins:<br />
* Hoogle<br />
<br />
[[Category:Tools]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Pointfree&diff=30711Pointfree2009-10-10T05:46:59Z<p>Steve C: Update lambdabot link to the redirected page</p>
<hr />
<div>__TOC__<br />
<br />
'''Pointfree Style'''<br />
<br />
It is very common for functional programmers to write functions as a<br />
composition of other functions, never mentioning the actual arguments<br />
they will be applied to. For example, compare:<br />
<br />
<haskell><br />
sum = foldr (+) 0<br />
</haskell><br />
<br />
with:<br />
<br />
<haskell><br />
sum' xs = foldr (+) 0 xs<br />
</haskell><br />
<br />
These functions perform the same operation, however, the former is more<br />
compact, and is considered cleaner. This is closely related to function<br />
pipelines (and to [http://www.vex.net/~trebla/weblog/pointfree.html unix shell scripting]<br />
): it is clearer to write <hask>let fn = f . g . h</hask> than to<br />
write <hask>let fn x = f (g (h x))</hask>.<br />
<br />
This style is particularly useful when deriving efficient programs by<br />
calculation and, in general, constitutes good discipline. It helps the writer<br />
(and reader) think about composing functions (high level), rather than<br />
shuffling data (low level).<br />
<br />
It is a common experience when rewriting expressions in pointfree style<br />
to derive more compact, clearer versions of the code -- explicit points<br />
often obscure the underlying algorithm.<br />
<br />
Point-free map fusion:<br />
<br />
<haskell><br />
foldr f e . map g == foldr (f . g) e<br />
</haskell><br />
<br />
versus pointful map fusion:<br />
<br />
<haskell><br />
foldr f e . map g == foldr f' e<br />
where f' a b = f (g a) b<br />
</haskell><br />
<br />
Some more examples:<br />
<br />
<haskell><br />
-- point-wise, and point-free member<br />
mem, mem' :: Eq a => a -> [a] -> Bool<br />
<br />
mem x lst = any (== x) lst<br />
mem' = any . (==)<br />
</haskell><br />
<br />
== But pointfree has more points! ==<br />
<br />
A common misconception is that the 'points' of pointfree style are the<br />
<hask>(.)</hask> operator (function composition, as an ASCII symbol),<br />
which uses the same identifier as the decimal point. This is wrong. The<br />
term originated in topology, a branch of mathematics which works with<br />
spaces composed of points, and functions between those spaces. So a<br />
'points-free' definition of a function is one which does not explicitly<br />
mention the points (values) of the space on which the function acts. In<br />
Haskell, our 'space' is some type, and 'points' are values. In the<br />
declaration:<br />
<haskell><br />
f x = x + 1<br />
</haskell><br />
we define the function <hask>f</hask> in terms of its action on an<br />
arbitrary point <hask>x</hask>. Contrast this with the points-free<br />
version:<br />
<haskell><br />
f = (+ 1)<br />
</haskell><br />
where there is no mention of the value on which the function is acting.<br />
<br />
== Background ==<br />
<br />
To find out more about this style, search for Squiggol and the<br />
Bird-Meertens Formalism, a style of functional programming by<br />
calculation that was developed by [http://web.comlab.ox.ac.uk/oucl/work/richard.bird/publications.html Richard Bird], [http://www.kestrel.edu/home/people/meertens/ Lambert Meertens], and<br />
others at Oxford University. [http://web.comlab.ox.ac.uk/oucl/work/jeremy.gibbons/publications/ Jeremy Gibbons] has also written a number of<br />
papers about the topic, which are cited below.<br />
<br />
== Tool support ==<br />
<br />
Thomas Yaeger has<br />
[http://www.cse.unsw.edu.au/~dons/code/lambdabot/Plugins/Pl/ written] a <br />
[http://haskell.org/haskellwiki/Lambdabot Lambdabot] <br />
plugin to automatically convert a large subset of Haskell expressions to<br />
pointfree form. This tool has made it easier to use the more abstract<br />
pointfree encodings (as it saves some mental gymnastics on the part of<br />
the programmer). You can experiment with this in the [[IRC channel|Haskell IRC channel]]. A stand-alone command-line version is available at [http://hackage.haskell.org/packages/archive/pkg-list.html HackageDB] (package pointfree).<br />
<br />
The @pl (point-less) plugin is rather infamous for using the <hask>(->) a</hask> <br />
[[Monad|monad]] to obtain concise code. It also makes use of [[Arrow|Arrows]]. <br />
It also sometimes produces (amusing) code blow ups with the<br />
<hask>(.)</hask> operator. Recently, @unpl has been written, which<br />
(attempts) to unscramble @pl-ified code.<br />
<br />
A transcript:<br />
<br />
<haskell><br />
> pl \x y -> x y<br />
id<br />
<br />
> unpl id<br />
(\ a -> a)<br />
<br />
> pl \x y -> x + 1<br />
const . (1 +)<br />
<br />
> unpl const . (1 +)<br />
(\ e _ -> 1 + e)<br />
<br />
> pl \v1 v2 -> sum (zipWith (*) v1 v2)<br />
(sum .) . zipWith (*)<br />
<br />
> unpl (sum .) . zipWith (*)<br />
(\ d g -> sum (zipWith (*) d g))<br />
<br />
> pl \x y z -> f (g x y z)<br />
((f .) .) . g<br />
<br />
> unpl ((f .) .) . g<br />
(\ e j m -> f (g e j m))<br />
<br />
> pl \x y z -> f (g x y) z<br />
(f .) . g<br />
<br />
> unpl (f .) . g<br />
(\ d i -> f (g d i))<br />
<br />
> pl \x y z -> f z (g x y)<br />
(flip f .) . g<br />
<br />
> unpl (flip f .) . g<br />
(\ i l c -> f c (g i l))<br />
<br />
> pl \(a,b) -> (b,a)<br />
uncurry (flip (,))<br />
<br />
> pl f a b = b a<br />
f = flip id<br />
<br />
> pl \ x -> x * x<br />
join (*)<br />
<br />
> pl \a b -> a:b:[]<br />
(. return) . (:)<br />
<br />
> pl \x -> x+x+x<br />
(+) =<< join (+)<br />
<br />
> pl \a b -> Nothing<br />
const (const Nothing)<br />
<br />
> pl \(a,b) -> (f a, g b)<br />
f *** g<br />
<br />
> pl \f g h x -> f x `h` g x<br />
flip . (ap .) . flip (.)<br />
<br />
> pl \x y -> x . f . y<br />
(. (f .)) . (.)<br />
<br />
> pl \f xs -> xs >>= return . f<br />
fmap<br />
<br />
> pl \h f g x -> f x `h` g x<br />
liftM2<br />
<br />
> pl \f a b c d -> f b c d a<br />
flip . ((flip . (flip .)) .)<br />
<br />
> pl \a (b,c) -> a c b<br />
(`ap` snd) . (. fst) . flip<br />
<br />
> pl \x y -> compare (f x) (f y)<br />
((. f) . compare .)<br />
</haskell><br />
<br />
For many many more examples, google for the results of '@pl' in the<br />
[[IRC_channel|#haskell]] logs. (Or join #haskell on FreeNode and try it<br />
yourself!) It can, of course, get out of hand:<br />
<br />
<haskell><br />
> pl \(a,b) -> a:b:[]<br />
uncurry ((. return) . (:))<br />
<br />
> pl \a b c -> a*b+2+c<br />
((+) .) . flip flip 2 . ((+) .) . (*)<br />
<br />
> pl \f (a,b) -> (f a, f b)<br />
(`ap` snd) . (. fst) . (flip =<< (((.) . (,)) .))<br />
<br />
> pl \f g (a,b) -> (f a, g b)<br />
flip flip snd . (ap .) . flip flip fst . ((.) .) . flip . (((.) . (,)) .)<br />
<br />
> unpl flip flip snd . (ap .) . flip flip fst . ((.) .) . flip . (((.) . (,)) .)<br />
(\ aa f -><br />
(\ p w -> ((,)) (aa (fst p)) (f w)) >>=<br />
\ ao -> snd >>= \ an -> return (ao an))<br />
</haskell><br />
<br />
== Combinator discoveries ==<br />
<br />
Some fun combinators have been found via @pl. Here we list some of the<br />
best:<br />
<br />
=== The owl ===<br />
<br />
<haskell><br />
((.)$(.))<br />
</haskell><br />
<br />
The owl has type <hask>(a -> b -> c) -> a -> (a1 -> b) -> a1 -><br />
c</hask>, and in pointful style can be written as <hask> f a b c d = a b<br />
(c d)</hask>.<br />
<br />
Example <br />
<haskell><br />
> ((.)$(.)) (==) 1 (1+) 0<br />
True<br />
</haskell><br />
<br />
=== Dot ===<br />
<br />
<haskell><br />
dot = ((.).(.))<br />
<br />
dot :: (b -> c) -> (a -> a1 -> b) -> a -> a1 -> c<br />
</haskell><br />
<br />
Example:<br />
<br />
<haskell><br />
sequence `dot` replicate == <br />
(sequence .) . replicate ==<br />
replicateM<br />
<br />
(=<<) == join `dot` fmap<br />
</haskell><br />
<br />
=== Swing ===<br />
<br />
-- Note: @pl had nothing to do with the invention of this combinator. I constructed it by hand after noticing a common pattern. -- Cale<br />
<br />
<haskell><br />
swing :: (((a -> b) -> b) -> c -> d) -> c -> a -> d<br />
swing = flip . (. flip id)<br />
swing f c a = f ($ a) c<br />
</haskell><br />
<br />
Some examples of use:<br />
<br />
<haskell><br />
swing map :: forall a b. [a -> b] -> a -> [b]<br />
swing any :: forall a. [a -> Bool] -> a -> Bool<br />
swing foldr :: forall a b. b -> a -> [a -> b -> b] -> b<br />
swing zipWith :: forall a b c. [a -> b -> c] -> a -> [b] -> [c]<br />
swing find :: forall a. [a -> Bool] -> a -> Maybe (a -> Bool)<br />
-- applies each of the predicates to the given value, returning the first predicate which succeeds, if any<br />
swing partition :: forall a. [a -> Bool] -> a -> ([a -> Bool], [a -> Bool])<br />
</haskell><br />
<br />
=== Squish ===<br />
<br />
<haskell><br />
f >>= a . b . c =<< g<br />
</haskell><br />
<br />
Example:<br />
<br />
<haskell><br />
(readFile y >>=) . ((a . b) .) . c =<< readFile x<br />
</haskell><br />
<br />
[[/Combine|An actually useful example]], numbering lines of a file.<br />
<br />
== Problems with pointfree ==<br />
<br />
Point-free style can (clearly) lead to [[Obfuscation]] when used unwisely.<br />
As higher-order functions are chained together, it can become harder to<br />
mentally infer the types of expressions. The mental cues to an<br />
expression's type (explicit function arguments, and the number of<br />
arguments) go missing.<br />
<br />
Point-free style often times leads to code which is difficult to modify. A function written in a pointfree style may have to be radically changed to make minor changes in functionality. This is because the function becomes more complicated than a composition of lambdas and other functions, and compositions must be changed to application for a pointful function.<br />
<br />
Perhaps these are why pointfree style is sometimes (often?) referred to as<br />
''pointless style''.<br />
<br />
== References ==<br />
<br />
One early reference is<br />
<br />
* Backus, J. 1978. "Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs," Communications of the Association for Computing Machinery 21:613-641.<br />
<br />
which appears to be available (as a scan) at http://www.stanford.edu/class/cs242/readings/backus.pdf<br />
<br />
A paper specifically about point-free style: <br />
* http://web.comlab.ox.ac.uk/oucl/work/jeremy.gibbons/publications/index.html#radix<br />
<br />
This style underlies a lot of expert Haskeller's intuitions. <br />
A rather infamous paper (for all the cute symbols) is Erik Meijer et. al's:<br />
<br />
* Functional Programming with Bananas, Lenses, and Barbed Wire, http://wwwhome.cs.utwente.nl/~fokkinga/mmf91m.ps.<br />
<br />
[http://en.wikipedia.org/wiki/Squiggol Squiggol], and the Bird-Meertens Formalism:<br />
* http://web.comlab.ox.ac.uk/oucl/work/jeremy.gibbons/publications/index.html#squiggolintro.<br />
* A Calculus of Functions for Program Derivation, R.S. Bird, in Res Topics in Fnl Prog, D. Turner ed, A-W 1990.<br />
* The Squiggolist, ed Johan Jeuring, published irregularly by CWI Amsterdam.<br />
<br />
[http://wiki.di.uminho.pt/twiki/bin/view/Personal/Alcino/PointlessHaskell Pointless Haskell] is a library for point-free programming with recursion patterns defined as hylomorphisms. It also allows the visualization of the intermediate data structure of the hylomorphisms with GHood. This feature together with the DrHylo tool allows us to easily visualize recursion trees of Haskell functions. [http://wiki.di.uminho.pt/wiki/pub/Ze/Bic/report.pdf Haskell Manipulation] by Jose Miguel Paiva Proenca discusses this tool based approach to re-factoring.<br />
<br />
This project is written by [http://www.di.uminho.pt/~mac/ Manuel Alcino Cunha], see his homepage for more related materials on the topic.<br />
An extended verson of his paper ''Point-free Programming with Hylomorphisms'' can be found [http://web.comlab.ox.ac.uk/oucl/research/pdt/ap/dgp/workshop2004/cunha.pdf here].<br />
<br />
== Other areas ==<br />
<br />
[[Combinatory logic]] and also [[Recursive function theory]] can be said in some sense pointfree.<br />
<br />
Are there pointfree approaches to [[relational algebra]]?<br />
See [http://www.di.uminho.pt/~jno/ps/_.pdf First Steps in Pointfree Functional Dependency Theory] written by Jos Nuno Oliveira. A concise and deep approach. See also [http://www.di.uminho.pt/~jno/html/ the author's homepage] and also [http://www.di.uminho.pt/~jno/html/jnopub.html his many other papers] -- many materials related to this topic can be found there.<br />
<br />
[[Category:Idioms]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Parsec&diff=30662Parsec2009-10-08T02:00:22Z<p>Steve C: Update ghc/parsec link to point to latest version of ghc</p>
<hr />
<div>== Parsec ==<br />
<br />
[[Category:Compiler tools]]<br />
[[Category:Combinators]]<br />
<br />
Parsec is an industrial strength, monadic parser combinator library for<br />
Haskell. It can parse context-sensitive, infinite look-ahead grammars<br />
but it performs best on predictive (LL[1]) grammars. <br />
<br />
For downloads and documentation see:<br />
* [http://legacy.cs.uu.nl/daan/parsec.html the Parsec site]<br />
* [http://hackage.haskell.org/package/parsec hackage]<br />
* [http://www.haskell.org/ghc/docs/latest/html/libraries/parsec/Text-ParserCombinators-Parsec.html GHC library documentation]<br />
<br />
=== Usage ===<br />
<br />
Parsec lets you construct parsers by combining higher-order<br />
[[Combinator]]s to create larger expressions. Combinator parsers are<br />
written and used within the same programming language as the rest of the<br />
program. The parsers are first-class citizens of the language , unlike<br />
[[Happy]] parsers, which must be generated via a preprocessor.<br />
<br />
Much more documentation can be found on the parsec site.<br />
<br />
{{Template:Stub}}<br />
<br />
=== Parsec clones in other languages ===<br />
<br />
* PCL for O'Caml http://lprousnth.files.wordpress.com/2007/08/pcl.pdf<br />
* JParsec for Java http://jparsec.codehaus.org/JParsec+Overview<br />
* NParsec, JParsec ported to C# http://jparsec.codehaus.org/NParsec+Tutorial<br />
* Ruby Parsec, JParsec ported to Ruby http://jparsec.codehaus.org/Ruby+Parsec<br />
* FParsec for F# http://www.quanttec.com/fparsec/<br />
* A small Parsec in Erlang http://www.engr.uconn.edu/~jeffm/Source/Erlang/<br />
* AliceParsec for Alice ML http://www.ps.uni-sb.de/alice/contribs.html<br />
* Parsnip for C++ http://parsnip-parser.sourceforge.net/<br />
* Somewhere there is a Nemerle port<br />
* Pysec for Python http://www.valuedlessons.com/2008/02/pysec-monadic-combinatoric-parsing-in.html<br />
<br />
Interesting non-Parsec parser combinator libraries:<br />
* Spirit for C++ http://spirit.sourceforge.net/documentation.html<br />
* Scala http://www.scala-lang.org/docu/files/api/scala/util/parsing/combinator$content.html</div>Steve Chttps://wiki.haskell.org/index.php?title=Introduction_to_QuickCheck1&diff=30625Introduction to QuickCheck12009-10-06T11:57:22Z<p>Steve C: Note that links are to QuickCheck version 1</p>
<hr />
<div>A quick introduction to QuickCheck, and testing Haskell code.<br />
<br />
== Motivation ==<br />
<br />
In September 2006, Bruno MartÃnez<br />
[http://www.haskell.org/pipermail/haskell-cafe/2006-September/018302.html asked] <br />
the following question:<br />
<br />
<haskell><br />
-- I've written a function that looks similar to this one<br />
<br />
getList = find 5 where<br />
find 0 = return []<br />
find n = do<br />
ch <- getChar<br />
if ch `elem` ['a'..'e'] then do<br />
tl <- find (n-1)<br />
return (ch : tl) else<br />
find n<br />
<br />
-- I want to test this function, without hitting the filesystem. In C++ I<br />
-- would use a istringstream. I couldn't find a function that returns a<br />
-- Handle from a String. The closer thing that may work that I could find<br />
-- was making a pipe and convertind the file descriptor. Can I simplify<br />
-- that function to take it out of the IO monad?<br />
</haskell><br />
<br />
So the problem is: how to effectively test this function in Haskell? The<br />
solution we turn to is refactoring and QuickCheck.<br />
<br />
== Keeping things pure ==<br />
<br />
The reason your getList is hard to test, is that the side effecting monadic code <br />
is mixed in with the pure computation, making it difficult to test<br />
without moving entirely into a "black box" IO-based testing model.<br />
Such a mixture is not good for reasoning about code.<br />
<br />
Let's untangle that, and then test the referentially transparent<br />
parts simply with QuickCheck. We can take advantage of lazy IO firstly,<br />
to avoid all the unpleasant low-level IO handling. <br />
<br />
So the first step is to factor out the IO part of the function into a<br />
thin "skin" layer:<br />
<br />
<haskell><br />
-- A thin monadic skin layer<br />
getList :: IO [Char]<br />
getList = fmap take5 getContents<br />
<br />
-- The actual worker<br />
take5 :: [Char] -> [Char]<br />
take5 = take 5 . filter (`elem` ['a'..'e'])<br />
</haskell><br />
<br />
== Testing with QuickCheck ==<br />
<br />
Now we can test the 'guts' of the algorithm, the take5 function, in<br />
isolation. Let's use QuickCheck. First we need an Arbitrary instance for<br />
the Char type -- this takes care of generating random Chars for us to<br />
test with. I'll restrict it to a range of nice chars just for<br />
simplicity:<br />
<br />
<haskell><br />
import Data.Char<br />
import Test.QuickCheck<br />
<br />
instance Arbitrary Char where<br />
arbitrary = choose ('\32', '\128')<br />
coarbitrary c = variant (ord c `rem` 4)<br />
</haskell><br />
<br />
Let's fire up GHCi (or Hugs) and try some generic properties (its nice<br />
that we can use the QuickCheck testing framework directly from the<br />
Haskell prompt). An easy one first, a [Char] is equal to itself:<br />
<br />
<haskell><br />
*A> quickCheck ((\s -> s == s) :: [Char] -> Bool)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
What just happened? QuickCheck generated 100 random [Char] values, and<br />
applied our property, checking the result was True for all cases.<br />
QuickCheck ''generated the test sets for us''!<br />
<br />
A more interesting property now: reversing twice is the identity:<br />
<br />
<haskell><br />
*A> quickCheck ((\s -> (reverse.reverse) s == s) :: [Char] -> Bool)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Great!<br />
<br />
== Testing take5 ==<br />
<br />
The first step to testing with QuickCheck is to work out some properties<br />
that are true of the function, for all inputs. That is, we need to find<br />
''invariants''.<br />
<br />
A simple invariant might be:<br />
<math>\forall~s~.~length~(take5~s)~=~5</math><br />
<br />
So let's write that as a QuickCheck property:<br />
<haskell><br />
\s -> length (take5 s) == 5<br />
</haskell><br />
<br />
Which we can then run in QuickCheck as:<br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) == 5)<br />
Falsifiable, after 0 tests:<br />
""<br />
</haskell><br />
<br />
Ah! QuickCheck caught us out. If the input string contains less than 5<br />
filterable characters, the resulting string will be less than 5<br />
characters long. So let's weaken the property a bit:<br />
<math>\forall~s~.~length~(take5~s)~\le~5</math><br />
<br />
That is, take5 returns a string of at most 5 characters long. Let's test<br />
this: <br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) <= 5)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Good!<br />
<br />
== Another property ==<br />
<br />
Another thing to check would be that the correct characters are<br />
returned. That is, for all returned characters, those characters are<br />
members of the set ['a','b','c','d','e'].<br />
<br />
We can specify that as:<br />
<math>\forall~s~.~\forall~e~.~e~\in~take5~s~\to~e~\in~[abcde] </math><br />
<br />
And in QuickCheck:<br />
<haskell><br />
*A> quickCheck (\s -> all (`elem` ['a'..'e']) (take5 s))<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Excellent. So we can have some confidence that the function neither<br />
returns strings that are too long, nor includes invalid characters.<br />
<br />
== Coverage ==<br />
<br />
One issue with the default QuickCheck configuration, when testing<br />
[Char], is that the standard 100 tests isn't enough for our situation.<br />
In fact, QuickCheck never generates a String greater than 5 characters<br />
long, when using the supplied Arbtrary instance for Char! We can confirm<br />
this:<br />
<br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) < 5)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
QuickCheck wastes its time generating different Chars, when what we<br />
really need is longer strings. One solution to this is to modify<br />
QuickCheck's default configuration to test deeper:<br />
<br />
<haskell><br />
deepCheck p = check (defaultConfig { configMaxTest = 10000}) p<br />
</haskell><br />
<br />
This instructs the system to find at least 10000 test cases before<br />
concluding that all is well. Let's check that it is generating longer<br />
strings:<br />
<br />
<haskell><br />
*A> deepCheck (\s -> length (take5 s) < 5)<br />
Falsifiable, after 125 tests:<br />
";:iD^*NNi~Y\\RegMob\DEL@krsx/=dcf7kub|EQi\DELD*"<br />
</haskell><br />
<br />
We can check the test data QuickCheck is generating using the<br />
'verboseCheck' hook. Here, testing on integers lists:<br />
<br />
<haskell><br />
*A> verboseCheck (\s -> length s < 5)<br />
0: []<br />
1: [0]<br />
2: []<br />
3: []<br />
4: []<br />
5: [1,2,1,1]<br />
6: [2]<br />
7: [-2,4,-4,0,0]<br />
Falsifiable, after 7 tests:<br />
[-2,4,-4,0,0]<br />
</haskell><br />
<br />
== Going further ==<br />
<br />
QuickCheck is effectively an embedded domain specific language for<br />
testing Haskell code, and allows for much more complex properties than<br />
those you've seen here to be tested. Some sources for further reading<br />
are:<br />
* [http://www.cse.unsw.edu.au/~dons/data/QuickCheck.html The QuickCheck v1 source]<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/QuickCheck/Test-QuickCheck.html QuickCheck v1 Library documentation]<br />
* [http://www.cs.chalmers.se/~rjmh/QuickCheck/manual.html QuickCheck v1 Manual]<br />
* [http://www.cse.unsw.edu.au/~dons/code/fps/tests/Properties.hs A large testsuite of QuickCheck code]<br />
* Tutorial: [[QuickCheck as a test set generator]]<br />
* Tutorial: [[QuickCheck / GADT]]<br />
* More [http://haskell.org/haskellwiki/Research_papers/Testing_and_correctness research on correctness and testing] in Haskell<br />
<br />
* 2009 Blog article: [http://koweycode.blogspot.com/2009/07/some-ideas-for-practical-quickcheck.html some ideas for practical QuickCheck]<br />
* 2004 Paper [http://www.math.chalmers.se/~koen/pubs/entry-tt04-quickcheck.html QuickCheck: Specification-based Random Testing], Koen Claessen. Presentation at Summer Institute on Trends in Testing: Theory, Techniques and Tools, August 2004.<br />
* 2003 Paper [http://www.math.chalmers.se/~koen/pubs/entry-fop-quickcheck.html Specification Based Testing with QuickCheck], Koen Claessen and John Hughes. In Jeremy Gibbons and Oege de Moor (eds.), The Fun of Programming, Cornerstones of Computing, pp. 17--40, Palgrave, 2003.<br />
* 2002 Paper [http://www.cs.chalmers.se/~rjmh/Papers/QuickCheckST.ps Testing Monadic Programs with QuickCheck], Koen Claessen, John Hughes. SIGPLAN Notices 37(12): 47-59 (2002):<br />
* 2000 Paper [http://www.cs.chalmers.se/~koen/pubs/icfp00-quickcheck.ps QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs], Koen Claessen and John Hughes. In Proc. of International Conference on Functional Programming (ICFP), ACM SIGPLAN, 2000.<br />
<br />
Note, QuickCheck doesn't need to just be an embedded domain specific language for testing ''Haskell'' code. By making instances of Arbitrary for FFI types you can use Haskell and QuickCheck to check code in other languages.<br />
<br />
[[Category:Tutorials]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Introduction_to_QuickCheck1&diff=30624Introduction to QuickCheck12009-10-06T11:54:00Z<p>Steve C: Delete link to a 'permission denied' document.</p>
<hr />
<div>A quick introduction to QuickCheck, and testing Haskell code.<br />
<br />
== Motivation ==<br />
<br />
In September 2006, Bruno MartÃnez<br />
[http://www.haskell.org/pipermail/haskell-cafe/2006-September/018302.html asked] <br />
the following question:<br />
<br />
<haskell><br />
-- I've written a function that looks similar to this one<br />
<br />
getList = find 5 where<br />
find 0 = return []<br />
find n = do<br />
ch <- getChar<br />
if ch `elem` ['a'..'e'] then do<br />
tl <- find (n-1)<br />
return (ch : tl) else<br />
find n<br />
<br />
-- I want to test this function, without hitting the filesystem. In C++ I<br />
-- would use a istringstream. I couldn't find a function that returns a<br />
-- Handle from a String. The closer thing that may work that I could find<br />
-- was making a pipe and convertind the file descriptor. Can I simplify<br />
-- that function to take it out of the IO monad?<br />
</haskell><br />
<br />
So the problem is: how to effectively test this function in Haskell? The<br />
solution we turn to is refactoring and QuickCheck.<br />
<br />
== Keeping things pure ==<br />
<br />
The reason your getList is hard to test, is that the side effecting monadic code <br />
is mixed in with the pure computation, making it difficult to test<br />
without moving entirely into a "black box" IO-based testing model.<br />
Such a mixture is not good for reasoning about code.<br />
<br />
Let's untangle that, and then test the referentially transparent<br />
parts simply with QuickCheck. We can take advantage of lazy IO firstly,<br />
to avoid all the unpleasant low-level IO handling. <br />
<br />
So the first step is to factor out the IO part of the function into a<br />
thin "skin" layer:<br />
<br />
<haskell><br />
-- A thin monadic skin layer<br />
getList :: IO [Char]<br />
getList = fmap take5 getContents<br />
<br />
-- The actual worker<br />
take5 :: [Char] -> [Char]<br />
take5 = take 5 . filter (`elem` ['a'..'e'])<br />
</haskell><br />
<br />
== Testing with QuickCheck ==<br />
<br />
Now we can test the 'guts' of the algorithm, the take5 function, in<br />
isolation. Let's use QuickCheck. First we need an Arbitrary instance for<br />
the Char type -- this takes care of generating random Chars for us to<br />
test with. I'll restrict it to a range of nice chars just for<br />
simplicity:<br />
<br />
<haskell><br />
import Data.Char<br />
import Test.QuickCheck<br />
<br />
instance Arbitrary Char where<br />
arbitrary = choose ('\32', '\128')<br />
coarbitrary c = variant (ord c `rem` 4)<br />
</haskell><br />
<br />
Let's fire up GHCi (or Hugs) and try some generic properties (its nice<br />
that we can use the QuickCheck testing framework directly from the<br />
Haskell prompt). An easy one first, a [Char] is equal to itself:<br />
<br />
<haskell><br />
*A> quickCheck ((\s -> s == s) :: [Char] -> Bool)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
What just happened? QuickCheck generated 100 random [Char] values, and<br />
applied our property, checking the result was True for all cases.<br />
QuickCheck ''generated the test sets for us''!<br />
<br />
A more interesting property now: reversing twice is the identity:<br />
<br />
<haskell><br />
*A> quickCheck ((\s -> (reverse.reverse) s == s) :: [Char] -> Bool)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Great!<br />
<br />
== Testing take5 ==<br />
<br />
The first step to testing with QuickCheck is to work out some properties<br />
that are true of the function, for all inputs. That is, we need to find<br />
''invariants''.<br />
<br />
A simple invariant might be:<br />
<math>\forall~s~.~length~(take5~s)~=~5</math><br />
<br />
So let's write that as a QuickCheck property:<br />
<haskell><br />
\s -> length (take5 s) == 5<br />
</haskell><br />
<br />
Which we can then run in QuickCheck as:<br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) == 5)<br />
Falsifiable, after 0 tests:<br />
""<br />
</haskell><br />
<br />
Ah! QuickCheck caught us out. If the input string contains less than 5<br />
filterable characters, the resulting string will be less than 5<br />
characters long. So let's weaken the property a bit:<br />
<math>\forall~s~.~length~(take5~s)~\le~5</math><br />
<br />
That is, take5 returns a string of at most 5 characters long. Let's test<br />
this: <br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) <= 5)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Good!<br />
<br />
== Another property ==<br />
<br />
Another thing to check would be that the correct characters are<br />
returned. That is, for all returned characters, those characters are<br />
members of the set ['a','b','c','d','e'].<br />
<br />
We can specify that as:<br />
<math>\forall~s~.~\forall~e~.~e~\in~take5~s~\to~e~\in~[abcde] </math><br />
<br />
And in QuickCheck:<br />
<haskell><br />
*A> quickCheck (\s -> all (`elem` ['a'..'e']) (take5 s))<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Excellent. So we can have some confidence that the function neither<br />
returns strings that are too long, nor includes invalid characters.<br />
<br />
== Coverage ==<br />
<br />
One issue with the default QuickCheck configuration, when testing<br />
[Char], is that the standard 100 tests isn't enough for our situation.<br />
In fact, QuickCheck never generates a String greater than 5 characters<br />
long, when using the supplied Arbtrary instance for Char! We can confirm<br />
this:<br />
<br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) < 5)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
QuickCheck wastes its time generating different Chars, when what we<br />
really need is longer strings. One solution to this is to modify<br />
QuickCheck's default configuration to test deeper:<br />
<br />
<haskell><br />
deepCheck p = check (defaultConfig { configMaxTest = 10000}) p<br />
</haskell><br />
<br />
This instructs the system to find at least 10000 test cases before<br />
concluding that all is well. Let's check that it is generating longer<br />
strings:<br />
<br />
<haskell><br />
*A> deepCheck (\s -> length (take5 s) < 5)<br />
Falsifiable, after 125 tests:<br />
";:iD^*NNi~Y\\RegMob\DEL@krsx/=dcf7kub|EQi\DELD*"<br />
</haskell><br />
<br />
We can check the test data QuickCheck is generating using the<br />
'verboseCheck' hook. Here, testing on integers lists:<br />
<br />
<haskell><br />
*A> verboseCheck (\s -> length s < 5)<br />
0: []<br />
1: [0]<br />
2: []<br />
3: []<br />
4: []<br />
5: [1,2,1,1]<br />
6: [2]<br />
7: [-2,4,-4,0,0]<br />
Falsifiable, after 7 tests:<br />
[-2,4,-4,0,0]<br />
</haskell><br />
<br />
== Going further ==<br />
<br />
QuickCheck is effectively an embedded domain specific language for<br />
testing Haskell code, and allows for much more complex properties than<br />
those you've seen here to be tested. Some sources for further reading<br />
are:<br />
* [http://www.cse.unsw.edu.au/~dons/data/QuickCheck.html The QuickCheck source]<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/QuickCheck/Test-QuickCheck.html Library documentation]<br />
* [http://www.cse.unsw.edu.au/~dons/code/fps/tests/Properties.hs A large testsuite of QuickCheck code]<br />
* [http://www.cs.chalmers.se/~rjmh/QuickCheck/manual.html QuickCheck Manual]<br />
* Tutorial: [[QuickCheck as a test set generator]]<br />
* Tutorial: [[QuickCheck / GADT]]<br />
* More [http://haskell.org/haskellwiki/Research_papers/Testing_and_correctness research on correctness and testing] in Haskell<br />
<br />
* 2009 Blog article: [http://koweycode.blogspot.com/2009/07/some-ideas-for-practical-quickcheck.html some ideas for practical QuickCheck]<br />
* 2004 Paper [http://www.math.chalmers.se/~koen/pubs/entry-tt04-quickcheck.html QuickCheck: Specification-based Random Testing], Koen Claessen. Presentation at Summer Institute on Trends in Testing: Theory, Techniques and Tools, August 2004.<br />
* 2003 Paper [http://www.math.chalmers.se/~koen/pubs/entry-fop-quickcheck.html Specification Based Testing with QuickCheck], Koen Claessen and John Hughes. In Jeremy Gibbons and Oege de Moor (eds.), The Fun of Programming, Cornerstones of Computing, pp. 17--40, Palgrave, 2003.<br />
* 2002 Paper [http://www.cs.chalmers.se/~rjmh/Papers/QuickCheckST.ps Testing Monadic Programs with QuickCheck], Koen Claessen, John Hughes. SIGPLAN Notices 37(12): 47-59 (2002):<br />
* 2000 Paper [http://www.cs.chalmers.se/~koen/pubs/icfp00-quickcheck.ps QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs], Koen Claessen and John Hughes. In Proc. of International Conference on Functional Programming (ICFP), ACM SIGPLAN, 2000.<br />
<br />
Note, QuickCheck doesn't need to just be an embedded domain specific language for testing ''Haskell'' code. By making instances of Arbitrary for FFI types you can use Haskell and QuickCheck to check code in other languages.<br />
<br />
[[Category:Tutorials]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Performance/Overloading&diff=30623Performance/Overloading2009-10-06T07:29:13Z<p>Steve C: Replace JHC with Jhc</p>
<hr />
<div>{{Performance infobox}}<br />
[[Category:Performance|Overloading]]<br />
== Overloaded functions are not your friend ==<br />
<br />
Haskell's [[Ad-hoc_polymorphism | overloading]] (using type classes) is elegant, neat, etc., etc., but it is death to performance if left to linger in an inner loop. Each type class constraint on an overloaded function corresponds to an extra argument passed at runtime (called a ''dictionary'') (except in [[Jhc]]!), and a reference to a class method is implemented by extracting a field from the dictionary. The presence of overloading can prevent many other optimisations, such as [[Performance/Strictness|Strictness]], from kicking in.<br />
<br />
How can you squash overloading?<br />
<br />
<b>Give explicit type signatures</b>: Signatures are the basic trick; putting them on exported, top-level functions is good software-engineering practice, anyway. (GHC Tip: using <tt>-fwarn-missing-signatures</tt> can help enforce good signature-practice).<br />
<br />
The automatic specialisation of overloaded functions (with <tt>-O</tt>) should take care of overloaded local and/or unexported functions.<br />
<br />
<b>Use <tt>SPECIALIZE</tt> pragmas</b>: Specialize the overloading on key functions in your program. (Supported by: [[GHC]], [[Jhc]]).<br />
<br />
<b>Alternatively, use <tt>INLINE</tt> pragmas</b>: This will just include function body at each call place. You can see sources of [[Library/Streams]] that heavily uses inlining to get highest performance marks even for polymorphism married to complex class structure. The back side of this heavy inlining is fast growth of compiled code. Don't use INLINE and SPECIALIZE on the same function. (Supported by: [[GHC]], [[Jhc]]).<br />
<br />
<b>Use a module export list</b>:<br />
If you omit the export list for the module, then everything in the module is exported. Also, any functions that lack type signatures will have the most general i.e. <i>polymorphic</i> types inferred for them. If those same functions are <i>only</i> used inside that module, then they are likely to be more polymorphic than necessary. Removing them from the module export list (whether explicit or implicit) will allow the compiler to specialise them to just the type(s) used in the module.</div>Steve Chttps://wiki.haskell.org/index.php?title=Performance/Overloading&diff=30622Performance/Overloading2009-10-06T07:23:36Z<p>Steve C: Add link to Ad-hoc_polymorphism page</p>
<hr />
<div>{{Performance infobox}}<br />
[[Category:Performance|Overloading]]<br />
== Overloaded functions are not your friend ==<br />
<br />
Haskell's [[Ad-hoc_polymorphism | overloading]] (using type classes) is elegant, neat, etc., etc., but it is death to performance if left to linger in an inner loop. Each type class constraint on an overloaded function corresponds to an extra argument passed at runtime (called a ''dictionary'') (except in [[JHC]]!), and a reference to a class method is implemented by extracting a field from the dictionary. The presence of overloading can prevent many other optimisations, such as [[Performance/Strictness|Strictness]], from kicking in.<br />
<br />
How can you squash overloading?<br />
<br />
<b>Give explicit type signatures</b>: Signatures are the basic trick; putting them on exported, top-level functions is good software-engineering practice, anyway. (GHC Tip: using <tt>-fwarn-missing-signatures</tt> can help enforce good signature-practice).<br />
<br />
The automatic specialisation of overloaded functions (with <tt>-O</tt>) should take care of overloaded local and/or unexported functions.<br />
<br />
<b>Use <tt>SPECIALIZE</tt> pragmas</b>: Specialize the overloading on key functions in your program. (Supported by: [[GHC]], [[JHC]]).<br />
<br />
<b>Alternatively, use <tt>INLINE</tt> pragmas</b>: This will just include function body at each call place. You can see sources of [[Library/Streams]] that heavily uses inlining to get highest performance marks even for polymorphism married to complex class structure. The back side of this heavy inlining is fast growth of compiled code. Don't use INLINE and SPECIALIZE on the same function. (Supported by: [[GHC]], [[JHC]]).<br />
<br />
<b>Use a module export list</b>:<br />
If you omit the export list for the module, then everything in the module is exported. Also, any functions that lack type signatures will have the most general i.e. <i>polymorphic</i> types inferred for them. If those same functions are <i>only</i> used inside that module, then they are likely to be more polymorphic than necessary. Removing them from the module export list (whether explicit or implicit) will allow the compiler to specialise them to just the type(s) used in the module.</div>Steve Chttps://wiki.haskell.org/index.php?title=Ad-hoc_polymorphism&diff=30621Ad-hoc polymorphism2009-10-06T07:20:51Z<p>Steve C: add 'or overloading'</p>
<hr />
<div>[[Category:Glossary]]<br />
A value is [[polymorphism|polymorphic]] if, depending on its context, it can assume more than one type. If the possible types are limited and must be individually specified before use, this is called ad-hoc polymorphism (or overloading).<br />
<br />
In object-oriented languages, ad-hoc polymorphism is often called ''overloading.'' For instance, in C++, the operator '''<hask>+</hask>''' may have type (in Haskell notation) <hask>Int -> Int -> Int</hask> or <hask>String -> String -> String</hask>, and may be applied to new types provided you write a definition for it using those types as arguments. The range of possible types of '''<hask>+</hask>''' in C++ is limited to those built-in and those explicitly defined.<br />
<br />
Many non-object-oriented languages use ad-hoc polymorphism too. In C, the operator '''<hask>+</hask>''' may have the types <hask>Int -> Int -> Int</hask> or <hask>Float -> Float -> Float</hask> or <hask>Double -> Int -> Double</hask> or <hask>Ptr -> Int -> Ptr</hask>. The range of possible types of <hask>+</hask> in C is limited to those built-in.<br />
<br />
In Haskell, ad-hoc polymorphism is achieved by type classes. For instance, the function <hask>show</hask> may have type of the form <hask>Int -> String</hask> or <hask>Float -> String</hask> or <hask>(Maybe Int, String, Either Char Float) -> String</hask> depending on its context, but the possible types of <hask>show</hask> are limited. The user must ''specify'' the type <hask>show</hask> will take by defining the type of its first argument as an instance of <hask>Show</hask>. (This is reflected in its signature <hask>show :: Show a => a -> String</hask>.) <br />
<br />
Contrast ad-hoc polymorphism with [[parametric polymorphism]] of the function <hask>length :: [a] -> Int</hask>. <hask>length</hask> may be applied to lists of ''any'' type--that is, may take on an unlimited number of types (of the form <hask>[a] -> Int</hask>)--and the user may apply it to any list without needing to specify the type beforehand.</div>Steve Chttps://wiki.haskell.org/index.php?title=Polymorphism&diff=30620Polymorphism2009-10-06T07:02:14Z<p>Steve C: Improve wording, and add Wikipedia link.</p>
<hr />
<div>[[Category:Glossary]]<br />
A value is polymorphic if, depending on the context where it's used, it can take on more than one type.<br />
<br />
There are different kinds of polymorphism.<br />
<br />
#[[Parametric polymorphism]]; mostly found in functional languages<br />
#[[Ad-hoc polymorphism]] or overloading<br />
#[[Inclusion polymorphism]]; mostly found in object oriented languages<br />
<br />
== Examples ==<br />
<haskell><br />
foldr :: forall a b. (a -> b -> b) -> b -> [a] -> b<br />
</haskell><br />
<hask>foldr</hask> is a parametric polymorphic [[function]]. When actually used, it may take on any of a variety of types, for example:<br />
<haskell><br />
::(Char ->Int->Int)->Int->String->Int<br />
::(String->String->String)->String->[String]->String<br />
</haskell><br />
An "integer literal" is a parametric polymorphic data type:<br />
<haskell><br />
1 :: forall t. (Num t) => t<br />
</haskell><br />
<br />
== References ==<br />
*[http://citeseer.nj.nec.com/cardelli85understanding.html On Understanding Types, Data Abstraction, and Polymorphism (1985)], by Luca Cardelli, Peter Wegner in ACM Computing Surveys.<br />
<br />
*[http://en.wikipedia.org/wiki/Type_polymorphism Type polymorphism] at Wikipedia</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=30619Prime numbers2009-10-06T05:51:33Z<p>Steve C: Add a new 'Prime Number Resources' section.</p>
<hr />
<div>== Prime Number Resources ==<br />
In mathematics, a prime number (or a prime) is a natural number which has exactly two distinct natural number divisors: 1 and itself.<br />
<br />
[http://www.haskell.org/haskellwiki/Prime_numbers Prime Numbers] at Wikipedia.<br />
<br />
[http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes Sieve of Eratosthenes] at Wikipedia.<br />
<br />
HackageDB packages:<br />
<br />
[http://hackage.haskell.org/package/Numbers Numbers]: An assortment of number theoretic functions.<br />
<br />
[http://hackage.haskell.org/package/NumberSieves NumberSieves]: Number Theoretic Sieves: primes, factorization, and Euler's Totient.<br />
<br />
[http://hackage.haskell.org/package/primes primes]: Efficient, purely functional generation of prime numbers.<br />
<br />
== Simple Prime Sieve ==<br />
The following is an elegant way to generate a list of all the prime numbers in the universe:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0]<br />
</haskell><br />
<br />
While simple, this method is rather inefficient and not recommended for more than the first 1000 prime numbers. For every number <hask>x</hask>, it will test <hask>x `mod` p /= 0</hask> for all prime numbers <hask>p</hask> that are smaller than the smallest prime factor of <hask>x</hask>.<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
== Simple Prime Sieve II ==<br />
The following method is slightly faster and works well for the first 5000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:filter isPrime [3,5..]<br />
where<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes<br />
divides n p = n `mod` p == 0 <br />
</haskell><br />
<br />
Compared to the previous sieve, it only tests odd numbers and avoids testing for prime factors of <math>n</math> that are larger than <math>\sqrt{n}</math>.<br />
<br />
== Simple Prime Sieve III ==<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:go 5 [] (tail primes)<br />
where<br />
divisibleBy d n = n `mod` d == 0<br />
go start ds (p:ps) = let pSq = p*p in<br />
foldr (\d -> filter (not . divisibleBy d)) [start, start + 2..pSq - 2] ds<br />
++ go (pSq + 2) (p:ds) ps<br />
</haskell><br />
<br />
Compared to the previous sieve, this keeps a list of the primes needed instead of repeatedly generating it with takeWhile.<br />
<br />
== Prime Wheels ==<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers that avoid the first a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
== Implicit Heap ==<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd ~(x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = f x . foldTree f . pairs $ xs<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*k | k <- [p,p+2..]]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
== Bitwise prime sieve ==<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
== Miller-Rabin Primality Test ==<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== Using IntSet for a traditional sieve ==<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Prime_numbers&diff=30615Prime numbers2009-10-06T02:09:32Z<p>Steve C: Add link to Template Haskell page</p>
<hr />
<div>== Simple Prime Sieve ==<br />
The following is an elegant way to generate a list of all the prime numbers in the universe:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = sieve [2..]<br />
where sieve (p:xs) = p : sieve [x | x<-xs, x `mod` p /= 0]<br />
</haskell><br />
<br />
While simple, this method is rather inefficient and not recommended for more than the first 1000 prime numbers. For every number <hask>x</hask>, it will test <hask>x `mod` p /= 0</hask> for all prime numbers <hask>p</hask> that are smaller than the smallest prime factor of <hask>x</hask>.<br />
<br />
Given an infinite list of prime numbers, we can implement primality tests and integer factorization:<br />
<br />
<haskell><br />
isPrime n = n > 1 && n == head (primeFactors n)<br />
<br />
primeFactors 1 = []<br />
primeFactors n = go n primes<br />
where<br />
go n ps@(p:pt)<br />
| p*p > n = [n]<br />
| n `rem` p == 0 = p : go (n `quot` p) ps<br />
| otherwise = go n pt<br />
</haskell><br />
<br />
== Simple Prime Sieve II ==<br />
The following method is slightly faster and works well for the first 5000 primes:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:filter isPrime [3,5..]<br />
where<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes<br />
divides n p = n `mod` p == 0 <br />
</haskell><br />
<br />
Compared to the previous sieve, it only tests odd numbers and avoids testing for prime factors of <math>n</math> that are larger than <math>\sqrt{n}</math>.<br />
<br />
== Simple Prime Sieve III ==<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:go 5 [] (tail primes)<br />
where<br />
divisibleBy d n = n `mod` d == 0<br />
go start ds (p:ps) = let pSq = p*p in<br />
foldr (\d -> filter (not . divisibleBy d)) [start, start + 2..pSq - 2] ds<br />
++ go (pSq + 2) (p:ds) ps<br />
</haskell><br />
<br />
Compared to the previous sieve, this keeps a list of the primes needed instead of repeatedly generating it with takeWhile.<br />
<br />
== Prime Wheels ==<br />
<br />
The idea of only testing odd numbers can be extended further. For instance, it is a useful fact that every prime number other than 2 and 3 must be of the form <math>6k+1</math> or <math>6k+5</math>. Thus, we only need to test these numbers:<br />
<br />
<haskell><br />
primes :: [Integer]<br />
primes = 2:3:primes'<br />
where<br />
1:p:candidates = [6*k+r | k <- [0..], r <- [1,5]]<br />
primes' = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) primes'<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
<br />
Here, <hask>primes'</hask> is the list of primes greater than 3 and <hask>isPrime</hask> does not test for divisibility by 2 or 3 because the <hask>candidates</hask> by construction don't have these numbers as factors. We also need to exclude 1 from the candidates and mark the next one as prime to start the recursion.<br />
<br />
Such a scheme to generate candidate numbers that avoid the first a given set of primes as divisors is called a '''prime wheel'''. Imagine that you had a wheel of circumference 6 to be rolled along the number line. With spikes positioned 1 and 5 units around the circumference, rolling the wheel will prick holes exactly in those positions on the line whose numbers are not divisible by 2 and 3.<br />
<br />
A wheel can be represented by its circumference and the spiked positions.<br />
<haskell><br />
data Wheel = Wheel Integer [Integer]<br />
</haskell><br />
We prick out numbers by rolling the wheel.<br />
<haskell><br />
roll (Wheel n rs) = [n*k+r | k <- [0..], r <- rs]<br />
</haskell><br />
The smallest wheel is the unit wheel with one spike, it will prick out every number.<br />
<haskell><br />
w0 = Wheel 1 [1]<br />
</haskell><br />
We can create a larger wheel by rolling a smaller wheel of circumference <hask>n</hask> along a rim of circumference <hask>p*n</hask> while excluding spike positions at multiples of <hask>p</hask>.<br />
<haskell><br />
nextSize (Wheel n rs) p =<br />
Wheel (p*n) [r' | k <- [0..(p-1)], r <- rs, let r' = n*k+r, r' `mod` p /= 0]<br />
</haskell><br />
Combining both, we can make wheels that prick out numbers that avoid a given list <hask>ds</hask> of divisors.<br />
<haskell><br />
mkWheel ds = foldl nextSize w0 ds<br />
</haskell><br />
<br />
Now, we can generate prime numbers with a wheel that for instance avoids all multiples of 2, 3, 5 and 7.<br />
<haskell><br />
primes :: [Integer]<br />
primes = small ++ large<br />
where<br />
1:p:candidates = roll $ mkWheel small<br />
small = [2,3,5,7]<br />
large = p : filter isPrime candidates<br />
isPrime n = all (not . divides n) $ takeWhile (\p -> p*p <= n) large<br />
divides n p = n `mod` p == 0<br />
</haskell><br />
It's a pretty big wheel with a circumference of 210 and allows us to calculate the first 10000 primes in convenient time.<br />
<br />
A fixed size wheel is fine, but how about adapting the wheel size while generating prime numbers? See the [[Research papers/Functional pearls|functional pearl]] titled [http://citeseer.ist.psu.edu/runciman97lazy.html Lazy wheel sieves and spirals of primes] for more.<br />
<br />
== Implicit Heap ==<br />
<br />
The following is a more efficient prime generator, implementing the sieve of<br />
Eratosthenes.<br />
<br />
See also the message threads [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/25270/focus=25312 Re: "no-coding" functional data structures via lazyness] for more about how merging ordered lists amounts to creating an implicit heap and [http://thread.gmane.org/gmane.comp.lang.haskell.cafe/26426/focus=26493 Re: Code and Perf. Data for Prime Finders] for an explanation of the <hask>People a</hask> structure that makes it work when tying a knot.<br />
<br />
<haskell><br />
data People a = VIP a (People a) | Crowd [a]<br />
<br />
mergeP :: Ord a => People a -> People a -> People a<br />
mergeP (VIP x xt) ys = VIP x $ mergeP xt ys<br />
mergeP (Crowd xs) (Crowd ys) = Crowd $ merge xs ys<br />
mergeP xs@(Crowd ~(x:xt)) ys@(VIP y yt) = case compare x y of<br />
LT -> VIP x $ mergeP (Crowd xt) ys<br />
EQ -> VIP x $ mergeP (Crowd xt) yt<br />
GT -> VIP y $ mergeP xs yt<br />
<br />
<br />
merge :: Ord a => [a] -> [a] -> [a]<br />
merge xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : merge xt ys<br />
EQ -> x : merge xt yt<br />
GT -> y : merge xs yt<br />
<br />
diff xs@(x:xt) ys@(y:yt) = case compare x y of<br />
LT -> x : diff xt ys<br />
EQ -> diff xt yt<br />
GT -> diff xs yt<br />
<br />
foldTree :: (a -> a -> a) -> [a] -> a<br />
foldTree f ~(x:xs) = f x . foldTree f . pairs $ xs<br />
where pairs ~(x: ~(y:ys)) = f x y : pairs ys<br />
<br />
primes, nonprimes :: [Integer]<br />
primes = 2:3:diff [5,7..] nonprimes<br />
nonprimes = serve . foldTree mergeP . map multiples $ tail primes<br />
where<br />
multiples p = vip [p*k | k <- [p,p+2..]]<br />
<br />
vip (x:xs) = VIP x $ Crowd xs<br />
serve (VIP x xs) = x:serve xs<br />
serve (Crowd xs) = xs<br />
</haskell><br />
<br />
<hask>nonprimes</hask> effectively implements a heap, exploiting lazy evaluation.<br />
<br />
== Bitwise prime sieve ==<br />
<br />
Count the number of prime below a given 'n'. Shows fast bitwise arrays,<br />
and an example of [[Template Haskell]] to defeat your enemies.<br />
<br />
<haskell><br />
{-# OPTIONS -O2 -optc-O -XBangPatterns #-}<br />
module Primes (nthPrime) where<br />
<br />
import Control.Monad.ST<br />
import Data.Array.ST<br />
import Data.Array.Base<br />
import System<br />
import Control.Monad<br />
import Data.Bits<br />
<br />
nthPrime :: Int -> Int<br />
nthPrime n = runST (sieve n)<br />
<br />
sieve n = do<br />
a <- newArray (3,n) True :: ST s (STUArray s Int Bool)<br />
let cutoff = truncate (sqrt $ fromIntegral n) + 1<br />
go a n cutoff 3 1<br />
<br />
go !a !m cutoff !n !c<br />
| n >= m = return c<br />
| otherwise = do<br />
e <- unsafeRead a n<br />
if e then<br />
if n < cutoff then<br />
let loop !j<br />
| j < m = do<br />
x <- unsafeRead a j<br />
when x $ unsafeWrite a j False<br />
loop (j+n)<br />
| otherwise = go a m cutoff (n+2) (c+1)<br />
in loop ( if n < 46340 then n * n else n `shiftL` 1)<br />
else go a m cutoff (n+2) (c+1)<br />
else go a m cutoff (n+2) c<br />
</haskell><br />
<br />
And places in a module:<br />
<br />
<haskell><br />
{-# OPTIONS -fth #-}<br />
import Primes<br />
<br />
main = print $( let x = nthPrime 10000000 in [| x |] )<br />
</haskell><br />
<br />
Run as:<br />
<br />
<haskell><br />
$ ghc --make -o primes Main.hs<br />
$ time ./primes<br />
664579<br />
./primes 0.00s user 0.01s system 228% cpu 0.003 total<br />
</haskell><br />
<br />
== Miller-Rabin Primality Test ==<br />
<haskell><br />
find2km :: Integral a => a -> (a,a)<br />
find2km n = f 0 n<br />
where <br />
f k m<br />
| r == 1 = (k,m)<br />
| otherwise = f (k+1) q<br />
where (q,r) = quotRem m 2 <br />
<br />
millerRabinPrimality :: Integer -> Integer -> Bool<br />
millerRabinPrimality n a<br />
| a <= 1 || a >= n-1 = <br />
error $ "millerRabinPrimality: a out of range (" <br />
++ show a ++ " for "++ show n ++ ")" <br />
| n < 2 = False<br />
| even n = False<br />
| b0 == 1 || b0 == n' = True<br />
| otherwise = iter (tail b)<br />
where<br />
n' = n-1<br />
(k,m) = find2km n'<br />
b0 = powMod n a m<br />
b = take (fromIntegral k) $ iterate (squareMod n) b0<br />
iter [] = False<br />
iter (x:xs)<br />
| x == 1 = False<br />
| x == n' = True<br />
| otherwise = iter xs<br />
<br />
pow' :: (Num a, Integral b) => (a -> a -> a) -> (a -> a) -> a -> b -> a<br />
pow' _ _ _ 0 = 1<br />
pow' mul sq x' n' = f x' n' 1<br />
where <br />
f x n y<br />
| n == 1 = x `mul` y<br />
| r == 0 = f x2 q y<br />
| otherwise = f x2 q (x `mul` y)<br />
where<br />
(q,r) = quotRem n 2<br />
x2 = sq x<br />
<br />
mulMod :: Integral a => a -> a -> a -> a<br />
mulMod a b c = (b * c) `mod` a<br />
squareMod :: Integral a => a -> a -> a<br />
squareMod a b = (b * b) `rem` a<br />
powMod :: Integral a => a -> a -> a -> a<br />
powMod m = pow' (mulMod m) (squareMod m)<br />
</haskell><br />
<br />
== Using IntSet for a traditional sieve ==<br />
<haskell><br />
<br />
module Sieve where<br />
import qualified Data.IntSet as I<br />
<br />
-- findNext - finds the next member of an IntSet.<br />
findNext c is | I.member c is = c<br />
| c > I.findMax is = error "Ooops. No next number in set." <br />
| otherwise = findNext (c+1) is<br />
<br />
-- mark - delete all multiples of n from n*n to the end of the set<br />
mark n is = is I.\\ (I.fromAscList (takeWhile (<=end) (map (n*) [n..])))<br />
where<br />
end = I.findMax is<br />
<br />
-- primes - gives all primes up to n <br />
primes n = worker 2 (I.fromAscList [2..n])<br />
where<br />
worker x is <br />
| (x*x) > n = is<br />
| otherwise = worker (findNext (x+1) is) (mark x is)<br />
</haskell><br />
<br />
== External links ==<br />
* http://www.cs.hmc.edu/~oneill/code/haskell-primes.zip<br />
: A collection of prime generators; the file "ONeillPrimes.hs" contains one of the fastest pure-Haskell prime generators. WARNING: Don't use the priority queue from that file for your projects: it's broken and works for primes only by a lucky chance.<br />
<br />
[[Category:Code]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30316SPOJ2009-09-28T03:03:06Z<p>Steve C: Add a System.IO solution to INTEST</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell. Solving the problems can be a interesting way to learn Haskell.<br />
<br />
Revealing answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as it encourages newcomers to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Introduction ===<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of list of Ints.<br />
<br />
The standard Haskell IO using System.IO and [Char] (or String) is slow and will make answering some problems difficult or impossible. Luckily, there is a faster IO alternative - [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring Data.ByteString].<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, Strict ByteStrings are more than adequate. There are cases where Lazy ByteStrings are better - when a file is enormous memory usage will be lower using Lazy ByteStrings because it does not read in all the file at once.<br />
You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. A useful part of Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solutions to INTEST ===<br />
<br />
[http://www.spoj.pl/problems/INTEST/ INTEST] description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Solution 0: Use System.IO getContents with [Char]/String - is very slow.<br />
<br />
<haskell><br />
import qualified Data.List as DLi<br />
<br />
main :: IO ()<br />
main = do<br />
(w1:w2:ws) <- words `fmap` getContents -- all IO in one go<br />
let n = readInt w1<br />
k = readInt w2<br />
print $ DLi.foldl' (f1 k) 0 $ map readInt $ take n ws<br />
where<br />
f1 :: Int -> Int -> Int -> Int<br />
f1 k acc ti<br />
| mod ti k == 0 = acc+1<br />
| otherwise = acc<br />
<br />
readInt :: String -> Int<br />
readInt = read<br />
</haskell><br />
<br />
Solution 1: A straightforward ByteString solution which achieves a reasonable time.<br />
<br />
<haskell><br />
import qualified Data.ByteString.Lazy.Char8 as BS<br />
<br />
main :: IO ()<br />
main = do<br />
(l:ls) <- BS.lines `fmap` BS.getContents -- all IO in one go<br />
let [n,k] = map readInt (BS.split ' ' l)<br />
print . length . filter ((== 0) . (`mod` k) . readInt) $ take n ls<br />
<br />
readInt :: BS.ByteString -> Int<br />
readInt x =<br />
case BS.readInt x of Just (i,_) -> i<br />
Nothing -> error "Unparsable Int"<br />
</haskell><br />
<br />
Solution 2: A faster version which reads the integers directly from the ByteString (thanks to Eugene on Haskell-cafe).<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import qualified Data.ByteString.Lazy as BLW<br />
import qualified Data.Word as DW<br />
<br />
main :: IO ()<br />
main = do<br />
(l, ls) <- BLW.break (==10) `fmap` BLW.getContents<br />
let [_, k] = map readInt . BLW.split 32 $ l<br />
print $ answer k ls<br />
<br />
answer :: Int -> BLW.ByteString -> Int<br />
answer k = fst . BLW.foldl' f (-1, 0)<br />
where<br />
f :: (Int,Int) -> DW.Word8 -> (Int,Int)<br />
f (!acc, !x) 10<br />
| x `mod` k==0 = (acc+1, 0)<br />
| otherwise = (acc, 0)<br />
f (!acc, !x) c = (acc, 10 * x + fromIntegral c - 48)<br />
<br />
readInt :: BLW.ByteString -> Int<br />
readInt = BLW.foldl' (\x c -> 10 * x + fromIntegral c - 48) 0<br />
</haskell><br />
<br />
Solution 3: A faster version which uses ByteString.Lazy.Char8 chunks (thanks to Don's original post on Haskell-cafe, and a speed-up patch from Eugene)<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import Data.Char<br />
import Data.Maybe<br />
import qualified Data.ByteString.Unsafe as BU<br />
import qualified Data.ByteString.Char8 as S -- 'Strict'<br />
import qualified Data.ByteString.Lazy.Char8 as L -- 'Lazy'<br />
<br />
main :: IO ()<br />
main = do<br />
ss <- L.getContents -- done with IO now.<br />
let (l,ls) = L.break (=='\n') ss<br />
-- don't need count, we're allocating lazily<br />
k = fst . fromJust . L.readInt . last . L.split ' ' $ l<br />
file = L.toChunks (L.tail ls) -- a lazy list of strict cache chunks<br />
print $ process k 0 file<br />
<br />
-- Optimised parsing of strict bytestrings representing \n separated numbers.<br />
--<br />
-- we have the file as a list of cache chunks align them on \n boundaries, and<br />
-- process each chunk separately when the next chunk is demanded, it will be<br />
-- read in.<br />
process :: Int -> Int -> [S.ByteString] -> Int<br />
process _ i [] = i<br />
process k !i (s:t:ts)<br />
| S.last s /= '\n' = process k (add k i s') ts'<br />
where<br />
(s',r) = S.breakEnd (=='\n') s<br />
(r',rs) = S.break (=='\n') t<br />
ts' = S.concat [r,r',S.singleton '\n'] : BU.unsafeTail rs : ts<br />
process k i (s: ss) = process k (add k i s) ss<br />
<br />
-- process a single cache-sized chunk of numbers, \n aligned<br />
add :: Int -> Int -> S.ByteString -> Int<br />
add k i s = fst $ S.foldl' f (i, 0) s<br />
where f (!a, !n) '\n' | mod n k == 0 = (a+1, 0)<br />
| otherwise = (a, 0)<br />
f (!a, !n) w = (a, 10*n+ord w-ord '0')<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30315SPOJ2009-09-28T02:53:45Z<p>Steve C: Replace 'Solution 1' by a simpler/faster method.</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell. Solving the problems can be a interesting way to learn Haskell.<br />
<br />
Revealing answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as it encourages newcomers to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Introduction ===<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of list of Ints.<br />
<br />
The standard Haskell IO using System.IO and [Char] (or String) is slow and will make answering some problems difficult or impossible. Luckily, there is a faster IO alternative - [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring Data.ByteString].<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, Strict ByteStrings are more than adequate. There are cases where Lazy ByteStrings are better - when a file is enormous memory usage will be lower using Lazy ByteStrings because it does not read in all the file at once.<br />
You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. A useful part of Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solutions to INTEST ===<br />
<br />
[http://www.spoj.pl/problems/INTEST/ INTEST] description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Solution 1: A straightforward method which achieves a reasonable time.<br />
<br />
<haskell><br />
import qualified Data.ByteString.Lazy.Char8 as BS<br />
<br />
main :: IO ()<br />
main = do<br />
(l:ls) <- BS.lines `fmap` BS.getContents -- all IO in one go<br />
let [n,k] = map readInt (BS.split ' ' l)<br />
print . length . filter ((== 0) . (`mod` k) . readInt) $ take n ls<br />
<br />
readInt :: BS.ByteString -> Int<br />
readInt x =<br />
case BS.readInt x of Just (i,_) -> i<br />
Nothing -> error "Unparsable Int"<br />
</haskell><br />
<br />
Solution 2: A faster version which reads the integers directly from the ByteString (thanks to Eugene on Haskell-cafe).<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import qualified Data.ByteString.Lazy as BLW<br />
import qualified Data.Word as DW<br />
<br />
main :: IO ()<br />
main = do<br />
(l, ls) <- BLW.break (==10) `fmap` BLW.getContents<br />
let [_, k] = map readInt . BLW.split 32 $ l<br />
print $ answer k ls<br />
<br />
answer :: Int -> BLW.ByteString -> Int<br />
answer k = fst . BLW.foldl' f (-1, 0)<br />
where<br />
f :: (Int,Int) -> DW.Word8 -> (Int,Int)<br />
f (!acc, !x) 10<br />
| x `mod` k==0 = (acc+1, 0)<br />
| otherwise = (acc, 0)<br />
f (!acc, !x) c = (acc, 10 * x + fromIntegral c - 48)<br />
<br />
readInt :: BLW.ByteString -> Int<br />
readInt = BLW.foldl' (\x c -> 10 * x + fromIntegral c - 48) 0<br />
</haskell><br />
<br />
Solution 3: A faster version which uses ByteString.Lazy.Char8 chunks (thanks to Don's original post on Haskell-cafe, and a speed-up patch from Eugene)<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import Data.Char<br />
import Data.Maybe<br />
import qualified Data.ByteString.Unsafe as BU<br />
import qualified Data.ByteString.Char8 as S -- 'Strict'<br />
import qualified Data.ByteString.Lazy.Char8 as L -- 'Lazy'<br />
<br />
main :: IO ()<br />
main = do<br />
ss <- L.getContents -- done with IO now.<br />
let (l,ls) = L.break (=='\n') ss<br />
-- don't need count, we're allocating lazily<br />
k = fst . fromJust . L.readInt . last . L.split ' ' $ l<br />
file = L.toChunks (L.tail ls) -- a lazy list of strict cache chunks<br />
print $ process k 0 file<br />
<br />
-- Optimised parsing of strict bytestrings representing \n separated numbers.<br />
--<br />
-- we have the file as a list of cache chunks align them on \n boundaries, and<br />
-- process each chunk separately when the next chunk is demanded, it will be<br />
-- read in.<br />
process :: Int -> Int -> [S.ByteString] -> Int<br />
process _ i [] = i<br />
process k !i (s:t:ts)<br />
| S.last s /= '\n' = process k (add k i s') ts'<br />
where<br />
(s',r) = S.breakEnd (=='\n') s<br />
(r',rs) = S.break (=='\n') t<br />
ts' = S.concat [r,r',S.singleton '\n'] : BU.unsafeTail rs : ts<br />
process k i (s: ss) = process k (add k i s) ss<br />
<br />
-- process a single cache-sized chunk of numbers, \n aligned<br />
add :: Int -> Int -> S.ByteString -> Int<br />
add k i s = fst $ S.foldl' f (i, 0) s<br />
where f (!a, !n) '\n' | mod n k == 0 = (a+1, 0)<br />
| otherwise = (a, 0)<br />
f (!a, !n) w = (a, 10*n+ord w-ord '0')<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Mailing_lists&diff=30249Mailing lists2009-09-26T07:23:59Z<p>Steve C: Link mail lists to the 'Archives' section.</p>
<hr />
<div>There are three mailing lists to discuss issues related to Haskell in<br />
general, and several additional mailing lists for more detailed<br />
discussion topics, including one for each particular implementation of<br />
Haskell.<br />
<br />
* [http://haskell.org/mailman/listinfo/haskell Subscribe to haskell@haskell.org] (announces only, low traffic)<br />
* [http://haskell.org/mailman/listinfo/haskell-cafe Subscribe to haskell-cafe@haskell.org] (very busy, daily community discussion)<br />
* [http://haskell.org/mailman/listinfo/beginners Subscribe to beginners@haskell.org] (busy, daily community discussion)<br />
* [http://haskell.org/mailman/listinfo A comprehensive list of all mailing lists hosted at haskell.org]<br />
<br />
==Mailing lists in detail==<br />
<br />
<dl><dt>[mailto:haskell@haskell.org haskell@haskell.org] ([[#Archives|archives]])</dt><br />
<dd>Announcements, discussion openers, technical questions. <br> [mailto:haskell@haskell.org haskell@haskell.org] is intended to be a low-bandwidth list, to which it is safe to subscribe without risking being buried in email. If a thread becomes longer than a handful of messages, please transfer to [mailto:haskell-cafe@haskell.org haskell-cafe@haskell.org].</dd><br />
<dt>[mailto:haskell-cafe@haskell.org haskell-cafe@haskell.org] ([[#Archives|archives]])</dt><br />
<dd>General Haskell questions; extended discussions.<br> In Simon Peyton Jones' words: "forum in which it's acceptable to ask anything, no matter how naive, and get polite replies."<br />
<dt>[mailto:beginners@haskell.org beginners@haskell.org] ([[#Archives|archives]])</dt><br />
<dd>Beginner-level, i.e., elementary, Haskell questions and discussions.<br> In the words of Benjamin L. Russell (the one who first suggested creating the mailing list and the current administrator): "Here, there is no such thing as a 'stupid question.'"</dd><br />
</dl><br />
<br />
===Mailing list tone===<br />
<br />
The division of the general list was introduced for people who want<br />
to stay in touch with what's happening in the Haskell world, but who don't want to be swamped with mail. Discussions of any kind can start on 'haskell', but should transfer to 'haskell-cafe' if they go beyond a few 'rounds'. '''Alternatively, if you are new to Haskell, then you have a choice: either haskell-cafe, or haskell-beginners.'''<br />
<br />
In practice, 'haskell' tends to be devoted mainly to announcements, 'haskell-cafe' tends to be devoted mainly to freeform discussion, and 'haskell-beginners' tends to be devoted mainly to beginner-level Haskell language discussions.<br />
<br />
The readership of the three mailing lists also varies. Whereas both 'haskell' and 'haskell-cafe' tend to be frequented by either language designers or researchers, 'haskell-beginners' tends to be frequented by beginner-level students and educators. 'Haskell-beginners' was created to address the needs of readers of 'haskell-cafe' who felt that the discussion there was either too academic, or too mathematical.<br />
<br />
When posting on 'haskell-cafe', remember:<br />
<br />
* Respect others. This is a civil discussion forum. Remember, the person on the other side of the keyboard is a person too, and is probably well-intentioned.<br />
<br />
* Try to keep discussions on-topic. Threads that have lost any relevance to the Haskell language should be moved elsewhere, including tangential or joking posts (though humor in the context of on-topic discussion is welcome.)<br />
<br />
* Think before sending. Avoid content-free posts, such as a message consisting merely of the phrase "+1." The etiquette for academic mailing list discussions is different from the etiquette for other Internet fora or for ordinary conversation. Remember that your posting will be sent to thousands of people, some of whom are very busy. Ask yourself whether your contribution adds anything of value to any of them.<br />
<br />
In the case of 'haskell-beginners', please keep in mind the following pointers when posting:<br />
<br />
* Since many readers of this mailing list are beginner-level students of Haskell, try to keep the discussion at a level that allows students of all backgrounds to participate in the discussion. I.e., when explaining difficult concepts, be careful not to assume an advanced background of the reader. For example, don't start a discussion on monads by saying: "A monad is a category theory-based data structure used to supplement pure computations with features like state, common environment or I/O." Instead, say: "A monad is a tool used in Haskell when we want to allow a program to do anything other than just return a value."<br />
<br />
* Again, since many readers of this mailing list are beginner-level students of Haskell, do not assume that readers have an advanced mathematics background, or that they know everything that may seem elementary to a computer science student. For example, if a student here asks whether the screen resolution is important in determining the precision of an algorithm to compute prime numbers by picking points randomly from a square, do not accuse the student of "polluting" the newsgroup by asking a question that "has nothing to do with Haskell." Understand that the student may not have enough mathematical or programming background to realize that screen resolution may be independent of the precision of the actual algorithm used to compute the prime numbers, which may then be represented on the screen independently of the precision of the algorithm itself. If beginner-level students are required to worry about offending somebody with a question that is too elementary every time they need an answer, they will stay beginners.<br />
<br />
===Subscription information===<br />
<br />
Haskell mailing lists are managed by [http://www.gnu.org/software/mailman/mailman.html mailman] -<br />
each list has a web interface. To subscribe, unsubscribe, or view the<br />
archives of a list visit the home page of the list, such as the [http://haskell.org/mailman/listinfo/haskell Haskell mailing list home page], the [http://haskell.org/mailman/listinfo/haskell-cafe Haskell Cafe mailing list home page], or the [http://haskell.org/mailman/listinfo/beginners Haskell-Beginners mailing list home page]. <br />
<br />
===Archiving===<br />
<br />
mail-archive.com provides an archive of all messages sent to the haskell list since March 1997. This includes messages from before the list was converted to mailman. You may search these archives: [http://www.mail-archive.com/haskell@haskell.org/ haskell archive], [http://www.mail-archive.com/haskell-cafe@haskell.org/ haskell-cafe archive], and [http://www.mail-archive.com/beginners@haskell.org/ haskell-beginners archive].<br />
<br />
MarkMail has a [http://haskell.markmail.org/ searchable archive] of all Haskell lists going back to around 2000.<br />
<br />
Also, the archives of the Haskell mailing list from September 1990 until 2006, before and after the list was converted to mailman, are [http://www.cse.unsw.edu.au/~dons/haskell-1990-2006/threads.html hosted here] (and as a [http://www.cse.unsw.edu.au/~dons/haskell-1990-2006.tar.bz2 tar file]).<br />
Related to this is the archives of [http://groups.google.com/group/comp.lang.functional/about?hl=en comp.lang.functional] going back to 1990.<br />
<br />
You may also [http://www.google.com/coop/cse?cx=015832023690232952875%3Acunmubfghzq search the mailing list] using the Google Coop Haskell Search Engine.<br />
<br />
====Archives====<br />
<br />
The following archives exist:<br />
<br />
haskell<br />
<br />
* [http://news.gmane.org/gmane.comp.lang.haskell.general gmane] ([http://dir.gmane.org/gmane.comp.lang.haskell.general info]) 2006/12-present<br />
* [http://www.haskell.org/pipermail/haskell/ mailman] 2000/10-present<br />
* [http://www.mail-archive.com/haskell@haskell.org/ mail-archive] 1997/03-present<br />
* [http://www.cse.unsw.edu.au/~dons/haskell-1990-2006/threads.html dons archive] ([http://www.cse.unsw.edu.au/~dons/haskell-1990-2006.tar.bz2 tar]) 1990/09-2006/08<br />
<br />
haskell-cafe<br />
<br />
* [http://news.gmane.org/gmane.comp.lang.haskell.cafe gmane] ([http://dir.gmane.org/gmane.comp.lang.haskell.cafe info]) 2002/04-present<br />
* [http://www.haskell.org/pipermail/haskell-cafe/ mailman] 2000/10-present<br />
* [http://www.mail-archive.com/haskell-cafe@haskell.org/ mail-archive] 1997/03-present<br />
<br />
haskell-beginners<br />
<br />
* [http://news.gmane.org/gmane.comp.lang.haskell.beginners gmane] ([http://dir.gmane.org/gmane.comp.lang.haskell.beginners info]) 2008/07-present<br />
* [http://www.haskell.org/pipermail/beginners/ mailman] 2008/07-present<br />
* [http://www.mail-archive.com/beginners@haskell.org/ mail-archive] 2008/07-present<br />
<br />
Any problems with haskell or haskell-cafe should be reported to [mailto:haskell-admin@haskell.org haskell-admin@haskell.org], and any problems with haskell-beginners should be reported to [mailto:DekuDekuplex@Yahoo.com DekuDekuplex@Yahoo.com].<br />
<br />
==More specific lists==<br />
<br />
* [http://haskell.org/mailman/listinfo A comprehensive list of all Mailing lists hosted at haskell.org]<br />
* [http://gmane.org/find.php?list=haskell Haskell lists at gmane]<br />
<br />
There are mailing lists for each implementation of Haskell,<br />
and for more detailed discussion topics. Questions, comments, and bug<br />
reports regarding a specific implementation should be sent directly<br />
to the appropriate list instead of the entire Haskell community.<br />
Separate topics such as documentation tools, the common FFI, and<br />
libraries, also have lists of their own.<br />
<br />
==Outside haskell.org==<br />
<br />
There are also Haskell related mailing lists that are not hosted at haskell.org.<br />
<br />
* [[Haskell art]]<br />
<br />
[[Category:Community]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30226SPOJ2009-09-25T03:09:23Z<p>Steve C: typo</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell. Solving the problems can be a interesting way to learn Haskell.<br />
<br />
Revealing answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as it encourages newcomers to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Introduction ===<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of list of Ints.<br />
<br />
The standard Haskell IO using System.IO and [Char] (or String) is slow and will make answering some problems difficult or impossible. Luckily, there is a faster IO alternative - [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring Data.ByteString].<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, Strict ByteStrings are more than adequate. There are cases where Lazy ByteStrings are better - when a file is enormous memory usage will be lower using Lazy ByteStrings because it does not read in all the file at once.<br />
You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. A useful part of Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solutions to INTEST ===<br />
<br />
[http://www.spoj.pl/problems/INTEST/ INTEST] description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Solution 1: A simple, reasonably efficient use of ByteString to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
Solution 2: A faster version which reads the integers directly from the ByteString (thanks to Eugene on Haskell-cafe).<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import qualified Data.ByteString.Lazy as BLW<br />
import qualified Data.Word as DW<br />
<br />
main :: IO ()<br />
main = do<br />
(l, ls) <- BLW.break (==10) `fmap` BLW.getContents<br />
let [_, k] = map readInt . BLW.split 32 $ l<br />
print $ answer k ls<br />
<br />
answer :: Int -> BLW.ByteString -> Int<br />
answer k = fst . BLW.foldl' f (-1, 0)<br />
where<br />
f :: (Int,Int) -> DW.Word8 -> (Int,Int)<br />
f (!acc, !x) 10<br />
| x `mod` k==0 = (acc+1, 0)<br />
| otherwise = (acc, 0)<br />
f (!acc, !x) c = (acc, 10 * x + fromIntegral c - 48)<br />
<br />
readInt :: BLW.ByteString -> Int<br />
readInt = BLW.foldl' (\x c -> 10 * x + fromIntegral c - 48) 0<br />
</haskell><br />
<br />
Solution 3: A faster version which uses ByteString.Lazy.Char8 chunks (thanks to Don's original post on Haskell-cafe, and a speed-up patch from Eugene)<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import Data.Char<br />
import Data.Maybe<br />
import qualified Data.ByteString.Unsafe as BU<br />
import qualified Data.ByteString.Char8 as S -- 'Strict'<br />
import qualified Data.ByteString.Lazy.Char8 as L -- 'Lazy'<br />
<br />
main :: IO ()<br />
main = do<br />
ss <- L.getContents -- done with IO now.<br />
let (l,ls) = L.break (=='\n') ss<br />
-- don't need count, we're allocating lazily<br />
k = fst . fromJust . L.readInt . last . L.split ' ' $ l<br />
file = L.toChunks (L.tail ls) -- a lazy list of strict cache chunks<br />
print $ process k 0 file<br />
<br />
-- Optimised parsing of strict bytestrings representing \n separated numbers.<br />
--<br />
-- we have the file as a list of cache chunks align them on \n boundaries, and<br />
-- process each chunk separately when the next chunk is demanded, it will be<br />
-- read in.<br />
process :: Int -> Int -> [S.ByteString] -> Int<br />
process _ i [] = i<br />
process k !i (s:t:ts)<br />
| S.last s /= '\n' = process k (add k i s') ts'<br />
where<br />
(s',r) = S.breakEnd (=='\n') s<br />
(r',rs) = S.break (=='\n') t<br />
ts' = S.concat [r,r',S.singleton '\n'] : BU.unsafeTail rs : ts<br />
process k i (s: ss) = process k (add k i s) ss<br />
<br />
-- process a single cache-sized chunk of numbers, \n aligned<br />
add :: Int -> Int -> S.ByteString -> Int<br />
add k i s = fst $ S.foldl' f (i, 0) s<br />
where f (!a, !n) '\n' | mod n k == 0 = (a+1, 0)<br />
| otherwise = (a, 0)<br />
f (!a, !n) w = (a, 10*n+ord w-ord '0')<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30221SPOJ2009-09-24T15:01:18Z<p>Steve C: typos and small corrections.</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell. Solving the problems can be a interesting way to learn Haskell.<br />
<br />
Revealing answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is encourages newcomers to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Introduction ===<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of list of Ints.<br />
<br />
The standard Haskell IO using System.IO and [Char] (or String) is slow and will make answering some problems difficult or impossible. Luckily, there is a faster IO alternative - [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring Data.ByteString].<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, Strict ByteStrings are more than adequate. There are cases where Lazy ByteStrings are better - when a file is enormous memory usage will be lower using Lazy ByteStrings because it does not read in all the file at once.<br />
You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. A useful part of Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solutions to INTEST ===<br />
<br />
[http://www.spoj.pl/problems/INTEST/ INTEST] description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Solution 1: A simple, reasonably efficient use of ByteString to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
Solution 2: A faster version which reads the integers directly from the ByteString (thanks to Eugene on Haskell-cafe).<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import qualified Data.ByteString.Lazy as BLW<br />
import qualified Data.Word as DW<br />
<br />
main :: IO ()<br />
main = do<br />
(l, ls) <- BLW.break (==10) `fmap` BLW.getContents<br />
let [_, k] = map readInt . BLW.split 32 $ l<br />
print $ answer k ls<br />
<br />
answer :: Int -> BLW.ByteString -> Int<br />
answer k = fst . BLW.foldl' f (-1, 0)<br />
where<br />
f :: (Int,Int) -> DW.Word8 -> (Int,Int)<br />
f (!acc, !x) 10<br />
| x `mod` k==0 = (acc+1, 0)<br />
| otherwise = (acc, 0)<br />
f (!acc, !x) c = (acc, 10 * x + fromIntegral c - 48)<br />
<br />
readInt :: BLW.ByteString -> Int<br />
readInt = BLW.foldl' (\x c -> 10 * x + fromIntegral c - 48) 0<br />
</haskell><br />
<br />
Solution 3: A faster version which uses ByteString.Lazy.Char8 chunks (thanks to Don's original post on Haskell-cafe, and a speed-up patch from Eugene)<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import Data.Char<br />
import Data.Maybe<br />
import qualified Data.ByteString.Unsafe as BU<br />
import qualified Data.ByteString.Char8 as S -- 'Strict'<br />
import qualified Data.ByteString.Lazy.Char8 as L -- 'Lazy'<br />
<br />
main :: IO ()<br />
main = do<br />
ss <- L.getContents -- done with IO now.<br />
let (l,ls) = L.break (=='\n') ss<br />
-- don't need count, we're allocating lazily<br />
k = fst . fromJust . L.readInt . last . L.split ' ' $ l<br />
file = L.toChunks (L.tail ls) -- a lazy list of strict cache chunks<br />
print $ process k 0 file<br />
<br />
-- Optimised parsing of strict bytestrings representing \n separated numbers.<br />
--<br />
-- we have the file as a list of cache chunks align them on \n boundaries, and<br />
-- process each chunk separately when the next chunk is demanded, it will be<br />
-- read in.<br />
process :: Int -> Int -> [S.ByteString] -> Int<br />
process _ i [] = i<br />
process k !i (s:t:ts)<br />
| S.last s /= '\n' = process k (add k i s') ts'<br />
where<br />
(s',r) = S.breakEnd (=='\n') s<br />
(r',rs) = S.break (=='\n') t<br />
ts' = S.concat [r,r',S.singleton '\n'] : BU.unsafeTail rs : ts<br />
process k i (s: ss) = process k (add k i s) ss<br />
<br />
-- process a single cache-sized chunk of numbers, \n aligned<br />
add :: Int -> Int -> S.ByteString -> Int<br />
add k i s = fst $ S.foldl' f (i, 0) s<br />
where f (!a, !n) '\n' | mod n k == 0 = (a+1, 0)<br />
| otherwise = (a, 0)<br />
f (!a, !n) w = (a, 10*n+ord w-ord '0')<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30220SPOJ2009-09-24T14:52:55Z<p>Steve C: Add 2 faster solutions to INTEST</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell.<br />
<br />
Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Introduction ===<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of list of Ints.<br />
<br />
The standard Haskell IO using System.IO and [Char] (or String) is slow and will make answering some problems difficult or impossible. Luckily, there is a faster IO alternative - [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring Data.ByteString].<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, Strict ByteStrings are more than adequate. There are cases where Lazy ByteStrings are better - when a file is enormous memory usage will be lower using Lazy ByteStrings because it does not read in all the file at once.<br />
You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. A useful part of Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solution to INTEST ===<br />
<br />
[http://www.spoj.pl/problems/INTEST/ INTEST] description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Solution 1: A simple, reasonably efficient use of ByteString to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
Solution 2: A faster version which reads the integers directly from the ByteString (thanks to Eugene on Haskell-cafe).<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import qualified Data.ByteString.Lazy as BLW<br />
import qualified Data.Word as DW<br />
<br />
main :: IO ()<br />
main = do<br />
(l, ls) <- BLW.break (==10) `fmap` BLW.getContents<br />
let [_, k] = map readInt . BLW.split 32 $ l<br />
print $ answer k ls<br />
<br />
answer :: Int -> BLW.ByteString -> Int<br />
answer k = fst . BLW.foldl' f (-1, 0)<br />
where<br />
f :: (Int,Int) -> DW.Word8 -> (Int,Int)<br />
f (!acc, !x) 10<br />
| x `mod` k==0 = (acc+1, 0)<br />
| otherwise = (acc, 0)<br />
f (!acc, !x) c = (acc, 10 * x + fromIntegral c - 48)<br />
<br />
readInt :: BLW.ByteString -> Int<br />
readInt = BLW.foldl' (\x c -> 10 * x + fromIntegral c - 48) 0<br />
</haskell><br />
<br />
Solution 3: A faster version which uses ByteString.Lazy.Char8 chunks (thanks to Don's original post on Haskell-cafe, and a speed-up patch from Eugene)<br />
<haskell><br />
{-# LANGUAGE BangPatterns #-}<br />
{-# OPTIONS_GHC -O2 -optc-O2 #-}<br />
import Data.Char<br />
import Data.Maybe<br />
import qualified Data.ByteString.Unsafe as BU<br />
import qualified Data.ByteString.Char8 as S -- 'Strict'<br />
import qualified Data.ByteString.Lazy.Char8 as L -- 'Lazy'<br />
<br />
main :: IO ()<br />
main = do<br />
ss <- L.getContents -- done with IO now.<br />
let (l,ls) = L.break (=='\n') ss<br />
-- don't need count, we're allocating lazily<br />
k = fst . fromJust . L.readInt . last . L.split ' ' $ l<br />
file = L.toChunks (L.tail ls) -- a lazy list of strict cache chunks<br />
print $ process k 0 file<br />
<br />
-- Optimised parsing of strict bytestrings representing \n separated numbers.<br />
--<br />
-- we have the file as a list of cache chunks align them on \n boundaries, and<br />
-- process each chunk separately when the next chunk is demanded, it will be<br />
-- read in.<br />
process :: Int -> Int -> [S.ByteString] -> Int<br />
process _ i [] = i<br />
process k !i (s:t:ts)<br />
| S.last s /= '\n' = process k (add k i s') ts'<br />
where<br />
(s',r) = S.breakEnd (=='\n') s<br />
(r',rs) = S.break (=='\n') t<br />
ts' = S.concat [r,r',S.singleton '\n'] : BU.unsafeTail rs : ts<br />
process k i (s: ss) = process k (add k i s) ss<br />
<br />
-- process a single cache-sized chunk of numbers, \n aligned<br />
add :: Int -> Int -> S.ByteString -> Int<br />
add k i s = fst $ S.foldl' f (i, 0) s<br />
where f (!a, !n) '\n' | mod n k == 0 = (a+1, 0)<br />
| otherwise = (a, 0)<br />
f (!a, !n) w = (a, 10*n+ord w-ord '0')<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30219SPOJ2009-09-24T14:22:07Z<p>Steve C: Rename IO/Brief Details to IO/Introduction and update the section.</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell.<br />
<br />
Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Introduction ===<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of list of Ints.<br />
<br />
The standard Haskell IO using System.IO and [Char] (or String) is slow and will make answering some problems difficult or impossible. Luckily, there is a faster IO alternative - [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bytestring Data.ByteString].<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solution to INTEST ===<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30218SPOJ2009-09-24T14:04:02Z<p>Steve C: Remove out of date paragraph about GHC 6.6.1</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell.<br />
<br />
Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
=== Brief Details ===<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of<br />
list of Ints. The code for INTEST above demonstrates one simple way<br />
to take advantage of ByteString. Using a similar technique may suffice<br />
for many problems. However, there are more advanced possibilities which<br />
could offer improvements in some cases.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solution to INTEST ===<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30217SPOJ2009-09-24T14:00:47Z<p>Steve C: Move 'Solutions to INTEST' into the IO section</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problem set archive, and accepts solutions in many languages, including Haskell.<br />
<br />
Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ. This page covers techniques for dealing with problems efficiently. It accumulates some earned wisdom about writing Haskell programs which overcome some of the technical obstacles in SPOJ. These are not spoilers, hopefully, but useful tips in general about auxiliary issues.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== I/O ==<br />
<br />
SPOJ has finally installed the [[GHC]] version 6.6.1. This makes the new,<br />
alternative I/O module available: [[Data.ByteString]]. Hopefully, it will be<br />
possible to find solutions for many more problems -- which could not be solved<br />
efficiently in the past because of mundane issues like I/O.<br />
<br />
=== Brief Details ===<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of<br />
list of Ints. The code for INTEST above demonstrates one simple way<br />
to take advantage of ByteString. Using a similar technique may suffice<br />
for many problems. However, there are more advanced possibilities which<br />
could offer improvements in some cases.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
=== Span and Break ===<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
=== Lazy ByteStrings ===<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Enormous Input: Solution to INTEST ===<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Arrays ==<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
=== Creating a mutable array ===<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
=== Generalized mutable arrays ===<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
=== Unsafe Array Access ===<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
== Garbage Collection ==<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30214SPOJ2009-09-24T13:41:20Z<p>Steve C: Update section on TEST problem</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problemset archive, and accept solutions in many languages, including Haskell. Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Getting Started: Solution to TEST ==<br />
<br />
[http://www.spoj.pl/problems/TEST/ TEST] description: the input consists of one number per line, the program should echo each number until <code>42</code> is reached, at which point the program should exit.<br />
<br />
<haskell><br />
main :: IO ()<br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
or<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact f<br />
where f = unlines . takeWhile (/="42") . words<br />
</haskell><br />
<br />
== Solution to INTEST ==<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Techniques for dealing with problems efficiently ==<br />
<br />
This section accumulates some earned wisdom about writing Haskell<br />
programs which overcome some of the technical obstacles in SPOJ.<br />
These are not spoilers, hopefully, but useful tips in general about<br />
auxiliary issues.<br />
<br />
=== I/O ===<br />
<br />
SPOJ has finally installed the [[GHC]] version 6.6.1. This makes the new,<br />
alternative I/O module available: [[Data.ByteString]]. Hopefully, it will be<br />
possible to find solutions for many more problems -- which could not be solved<br />
efficiently in the past because of mundane issues like I/O.<br />
<br />
==== Brief Details ====<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of<br />
list of Ints. The code for INTEST above demonstrates one simple way<br />
to take advantage of ByteString. Using a similar technique may suffice<br />
for many problems. However, there are more advanced possibilities which<br />
could offer improvements in some cases.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
==== Span and Break ====<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
==== Lazy ByteStrings ====<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Arrays ===<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
==== Creating a mutable array ====<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
==== Generalized mutable arrays ====<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
==== Unsafe Array Access ====<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
=== Garbage Collection ===<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30213SPOJ2009-09-24T13:35:54Z<p>Steve C: Haskell and SPOJ section</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problemset archive, and accept solutions in many languages, including Haskell. Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ.<br />
<br />
== Haskell and SPOJ ==<br />
Many [http://en.wikipedia.org/wiki/Online_judge Online Judges] support just a few languages such as C, C++, Pascal and Java. SPOJ is unique in that it supports many (currently over 30) languages, including Haskell. Most problems have a time limit which is calculated to be a comfortable time when solving the problem in C. But it can be a challenge (and sometimes impossible) to solve the problem, within this time limit, using other languages.<br />
<br />
== Solution to TEST ==<br />
<br />
TEST description: the input consists of one number per line, the<br />
program should echo each number until <code>42</code> is reached, at<br />
which point the program should exit.<br />
<br />
<haskell><br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
== Solution to INTEST ==<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Techniques for dealing with problems efficiently ==<br />
<br />
This section accumulates some earned wisdom about writing Haskell<br />
programs which overcome some of the technical obstacles in SPOJ.<br />
These are not spoilers, hopefully, but useful tips in general about<br />
auxiliary issues.<br />
<br />
=== I/O ===<br />
<br />
SPOJ has finally installed the [[GHC]] version 6.6.1. This makes the new,<br />
alternative I/O module available: [[Data.ByteString]]. Hopefully, it will be<br />
possible to find solutions for many more problems -- which could not be solved<br />
efficiently in the past because of mundane issues like I/O.<br />
<br />
==== Brief Details ====<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of<br />
list of Ints. The code for INTEST above demonstrates one simple way<br />
to take advantage of ByteString. Using a similar technique may suffice<br />
for many problems. However, there are more advanced possibilities which<br />
could offer improvements in some cases.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
==== Span and Break ====<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
==== Lazy ByteStrings ====<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Arrays ===<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
==== Creating a mutable array ====<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
==== Generalized mutable arrays ====<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
==== Unsafe Array Access ====<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
=== Garbage Collection ===<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30212SPOJ2009-09-24T13:19:10Z<p>Steve C: Add note about posting solutions.</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problemset archive, and accept solutions in many languages, including Haskell. Posting answers to questions is generally frowned upon as it enables cheating, but giving example solutions to a few of the simpler questions is OK as is allows beginners to get started on SPOJ.<br />
<br />
== Solution to TEST ==<br />
<br />
TEST description: the input consists of one number per line, the<br />
program should echo each number until <code>42</code> is reached, at<br />
which point the program should exit.<br />
<br />
<haskell><br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
== Solution to INTEST ==<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Techniques for dealing with problems efficiently ==<br />
<br />
This section accumulates some earned wisdom about writing Haskell<br />
programs which overcome some of the technical obstacles in SPOJ.<br />
These are not spoilers, hopefully, but useful tips in general about<br />
auxiliary issues.<br />
<br />
=== I/O ===<br />
<br />
SPOJ has finally installed the [[GHC]] version 6.6.1. This makes the new,<br />
alternative I/O module available: [[Data.ByteString]]. Hopefully, it will be<br />
possible to find solutions for many more problems -- which could not be solved<br />
efficiently in the past because of mundane issues like I/O.<br />
<br />
==== Brief Details ====<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of<br />
list of Ints. The code for INTEST above demonstrates one simple way<br />
to take advantage of ByteString. Using a similar technique may suffice<br />
for many problems. However, there are more advanced possibilities which<br />
could offer improvements in some cases.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
==== Span and Break ====<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
==== Lazy ByteStrings ====<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Arrays ===<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
==== Creating a mutable array ====<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
==== Generalized mutable arrays ====<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
==== Unsafe Array Access ====<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
=== Garbage Collection ===<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=SPOJ&diff=30211SPOJ2009-09-24T13:11:55Z<p>Steve C: Update SPOJ description</p>
<hr />
<div>[http://www.spoj.pl SPOJ] (Sphere Online Judge) is an automated<br />
programming contest website. It has a large problemset archive, and accept solutions in many languages, including Haskell.<br />
<br />
== Solution to TEST ==<br />
<br />
TEST description: the input consists of one number per line, the<br />
program should echo each number until <code>42</code> is reached, at<br />
which point the program should exit.<br />
<br />
<haskell><br />
main = mapM_ putStrLn . takeWhile (/="42") . lines =<< getContents<br />
</haskell><br />
<br />
== Solution to INTEST ==<br />
<br />
INTEST description: the first line of input contains two numbers: n and k.<br />
The input then consists of n lines of numbers. The output of the program<br />
should be the count of the numbers which are divisible by k.<br />
<br />
Prior to the installation of GHC 6.6.1, it was quite difficult, if not<br />
impossible, to pass this demonstration. This solution shows a simple,<br />
reasonably efficient manner of using the new ByteString library to achieve<br />
acceptable times.<br />
<br />
<haskell><br />
import Data.List (unfoldr)<br />
import qualified Data.ByteString.Char8 as SS<br />
<br />
divisibleBy :: Int -> Int -> Bool<br />
a `divisibleBy` n = a `rem` n == 0<br />
<br />
readInt1 :: SS.ByteString -> Maybe (Int, SS.ByteString)<br />
readInt1 cs = do<br />
(n, cs') <- SS.readInt cs<br />
return (n, SS.tail cs')<br />
<br />
main = do<br />
cs <- SS.getContents -- This is the only line of I/O<br />
let n:k:ns = unfoldr readInt1 cs<br />
count = length $ filter (`divisibleBy` k) ns<br />
print count<br />
</haskell><br />
<br />
== Techniques for dealing with problems efficiently ==<br />
<br />
This section accumulates some earned wisdom about writing Haskell<br />
programs which overcome some of the technical obstacles in SPOJ.<br />
These are not spoilers, hopefully, but useful tips in general about<br />
auxiliary issues.<br />
<br />
=== I/O ===<br />
<br />
SPOJ has finally installed the [[GHC]] version 6.6.1. This makes the new,<br />
alternative I/O module available: [[Data.ByteString]]. Hopefully, it will be<br />
possible to find solutions for many more problems -- which could not be solved<br />
efficiently in the past because of mundane issues like I/O.<br />
<br />
==== Brief Details ====<br />
<br />
ByteStrings come in two varieties: Strict (default), and Lazy. Strict<br />
ByteStrings, in brief, are implemented by a foreign pointer to a block<br />
of memory which is filled by low level operations. The ByteString<br />
data type points to this memory region and also contains two Ints<br />
which track relative position and length. This means that many<br />
operations on ByteStrings can re-use memory and merely manipulate the<br />
two Ints.<br />
<br />
Data.ByteString.Lazy is a layer on top which provides the ByteString<br />
API, but now the data is only read in chunks of Strict ByteStrings<br />
when necessary.<br />
<br />
Don Stewart and others have put a great deal of excellent<br />
work into the library to ensure that the high-level interface will be<br />
optimized into efficient code. However, it may behoove you to inspect<br />
the source code for yourself, which is quite accessible and available<br />
in the GHC repository under libraries/base/Data/.<br />
<br />
For SPOJ purposes, the most common operation is reading some kind of<br />
list of Ints. The code for INTEST above demonstrates one simple way<br />
to take advantage of ByteString. Using a similar technique may suffice<br />
for many problems. However, there are more advanced possibilities which<br />
could offer improvements in some cases.<br />
<br />
The module is normally imported qualified because it shares many names<br />
with standard Prelude functions. I use the prefix <code>SS</code> in<br />
my examples, for Strict byteString.<br />
<br />
==== Span and Break ====<br />
<br />
The <code>span</code> and <code>break</code> functions available<br />
from ByteString can be used to parse the occasional non-numeric value<br />
in input. For example, the following code skips a line:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as SS<br />
-- ...<br />
skipLine cs = snd $ SS.span (== '\n') cs'<br />
where (_, cs') = SS.break (== '\n') cs<br />
</haskell><br />
<br />
The nice part about <code>span</code> and <code>break</code> is that<br />
when it is found in the form <code>span (== c)</code> for all<br />
characters c, it is optimized down to a simple <code>memchr</code>.<br />
<br />
==== Lazy ByteStrings ====<br />
<br />
In general, I find that the Strict ByteStrings are more than adequate,<br />
and sometimes more complicated usage does not result in better<br />
performance. There are cases where Lazy ByteStrings do seem handle<br />
better. You can use Lazy ByteStrings as a drop-in replacement for<br />
Strict, mostly. But that does not generally confer any advantage.<br />
The best part about Lazy ByteStrings is the function called<br />
<code>toChunks</code>. This function provides an interface to the<br />
lower-level portions which actually operate by reading chunks of data<br />
into Strict ByteStrings acting as buffers. The chunks are kept in a<br />
lazy list, so each chunk is only read on-demand. However, you are<br />
getting the data as efficiently as the library deems possible.<br />
<br />
[http://www.haskell.org/pipermail/haskell-cafe/2007-June/026654.html Don Stewart Discusses Chunked ByteString Input]<br />
<br />
=== Arrays ===<br />
<br />
From time to time you may want to write an algorithm using<br />
Mutable [[Arrays]] in Haskell. While it may be tempting to jump<br />
right into arrays, you should consider that the [[Data.IntMap]]<br />
and [[Data.IntSet]] libraries are actually very fast and convenient.<br />
You may be surprised to find that they are more than sufficient.<br />
In fact, I would argue that unless you plan to use Unboxed Arrays,<br />
you should stick with these.<br />
<br />
If you are set on using a mutable array, go ahead and read the<br />
wiki page [[Arrays]] for a summary of the choices available.<br />
I am going to assume that you know the basics as imparted on<br />
that page. In particular, I'll be talking about ST and IO<br />
arrays.<br />
<br />
IO arrays are probably the easiest mutable arrays to use. They are<br />
fairly straightforward, but require you to stay in the IO monad.<br />
I made the mistake of avoiding IO arrays originally, but through some<br />
strange fluke, it turns out there are many cases where IO arrays <br />
still have the best performance.<br />
<br />
ST arrays have the advantage that they can be easily constructed in<br />
monadic code, but then returned to the functional world (without<br />
invoking voodoo like <code>unsafePerformIO</code>). However, ST<br />
itself can make the resulting type expressions confusing. In fact, it<br />
appears that you cannot type-annotate certain expressions in ST monadic<br />
code without going outside Haskell'98. <br />
<br />
Personally, I try to write my mutable array code monad-agnostic.<br />
This way I can compare IO array performance to ST array performance.<br />
However this can really make the type annotations confusing if you<br />
haven't seen it before!<br />
<br />
After seeing a few examples, though, I think anyone can pick up<br />
on the pattern. The usage isn't that difficult, once you see<br />
how the types fit together.<br />
<br />
==== Creating a mutable array ====<br />
<br />
There is a set of functions defined to work with mutable arrays<br />
separate from those that work on immutable arrays. You will want<br />
to import Data.Array.{ST,IO} depending on what you use.<br />
<br />
<haskell><br />
a <- newArray (1, 10) 0<br />
</haskell><br />
<br />
<code>newArray</code> expects bounds to be supplied just like for<br />
immutable arrays. The second argument is the initial value with which<br />
to initialize all the elements. A simple example of usage:<br />
<br />
<haskell><br />
runSTArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
The <code>runSTArray</code> function is an efficient helper function<br />
which you use when you want to create an array in ST but return an<br />
immutable array. It also happens to be a special case where <br />
<code>newArray</code> doesn't need a type-annotation.<br />
<br />
In general, you will have to supply some kind of type-annotation<br />
for <code>newArray</code>. Let's suppose you are using Unboxed<br />
Arrays and want to write the same code.<br />
<br />
<haskell><br />
runSTUArray (newArray (1, 10) 0)<br />
</haskell><br />
<br />
But now GHC will spit at you an incomprehensible error message.<br />
Well it's not so bad actually, it says,<br />
<br />
<haskell><br />
No instance for (MArray (STUArray s) t (ST s))<br />
</haskell><br />
<br />
GHC has the wrong idea about our array. We need to inform<br />
it that our array is indexed by Ints and contains Ints.<br />
<br />
<haskell><br />
runSTUArray (newArray (1,10) 0 :: ST s (STUArray s Int Int))<br />
</haskell><br />
<br />
Remember the type of <code>newArray</code>:<br />
<br />
<haskell><br />
newArray :: (MArray a e m, Ix i) => (i, i) -> e -> m (a i e)<br />
</haskell><br />
<br />
The result must be in a monad. That monad is <code>ST s</code>.<br />
<code>s</code> is the type-variable which represents the "state" which<br />
must remain contained within the monad. But the <code>STUArray</code><br />
needs to know about that "state." The type-signature expresses that<br />
the monad and the array are sharing this "state."<br />
<br />
At this point, you are pretty much ready to do anything with arrays.<br />
I'll show you one more example which uses the more general function<br />
<code>runST</code>.<br />
<br />
<haskell><br />
runST (do <br />
a <- newArray (1, 10) 1 :: ST s (STUArray s Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a<br />
return (v, a' :: Array Int Int))<br />
</haskell><br />
<br />
This code computes the first 10 Fibonacci numbers and returns a pair<br />
containing the 10th and array turned immutable. Some notes:<br />
<br />
* <code>unsafeFreeze</code> turns the array immutable in-place and therefore renders it unsafe to perform any mutations after the fact. If you don't change the array after <code>unsafeFreeze</code>, it's perfectly fine. <code>runSTArray</code> uses this internally.<br />
<br />
* Second, I had to put the type-annotation for <code>a'</code> on the final line instead of the binding line. This is one of those little annoyances about <code>ST</code> which I mentioned earlier. The reason I did not say <code>a' <- unsafeFreeze a :: ST s (Array Int Int)</code> is because GHC views this <code>s</code> as being different from the one on the first line. Clearly, GHC can't allow you to mix "state" from different places. Of course, we know that's not happening, but without lexically scoped type variables, GHC can't discern.<br />
<br />
* Don't do <code>runST $ do ...</code>. It doesn't work (at the moment). The reasons are obscure, having to do with unification and higher-rank polymorphism. Do <code>runST (do ...)</code> instead.<br />
<br />
Hopefully at this point you can see that it isn't difficult to<br />
get mutable arrays, so long as you know how the type annotations<br />
are supposed to go. IO arrays have simpler type annotations,<br />
because the type-variable <code>s</code> is not necessary.<br />
So you can write the above example as:<br />
<br />
<haskell><br />
do <br />
a <- newArray (1, 10) 1 :: IO (IOUArray Int Int)<br />
mapM_ (\ i -> do<br />
x <- readArray a (i - 2)<br />
y <- readArray a (i - 1)<br />
writeArray a i (x + y))<br />
[3 .. 10]<br />
v <- readArray a 10<br />
a' <- unsafeFreeze a :: IO (Array Int Int)<br />
return (v, a')<br />
</haskell><br />
<br />
==== Generalized mutable arrays ====<br />
<br />
Mutable arrays are instances of the typeclass <code>MArray</code>.<br />
<code>MArray a e m</code> is a multi-parameter type-class<br />
where <code>a</code> is the array-type, <code>e</code> is the<br />
element-type, and <code>m</code> is the monad.<br />
<br />
Sometimes you will want to write your code so that it can<br />
operate on any mutable array. Generally, type-inference will<br />
be more than capable. The types generated can be a bit confusing,<br />
though. Other times, you may want to create a datatype, in which<br />
case you will need to know how to specify it for MArrays.<br />
<br />
Here is an example which sums up an array of Ints:<br />
<br />
<haskell><br />
sumArray a = do<br />
(s,e) <- getBounds a<br />
let loop sum i <br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
when we ask the type of this function we get a pretty hefty answer:<br />
<br />
<haskell><br />
sumArray :: (MArray a e m, Num e, Ix i, Num i) => a i e -> m e<br />
</haskell><br />
<br />
Our function works on any mutable array type <code>a</code>, with<br />
index type <code>i</code>, element type <code>e</code>, and in monad<br />
<code>m</code>.<br />
<br />
We can create a datatype which contains a mutable array by simply<br />
satisfying the kind requirements for MArray:<br />
<br />
<haskell><br />
data IntArray a = IA (a Int Int)<br />
<br />
sumIntArray (IA a) = do<br />
(s,e) <- getBounds a<br />
let loop sum i<br />
| i > e = return sum<br />
| otherwise = do<br />
x <- readArray a i<br />
loop (sum + x) (i + 1)<br />
loop 0 s<br />
</haskell><br />
<br />
But keep in mind, you still may need to supply some kind of type<br />
annotation when you construct the datatype, so that<br />
<code>newArray</code> knows which kind of array to create.<br />
<br />
==== Unsafe Array Access ====<br />
<br />
There are two lower-level array access functions which can help you<br />
squeeze out a bit more performance: <hask>unsafeWrite</hask> and<br />
<hask>unsafeRead</hask>.<br />
<br />
Just <hask>import Data.Array.Base (unsafeWrite, unsafeRead)</hask> and use<br />
these functions in place of <hask>writeArray</hask> and <hask>readArray</hask><br />
when you feel it is safe. The first difference is that the normal array<br />
functions check array-bounds before doing anything. You can crash your<br />
program using the unsafe ones, if you access an index out-of-bounds.<br />
The second difference is that the unsafe functions do not perform any<br />
Index-arithmetic. They expect to be handed an array index that has <br />
already been converted into a zero-based, flat system. If you already<br />
use zero-based flat arrays, great. Otherwise, you may have to do a small<br />
bit of arithmetic first.<br />
<br />
=== Garbage Collection ===<br />
<br />
SPOJ has a fairly rigid framework for running programs, naturally.<br />
While it is, thankfully, possible to specify compiler options inside<br />
Haskell programs, the same is not true for RTS options. Unfortunately,<br />
sometimes there are circumstances where you would like to tweak runtime<br />
options like on the Garbage Collector. There is one technique that I<br />
have found which will allow you to tune the GC for your programs when<br />
they run on SPOJ.<br />
<br />
The gist of it is to restart the program using System.Posix tools.<br />
The major hurdle is finding the location of the program executable<br />
so that it may be supplied to the <hask>executeFile</hask> function.<br />
The following code achieves this:<br />
<br />
<haskell><br />
{-# OPTIONS -ffi #-}<br />
-- or -fglasgow-exts<br />
import System.Environment (getArgs)<br />
import System.Posix.Process (executeFile)<br />
import Foreign.C.Types (CInt)<br />
import Foreign.C.String (CString, peekCString)<br />
import Foreign (peek, alloca, peekElemOff, Ptr)<br />
<br />
main = do<br />
flags <- getArgs<br />
progname <- getFullProgName<br />
if null flags<br />
then <br />
-- Supply an "argument" so that flags will not be null.<br />
-- RTS option -A100m will increase the allocation area size<br />
-- to 100 megabytes.<br />
executeFile progname False ["r","+RTS","-A100m"] Nothing<br />
else<br />
realMain<br />
<br />
-- Now the trickier part: getProgName in GHC does not return the<br />
-- full path, for "portability" reasons. SPOJ does not run<br />
-- programs from the current directory. That means we need to<br />
-- find the full path to the program some other way.<br />
<br />
foreign import ccall unsafe "getProgArgv"<br />
getProgArgv :: Ptr CInt -> Ptr (Ptr CString) -> IO ()<br />
<br />
-- As it turns out, the C function which getProgName uses actually<br />
-- does return the full path. But then getProgName cuts it out<br />
-- before returning it. This is a version of getProgName which<br />
-- leaves the full path intact.<br />
<br />
getFullProgName :: IO String<br />
getFullProgName =<br />
alloca $ \ p_argc -><br />
alloca $ \ p_argv -> do<br />
getProgArgv p_argc p_argv<br />
argv <- peek p_argv<br />
s <- peekElemOff argv 0 >>= peekCString<br />
return s<br />
</haskell><br />
<br />
[[Category:Contests]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Exact_real_arithmetic&diff=30063Exact real arithmetic2009-09-14T03:59:58Z<p>Steve C: fix typo</p>
<hr />
<div>__TOC__<br />
<br />
== Introduction ==<br />
Exact real arithmetic is an interesting area: it is a deep connection between<br />
* numeric methods<br />
* and deep theoretic foundations of algorithms (and mathematics).<br />
Its topic: computable real numbers raise a lot of interesting questions rooted in mathematical analysis, arithmetic, but also [[Computer science#Computability theory|Computability theory]] (see numbers-as-programs approaches).<br />
<br />
Computable reals can be achieved by many approaches -- it is not one single theory.<br />
<br />
=== What it is ''not'' ===<br />
Exact real arithmetic is not the same as ''fixed'' arbitrary precision reals (see <code>Precision(n)</code> of [http://yacas.sourceforge.net/ Yacas]).<br />
<br />
Exact reals must allow us to run a huge series of computations, prescribing only the precision of the end result. Intermediate computations, and determining their necessary precision must be achieved automatically, dynamically.<br />
<br />
Maybe another problem, but it was that lead me to think on exact real arithmetic:<br />
using some Mandelbrot-plotting programs, the number of iterations must be prescribed by the user at the beginning. And when we zoom too deep into these Mandelbrot worlds, it will become ragged or smooth. Maybe solving this particular problem does not need necessarily the concept of exact real arithmetic, but it was the first time I began to think on such problems.<br />
<br />
See other numeric algorithms at [[Libraries and tools/Mathematics]].<br />
<br />
=== Why, are there reals at all, which are defined exactly, but are not computable? ===<br />
<br />
See e.g. [[Chaitin's construction]].<br />
<br />
== Theory ==<br />
<br />
* Jean Vuillemin's [http://www.inria.fr/rrrt/rr-0760.html Exact real computer arithmetic with continued fractions] is very good article on the topic itself. It can serve also as a good introductory article, too, because it presents the connections to both mathematical analysis and [[Computer science#Computability theory|Computability theory]]. It discusses several methods, and it describes some of them in more details.<br />
<br />
* [http://www.cs.bham.ac.uk/~mhe/ MartÃn EscardÃ³]'s project [http://www.dcs.ed.ac.uk/home/mhe/plume/ A Calculator for Exact Real Number Computation] -- its chosen functional language is Haskell, mainly because of its purity, lazyness, presence of lazy lists, pattern matching. MartÃn EscardÃ³ has many exact real arithetic materials also among his many [http://www.cs.bham.ac.uk/~mhe/papers/index.html papers]. <br />
<br />
* [http://users.info.unicaen.fr/~karczma/arpap/ Jerzy Karczmarczuk]'s paper with the funny title [http://users.info.unicaen.fr/~karczma/arpap/lazypi.ps.gz The Most Unreliable Technique in the World to compute pi] describes how to compute Pi as a lazy list of digits.<br />
<br />
== Implementations ==<br />
<br />
See [[Libraries and tools/Mathematics]]<br />
<br />
<br />
== Portal-like homepages ==<br />
<br />
* [http://wwwhomes.doc.ic.ac.uk/~ae/exact-computation/ Exact Computation]: There are functional programming materials too, even with downloadable Haskell source.<br />
<br />
<br />
[[Category:Mathematics]]<br />
[[Category:theoretical foundations]]</div>Steve Chttps://wiki.haskell.org/index.php?title=List_function_suggestions&diff=29904List function suggestions2009-09-02T09:08:54Z<p>Steve C: Remove references to the deprecated PackedString.</p>
<hr />
<div>This page lists proposed extensions to the Haskell list functions, whether in the [http://www.haskell.org/ghc/docs/latest/html/libraries/base/Prelude.html Prelude] or [http://www.haskell.org/ghc/docs/latest/html/libraries/base/Data-List.html Data.List].<br />
Please discuss the proposals on the Talk Page or the libraries list, and use this page to record the results of discussions.<br />
However, since the advent of [[Hackage]]DB and [[Cabal-Install]] it is preferred to provide such functionality in specialised packages, rather than extending the already large base library.<br />
<br />
== Splitting on a separator, etc ==<br />
<br />
We need these useful functions in Data.List; I'll call them 'split' (and variants) and 'replace'. These are easily implemented but everyone always reinvents them. Various versions have been proposed, but there was no consensus on which was best, e.g.<br />
<br />
* [http://www.haskell.org/pipermail/haskell-cafe/2006-July/thread.html#16559 haskell-cafe thread July 2006]<br />
* [http://www.haskell.org/pipermail/libraries/2004-July/thread.html#2342 libraries thread July 2004]<br />
<br />
Note: a lot of good points (diverging opinions!) are covered in the mailing lists, but if we include all these various cases, split* will have 9 variants! The goal is to reach some kind of reasonable consensus, specifically on naming and semantics. Even if we need pairs of functions to satisfy various usage and algebraic needs. Failing to accommodate every possible use of these functions should not be a sufficient reason to abandon the whole project.<br />
<br />
The goal is clarity/uniformity (everyone uses them widely and recognizes them) and portability (I don't have to keep reimplementing these or copying that one file UsefulMissingFunctions.hs).<br />
<br />
Note: I (Jared Updike) am working with the belief that efficiency should not be a valid argument to bar these otherwise universally useful functions from the libraries; regexes are overkill for 'split' and 'replace' for common simple situations. Let's assume people will know (or learn) when they need heavier machinery (regexes, ByteString) and will use it when efficiency is important. We can try to facilitate this by reusing any names from ByteString, etc.<br />
<br />
=== split (working name) ===<br />
<br />
First of all: Check out whether the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/split split] package provides, what you need.<br />
<br />
We need a few of these:<br />
<br />
<haskell><br />
split :: Eq a => a -> [a] -> [[a]]<br />
splitWith :: (a -> Bool) -> [a] -> [[a]]<br />
tokens :: (a -> Bool) -> [a] -> [[a]]<br />
</haskell><br />
<br />
That preserve:<br />
<br />
<haskell><br />
join sep . split sep = id<br />
</haskell><br />
<br />
See below for 'join'<br />
<br />
And some that use above split but filter to remove empty elements (but do not preserve above property). Easy enough:<br />
<br />
<haskell><br />
split' :: Eq a => a -> [a] -> [[a]]<br />
splitWith' :: (a -> Bool) -> [a] -> [[a]]<br />
tokens' :: (a -> Bool) -> [a] -> [[a]]<br />
</haskell><br />
<br />
i.e.<br />
<br />
<haskell><br />
split' sep = filter (not . null) . split sep<br />
</haskell><br />
<br />
Usage would be:<br />
<br />
<haskell><br />
tokensws = tokens' (`elem` " \f\v\t\n\r\b")<br />
<br />
tokensws "Hello there\n \n Haskellers! " ==<br />
["Hello", "there", "Haskellers!"]<br />
</haskell><br />
<br />
'''TODO: add version like python with multi-element separator'''<br />
<br />
'''TODO: give code, copy-paste from threads mentioned above'''<br />
<br />
'''TODO: list names and reasons for/against'''<br />
<br />
=== replace (working name) ===<br />
<br />
<haskell><br />
replace :: [a] -> [a] -> [a] -> [a]<br />
</haskell><br />
<br />
like Python replace:<br />
<br />
<haskell><br />
replace "the" "a" "the quick brown fox jumped over the lazy black dog"<br />
===><br />
"a quick brown fox jumped over a lazy black dog"<br />
</haskell><br />
<br />
'''TODO: give code, copy-paste from threads mentioned above'''<br />
<br />
'''TODO: list names and reasons for/against'''<br />
<br />
Implemented for instance in [http://hackage.haskell.org/packages/archive/utility-ht/0.0.1/doc/html/Data-List-HT.html#v%3Areplace utility-ht].<br />
<br />
=== join (working name) ===<br />
<br />
<haskell><br />
join :: [a] -> [[a]] -> [a]<br />
</haskell><br />
<br />
<haskell><br />
join sep = concat . intersperse sep<br />
</haskell><br />
<br />
Note: this function has been implemented as 'intercalate' in Data.List.<br />
<br />
'''TODO: copy-paste things from threads mentioned above'''<br />
<br />
'''TODO: list names and reasons for/against'''<br />
<br />
== Sorted lists ==<br />
<br />
The following are versions of standard prelude functions, but intended for sorted lists. The advantage is that they frequently reduce execution time by an O(n). The disadvantage is that the elements have to be members of Ord, and the lists have to be already sorted.<br />
<br />
<haskell><br />
-- Eliminates duplicate entries from the list, where duplication is defined<br />
-- by the 'eq' function. The last value is kept.<br />
sortedNubBy :: (a -> a -> Bool) -> [a] -> [a]<br />
sortedNubBy eq (x1 : xs@(x2 : _)) =<br />
if eq x1 x2 then sortedNubBy eq xs else x1 : sortedNubBy eq xs<br />
sortedNubBy _ xs = xs<br />
<br />
sortedNub :: (Eq a) => [a] -> [a]<br />
sortedNub = sortedNubBy (==)<br />
<br />
-- Merge two sorted lists into a new sorted list. Where elements are equal<br />
-- the element from the first list is taken first.<br />
mergeBy :: (a -> a -> Ordering) -> [a] -> [a] -> [a]<br />
mergeBy cmp xs@(x1:xs1) ys@(y1:ys1) =<br />
if cmp x1 y1 == GT<br />
then y1 : mergeBy cmp xs ys1<br />
else x1 : mergeBy cmp xs1 ys<br />
mergeBy _ [] ys = ys<br />
mergeBy _ xs [] = xs<br />
<br />
merge :: (Ord a) => [a] -> [a] -> [a]<br />
merge = mergeBy compare<br />
</haskell><br />
<br />
<hask>mergeBy</hask> is implemented in [http://hackage.haskell.org/packages/archive/utility-ht/0.0.1/doc/html/Data-List-HT.html#v%3AmergeBy utility-ht].<br />
<br />
== Generalize groupBy and friends ==<br />
<br />
In the Haskell 98 List library, <hask>groupBy</hask> assumes that its argument function defines an equivalence, and the reference definition returns sublists where each element is equivalent to the first. The following definition, comparing adjacent elements, does the same thing on equivalence relations:<br />
<haskell><br />
groupBy :: (a -> a -> Bool) -> [a] -> [[a]]<br />
groupBy rel [] = []<br />
groupBy rel (x:xs) = (x:ys) : groupBy rel zs<br />
where (ys,zs) = groupByAux x xs<br />
groupByAux x0 (x:xs) | rel x0 x = (x:ys, zs)<br />
where (ys,zs) = groupByAux x xs<br />
groupByAux y xs = ([], xs)<br />
</haskell><br />
However it is also useful on other relations, e.g.<br />
* Picking out maximal ascending sublists (runs):<br />
<haskell><br />
> groupBy (<=) [7,3,5,9,6,8,3,5,4]<br />
[[7],[3,5,9],[6,8],[3,5],[4]]<br />
</haskell><br />
* Picking out contiguous sublists from an ascending sequence:<br />
<haskell><br />
> groupBy (\a b -> a+1 == b) [1,2,3,4,6]<br />
[[1,2,3,4],[6]]<br />
</haskell><br />
* Splitting a line at the start of each word:<br />
<haskell><br />
> groupBy (\ c1 c2 -> isLetter c1 || not (isLetter c2)) "This is a line"<br />
["This ","is ","a ","line"]<br />
</haskell><br />
Since this more useful definition agrees with the Haskell 98 one on its specified domain, it should be a backwards-compatible replacement.<br />
<br />
The same applies to <hask>nubBy</hask>, and possibly <hask>deleteBy</hask>, <hask>deleteFirstsBy</hask> and <hask>intersectBy</hask> (which could have more general types to make this clear).<br />
<br />
See:<br />
* Libraries list on [http://www.haskell.org/pipermail/libraries/2007-August/008028.html Data.List.groupBy with non-transitive equality predicate]<br />
* Implementation in [http://hackage.haskell.org/packages/archive/utility-ht/0.0.1/doc/html/Data-List-HT.html#v%3AgroupBy utility-ht]<br />
<br />
== groupOn and sortOn ==<br />
<br />
Almost all uses of <hask>groupBy</hask> and <hask>sortBy</hask> actually use a specific compare function. This can (using a recent version of base) as<br />
<hask>sortBy (comparing fst)</hask><br />
or<br />
<hask>sortBy (compare `on` fst)</hask>.<br />
Since this use is so common, it might be worthwhile to add separate functions for this:<br />
<haskell><br />
sortOn :: Ord b => (a -> b) -> [a] -> [a]<br />
sortOn = sortBy . comparing<br />
</haskell><br />
The same goes for <hask>groupBy</hask><br />
<haskell><br />
groupOn :: Eq b => (a -> b) -> [a] -> [a]<br />
groupOn f = groupBy (\x y -> f x == f y)<br />
</haskell><br />
The names could be better, the idea behind 'on' comes from the 'on' function.<br />
<br />
See [http://hackage.haskell.org/packages/archive/utility-ht/0.0.1/doc/html/Data-List-Key.html utility-ht] package.<br />
<br />
== See also ==<br />
* [[Prelude extensions]]<br />
* [[Hackage]]<br />
<br />
[[Category:Proposals]]<br />
[[Category:Standard libraries]]<br />
[[Category:Idioms]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Performance/GHC&diff=29880Performance/GHC2009-08-31T16:23:12Z<p>Steve C: Delete -O3 comment - the bug cited has now been fixed.</p>
<hr />
<div>{{Performance infobox}}<br />
[[Category:Performance|GHC]] [[Category:GHC]]<br />
Please report any overly-slow GHC-compiled programs. Since [[GHC]] doesn't have any credible competition in the performance department these days it's hard to say what overly-slow means, so just use your judgement! Of course, if a GHC compiled program runs slower than the same program compiled with another Haskell compiler, then it's definitely a bug. Furthermore, if an equivalent OCaml, SML or Clean program is faster, this ''might'' be a bug.<br />
<br />
== Use optimisation ==<br />
<br />
Optimise, using <tt>-O</tt> or <tt>-O2</tt>: this is the most basic way to make your program go faster. Compilation time will be slower, especially with <tt>-O2</tt>.<br />
<br />
At present, <tt>-O2</tt> is nearly indistinguishable from <tt>-O</tt>.<br />
<br />
GHCi cannot optimise interpreted code, so when using GHCi, compile critical modules using <tt>-O</tt> or <tt>-O2</tt>, then load them into GHCi.<br />
<br />
Here is a short summary of useful compile time flags:<br />
* <tt>-O</tt>:<br />
* <tt>-O2</tt>:<br />
* <tt>-funfolding-use-threshold=16</tt>: demand more inlining.<br />
* <tt>-fexcess-precision</tt>: see [[Performance/Floating_point]]<br />
* <tt>-optc-O3</tt>: Enables a suite of optimizations in the GCC compiler. See the [http://www.openbsd.org/cgi-bin/man.cgi?query=gcc&sektion=1 gcc(1) man-page] for details. (a C-compiler option).<br />
* <tt>-optc-ffast-math</tt>: A C-compiler option which allows it to be less strict with respect to the standard when compiling IEEE 754 floating point arithmetic. Math operations will not trap if something goes wrong and math operations will assume that NaN and +- Infinity are not in arguments or results. For most practical floating point processing, this is a non-issue and enabling the flag can speed up FP arithmetic by a considerable amount. Also see the gcc(1) man-page. (a C-compiler option).<br />
<br />
Other useful flags:<br />
* <tt>-ddump-simpl > core.txt</tt>: generate core.txt file (see below).<br />
<br />
<br />
== Measuring performance ==<br />
<br />
The first thing to do is measure the performance of your program, and find out whether all the time is being spent in the garbage collector or not. Run your program with the <tt>+RTS -sstderr</tt> option:<br />
<br />
$ ./clausify 20 +RTS -sstderr<br />
42,764,972 bytes allocated in the heap<br />
6,915,348 bytes copied during GC (scavenged)<br />
360,448 bytes copied during GC (not scavenged)<br />
36,616 bytes maximum residency (7 sample(s))<br />
<br />
81 collections in generation 0 ( 0.07s)<br />
7 collections in generation 1 ( 0.00s)<br />
<br />
2 Mb total memory in use<br />
<br />
INIT time 0.00s ( 0.00s elapsed)<br />
MUT time 0.65s ( 0.94s elapsed)<br />
GC time 0.07s ( 0.06s elapsed)<br />
EXIT time 0.00s ( 0.00s elapsed)<br />
Total time 0.72s ( 1.00s elapsed)<br />
<br />
%GC time 9.7% (6.0% elapsed)<br />
<br />
Alloc rate 65,792,264 bytes per MUT second<br />
<br />
Productivity 90.3% of total user, 65.1% of total elapsed<br />
<br />
This tells you how much time is being spent running the program itself (MUT time), and how much time spent in the garbage collector (GC time). <br />
<br />
If your program is doing a lot of GC, then your first priority should be to check for [[Performance:Space Leaks|Space Leaks]] using [http://www.haskell.org/ghc/docs/latest/html/users_guide/prof-heap.html heap profiling], and then to try to reduce allocations by [http://www.haskell.org/ghc/docs/latest/html/users_guide/prof-time-options.html time and allocation profiling]. <br />
<br />
If you can't reduce the GC cost any further, then using more memory by tweaking the [http://www.haskell.org/ghc/docs/latest/html/users_guide/runtime-control.html#rts-options-gc GC options] will probably help. For example, increasing the default heap size with <tt>+RTS -H128m</tt> will reduce the number of GCs.<br />
<br />
If your program isn't doing too much GC, then you should proceed to [http://www.haskell.org/ghc/docs/latest/html/users_guide/prof-time-options.html time and allocation profiling] to see where the big hitters are.<br />
<br />
== Modules and separate compilation ==<br />
<br />
In general, splitting code across modules should not make programs less efficient. GHC does quite aggressive cross-module inlining: when you import a function f from another module M, GHC consults the "interface file" M.hi to get f's definition.<br />
<br />
For best results, ''use an explicit export list''. If you do, GHC can inline any non-exported functions that are only called once, even if they are very big. Without an explicit export list, GHC must assume that every function is exported, and hence (to avoid code bloat) is more conservative about inlining.<br />
<br />
There is one exception to the general rule that splitting code across modules does not harm performance. As mentioned above, if a non-exported non-recursive function is called exactly once, then it is inlined ''regardless of size'', because doing so does not cause code duplication. But if it's exported and is large, then its inlining is not exposed -- and even if it were it might not be inlined, because doing so duplicates its code an unknown number of times. You can change the threshold for (a) exposing and (b) using an inlining, with flags <tt>-funfolding-creation-threshold</tt> and <tt>-funfolding-use-threshold</tt> respectively.<br />
<br />
== Unboxed types ==<br />
<br />
When you are ''really'' desperate for speed, and you want to get right down to the &ldquo;raw bits.&rdquo; Please see [http://www.haskell.org/ghc/docs/latest/html/users_guide/primitives.html GHC Primitives] for some information about using unboxed types.<br />
<br />
This should be a last resort, however, since unboxed types and primitives are non-portable. Fortunately, it is usually not necessary to resort to using explicit unboxed types and primitives, because GHC's optimiser can do the work for you by inlining operations it knows about, and unboxing strict function arguments (see [[Performance/Strictness]]). Strict and unpacked constructor fields can also help a lot (see [[Performance/Data Types]]). Sometimes GHC needs a little help to generate the right code, so you might have to look at the Core output to see whether your tweaks are actually having the desired effect.<br />
<br />
One thing that can be said for using unboxed types and primitives is that you ''know'' you're writing efficient code, rather than relying on GHC's optimiser to do the right thing, and being at the mercy of changes in GHC's optimiser down the line. This may well be important to you, in which case go for it.<br />
<br />
=== An example ===<br />
<br />
Usually unboxing is not explicitly required (see the Core tutorial below), however there<br />
are circumstances where you require precise control over how your code is<br />
unboxed. The following program was at one point an entry in the<br />
[http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=ghc&lang2=ghc Great Language Shootout]. <br />
GHC did a good job unboxing the loop, but wouldn't generate the best loop. The<br />
solution was to unbox the loop function by hand, resulting in better code.<br />
<br />
The original code:<br />
<br />
loop :: Int -> Double -> Double<br />
loop d s = if d == 0 then s<br />
else loop (d-1) (s + 1/fromIntegral d)<br />
The hand-unboxed code (note that it is uglier, and harder to read):<br />
<br />
import GHC.Base<br />
import GHC.Float<br />
<br />
loop :: Int# -> Double# -> Double#<br />
loop d s = if d ==# 0# then s <br />
else loop (d -# 1#) (s +## (1.0## /## int2Double# d))<br />
<br />
GHC 6.4.1 compiles the first loop to:<br />
<br />
$wloop :: Int# -> Double# -> Double#<br />
$wloop = \ (ww_s2Ga :: Int#) (ww1_s2Ge :: Double#) -><br />
case Double# ww_s2Ga of wild_XC {<br />
__DEFAULT -><br />
case /## 1.0 (int2Double# wild_XC) of y_a2Cd { <br />
__DEFAULT -> $wloop (-# wild_XC 1) (+## ww1_s2Ge y_a2Cd)<br />
};<br />
0 -> ww1_s2Ge<br />
}<br />
<br />
And the second, unboxed loop is translated to<br />
<br />
loop1 :: Int# -> Double# -> Double#<br />
loop1 = \ (d_a1as :: Int#) (s_a1at :: Double#) -><br />
case Double# d_a1as of wild_B1 {<br />
__DEFAULT -> loop1 (-# wild_B1 1) (+## s_a1at (/## 1.0 (int2Double# wild_B1)));<br />
0 -> s_a1at<br />
}<br />
<br />
which contains 1 less case statement. The second version runs as fast as C, the<br />
first a bit slower. A similar problem was also solved with explicit unboxing in the [http://shootout.alioth.debian.org/gp4/benchmark.php?test=recursive&lang=all recursive benchmark entry].<br />
<br />
== Primops ==<br />
<br />
If you really, really need the speed, and other techniques don't seem to<br />
be helping, programming your code in raw GHC primops can sometimes do<br />
the job. As for unboxed types, you get some guarantees that your code's<br />
performance isn't subject to changes to the GHC optimisations, at the<br />
cost of more unreadable code.<br />
<br />
For example, in an imperative benchmark program a bottleneck was<br />
swapping two values. Raw primops solved the problem:<br />
<br />
swap i j a s =<br />
if i <# j then case readIntOffAddr# a i s of { (# s, x #) -><br />
case readIntOffAddr# a j s of { (# s, y #) -><br />
case writeIntOffAddr# a j x s of { s -><br />
case writeIntOffAddr# a i y s of { s -><br />
swap (i +# 1#) (j -# 1#) a s<br />
}}}}<br />
else (# s, () #)<br />
{-# INLINE swap #-}<br />
<br />
== Inlining ==<br />
<br />
GHC does a lot of inlining, which has a dramatic effect on performance.<br />
<br />
Without -O, GHC does inlining ''within'' a module, but no ''cross-module'' inlining. <br />
<br />
With -O, it does a lot of cross-module inlining. Indeed, generally<br />
speaking GHC will inline ''across'' modules just as much as it does<br />
''within'' modules, with a single large exception. If GHC sees that a<br />
function 'f' is called just once, it inlines it regardless of how big<br />
'f' is. But once 'f' is exported, GHC can never see that it's called<br />
exactly once, even if that later turns out to be the case. This<br />
inline-once optimisation is pretty important in practice. <br />
<br />
So: if you care about performance, do not export functions that are not used outside the module (i.e. use an explicit export list, and keep it as small as possible).<br />
<br />
Sometimes ''explicitly'' inlining critical chunks of code can help.<br />
The INLINE pragma can be used for this purpose; but not for recursive functions, since inlining them forever would obviously be a bad idea.<br />
<br />
If a function you want inlined contains a slow path, it can help a<br />
good deal to separate the slow path into its own function and NOINLINE<br />
it. <br />
<br />
== Looking at the Core ==<br />
<br />
GHC's compiler intermediate language can be very useful for improving<br />
the performance of your code. Core is a functional language much like a very<br />
stripped down Haskell (by design), so it's still readable, and still purely<br />
functional. The general technique is to iteratively inspect how the critical<br />
functions of your program are compiled to Core, checking that they're compiled<br />
in the most optimal manner. Sometimes GHC doesn't quite manage to unbox your<br />
function arguments, float out common subexpressions, or unfold loops ideally --<br />
but you'll only know if you read the Core.<br />
<br />
References:<br />
* [http://haskell.org/ghc/docs/papers/core.ps.gz An External Representation for the GHC Core Language], Andrew Tolmach<br />
* [http://research.microsoft.com/Users/simonpj/Papers/comp-by-trans-scp.ps.gz A transformation-based optimiser for Haskell], Simon L. Peyton Jones and Andre Santos<br />
* [http://research.microsoft.com/Users/simonpj/Papers/inlining/index.htm Secrets of the Glasgow Haskell Compiler Inliner], Simon L. Peyton Jones and Simon Marlow<br />
* [http://research.microsoft.com/users/simonpj/papers/spineless-tagless-gmachine.ps.gz Implementing lazy functional languages on stock hardware: the Spineless Tagless G-machine], Simon L. Peyton Jones<br />
<br />
== Core by example ==<br />
<br />
Here's a step-by-step guide to optimising a particular program, <br />
the [http://shootout.alioth.debian.org/gp4/benchmark.php?test=partialsums&lang=ghc&id=2 partial-sums problem] from the [http://shootout.alioth.debian.org Great Language Shootout]. We developed a number<br />
of examples on [http://haskell.org/haskellwiki/Shootout/Partial_sums Haskell shootout entry] page.<br />
<br />
Begin with the naive translation of the Clean entry (which was fairly quick):<br />
Lots of math in a tight loop.<br />
<br />
import System<br />
import Numeric<br />
<br />
main = do n <- getArgs = readIO . head<br />
let sums = loop 1 n 1 0 0 0 0 0 0 0 0 0<br />
fn (s,t) = putStrLn $ (showFFloat (Just 9) s []) ++ "\t" ++ t<br />
mapM_ (fn :: (Double, String) - IO ()) (zip sums names)<br />
<br />
names = ["(2/3)^k", "k^-0.5", "1/k(k+1)", "Flint Hills", "Cookson Hills"<br />
, "Harmonic", "Riemann Zeta", "Alternating Harmonic", "Gregory"]<br />
<br />
loop k n alt a1 a2 a3 a4 a5 a6 a7 a8 a9<br />
| k n = [ a1, a2, a3, a4, a5, a6, a7, a8, a9 ]<br />
| otherwise = loop (k+1) n (-alt)<br />
(a1 + (2/3) ** (k-1))<br />
(a2 + k ** (-0.5))<br />
(a3 + 1 / (k * (k + 1)))<br />
(a4 + 1 / (k*k*k * sin k * sin k))<br />
(a5 + 1 / (k*k*k * cos k * cos k))<br />
(a6 + 1 / k)<br />
(a7 + 1 / (k*k))<br />
(a8 + alt / k)<br />
(a9 + alt / (2 * k - 1))<br />
<br />
Compiled with '''-O2''' it runs. However, the performance is ''really'' bad.<br />
Somewhere greater than 128M heap -- in fact eventually running out of<br />
memory. A classic space leak. So look at the generated Core. <br />
<br />
=== Inspect the Core ===<br />
<br />
The best way to check the Core that GHC generates is with the<br />
'''-ddump-simpl''' flag (dump the results after code simplification, and<br />
after all optimisations are run). The result can be verbose, so pipe it into a pager.<br />
<br />
Looking for the 'loop', we find that it has been compiled to a function with<br />
the following type:<br />
<br />
$sloop_r2U6 :: GHC.Prim.Double#<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Float.Double<br />
-> GHC.Prim.Double#<br />
-> [GHC.Float.Double]<br />
<br />
Hmm, I certainly don't want boxed doubles in such a tight loop (boxed values<br />
are represented as pointers to closures on the heap, unboxed values are raw<br />
machine values).<br />
<br />
=== Strictify ===<br />
<br />
The next step then is to encourage GHC to unbox this loop, by providing some<br />
strictness annotations. So rewrite the loop like this:<br />
<br />
loop k n alt a1 a2 a3 a4 a5 a6 a7 a8 a9<br />
| () !k !n !alt !a1 !a2 !a3 !a4 !a5 !a6 !a7 !a8 !a9 !False = undefined<br />
| k > n = [ a1, a2, a3, a4, a5, a6, a7, a8, a9 ]<br />
| otherwise = loop (k+1) n (-alt)<br />
(a1 + (2/3) ** (k-1))<br />
(a2 + k ** (-0.5))<br />
(a3 + 1 / (k * (k + 1)))<br />
(a4 + 1 / (k*k*k * sin k * sin k))<br />
(a5 + 1 / (k*k*k * cos k * cos k))<br />
(a6 + 1 / k)<br />
(a7 + 1 / (k*k))<br />
(a8 + alt / k)<br />
(a9 + alt / (2 * k - 1)) where x ! y = x `seq` y<br />
<br />
Here the first guard is purely a syntactic trick to inform ghc that the<br />
arguments should be strictly evaluated. I've played a little game here, using<br />
'''!''' for '''`seq`''' is reminiscent of the new bang-pattern proposal for<br />
strictness. Let's see how this compiles. Strictifying all args GHC produces an<br />
inner loop of:<br />
<br />
$sloop_r2WS :: GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> GHC.Prim.Double#<br />
-> [GHC.Float.Double]<br />
<br />
Ah! perfect. Let's see how that runs:<br />
<br />
$ ghc Naive.hs -O2 -no-recomp<br />
$ time ./a.out 2500000<br />
3.000000000 (2/3)^k<br />
3160.817621887 k^-0.5<br />
0.999999600 1/k(k+1)<br />
30.314541510 Flint Hills<br />
42.995233998 Cookson Hills<br />
15.309017155 Harmonic<br />
1.644933667 Riemann Zeta<br />
0.693146981 Alternating Harmonic<br />
0.785398063 Gregory<br />
./a.out 2500000 4.45s user 0.02s system 99% cpu 4.482 total<br />
<br />
=== Crank up the gcc flags ===<br />
<br />
Not too bad. No space leak and quite zippy. But let's see what more can be<br />
done. First, double arithmetic usually (always?) benefits from<br />
-fexcess-precision, and cranking up the flags to gcc:<br />
<br />
paprika$ ghc Naive.hs -O2 -fexcess-precision -optc-O3 -optc-ffast-math -no-recomp<br />
paprika$ time ./a.out 2500000<br />
3.000000000 (2/3)^k<br />
3160.817621887 k^-0.5<br />
0.999999600 1/k(k+1)<br />
30.314541510 Flint Hills<br />
42.995233998 Cookson Hills<br />
15.309017155 Harmonic<br />
1.644933667 Riemann Zeta<br />
0.693146981 Alternating Harmonic<br />
0.785398063 Gregory<br />
./a.out 2500000 3.71s user 0.01s system 99% cpu 3.726 total<br />
<br />
Even better! Now, let's dive into the Core to see if there are any optimisation<br />
opportunites that GHC missed. So add '''-ddump-simpl''' and peruse the output.<br />
<br />
=== Common subexpressions ===<br />
<br />
Looking at the Core, I see firstly that some of the common subexpressions<br />
haven't been factored out:<br />
<br />
case [GHC.Float.Double] GHC.Prim./## 1.0<br />
(GHC.Prim.*## (GHC.Prim.*##<br />
(GHC.Prim.*## (GHC.Prim.*## sc10_s2VS sc10_s2VS) sc10_s2VS)<br />
(GHC.Prim.sinDouble# sc10_s2VS))<br />
(GHC.Prim.sinDouble# sc10_s2VS))<br />
<br />
Multiple calls to '''sin'''. Hmm... And similar for '''cos''' and '''k*k'''. <br />
Simon Peyton-Jones says:<br />
<br />
GHC doesn't do full CSE. It'd be a relatively easy pass for someone to<br />
add, but it can cause space leaks. And it can replace two<br />
strictly-evaluated calls with one lazy thunk:<br />
let { x = case e of ....; y = case e of .... } in ...<br />
==><br />
let { v = e; x = case v of ...; y = case v of ... } in ...<br />
<br />
Instead GHC does "opportunistic CSE". If you have<br />
let x = e in .... let y = e in ....<br />
then it'll discard the duplicate binding. But that's very weak.<br />
<br />
So it looks like we might have to float out the commmon subexpressions by hand.<br />
The inner loop now looks like:<br />
<br />
loop k n alt a1 a2 a3 a4 a5 a6 a7 a8 a9<br />
| () !k !n !alt !a1 !a2 !a3 !a4 !a5 !a6 !a7 !a8 !a9 !False = undefined<br />
| k > n = [ a1, a2, a3, a4, a5, a6, a7, a8, a9 ]<br />
| otherwise = loop (k+1) n (-alt)<br />
(a1 + (2/3) ** (k-1))<br />
(a2 + k ** (-0.5))<br />
(a3 + 1 / (k * (k + 1)))<br />
(a4 + 1 / (k3 * sk * sk))<br />
(a5 + 1 / (k3 * ck * ck))<br />
(a6 + 1 / k)<br />
(a7 + 1 / k2)<br />
(a8 + alt / k)<br />
(a9 + alt / (2 * k - 1))<br />
where sk = sin k<br />
ck = cos k<br />
k2 = k * k<br />
k3 = k2 * k<br />
x ! y = x `seq` y<br />
<br />
looking at the Core shows the sins are now allocated and shared:<br />
<br />
let a9_s2MI :: GHC.Prim.Double#<br />
a9_s2MI = GHC.Prim.sinDouble# sc10_s2Xa<br />
<br />
So the common expressions are floated out, and it now runs:<br />
<br />
paprika$ time ./a.out 2500000 <br />
3160.817621887 k^-0.5<br />
0.999999600 1/k(k+1)<br />
30.314541510 Flint Hills<br />
42.995233998 Cookson Hills<br />
15.309017155 Harmonic<br />
1.644933667 Riemann Zeta<br />
0.693146981 Alternating Harmonic<br />
0.785398063 Gregory<br />
./a.out 2500000 3.29s user 0.00s system 99% cpu 3.290 total<br />
<br />
Faster. So we gained 12% by floating out those common expressions.<br />
<br />
See also the [[GCD inlining strictness and CSE]] for another example of<br />
where CSE should be performed to improve performance.<br />
<br />
=== Strength reduction ===<br />
<br />
Finally, another trick -- manual <br />
[http://en.wikipedia.org/wiki/Strength_reduction strength reduction]. When I checked the C<br />
entry, it used an integer for the k parameter to the loop, and cast it<br />
to a double for the math each time around, so perhaps we can make it an<br />
Int parameter. Secondly, the alt parameter only has it's sign flipped<br />
each time, so perhaps we can factor out the alt / k arg (it's either 1 /<br />
k or -1 on k), saving a division. Thirdly, '''(k ** (-0.5))''' is just a<br />
slow way of doing a '''sqrt'''.<br />
<br />
The final loop looks like:<br />
<br />
loop i n alt a1 a2 a3 a4 a5 a6 a7 a8 a9<br />
| i !n !alt !a1 !a2 !a3 !a4 !a5 !a6 !a7 !a8 !a9 !False = undefined -- strict<br />
| k > n = [ a1, a2, a3, a4, a5, a6, a7, a8, a9 ]<br />
| otherwise = loop (i+1) n (-alt)<br />
(a1 + (2/3) ** (k-1))<br />
(a2 + 1 / sqrt k)<br />
(a3 + 1 / (k * (k + 1)))<br />
(a4 + 1 / (k3 * sk * sk))<br />
(a5 + 1 / (k3 * ck * ck))<br />
(a6 + dk)<br />
(a7 + 1 / k2)<br />
(a8 + alt * dk)<br />
(a9 + alt / (2 * k - 1))<br />
where k3 = k2*k; k2 = k*k; dk = 1/k; k = fromIntegral i :: Double<br />
sk = sin k; ck = cos k; x!y = x`seq`y<br />
<br />
Checking the generated C code (for another tutorial, perhaps) shows that the<br />
same C operations are generated as the C entry uses.<br />
<br />
And it runs:<br />
$ time ./i 2500000<br />
3.000000200 (2/3)^k<br />
3186.765000000 k^-0.5<br />
0.999852700 1/k(k+1)<br />
30.314493000 Flint Hills<br />
42.995068000 Cookson Hills<br />
15.403683000 Harmonic<br />
1.644725300 Riemann Zeta<br />
0.693137470 Alternating Harmonic<br />
0.785399100 Gregory<br />
./i 2500000 2.37s user 0.01s system 99% cpu 2.389 total<br />
<br />
A big speedup!<br />
<br />
This entry in fact <br />
[http://shootout.alioth.debian.org/gp4/benchmark.php?test=partialsums&lang=all runs] <br />
faster than hand optimised (and vectorised) GCC! And is only slower than<br />
optimised Fortran. Lesson: Haskell can be very, very fast.<br />
<br />
So, by carefully tweaking things, we first squished a space leak, and then<br />
gained another 45%.<br />
<br />
=== Summary ===<br />
<br />
* Manually inspect the Core that is generated<br />
* Use strictness annotations to ensure loops are unboxed<br />
* Watch out for optimisations such as CSE and strength reduction that are missed<br />
* Read the generated C for really tight loops.<br />
* Use -fexcess-precision and -optc-ffast-math for doubles<br />
<br />
== Parameters ==<br />
<br />
On x86 (possibly others), adding parameters to a loop is rather<br />
expensive, and it can be a large win to "hide" your parameters in a<br />
mutable array. (Note that this is the kind of thing quite likely to<br />
change between GHC versions, so measure before using this trick!)<br />
<br />
== Pattern matching ==<br />
<br />
On rare occasions pattern matching can give improvements in code that<br />
needs to repeatedly take apart data structures. This code:<br />
<br />
flop :: Int -> [Int] -> [Int]<br />
flop n xs = rs<br />
where (rs, ys) = fl n xs ys<br />
fl 0 xs ys = (ys, xs)<br />
fl n (x:xs) ys = fl (n-1) xs (x:ys)<br />
<br />
Can be rewritten to be faster (and more ugly) as:<br />
<br />
flop :: Int -> [Int] -> [Int]<br />
flop 2 (x1:x2:xs) = x2:x1:xs<br />
flop 3 (x1:x2:x3:xs) = x3:x2:x1:xs<br />
flop 4 (x1:x2:x3:x4:xs) = x4:x3:x2:x1:xs<br />
flop 5 (x1:x2:x3:x4:x5:xs) = x5:x4:x3:x2:x1:xs<br />
flop 6 (x1:x2:x3:x4:x5:x6:xs) = x6:x5:x4:x3:x2:x1:xs<br />
flop 7 (x1:x2:x3:x4:x5:x6:x7:xs) = x7:x6:x5:x4:x3:x2:x1:xs<br />
flop 8 (x1:x2:x3:x4:x5:x6:x7:x8:xs) = x8:x7:x6:x5:x4:x3:x2:x1:xs<br />
flop 9 (x1:x2:x3:x4:x5:x6:x7:x8:x9:xs) = x9:x8:x7:x6:x5:x4:x3:x2:x1:xs<br />
flop 10 (x1:x2:x3:x4:x5:x6:x7:x8:x9:x10:xs) = x10:x9:x8:x7:x6:x5:x4:x3:x2:x1:xs<br />
flop n xs = rs<br />
where (rs, ys) = fl n xs ys<br />
fl 0 xs ys = (ys, xs)<br />
fl n (x:xs) ys = fl (n-1) xs (x:ys)<br />
<br />
== Arrays ==<br />
<br />
If you are using array access and GHC primops, do not be too eager to<br />
use raw Addr#esses; MutableByteArray# is just as fast and frees you<br />
from memory management.<br />
<br />
== Memory allocation and arrays ==<br />
<br />
When you are allocating arrays, it may help to know a little about GHC's memory allocator. There are lots of deatils in [http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/Storage The GHC Commentary]), but here are some useful facts:<br />
<br />
* For larger objects ghc has an allocation granularity of 4k. That is it always uses a multiple of 4k bytes, which can lead to wasteage of up to 4k per array. Furthermore, a byte array has some overhead: it needs one word for the heap cell header and another for the length. So if you allocate a 4k byte array then it uses 8k. So the trick is to allocate 4k - overhead. This is what the Data.ByteString library does<br />
<br />
* GHC allocates memory from the OS in units of a "megablock", currently 1Mbyte. So if you allocate a 1Mb array, the storage manager has to allocate 1Mb + overhead, which will cause it to allocate a 2Mb megablock. The surplus will be returned to the system in the form of free blocks, but if all you do is allocate lots of 1Mb arrays, you'll waste about half the space because there's never enough contiguous free space to contain another 1Mb array. Similar problem for 512k arrays: the storage manager allocates a 1Mb block, and returns slightly less than half of it as free blocks, so each 512k allocation takes a whole new 1Mb block.<br />
<br />
== Rewrite rules ==<br />
<br />
Algebraic properties in your code might be missed by the GHC optimiser.<br />
You can use [[Playing by the rules|user-supplied rewrite rules]] to<br />
teach the compiler to optimise your code using domain-specific<br />
optimisations.</div>Steve Chttps://wiki.haskell.org/index.php?title=Performance/Strings&diff=29879Performance/Strings2009-08-31T15:59:14Z<p>Steve C: Change Data.FastPackedString do Data.ByteString</p>
<hr />
<div>{{Performance infobox}}<br />
[[Category:Performance|Strings]]<br />
==Strings==<br />
<br />
Sometimes the cost of representing strings as lists of ''Char'' can be<br />
too much. In this case, you can instead use packed strings. There are a<br />
number of options:<br />
<br />
* One of the packed string libraries, for example [http://www.cse.unsw.edu.au/~dons/fps.html Data.ByteString]<br />
* Unboxed arrays of Word8 or Char<br />
* Ptrs to foreign malloced Word8 buffers<br />
<br />
The packed string libraries have the benefit over arrays of Word8 or<br />
Char types, in that they provide the usual list-like operations.<br />
<br />
Some interesting results for Data.ByteString are documented <br />
[http://www.cse.unsw.edu.au/~dons/code/fps/README here]. In particular,<br />
it compares FPS against the existing PackedString and [Char] functions,<br />
and is used successfully with 1 terabyte strings.<br />
<br />
==Example==<br />
<br />
Pete Chown asked the question:<br />
<br />
I want to read a text file. As an example, let's use<br />
/usr/share/dict/words and try to print out the last line of the file.<br />
<br />
The python version completes in around 0.05s.<br />
<br />
=== Attempt 1 : [Char] ===<br />
<br />
<haskell><br />
import System.IO <br />
main = readFile "/usr/share/dict/words" >>= putStrLn.last.lines<br />
</haskell><br />
<br />
Run in hugs, this program took several seconds to complete. Problem:<br />
interpreted (solution, use a Haskell compiler). Compiled, the program<br />
completes in a fairly quick 0.2s. Still, we can do better.<br />
<br />
=== Attempt 2 : Data.ByteString ===<br />
<br />
Using [http://www.cse.unsw.edu.au/~dons/fps.html ByteString], we get:<br />
<br />
<haskell><br />
import qualified Data.ByteString as B <br />
import IO <br />
main = B.readFile "/usr/share/dict/words" >>= B.putStr . last . B.lines<br />
</haskell><br />
<br />
Runs in 0.063s<br />
<br />
=== Attempt 3 : No Lists ===<br />
<br />
Avoid splitting the file into lists at all, and just keep a single<br />
buffer (as a C programmer would perhaps do):<br />
<br />
<haskell><br />
import qualified Data.ByteString as P<br />
import Maybe<br />
import IO<br />
<br />
main = P.readFile "/usr/share/dict/words" >>= P.putStrLn . snd . fromJust . P.breakLast '\n'<br />
</haskell><br />
<br />
Runs in 0.013s<br />
<br />
=== Related work ===<br />
<br />
An extended tutorial on using PackedStrings/ByteStrings for high performance string manipulating code is [[Wc|here]].<br />
<br />
[http://groups.google.com/group/fa.haskell/browse_thread/thread/4133fa71ce97eb0e/fef34d1c3943bbe0#fef34d1c3943bbe0 A discussion] of the fastest way to parse a file of numbers, comparing various approaches using ByteStrings.</div>Steve Chttps://wiki.haskell.org/index.php?title=Performance/Strings&diff=29878Performance/Strings2009-08-31T15:50:37Z<p>Steve C: Remove reference to the deprecated Data.PackedString</p>
<hr />
<div>{{Performance infobox}}<br />
[[Category:Performance|Strings]]<br />
==Strings==<br />
<br />
Sometimes the cost of representing strings as lists of ''Char'' can be<br />
too much. In this case, you can instead use packed strings. There are a<br />
number of options:<br />
<br />
* One of the packed string libraries, for example [http://www.cse.unsw.edu.au/~dons/fps.html Data.ByteString]<br />
* Unboxed arrays of Word8 or Char<br />
* Ptrs to foreign malloced Word8 buffers<br />
<br />
The packed string libraries have the benefit over arrays of Word8 or<br />
Char types, in that they provide the usual list-like operations.<br />
<br />
Some interesting results for Data.ByteString are documented <br />
[http://www.cse.unsw.edu.au/~dons/code/fps/README here]. In particular,<br />
it compares FPS against the existing PackedString and [Char] functions,<br />
and is used successfully with 1 terabyte strings.<br />
<br />
==Example==<br />
<br />
Pete Chown asked the question:<br />
<br />
I want to read a text file. As an example, let's use<br />
/usr/share/dict/words and try to print out the last line of the file.<br />
<br />
The python version completes in around 0.05s.<br />
<br />
=== Attempt 1 : [Char] ===<br />
<br />
<haskell><br />
import System.IO <br />
main = readFile "/usr/share/dict/words" >>= putStrLn.last.lines<br />
</haskell><br />
<br />
Run in hugs, this program took several seconds to complete. Problem:<br />
interpreted (solution, use a Haskell compiler). Compiled, the program<br />
completes in a fairly quick 0.2s. Still, we can do better.<br />
<br />
=== Attempt 2 : Packed Strings ===<br />
<br />
Using [http://www.cse.unsw.edu.au/~dons/fps.html fast, packed strings], we get:<br />
<br />
<haskell><br />
import qualified Data.FastPackedString as P <br />
import IO <br />
main = P.readFile "/usr/share/dict/words" >>= P.putStr . last . P.lines<br />
</haskell><br />
<br />
Runs in 0.063s<br />
<br />
=== Attempt 3 : No Lists ===<br />
<br />
Avoid splitting the file into lists at all, and just keep a single<br />
buffer (as a C programmer would perhaps do):<br />
<br />
<haskell><br />
import qualified Data.ByteString as P<br />
import Maybe<br />
import IO<br />
<br />
main = P.readFile "/usr/share/dict/words" >>= P.putStrLn . snd . fromJust . P.breakLast '\n'<br />
</haskell><br />
<br />
Runs in 0.013s<br />
<br />
=== Related work ===<br />
<br />
An extended tutorial on using PackedStrings/ByteStrings for high performance string manipulating code is [[Wc|here]].<br />
<br />
[http://groups.google.com/group/fa.haskell/browse_thread/thread/4133fa71ce97eb0e/fef34d1c3943bbe0#fef34d1c3943bbe0 A discussion] of the fastest way to parse a file of numbers, comparing various approaches using ByteStrings.</div>Steve Chttps://wiki.haskell.org/index.php?title=Wc&diff=29821Wc2009-08-30T08:07:25Z<p>Steve C: Change Data.ByteString to use B.count which makes it twice as fast.</p>
<hr />
<div>[[Category:Code]]<br />
[[Category:Tutorials]]<br />
[[Category:Performance]]<br />
<br />
Some implementations of the 'wc -l' program in Haskell, with an eye to C-like<br />
performance. This illustrates the balance to be made between performance and<br />
elegance, over several increasingly fast (and more complex) examples.<br />
<br />
== Baseline ==<br />
<br />
The baseline is the C program 'wc'<br />
<br />
$ du -hsL /usr/share/dict/words<br />
912K /usr/share/dict/words<br />
<br />
$ time wc -l < /usr/share/dict/words <br />
98326<br />
wc -l < /usr/share/dict/words 0.00s user 0.00s system 27% cpu 0.015 total<br />
<br />
So the best we can probably hope to get is around 0.015s<br />
<br />
== Standard [Char] ==<br />
<br />
<haskell><br />
main :: IO ()<br />
main = print . length . lines =<< getContents<br />
</haskell><br />
<br />
$ ghc -O wc.hs<br />
$ time ./wc < /usr/share/dict/words<br />
98326<br />
./wc < /usr/share/dict/words 0.10s user 0.00s system 94% cpu 0.106 total<br />
<br />
Ok. About 10x C, as to be expected with a list representation.<br />
<br />
== Faster [Char] ==<br />
<br />
Perhaps writing our loop, rather than the duplication involved in length . lines, will improve things:<br />
<br />
<haskell><br />
main :: IO ()<br />
main = interact (count 0)<br />
where count i [] = show i<br />
count i ('\n':xs) = count (i+1) xs<br />
count i (_:xs) = count i xs<br />
</haskell><br />
<br />
$ ghc -O wc.hs<br />
$ time ./wc < /usr/share/dict/words<br />
98326./wc < /usr/share/dict/words 0.06s user 0.00s system 87% cpu 0.073 total<br />
<br />
Ok. Not too bad.<br />
<br />
== Data.ByteString ==<br />
<br />
Try to improve performance by using the<br />
[http://www.cse.unsw.edu.au/~dons/fps.html Data.ByteString] library. This uses packed byte arrays instead of heap-allocated [Char] to represent strings.<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as B<br />
<br />
main :: IO ()<br />
main = B.getContents >>= print . B.count '\n'<br />
</haskell><br />
<br />
$ time ./wc < /usr/share/dict/words<br />
98326<br />
./wc < /usr/share/dict/words 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
Much better, it is now becoming competitive with C. This (and the Data.ByteString.Lazy example below) is as fast as we'll get.<br />
<br />
== Data.ByteString.Lazy ==<br />
<br />
Or we could use the new lazy bytestring type, a lazy list of strict,<br />
L1-cache-sized chunks of bytes. This example due to Chad Scherrer:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Lazy.Char8 as L<br />
<br />
main :: IO ()<br />
main = L.getContents >>= print . L.count '\n'<br />
</haskell><br />
<br />
$ time ./a < /usr/share/dict/words<br />
98326<br />
./a < /usr/share/dict/british-english 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
== Line-by-line processing ==<br />
<br />
We can ask the bytestring library to hand us a string line at a time.<br />
<br />
<haskell><br />
import System.IO<br />
import Data.ByteString (hGetLines)<br />
main = hGetLines stdin >>= print . length<br />
</haskell><br />
<br />
$ time ./b < /usr/share/dict/british-english<br />
98326<br />
./b < /usr/share/dict/british-english 0.04s user 0.01s system 94% cpu 0.055 total<br />
<br />
Though this is a bit slower, since it needs to hang on to the lines for<br />
longer.<br />
<br />
== Ptr hacking ==<br />
<br />
ByteStrings give you access to the underlying pointers to bytes in memory,<br />
which can be used to optimise some particular code. So when the ByteString api<br />
doesn't provide what you want, you can step inside the ForeignPtr and go nuts.<br />
<br />
This example also makes use of a cpp macro to force strictness on a function,<br />
via a seq guard case.<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT4(f) f a b c d | a `seq` b `seq` c `seq` d `seq` False = undefined<br />
<br />
main = head `fmap` getArgs >>= B.readFile >>= \(B.PS x _ l) -><br />
withForeignPtr x $ \p -> go p l 0 0<br />
<br />
where go :: Ptr Word8 -> Int -> Int -> Int -> IO ()<br />
STRICT4(go)<br />
go p l n i | n >= l = print i<br />
| otherwise = do (w::Word8) <- peek (p `plusPtr` n)<br />
go p l (n+1) $ if w == 0x0a then (i+1) else i<br />
</haskell><br />
<br />
$ ghc -O -package fps -fglasgow-exts -cpp wc.hs<br />
$ time ./wc /usr/share/dict/words <br />
98326 <br />
./wc /usr/share/dict/words 0.00s user 0.01s system 67% cpu 0.018 total<br />
<br />
Ok, slower than using length . lines. Lets try some other things.<br />
<br />
== Use the FFI ==<br />
<br />
Try and step around the inefficent need to inspect each character in Haskell,<br />
by using memchr(3), the C function to find each newline for us.<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import Foreign.C.Types<br />
<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT4(f) f a b c d | a `seq` b `seq` c `seq` d `seq` False = undefined<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x _ l) -> withForeignPtr x $ \p -> go p l 0 0<br />
<br />
where<br />
go :: Ptr Word8 -> Int -> Int -> Int -> IO ()<br />
STRICT4(go)<br />
go p l n i<br />
| n >= l = print i<br />
| otherwise = do<br />
let p' = p `plusPtr` n<br />
q = memchr p' 0x0a (fromIntegral (l-n))<br />
if q == nullPtr<br />
then print i<br />
else do let k = q `minusPtr` p'<br />
go p l (n+k+1) (i+1)<br />
<br />
foreign import ccall unsafe "string.h memchr" memchr<br />
:: Ptr Word8 -> CInt -> CSize -> Ptr Word8<br />
</haskell><br />
<br />
$ time ./wc /usr/share/dict/words<br />
98326 <br />
./wc /usr/share/dict/words 0.00s user 0.00s system 47% cpu 0.017 total<br />
<br />
Slowly inching forwards.<br />
<br />
== Read the Core ==<br />
<br />
While we're here, we can check whether the strictness on the 'go' function<br />
makes any difference, by reading the GHC Core:<br />
<br />
$ ghc -O -package fps -cpp -ffi wc.hs -ddump-simpl | less<br />
<br />
Search for the 'go' function:<br />
<br />
Main.$wgo :: GHC.Prim.Addr#<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.IOBase.IO ()<br />
<br />
And without the strictness:<br />
<br />
Main.$wgo :: GHC.Ptr.Ptr GHC.Word.Word8<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.Base.Int<br />
-> GHC.IOBase.IO ()<br />
<br />
So GHC is helpfully unboxing the Ptr Word8 into a raw machine Addr#.<br />
<br />
== Avoid some code ==<br />
<br />
The guard that checks the length is unneeded, since memchr takes a length<br />
argument anyway. It also calculates the next pointer for us, so avoid<br />
recalculating it. (Note that this is equivalent to using the 'count'<br />
function, which has the same implementation).<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import Foreign.C.Types<br />
<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT3(f) f a b c | a `seq` b `seq` c `seq` False = undefined<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x s l) -> withForeignPtr x $ \p -> <br />
go (p `plusPtr` s) (fromIntegral l) 0<br />
where<br />
go :: Ptr Word8 -> CSize -> Int -> IO ()<br />
STRICT3(go)<br />
go p l i = do<br />
let q = memchr p 0x0a l<br />
if q == nullPtr<br />
then print i<br />
else do let k = fromIntegral $ q `minusPtr` p<br />
go (q `plusPtr` 1) (l-k) (i+1)<br />
<br />
foreign import ccall unsafe "string.h memchr" memchr<br />
:: Ptr Word8 -> CInt -> CSize -> Ptr Word8<br />
</haskell><br />
<br />
Checking the Core, 'go' is now:<br />
<br />
Main.$wgo :: GHC.Prim.Addr#<br />
-> GHC.Prim.Word#<br />
-> GHC.Prim.Int#<br />
-> GHC.IOBase.IO ()<br />
<br />
The code is certainly a bit simpler, at least.<br />
<br />
$ ghc -O -package fps -cpp -ffi wc.hs<br />
$ time ./wc /usr/share/dict/words<br />
98326<br />
./wc /usr/share/dict/words 0.00s user 0.01s system 70% cpu 0.017 total<br />
<br />
But we can't seem to squeeze any more out, at least on data this size.<br />
<br />
== Going via C ==<br />
<br />
We reach a point where I can't think of any more tricks, so we can always code<br />
up a little C and call into that, for this tight loop. Sometimes we just have<br />
to do this, and that's what the ffi is for, after all.<br />
<br />
<haskell><br />
-- wc.hs<br />
<br />
import Foreign<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x _ l) -> withForeignPtr x $ \p -> print (c_wc p l)<br />
<br />
foreign import ccall unsafe "wc.h wc" c_wc :: Ptr Word8 -> Int -> Int<br />
<br />
-- wc_c.c<br />
int wc(char *p, int len) {<br />
int c;<br />
for (c = 0; len--; ++p)<br />
if (*p == '\n')<br />
++c;<br />
return c;<br />
}<br />
<br />
-- wc.h<br />
int wc(char *p, int len);<br />
</haskell><br />
<br />
$ gcc -O3 -c wc_c.c<br />
$ ghc -O -package fps wc.hs -o wc -fglasgow-exts wc_c.o<br />
$ time ./wc /usr/share/dict/words<br />
98326<br />
./wc /usr/share/dict/words 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
And we are done. Note that the tight C loop didn't give us anything in the end over<br />
the naive ByteString code, which is a very satisfying result.</div>Steve Chttps://wiki.haskell.org/index.php?title=Wc&diff=29816Wc2009-08-30T07:18:05Z<p>Steve C: Remove Data.PackedString section - its deprecated and has been replaced by Data.ByteString</p>
<hr />
<div>[[Category:Code]]<br />
[[Category:Tutorials]]<br />
[[Category:Performance]]<br />
<br />
Some implementations of the 'wc -l' program in Haskell, with an eye to C-like<br />
performance. This illustrates the balance to be made between performance and<br />
elegance, over several increasingly fast (and more complex) examples.<br />
<br />
== Baseline ==<br />
<br />
The baseline is the C program 'wc'<br />
<br />
$ du -hsL /usr/share/dict/words<br />
912K /usr/share/dict/words<br />
<br />
$ time wc -l < /usr/share/dict/words <br />
98326<br />
wc -l < /usr/share/dict/words 0.00s user 0.00s system 27% cpu 0.015 total<br />
<br />
So the best we can probably hope to get is around 0.015s<br />
<br />
== Standard [Char] ==<br />
<br />
<haskell><br />
main = print . length . lines =<< getContents<br />
</haskell><br />
<br />
$ ghc -O wc.hs<br />
$ time ./wc < /usr/share/dict/words<br />
98326<br />
./wc < /usr/share/dict/words 0.10s user 0.00s system 94% cpu 0.106 total<br />
<br />
Ok. About 10x C, as to be expected with a list representation.<br />
<br />
== Faster [Char] ==<br />
<br />
Perhaps writing our loop, rather than the duplication involved in length . lines, will improve things:<br />
<br />
<haskell><br />
main = interact (count 0)<br />
where count i [] = show i<br />
count i ('\n':xs) = count (i+1) xs<br />
count i (_:xs) = count i xs<br />
</haskell><br />
<br />
$ ghc -O wc.hs<br />
$ time ./wc < /usr/share/dict/words<br />
98326./wc < /usr/share/dict/words 0.06s user 0.00s system 87% cpu 0.073 total<br />
<br />
Ok. Not too bad.<br />
<br />
== Data.ByteString ==<br />
<br />
Try to improve performance a bit by using the new<br />
[http://www.cse.unsw.edu.au/~dons/fps.html Data.ByteString] library. This uses packed byte arrays instead of heap-allocated [Char] to represent strings.<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as B <br />
<br />
main = print . length . B.lines =<< B.getContents<br />
</haskell><br />
<br />
$ time ./wc < /usr/share/dict/words<br />
98326<br />
./wc < /usr/share/dict/words 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
Excellent! Definitely competitive with C. This is in fact as fast as<br />
we'll get. It helps that the ByteString library is fusing length .<br />
lines, so that the intermediate list is not constructed.<br />
<br />
== Data.ByteString.Lazy ==<br />
<br />
Or we could use the new lazy bytestring type, a lazy list of strict,<br />
L1-cache-sized chunks of bytes. This example due to Chad Scherrer:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Lazy.Char8 as L<br />
main = L.getContents >>= print . L.count '\n'<br />
</haskell><br />
<br />
$ time ./a < /usr/share/dict/words<br />
98326<br />
./a < /usr/share/dict/british-english 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
== Line-by-line processing ==<br />
<br />
We can ask the bytestring library to hand us a string line at a time.<br />
<br />
<haskell><br />
import System.IO<br />
import Data.ByteString (hGetLines)<br />
main = hGetLines stdin >>= print . length<br />
</haskell><br />
<br />
$ time ./b < /usr/share/dict/british-english<br />
98326<br />
./b < /usr/share/dict/british-english 0.04s user 0.01s system 94% cpu 0.055 total<br />
<br />
Though this is a bit slower, since it needs to hang on to the lines for<br />
longer.<br />
<br />
== Ptr hacking ==<br />
<br />
ByteStrings give you access to the underlying pointers to bytes in memory,<br />
which can be used to optimise some particular code. So when the ByteString api<br />
doesn't provide what you want, you can step inside the ForeignPtr and go nuts.<br />
<br />
This example also makes use of a cpp macro to force strictness on a function,<br />
via a seq guard case.<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT4(f) f a b c d | a `seq` b `seq` c `seq` d `seq` False = undefined<br />
<br />
main = head `fmap` getArgs >>= B.readFile >>= \(B.PS x _ l) -><br />
withForeignPtr x $ \p -> go p l 0 0<br />
<br />
where go :: Ptr Word8 -> Int -> Int -> Int -> IO ()<br />
STRICT4(go)<br />
go p l n i | n >= l = print i<br />
| otherwise = do (w::Word8) <- peek (p `plusPtr` n)<br />
go p l (n+1) $ if w == 0x0a then (i+1) else i<br />
</haskell><br />
<br />
$ ghc -O -package fps -fglasgow-exts -cpp wc.hs<br />
$ time ./wc /usr/share/dict/words <br />
98326 <br />
./wc /usr/share/dict/words 0.00s user 0.01s system 67% cpu 0.018 total<br />
<br />
Ok, slower than using length . lines. Lets try some other things.<br />
<br />
== Use the FFI ==<br />
<br />
Try and step around the inefficent need to inspect each character in Haskell,<br />
by using memchr(3), the C function to find each newline for us.<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import Foreign.C.Types<br />
<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT4(f) f a b c d | a `seq` b `seq` c `seq` d `seq` False = undefined<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x _ l) -> withForeignPtr x $ \p -> go p l 0 0<br />
<br />
where<br />
go :: Ptr Word8 -> Int -> Int -> Int -> IO ()<br />
STRICT4(go)<br />
go p l n i<br />
| n >= l = print i<br />
| otherwise = do<br />
let p' = p `plusPtr` n<br />
q = memchr p' 0x0a (fromIntegral (l-n))<br />
if q == nullPtr<br />
then print i<br />
else do let k = q `minusPtr` p'<br />
go p l (n+k+1) (i+1)<br />
<br />
foreign import ccall unsafe "string.h memchr" memchr<br />
:: Ptr Word8 -> CInt -> CSize -> Ptr Word8<br />
</haskell><br />
<br />
$ time ./wc /usr/share/dict/words<br />
98326 <br />
./wc /usr/share/dict/words 0.00s user 0.00s system 47% cpu 0.017 total<br />
<br />
Slowly inching forwards.<br />
<br />
== Read the Core ==<br />
<br />
While we're here, we can check whether the strictness on the 'go' function<br />
makes any difference, by reading the GHC Core:<br />
<br />
$ ghc -O -package fps -cpp -ffi wc.hs -ddump-simpl | less<br />
<br />
Search for the 'go' function:<br />
<br />
Main.$wgo :: GHC.Prim.Addr#<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.IOBase.IO ()<br />
<br />
And without the strictness:<br />
<br />
Main.$wgo :: GHC.Ptr.Ptr GHC.Word.Word8<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.Base.Int<br />
-> GHC.IOBase.IO ()<br />
<br />
So GHC is helpfully unboxing the Ptr Word8 into a raw machine Addr#.<br />
<br />
== Avoid some code ==<br />
<br />
The guard that checks the length is unneeded, since memchr takes a length<br />
argument anyway. It also calculates the next pointer for us, so avoid<br />
recalculating it. (Note that this is equivalent to using the 'count'<br />
function, which has the same implementation).<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import Foreign.C.Types<br />
<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT3(f) f a b c | a `seq` b `seq` c `seq` False = undefined<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x s l) -> withForeignPtr x $ \p -> <br />
go (p `plusPtr` s) (fromIntegral l) 0<br />
where<br />
go :: Ptr Word8 -> CSize -> Int -> IO ()<br />
STRICT3(go)<br />
go p l i = do<br />
let q = memchr p 0x0a l<br />
if q == nullPtr<br />
then print i<br />
else do let k = fromIntegral $ q `minusPtr` p<br />
go (q `plusPtr` 1) (l-k) (i+1)<br />
<br />
foreign import ccall unsafe "string.h memchr" memchr<br />
:: Ptr Word8 -> CInt -> CSize -> Ptr Word8<br />
</haskell><br />
<br />
Checking the Core, 'go' is now:<br />
<br />
Main.$wgo :: GHC.Prim.Addr#<br />
-> GHC.Prim.Word#<br />
-> GHC.Prim.Int#<br />
-> GHC.IOBase.IO ()<br />
<br />
The code is certainly a bit simpler, at least.<br />
<br />
$ ghc -O -package fps -cpp -ffi wc.hs<br />
$ time ./wc /usr/share/dict/words<br />
98326<br />
./wc /usr/share/dict/words 0.00s user 0.01s system 70% cpu 0.017 total<br />
<br />
But we can't seem to squeeze any more out, at least on data this size.<br />
<br />
== Going via C ==<br />
<br />
We reach a point where I can't think of any more tricks, so we can always code<br />
up a little C and call into that, for this tight loop. Sometimes we just have<br />
to do this, and that's what the ffi is for, after all.<br />
<br />
<haskell><br />
-- wc.hs<br />
<br />
import Foreign<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x _ l) -> withForeignPtr x $ \p -> print (c_wc p l)<br />
<br />
foreign import ccall unsafe "wc.h wc" c_wc :: Ptr Word8 -> Int -> Int<br />
<br />
-- wc_c.c<br />
int wc(char *p, int len) {<br />
int c;<br />
for (c = 0; len--; ++p)<br />
if (*p == '\n')<br />
++c;<br />
return c;<br />
}<br />
<br />
-- wc.h<br />
int wc(char *p, int len);<br />
</haskell><br />
<br />
$ gcc -O3 -c wc_c.c<br />
$ ghc -O -package fps wc.hs -o wc -fglasgow-exts wc_c.o<br />
$ time ./wc /usr/share/dict/words<br />
98326<br />
./wc /usr/share/dict/words 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
And we are done. Note that the tight C loop didn't give us anything in the end over<br />
the naive ByteString code, which is a very satisfying result.</div>Steve Chttps://wiki.haskell.org/index.php?title=Wc&diff=29815Wc2009-08-30T06:36:48Z<p>Steve C: Add -L option to du - needed on Fedora 11</p>
<hr />
<div>[[Category:Code]]<br />
[[Category:Tutorials]]<br />
[[Category:Performance]]<br />
<br />
Some implementations of the 'wc -l' program in Haskell, with an eye to C-like<br />
performance. This illustrates the balance to be made between performance and<br />
elegance, over several increasingly fast (and more complex) examples.<br />
<br />
== Baseline ==<br />
<br />
The baseline is the C program 'wc'<br />
<br />
$ du -hsL /usr/share/dict/words<br />
912K /usr/share/dict/words<br />
<br />
$ time wc -l < /usr/share/dict/words <br />
98326<br />
wc -l < /usr/share/dict/words 0.00s user 0.00s system 27% cpu 0.015 total<br />
<br />
So the best we can probably hope to get is around 0.015s<br />
<br />
== Standard [Char] ==<br />
<br />
<haskell><br />
main = print . length . lines =<< getContents<br />
</haskell><br />
<br />
$ ghc -O wc.hs<br />
$ time ./wc < /usr/share/dict/words<br />
98326<br />
./wc < /usr/share/dict/words 0.10s user 0.00s system 94% cpu 0.106 total<br />
<br />
Ok. About 10x C, as to be expected with a list representation.<br />
<br />
== Faster [Char] ==<br />
<br />
Perhaps writing our loop, rather than the duplication involved in length . lines, will improve things:<br />
<br />
<haskell><br />
main = interact (count 0)<br />
where count i [] = show i<br />
count i ('\n':xs) = count (i+1) xs<br />
count i (_:xs) = count i xs<br />
</haskell><br />
<br />
$ ghc -O wc.hs<br />
$ time ./wc < /usr/share/dict/words<br />
98326./wc < /usr/share/dict/words 0.06s user 0.00s system 87% cpu 0.073 total<br />
<br />
Ok. Not too bad.<br />
<br />
== Data.PackedString ==<br />
<br />
Ok, lets try the old Data.PackedString library.<br />
<br />
My first attempt to directly use hGet failed, as hGet has a stack<br />
overflow for files > ~500k.<br />
<br />
<haskell><br />
import Data.PackedString<br />
import System.IO<br />
<br />
main = print . length . linesPS =<< getit "/usr/share/dict/words"<br />
where<br />
getit f = do<br />
h <- openFile f ReadMode<br />
s <- hGetContents h<br />
length s `seq` return ()<br />
hClose h<br />
return $! packString s<br />
</haskell><br />
<br />
$ time ./wc<br />
98326<br />
./wc < /usr/share/dict/words 0.14s user 0.02s system 95% cpu 0.168 total<br />
<br />
Hmm. Worse than [Char]. Unfortunately, this is not uncommon with Data.PackedString.<br />
<br />
== Data.ByteString ==<br />
<br />
Try to improve performance a bit by using the new<br />
[http://www.cse.unsw.edu.au/~dons/fps.html Data.ByteString] library, a<br />
replacement for Data.PackedString. This uses packed byte arrays instead<br />
of heap-allocated [Char] to represent strings.<br />
<br />
<haskell><br />
import qualified Data.ByteString.Char8 as B <br />
<br />
main = print . length . B.lines =<< B.getContents<br />
</haskell><br />
<br />
$ time ./wc < /usr/share/dict/words<br />
98326<br />
./wc < /usr/share/dict/words 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
Excellent! Definitely competitive with C. This is in fact as fast as<br />
we'll get. It helps that the ByteString library is fusing length .<br />
lines, so that the intermediate list is not constructed.<br />
<br />
== Data.ByteString.Lazy ==<br />
<br />
Or we could use the new lazy bytestring type, a lazy list of strict,<br />
L1-cache-sized chunks of bytes. This example due to Chad Scherrer:<br />
<br />
<haskell><br />
import qualified Data.ByteString.Lazy.Char8 as L<br />
main = L.getContents >>= print . L.count '\n'<br />
</haskell><br />
<br />
$ time ./a < /usr/share/dict/words<br />
98326<br />
./a < /usr/share/dict/british-english 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
== Line-by-line processing ==<br />
<br />
We can ask the bytestring library to hand us a string line at a time.<br />
<br />
<haskell><br />
import System.IO<br />
import Data.ByteString (hGetLines)<br />
main = hGetLines stdin >>= print . length<br />
</haskell><br />
<br />
$ time ./b < /usr/share/dict/british-english<br />
98326<br />
./b < /usr/share/dict/british-english 0.04s user 0.01s system 94% cpu 0.055 total<br />
<br />
Though this is a bit slower, since it needs to hang on to the lines for<br />
longer.<br />
<br />
== Ptr hacking ==<br />
<br />
ByteStrings give you access to the underlying pointers to bytes in memory,<br />
which can be used to optimise some particular code. So when the ByteString api<br />
doesn't provide what you want, you can step inside the ForeignPtr and go nuts.<br />
<br />
This example also makes use of a cpp macro to force strictness on a function,<br />
via a seq guard case.<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT4(f) f a b c d | a `seq` b `seq` c `seq` d `seq` False = undefined<br />
<br />
main = head `fmap` getArgs >>= B.readFile >>= \(B.PS x _ l) -><br />
withForeignPtr x $ \p -> go p l 0 0<br />
<br />
where go :: Ptr Word8 -> Int -> Int -> Int -> IO ()<br />
STRICT4(go)<br />
go p l n i | n >= l = print i<br />
| otherwise = do (w::Word8) <- peek (p `plusPtr` n)<br />
go p l (n+1) $ if w == 0x0a then (i+1) else i<br />
</haskell><br />
<br />
$ ghc -O -package fps -fglasgow-exts -cpp wc.hs<br />
$ time ./wc /usr/share/dict/words <br />
98326 <br />
./wc /usr/share/dict/words 0.00s user 0.01s system 67% cpu 0.018 total<br />
<br />
Ok, slower than using length . lines. Lets try some other things.<br />
<br />
== Use the FFI ==<br />
<br />
Try and step around the inefficent need to inspect each character in Haskell,<br />
by using memchr(3), the C function to find each newline for us.<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import Foreign.C.Types<br />
<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT4(f) f a b c d | a `seq` b `seq` c `seq` d `seq` False = undefined<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x _ l) -> withForeignPtr x $ \p -> go p l 0 0<br />
<br />
where<br />
go :: Ptr Word8 -> Int -> Int -> Int -> IO ()<br />
STRICT4(go)<br />
go p l n i<br />
| n >= l = print i<br />
| otherwise = do<br />
let p' = p `plusPtr` n<br />
q = memchr p' 0x0a (fromIntegral (l-n))<br />
if q == nullPtr<br />
then print i<br />
else do let k = q `minusPtr` p'<br />
go p l (n+k+1) (i+1)<br />
<br />
foreign import ccall unsafe "string.h memchr" memchr<br />
:: Ptr Word8 -> CInt -> CSize -> Ptr Word8<br />
</haskell><br />
<br />
$ time ./wc /usr/share/dict/words<br />
98326 <br />
./wc /usr/share/dict/words 0.00s user 0.00s system 47% cpu 0.017 total<br />
<br />
Slowly inching forwards.<br />
<br />
== Read the Core ==<br />
<br />
While we're here, we can check whether the strictness on the 'go' function<br />
makes any difference, by reading the GHC Core:<br />
<br />
$ ghc -O -package fps -cpp -ffi wc.hs -ddump-simpl | less<br />
<br />
Search for the 'go' function:<br />
<br />
Main.$wgo :: GHC.Prim.Addr#<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.IOBase.IO ()<br />
<br />
And without the strictness:<br />
<br />
Main.$wgo :: GHC.Ptr.Ptr GHC.Word.Word8<br />
-> GHC.Prim.Int#<br />
-> GHC.Prim.Int#<br />
-> GHC.Base.Int<br />
-> GHC.IOBase.IO ()<br />
<br />
So GHC is helpfully unboxing the Ptr Word8 into a raw machine Addr#.<br />
<br />
== Avoid some code ==<br />
<br />
The guard that checks the length is unneeded, since memchr takes a length<br />
argument anyway. It also calculates the next pointer for us, so avoid<br />
recalculating it. (Note that this is equivalent to using the 'count'<br />
function, which has the same implementation).<br />
<br />
<haskell><br />
import Foreign<br />
import Foreign.ForeignPtr<br />
import Foreign.C.Types<br />
<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
#define STRICT3(f) f a b c | a `seq` b `seq` c `seq` False = undefined<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x s l) -> withForeignPtr x $ \p -> <br />
go (p `plusPtr` s) (fromIntegral l) 0<br />
where<br />
go :: Ptr Word8 -> CSize -> Int -> IO ()<br />
STRICT3(go)<br />
go p l i = do<br />
let q = memchr p 0x0a l<br />
if q == nullPtr<br />
then print i<br />
else do let k = fromIntegral $ q `minusPtr` p<br />
go (q `plusPtr` 1) (l-k) (i+1)<br />
<br />
foreign import ccall unsafe "string.h memchr" memchr<br />
:: Ptr Word8 -> CInt -> CSize -> Ptr Word8<br />
</haskell><br />
<br />
Checking the Core, 'go' is now:<br />
<br />
Main.$wgo :: GHC.Prim.Addr#<br />
-> GHC.Prim.Word#<br />
-> GHC.Prim.Int#<br />
-> GHC.IOBase.IO ()<br />
<br />
The code is certainly a bit simpler, at least.<br />
<br />
$ ghc -O -package fps -cpp -ffi wc.hs<br />
$ time ./wc /usr/share/dict/words<br />
98326<br />
./wc /usr/share/dict/words 0.00s user 0.01s system 70% cpu 0.017 total<br />
<br />
But we can't seem to squeeze any more out, at least on data this size.<br />
<br />
== Going via C ==<br />
<br />
We reach a point where I can't think of any more tricks, so we can always code<br />
up a little C and call into that, for this tight loop. Sometimes we just have<br />
to do this, and that's what the ffi is for, after all.<br />
<br />
<haskell><br />
-- wc.hs<br />
<br />
import Foreign<br />
import System.Environment<br />
import qualified Data.ByteString as B<br />
<br />
main = do<br />
f <- head `fmap` getArgs<br />
B.readFile f >>= \(B.PS x _ l) -> withForeignPtr x $ \p -> print (c_wc p l)<br />
<br />
foreign import ccall unsafe "wc.h wc" c_wc :: Ptr Word8 -> Int -> Int<br />
<br />
-- wc_c.c<br />
int wc(char *p, int len) {<br />
int c;<br />
for (c = 0; len--; ++p)<br />
if (*p == '\n')<br />
++c;<br />
return c;<br />
}<br />
<br />
-- wc.h<br />
int wc(char *p, int len);<br />
</haskell><br />
<br />
$ gcc -O3 -c wc_c.c<br />
$ ghc -O -package fps wc.hs -o wc -fglasgow-exts wc_c.o<br />
$ time ./wc /usr/share/dict/words<br />
98326<br />
./wc /usr/share/dict/words 0.00s user 0.00s system 25% cpu 0.016 total<br />
<br />
And we are done. Note that the tight C loop didn't give us anything in the end over<br />
the naive ByteString code, which is a very satisfying result.</div>Steve Chttps://wiki.haskell.org/index.php?title=Performance/IO&diff=29814Performance/IO2009-08-30T06:16:28Z<p>Steve C: Expand on Library/Streams description.</p>
<hr />
<div>{{Performance infobox}}<br />
[[Category:Performance|IO]]<br />
==I/O==<br />
<br />
If the standard lazy IO operations are proving to be a bottleneck,<br />
buffer-based IO is an alternative (hGetBuf/hPutBuf). This can be<br />
particularly effective when combined with packed strings (see [[wc]]). <br />
<br />
Some external libraries also provide memory mapped IO. <br />
<br />
[[Library/Streams]] is a new (in 2006) approach to I/O, which claims to be 5-10 times faster on some operations than handle-based IO.</div>Steve Chttps://wiki.haskell.org/index.php?title=Library/Streams&diff=29813Library/Streams2009-08-30T06:09:39Z<p>Steve C: Add year of development - to give more context.</p>
<hr />
<div>[[Category:Libraries]]<br />
== Introduction ==<br />
<br />
=== Streams: the extensible I/O library ===<br />
<br />
I (Bulat Ziganshin) developed a new I/O library in 2006 that IMHO is so sharp that it can eventually replace the current I/O facilities based on using Handles. The main advantage of the new library is its strong modular design using<br />
typeclasses. The library consists of small independent modules, each<br />
implementing one type of stream (file, memory buffer, pipe) or one<br />
part of common stream functionality (buffering, Char encoding,<br />
locking). 3rd-party libs can easily add new stream types and new common<br />
functionality. Other benefits of the new library include support for<br />
streams functioning in any monad, Hugs and GHC compatibility, high<br />
speed and an easy migration path from the existing I/O library.<br />
<br />
The Streams library is heavily based on the HVIO module written by John<br />
Goerzen. I especially want to thank John for his clever design and<br />
implementation. Really, I just renamed HVIO to Stream and presented<br />
this as my own work. :) Further development direction was inspired<br />
by the "New I/O library" written by Simon Marlow.<br />
<br />
=== Simple Streams ===<br />
<br />
The key concept of the lib is the Stream class, whose interface mimics<br />
familiar interface for Handles, just with "h" replaced with "v" in<br />
function names:<br />
<br />
<haskell><br />
class (Monad m) => Stream m h where<br />
vPutStrLn :: h -> String -> m ()<br />
vGetContents :: h -> m String<br />
vIsEOF :: h -> m Bool<br />
vClose :: h -> m ()<br />
....................<br />
</haskell><br />
<br />
This means that you already know how to use any stream! The Stream interface<br />
currently has 8 implementations: a Handle itself, raw files, pipes,<br />
memory buffers and string buffers. Future plans include support for<br />
memory-mapped files, sockets, circular memory buffers for interprocess<br />
communication and UArray-based streams.<br />
<br />
By themselves, these Stream implementations are rather simple. Basically,<br />
to implement new Stream type, it's enough to provide vPutBuf/vGetBuf<br />
operations, or even vGetChar/vPutChar. The latter way, although<br />
inefficient, allows us to implement streams that can work in any monad.<br />
StringReader and StringBuffer streams use this to provide string-based<br />
Stream class implementations both for IO and ST monads. Yes, you can<br />
use the full power of Stream operations inside the ST monad!<br />
<br />
=== Layers of functionality ===<br />
<br />
All additional functionality is implemented via Stream Transformers,<br />
which are just parameterized Streams, whose parameters also<br />
implement the Stream interface. This allows you to apply any number of stream<br />
transformers to the raw stream and then use the result as an ordinary<br />
Stream. For example:<br />
<br />
<haskell><br />
h <- openRawFD "test" WriteMode<br />
>>= bufferBlockStream<br />
>>= withEncoding utf8<br />
>>= withLocking<br />
</haskell><br />
<br />
This code creates a new FD, which represents a raw file, and then adds<br />
to this Stream buffering, Char encoding and locking functionality. The<br />
result type of "h" is something like this:<br />
<br />
<haskell><br />
WithLocking (WithEncoding (BufferedBlockStream FD))<br />
</haskell><br />
<br />
The complete type, as well as all the intermediate types, implements the Stream<br />
interface. Each transformer intercepts operations corresponding to its<br />
nature, and passes the rest through. For example, the encoding transformer<br />
intercepts only vGetChar/vPutChar operations and translates them to<br />
the sequences of vGetByte/vPutByte calls of the lower-level stream.<br />
The locking transformer just wraps any operation in the locking wrapper.<br />
<br />
We can trace, for example, the execution of a "vPutBuf" operation on the<br />
above-constructed Stream. First, the locking transformer acquires a lock<br />
and then passes this call to the next level. Then the encoding transformer does<br />
nothing and passes this call to the next level. The buffering<br />
transformer flushes the current buffer and passes the call further.<br />
Finally, FD itself performs the operation after all these<br />
preparations and on the returning path, the locking transformer release<br />
its lock.<br />
<br />
As another example, the "vPutChar" call on this Stream is<br />
transformed (after locking) into several "vPutByte" calls by the<br />
encoding transformer, and these bytes go to the buffer in the<br />
buffering transformer, with or without a subsequent call to the FD's<br />
"vPutBuf".<br />
<br />
=== Modularity ===<br />
<br />
As you can see, stream transformers really are independent of each<br />
other. This allows you to use them on any stream and in any combination<br />
(but you should apply them in proper order - buffering, then Char<br />
encoding, then locking). As a result, you can apply to the stream<br />
only the transformers that you really need. If you don't use the<br />
stream in multiple threads, you don't need to apply the locking<br />
transformer. If you don't use any encodings other than Latin-1 -- or<br />
don't use text I/O at all -- you don't need an encoding transformer.<br />
Moreover, you may not even need to know anything about the UserData transformer<br />
until you actually need to use it :)<br />
<br />
Both streams and stream transformers can be implemented by 3rd-party<br />
libraries. Streams and transformers from arbitrary libraries will<br />
seamlessly work together as long as they properly implement the Stream<br />
interface. My future plans include implementation of an on-the-fly<br />
(de)compression transformer and I will be happy to see 3rd-party<br />
transformers that intercept vGetBuf/vPutBuf calls and use select(),<br />
kqueue() and other methods to overlap I/O operations.<br />
<br />
=== Speed ===<br />
<br />
A quick comment about speed: it's fast enough -- 10-50 MB/s (depending<br />
on the type of operation) on a 1GHz cpu. The Handle operations, for comparison,<br />
show speed of 1-10 mb/s on the same computer. But that don't means that each<br />
and any operation in new library is 10 times faster. Strict I/O (including<br />
vGetChar/vPutChar) is a LOT faster. I included a demonstration of this<br />
fascinating speed as "Examples/wc.hs". If you need a really high speed,<br />
don't forget to increase buffer size with "vSetBuffering".<br />
<br />
On the other side, lazy I/O (including any operations that receive or return<br />
strings) show only modest speedup. This is limited by Haskell/GHC itself and<br />
I can't do much to get around these limits. Instead, I plan to provide support<br />
for I/O using packed strings. This will allow to write I/O-intensive Haskell<br />
programs that are as fast as their C counterparts.<br />
<br />
Other sources of slowness includes using of locking transformer (if you need<br />
to do this, try use "lock" around speed-critical algorithms) and complex class<br />
structure, what may be avoided by using "forall" types (I'm not sure, Simon<br />
Marlow can enlighten this topic).<br />
<br />
The library includes benchmarking code in the file "Examples/StreamsBenchmark.hs"<br />
<br />
<br />
<br />
== Overview of Stream transformers ==<br />
<br />
=== Buffering ===<br />
<br />
There are three buffering transformers. Each buffering transformer<br />
implements support for vGetByte, vPutChar, vGetContents and other<br />
byte- and text-oriented operations for the streams, which by themselves<br />
support only vGetBuf/vPutBuf (or vReceiveBuf/vSendBuf) operations.<br />
<br />
The first transformer can be applied to any stream supporting<br />
vGetBuf/vPutBuf. This is applied by the operation "bufferBlockStream". The<br />
well-known vSetBuffering/vGetBuffering operations are intercepted by<br />
this transformer and used to control buffer size. At this moment, only<br />
BlockBuffering is implemented, while LineBuffering and NoBuffering are<br />
only in the planning stages.<br />
<br />
Two other transformers can be applied to streams that implement<br />
vReceiveBuf/vSendBuf operations -- that is, streams whose data<br />
reside in memory, including in-memory streams and memory-mapped<br />
files. In these cases, the buffering transformer doesn't need to allocate<br />
a buffer itself, it just requests from the underlying stream the address and<br />
size of the next available portion of data. Nevertheless, the final<br />
result is the same -- we get support for all byte- and text-oriented<br />
I/O operations. The "bufferMemoryStream" operation can be applied to any<br />
memory-based stream to add buffering to it. The "bufferMemoryStreamUnchecked"<br />
operation (which implements the third buffering transformer) can be used instead,<br />
if you can guarantee that I/O operations can't overflow the used buffer.<br />
<br />
=== Encoding ===<br />
<br />
The Char encoding transformer allows you to encode each Char written to the<br />
stream as a sequence of bytes, implementing UTF and other encodings.<br />
This transformer can be applied to any stream implementing<br />
vGetByte/vPutByte operations and in return it implements<br />
vGetChar/vPutChar and all other text-oriented operations. This<br />
transformer can be applied to a stream with the "withEncoding encoding"<br />
operation, where `encoding` may be `latin1`, `utf8` or any other<br />
encoding that you (or a 3rd-party lib) implement. Look at the<br />
"Data.CharEncoding" module to see how to implement new encodings.<br />
Encoding of streams created with the "withEncoding" operation can be<br />
queried with "vGetEncoding". See examples of their usage in the file<br />
"Examples/CharEncoding.hs"<br />
<br />
=== Locking ===<br />
<br />
The locking transformer ensures that the stream is properly shared by<br />
several threads. You already know enough about its basic usage --<br />
"withLocking" applies this transformer to the stream and all the<br />
required locking is performed automagically. You can also use "lock"<br />
operations to acquire the lock explicitly during multiple operations:<br />
<br />
<haskell><br />
lock h $ \h -> do<br />
savedpos <- vTell h<br />
vSeek h AbsoluteSeek 100<br />
vPutStr h ":-)"<br />
vSeek h AbsoluteSeek savedpos<br />
</haskell><br />
<br />
See the file "Examples/Locking.hs" for examples of using locking transformer.<br />
<br />
=== Attaching user data ===<br />
<br />
This transformer allows you to attach arbitrary data to any Stream. It does<br />
nothing extraordinary except that the stream with attached data is the proper<br />
Stream, again. See example of its usage in the file "Examples/UserData.hs"<br />
<br />
== Overview of Stream [[type]]s ==<br />
<br />
=== Handle (legacy way to access files/sockets) ===<br />
<br />
"Handle" is an instance of the Stream class, with a straightforward implementation.<br />
You can use the<br />
Char encoding transformer with Handles. Although Handles implement<br />
buffering and locking by themselves, you may also be interested in<br />
applying these transformers to the Handle type. This has<br />
benefits -- "bufferBlockStream" works faster than internal Handle<br />
buffering, and the locking transformer enables the use of a "lock" operation to<br />
create a lock around a sequence of operations. Moreover, the locking<br />
transformer should be used to ensure proper multi-threading operation<br />
of Handle with added encoding or buffering facilities.<br />
<br />
=== FD (new way to access files) ===<br />
<br />
The new method of using files, independent of the existing I/O<br />
library, is implemented with the FD type. FD is just an Int representing a<br />
POSIX file descriptor and the FD type implements only basic Stream I/O<br />
operations - vGetBuf and vPutBuf. So, to create a full-featured FD-based<br />
stream, you need to apply buffering transformers. Therefore, the library<br />
defines two ways to open files with FD - openRawFD/openRawBinaryFD<br />
just creates FD, while openFD/openBinaryFD creates FD and immediatelly<br />
apply buffering transformer (bufferBlockStream) to it. In most cases<br />
you will use the latter operations. Both pairs mimic the arguments and<br />
behaviour of well-known Handle operations openFile/openBinaryFile, so<br />
you already know how to use them. Other transformers may be used then<br />
as you need. So, abovementioned example can be abbreviated to:<br />
<br />
<haskell><br />
h <- openFD "test" WriteMode<br />
>>= withEncoding utf8<br />
>>= withLocking<br />
</haskell><br />
<br />
Thus, to switch from the existing I/O library to using Streams, you<br />
need only to replace "h" with "v" in the names of Handle operations, and<br />
replace openFile/openBinaryFile calls with openFD/openBinaryFD while<br />
adding the "withLocking" transformer to files used in multiple threads.<br />
That's all!<br />
<br />
For example, the following code:<br />
<br />
<haskell><br />
h <- openFile "test" ReadMode<br />
text <- hGetContents h<br />
hClose h<br />
</haskell><br />
<br />
should be translated to:<br />
<br />
<haskell><br />
h <- openFD "test" ReadMode<br />
-- >>= withLocking -- needed only for multi-threaded usage<br />
text <- vGetContents h<br />
vClose h<br />
</haskell><br />
<br />
<br />
File "Examples/FD.hs" will show you the FD usage.<br />
<br />
<br />
In order to work with stdin/stdout/stderr via FDs, you should open them in the same way:<br />
<br />
<haskell><br />
stdinStream <- bufferBlockStream fdStdIn <br />
>>= withEncoding utf8 -- optional, required only for using non-Latin1 encoding<br />
>>= withLocking -- optional, required only to use this Stream in concurrent Haskell threads<br />
<br />
stdoutStream <- bufferBlockStream fdStdOut<br />
>>= withEncoding utf8 -- see above<br />
>>= withLocking -- ...<br />
<br />
stderrStream <- bufferBlockStream fdStdErr<br />
>>= withEncoding utf8 -- ...<br />
>>= withLocking -- ...<br />
</haskell><br />
<br />
Please note that Streams currently supports only block buffering, there is no line buffering and no-buffering support.<br />
<br />
=== MemBuf (memory-resident stream) ===<br />
<br />
MemBuf is a stream type, that keeps its contents in memory buffer.<br />
There are two types of MemBufs you can create - you can either open<br />
existing memory buffer with "openMemBuf ptr size" or create new one<br />
with "createMemBuf initsize". MemBuf opened by "openMemBuf" will be<br />
never resized or moved in memory, and will not be freed by "vClose".<br />
MemBuf created by "createMemBuf" will grow as needed, can be manually<br />
resized by "vSetFileSize" operation, and is automatically freed by<br />
"vClose".<br />
<br />
Actually, raw MemBufs created by the "createRawMemBuf" and "openRawMemBuf"<br />
operations, while createMemBuf/openMemBuf incorporate an additional<br />
"bufferMemoryStream" call (as you should remember, buffering adds vGetChar,<br />
vPutStr and other text- and byte-I/O operations on top of vReceiveBuf<br />
and vSendBuf). You can also apply Char encoding and locking<br />
transformers to these streams. The "saveToFile" and "readFromFile" operations<br />
provide an easy way to save/restore buffer contents in a file.<br />
<br />
File "Examples/MemBuf.hs" demonstrates the usage of MemBuf.<br />
<br />
=== FunctionsMemoryStream ===<br />
<br />
This Stream type allows implementation of arbitrary streams, just by<br />
providing three functions that implement vReceiveBuf, vSendBuf and cleanup<br />
operations. It seems that this Stream type is of interest only for my<br />
own program and can be scrutinized only as example of creating 3rd-party<br />
Stream types. It is named "FunctionsMemoryStream", see the sources if you<br />
are interested.<br />
<br />
=== StringReader & StringBuffer (String-based streams) ===<br />
<br />
Four remaining Stream types were part of the HVIO module and I copied their<br />
description from there:<br />
<br />
In addition to Handle, there are several pre-defined stream types for<br />
your use. 'StringReader' is a particularly interesting one. At<br />
creation time, you pass it a String. Its contents are read lazily<br />
whenever a read call is made. It can be used, therefore, to implement<br />
filters (simply initialize it with the result from, say, a map over<br />
hGetContents from another Stream object), codecs, and simple I/O<br />
testing. Because it is lazy, it needs not hold the entire string in<br />
memory. You can create a 'StringReader' with a call to<br />
'newStringReader'.<br />
<br />
'StringBuffer' is a similar type, but with a different purpose. It<br />
provides a full interface like Handle (it supports read, write and<br />
seek operations). However, it maintains an in-memory buffer with the<br />
contents of the file, rather than an actual on-disk file. You can<br />
access the entire contents of this buffer at any time. This can be<br />
quite useful for testing I/O code, or for cases where existing APIs<br />
use I/O, but you prefer a String representation. Note however that<br />
this stream type is very inefficient. You can create a 'StringBuffer'<br />
with a call to 'newStringBuffer'.<br />
<br />
One significant improvement over the original HVIO library is that<br />
'StringReader' and 'StringBuffer' can work not only in IO, but also in<br />
ST monad.<br />
<br />
=== Pipes (passing data between Haskell threads) ===<br />
<br />
Finally, there are pipes. These pipes are analogous to the Unix pipes<br />
that are available from System.Posix, but don't require Unix and work<br />
only in Haskell. When you create a pipe, you actually get two Stream<br />
objects: a 'PipeReader' and a 'PipeWriter'. You must use the<br />
'PipeWriter' in one thread and the 'PipeReader' in another thread.<br />
Data that's written to the 'PipeWriter' will then be available for<br />
reading with the 'PipeReader'. The pipes are implemented completely<br />
with existing Haskell threading primitives, and require no special<br />
operating system support. Unlike Unix pipes, these pipes cannot be<br />
used across a fork(). Also unlike Unix pipes, these pipes are<br />
portable and interact well with Haskell threads. A new pipe can be<br />
created with a call to 'newHVIOPipe'.<br />
<br />
<br />
<br />
== Additional details ==<br />
<br />
=== Support for [[GHC]], [[Hugs]] and other compilers ===<br />
<br />
The library is compatible with [[GHC]] 6.4<br />
<br />
<br />
The library fully supports [[Hugs]] 2003-2006, but<br />
<br />
1) support for FD and MMFile is temporarily disabled because I don't know how<br />
to build DLLs<br />
<br />
2) Hugs 2003 doesn't include support for "instance Bits Word" and vGetBuf/vPutBuf,<br />
so you need to add these implementations manually or delete the lines that use it<br />
(look for "2003" in the sources)<br />
<br />
3) WinHugs doesn't support preprocessing, so I included the MakeHugs.cmd script<br />
to preprocess source files using cpphs<br />
<br />
<br />
Main disadvantage of the library is that it supports only Hugs and GHC<br />
because of using extensions in type classe system (namely, MPTC+FD). I think that it<br />
can be made H98-compatible at the cost of excluding support for non-IO<br />
monads. I will try to make such a stripped version for other compilers<br />
if people are interested.<br />
<br />
=== Downloading and installation ===<br />
<br />
To get Streams 0.1.7, you can download one of<br />
http://files.pupeno.com/software/streams/Streams-0.1.7.tar.bz2<br />
http://files.pupeno.com/software/streams/Streams-0.1.7.tar.gz<br />
or you can get it from its repository by running:<br />
<br />
darcs get --tag=0.1.7 http://software.pupeno.com/Streams-0.1 Streams-0.1.7<br />
<br />
You can also download and keep track of the 0.1 branch, which is<br />
supposed to remain stable and only get bug-fixes by running <br />
<br />
darcs get http://software.pupeno.com/Streams-0.1/<br />
<br />
and then run 'darcs pull' inside it to get further changes.<br />
<br />
To get the latest unstable and fluctuating version, the development<br />
version, run:<br />
<br />
darcs get http://software.pupeno.com/Streams/<br />
<br />
Note: as of this moment, while the project is being darcsified you are<br />
not going to find anything useful there, but we expect that to change.<br />
<br />
Preferably, you should send patches to code to <br />
[mailto:Bulat.Ziganshin@gmail.com Bulat.Ziganshin@gmail.com]<br />
and to other parts of library to Pupeno. Documentation may<br />
be edited right at the project homepage, which remains<br />
http://haskell.org/haskellwiki/Library/Streams<br />
<br />
Thanks to Jeremy Shaw, the library is now cabalized. To install it, run command:<br />
<br />
make install<br />
<br />
Directory "Examples" contains examples of using the library.<br />
<br />
=== Stage of development ===<br />
<br />
The library is currently at the beta stage. It contains a number of<br />
known minor problems and an unknown number of yet-to-be-discovered bugs.<br />
It is not properly documented, doesn't include QuickCheck tests, is not<br />
cabalized, and not all "h*" operations have their "v*" equivalents yet.<br />
If anyone wants to join this effort in order to help fix these oddities<br />
and prepare the lib for inclusion in the standard libraries suite, I would<br />
be really happy. :) I will also be happy (although much less ;) to see<br />
bug reports and suggestions about its interface and internal<br />
organization. It's just a first public version, so we still can change<br />
everything here!<br />
<br />
In particular, this wiki page is an official library documentation.<br />
Please continue to improve it and add more information about using the library.<br />
Feel free to ask me about library usage via email:<br />
[mailto:Bulat.Ziganshin@gmail.com Bulat.Ziganshin@gmail.com]<br />
<br />
=== Changelog ===<br />
<br />
User-visible improvements made in Streams library since version 0.1 (6 Feb 2006)<br />
<br />
0.1a (6 Feb 2006)<br />
- Fixed bug: System.MMFile was uncompilable on non-Windows systems<br />
<br />
0.1b (9 Feb 2006)<br />
- Fixed bug: very slow WithLocking.vGetLine<br />
- Fixed bug: System.FD was also uncompilable on non-Windows systems<br />
<br />
0.1c (12 Feb 2006)<br />
- Fixed bug: System.FD modified one more time to reach Unix compatibility<br />
<br />
0.1d (13 Feb 2006)<br />
- Fixed bug: BufferedBlockStream.vGetLine caused exception<br />
* CharEncoding transformer was made faster, but vSetEncoding no more supported<br />
<br />
0.1e (8 Jun 2006)<br />
- Fixed bug: "openFD name WriteMode" didn't truncate files on unixes<br />
* Full library now released under BSD3 license, thanks to John Goerzen<br />
+ Now cabalized, thanks to Jeremy Shaw<br />
<br />
0.1.6 (Oct 14 2006)<br />
* Added compatibility with just released GHC 6.6<br />
<br />
0.1.7 (Nov 24 2006)<br />
* true support for GHC 6.6<br />
* support of files larger than 4 gb on windows (see FD5gb.hs example)<br />
* files are now open in shared mode on all systems<br />
* haddock'ized internal docs<br />
* ready to be included in any unix packaging system</div>Steve Chttps://wiki.haskell.org/index.php?title=Library/Streams&diff=29812Library/Streams2009-08-30T06:05:33Z<p>Steve C: Add author's name to expand upon "I".</p>
<hr />
<div>[[Category:Libraries]]<br />
== Introduction ==<br />
<br />
=== Streams: the extensible I/O library ===<br />
<br />
I (Bulat Ziganshin) have developed a new I/O library that IMHO is so sharp that it can eventually replace the current I/O facilities based on using Handles.<br />
The main advantage of the new library is its strong modular design using<br />
typeclasses. The library consists of small independent modules, each<br />
implementing one type of stream (file, memory buffer, pipe) or one<br />
part of common stream functionality (buffering, Char encoding,<br />
locking). 3rd-party libs can easily add new stream types and new common<br />
functionality. Other benefits of the new library include support for<br />
streams functioning in any monad, Hugs and GHC compatibility, high<br />
speed and an easy migration path from the existing I/O library.<br />
<br />
The Streams library is heavily based on the HVIO module written by John<br />
Goerzen. I especially want to thank John for his clever design and<br />
implementation. Really, I just renamed HVIO to Stream and presented<br />
this as my own work. :) Further development direction was inspired<br />
by the "New I/O library" written by Simon Marlow.<br />
<br />
=== Simple Streams ===<br />
<br />
The key concept of the lib is the Stream class, whose interface mimics<br />
familiar interface for Handles, just with "h" replaced with "v" in<br />
function names:<br />
<br />
<haskell><br />
class (Monad m) => Stream m h where<br />
vPutStrLn :: h -> String -> m ()<br />
vGetContents :: h -> m String<br />
vIsEOF :: h -> m Bool<br />
vClose :: h -> m ()<br />
....................<br />
</haskell><br />
<br />
This means that you already know how to use any stream! The Stream interface<br />
currently has 8 implementations: a Handle itself, raw files, pipes,<br />
memory buffers and string buffers. Future plans include support for<br />
memory-mapped files, sockets, circular memory buffers for interprocess<br />
communication and UArray-based streams.<br />
<br />
By themselves, these Stream implementations are rather simple. Basically,<br />
to implement new Stream type, it's enough to provide vPutBuf/vGetBuf<br />
operations, or even vGetChar/vPutChar. The latter way, although<br />
inefficient, allows us to implement streams that can work in any monad.<br />
StringReader and StringBuffer streams use this to provide string-based<br />
Stream class implementations both for IO and ST monads. Yes, you can<br />
use the full power of Stream operations inside the ST monad!<br />
<br />
=== Layers of functionality ===<br />
<br />
All additional functionality is implemented via Stream Transformers,<br />
which are just parameterized Streams, whose parameters also<br />
implement the Stream interface. This allows you to apply any number of stream<br />
transformers to the raw stream and then use the result as an ordinary<br />
Stream. For example:<br />
<br />
<haskell><br />
h <- openRawFD "test" WriteMode<br />
>>= bufferBlockStream<br />
>>= withEncoding utf8<br />
>>= withLocking<br />
</haskell><br />
<br />
This code creates a new FD, which represents a raw file, and then adds<br />
to this Stream buffering, Char encoding and locking functionality. The<br />
result type of "h" is something like this:<br />
<br />
<haskell><br />
WithLocking (WithEncoding (BufferedBlockStream FD))<br />
</haskell><br />
<br />
The complete type, as well as all the intermediate types, implements the Stream<br />
interface. Each transformer intercepts operations corresponding to its<br />
nature, and passes the rest through. For example, the encoding transformer<br />
intercepts only vGetChar/vPutChar operations and translates them to<br />
the sequences of vGetByte/vPutByte calls of the lower-level stream.<br />
The locking transformer just wraps any operation in the locking wrapper.<br />
<br />
We can trace, for example, the execution of a "vPutBuf" operation on the<br />
above-constructed Stream. First, the locking transformer acquires a lock<br />
and then passes this call to the next level. Then the encoding transformer does<br />
nothing and passes this call to the next level. The buffering<br />
transformer flushes the current buffer and passes the call further.<br />
Finally, FD itself performs the operation after all these<br />
preparations and on the returning path, the locking transformer release<br />
its lock.<br />
<br />
As another example, the "vPutChar" call on this Stream is<br />
transformed (after locking) into several "vPutByte" calls by the<br />
encoding transformer, and these bytes go to the buffer in the<br />
buffering transformer, with or without a subsequent call to the FD's<br />
"vPutBuf".<br />
<br />
=== Modularity ===<br />
<br />
As you can see, stream transformers really are independent of each<br />
other. This allows you to use them on any stream and in any combination<br />
(but you should apply them in proper order - buffering, then Char<br />
encoding, then locking). As a result, you can apply to the stream<br />
only the transformers that you really need. If you don't use the<br />
stream in multiple threads, you don't need to apply the locking<br />
transformer. If you don't use any encodings other than Latin-1 -- or<br />
don't use text I/O at all -- you don't need an encoding transformer.<br />
Moreover, you may not even need to know anything about the UserData transformer<br />
until you actually need to use it :)<br />
<br />
Both streams and stream transformers can be implemented by 3rd-party<br />
libraries. Streams and transformers from arbitrary libraries will<br />
seamlessly work together as long as they properly implement the Stream<br />
interface. My future plans include implementation of an on-the-fly<br />
(de)compression transformer and I will be happy to see 3rd-party<br />
transformers that intercept vGetBuf/vPutBuf calls and use select(),<br />
kqueue() and other methods to overlap I/O operations.<br />
<br />
=== Speed ===<br />
<br />
A quick comment about speed: it's fast enough -- 10-50 MB/s (depending<br />
on the type of operation) on a 1GHz cpu. The Handle operations, for comparison,<br />
show speed of 1-10 mb/s on the same computer. But that don't means that each<br />
and any operation in new library is 10 times faster. Strict I/O (including<br />
vGetChar/vPutChar) is a LOT faster. I included a demonstration of this<br />
fascinating speed as "Examples/wc.hs". If you need a really high speed,<br />
don't forget to increase buffer size with "vSetBuffering".<br />
<br />
On the other side, lazy I/O (including any operations that receive or return<br />
strings) show only modest speedup. This is limited by Haskell/GHC itself and<br />
I can't do much to get around these limits. Instead, I plan to provide support<br />
for I/O using packed strings. This will allow to write I/O-intensive Haskell<br />
programs that are as fast as their C counterparts.<br />
<br />
Other sources of slowness includes using of locking transformer (if you need<br />
to do this, try use "lock" around speed-critical algorithms) and complex class<br />
structure, what may be avoided by using "forall" types (I'm not sure, Simon<br />
Marlow can enlighten this topic).<br />
<br />
The library includes benchmarking code in the file "Examples/StreamsBenchmark.hs"<br />
<br />
<br />
<br />
== Overview of Stream transformers ==<br />
<br />
=== Buffering ===<br />
<br />
There are three buffering transformers. Each buffering transformer<br />
implements support for vGetByte, vPutChar, vGetContents and other<br />
byte- and text-oriented operations for the streams, which by themselves<br />
support only vGetBuf/vPutBuf (or vReceiveBuf/vSendBuf) operations.<br />
<br />
The first transformer can be applied to any stream supporting<br />
vGetBuf/vPutBuf. This is applied by the operation "bufferBlockStream". The<br />
well-known vSetBuffering/vGetBuffering operations are intercepted by<br />
this transformer and used to control buffer size. At this moment, only<br />
BlockBuffering is implemented, while LineBuffering and NoBuffering are<br />
only in the planning stages.<br />
<br />
Two other transformers can be applied to streams that implement<br />
vReceiveBuf/vSendBuf operations -- that is, streams whose data<br />
reside in memory, including in-memory streams and memory-mapped<br />
files. In these cases, the buffering transformer doesn't need to allocate<br />
a buffer itself, it just requests from the underlying stream the address and<br />
size of the next available portion of data. Nevertheless, the final<br />
result is the same -- we get support for all byte- and text-oriented<br />
I/O operations. The "bufferMemoryStream" operation can be applied to any<br />
memory-based stream to add buffering to it. The "bufferMemoryStreamUnchecked"<br />
operation (which implements the third buffering transformer) can be used instead,<br />
if you can guarantee that I/O operations can't overflow the used buffer.<br />
<br />
=== Encoding ===<br />
<br />
The Char encoding transformer allows you to encode each Char written to the<br />
stream as a sequence of bytes, implementing UTF and other encodings.<br />
This transformer can be applied to any stream implementing<br />
vGetByte/vPutByte operations and in return it implements<br />
vGetChar/vPutChar and all other text-oriented operations. This<br />
transformer can be applied to a stream with the "withEncoding encoding"<br />
operation, where `encoding` may be `latin1`, `utf8` or any other<br />
encoding that you (or a 3rd-party lib) implement. Look at the<br />
"Data.CharEncoding" module to see how to implement new encodings.<br />
Encoding of streams created with the "withEncoding" operation can be<br />
queried with "vGetEncoding". See examples of their usage in the file<br />
"Examples/CharEncoding.hs"<br />
<br />
=== Locking ===<br />
<br />
The locking transformer ensures that the stream is properly shared by<br />
several threads. You already know enough about its basic usage --<br />
"withLocking" applies this transformer to the stream and all the<br />
required locking is performed automagically. You can also use "lock"<br />
operations to acquire the lock explicitly during multiple operations:<br />
<br />
<haskell><br />
lock h $ \h -> do<br />
savedpos <- vTell h<br />
vSeek h AbsoluteSeek 100<br />
vPutStr h ":-)"<br />
vSeek h AbsoluteSeek savedpos<br />
</haskell><br />
<br />
See the file "Examples/Locking.hs" for examples of using locking transformer.<br />
<br />
=== Attaching user data ===<br />
<br />
This transformer allows you to attach arbitrary data to any Stream. It does<br />
nothing extraordinary except that the stream with attached data is the proper<br />
Stream, again. See example of its usage in the file "Examples/UserData.hs"<br />
<br />
== Overview of Stream [[type]]s ==<br />
<br />
=== Handle (legacy way to access files/sockets) ===<br />
<br />
"Handle" is an instance of the Stream class, with a straightforward implementation.<br />
You can use the<br />
Char encoding transformer with Handles. Although Handles implement<br />
buffering and locking by themselves, you may also be interested in<br />
applying these transformers to the Handle type. This has<br />
benefits -- "bufferBlockStream" works faster than internal Handle<br />
buffering, and the locking transformer enables the use of a "lock" operation to<br />
create a lock around a sequence of operations. Moreover, the locking<br />
transformer should be used to ensure proper multi-threading operation<br />
of Handle with added encoding or buffering facilities.<br />
<br />
=== FD (new way to access files) ===<br />
<br />
The new method of using files, independent of the existing I/O<br />
library, is implemented with the FD type. FD is just an Int representing a<br />
POSIX file descriptor and the FD type implements only basic Stream I/O<br />
operations - vGetBuf and vPutBuf. So, to create a full-featured FD-based<br />
stream, you need to apply buffering transformers. Therefore, the library<br />
defines two ways to open files with FD - openRawFD/openRawBinaryFD<br />
just creates FD, while openFD/openBinaryFD creates FD and immediatelly<br />
apply buffering transformer (bufferBlockStream) to it. In most cases<br />
you will use the latter operations. Both pairs mimic the arguments and<br />
behaviour of well-known Handle operations openFile/openBinaryFile, so<br />
you already know how to use them. Other transformers may be used then<br />
as you need. So, abovementioned example can be abbreviated to:<br />
<br />
<haskell><br />
h <- openFD "test" WriteMode<br />
>>= withEncoding utf8<br />
>>= withLocking<br />
</haskell><br />
<br />
Thus, to switch from the existing I/O library to using Streams, you<br />
need only to replace "h" with "v" in the names of Handle operations, and<br />
replace openFile/openBinaryFile calls with openFD/openBinaryFD while<br />
adding the "withLocking" transformer to files used in multiple threads.<br />
That's all!<br />
<br />
For example, the following code:<br />
<br />
<haskell><br />
h <- openFile "test" ReadMode<br />
text <- hGetContents h<br />
hClose h<br />
</haskell><br />
<br />
should be translated to:<br />
<br />
<haskell><br />
h <- openFD "test" ReadMode<br />
-- >>= withLocking -- needed only for multi-threaded usage<br />
text <- vGetContents h<br />
vClose h<br />
</haskell><br />
<br />
<br />
File "Examples/FD.hs" will show you the FD usage.<br />
<br />
<br />
In order to work with stdin/stdout/stderr via FDs, you should open them in the same way:<br />
<br />
<haskell><br />
stdinStream <- bufferBlockStream fdStdIn <br />
>>= withEncoding utf8 -- optional, required only for using non-Latin1 encoding<br />
>>= withLocking -- optional, required only to use this Stream in concurrent Haskell threads<br />
<br />
stdoutStream <- bufferBlockStream fdStdOut<br />
>>= withEncoding utf8 -- see above<br />
>>= withLocking -- ...<br />
<br />
stderrStream <- bufferBlockStream fdStdErr<br />
>>= withEncoding utf8 -- ...<br />
>>= withLocking -- ...<br />
</haskell><br />
<br />
Please note that Streams currently supports only block buffering, there is no line buffering and no-buffering support.<br />
<br />
=== MemBuf (memory-resident stream) ===<br />
<br />
MemBuf is a stream type, that keeps its contents in memory buffer.<br />
There are two types of MemBufs you can create - you can either open<br />
existing memory buffer with "openMemBuf ptr size" or create new one<br />
with "createMemBuf initsize". MemBuf opened by "openMemBuf" will be<br />
never resized or moved in memory, and will not be freed by "vClose".<br />
MemBuf created by "createMemBuf" will grow as needed, can be manually<br />
resized by "vSetFileSize" operation, and is automatically freed by<br />
"vClose".<br />
<br />
Actually, raw MemBufs created by the "createRawMemBuf" and "openRawMemBuf"<br />
operations, while createMemBuf/openMemBuf incorporate an additional<br />
"bufferMemoryStream" call (as you should remember, buffering adds vGetChar,<br />
vPutStr and other text- and byte-I/O operations on top of vReceiveBuf<br />
and vSendBuf). You can also apply Char encoding and locking<br />
transformers to these streams. The "saveToFile" and "readFromFile" operations<br />
provide an easy way to save/restore buffer contents in a file.<br />
<br />
File "Examples/MemBuf.hs" demonstrates the usage of MemBuf.<br />
<br />
=== FunctionsMemoryStream ===<br />
<br />
This Stream type allows implementation of arbitrary streams, just by<br />
providing three functions that implement vReceiveBuf, vSendBuf and cleanup<br />
operations. It seems that this Stream type is of interest only for my<br />
own program and can be scrutinized only as example of creating 3rd-party<br />
Stream types. It is named "FunctionsMemoryStream", see the sources if you<br />
are interested.<br />
<br />
=== StringReader & StringBuffer (String-based streams) ===<br />
<br />
Four remaining Stream types were part of the HVIO module and I copied their<br />
description from there:<br />
<br />
In addition to Handle, there are several pre-defined stream types for<br />
your use. 'StringReader' is a particularly interesting one. At<br />
creation time, you pass it a String. Its contents are read lazily<br />
whenever a read call is made. It can be used, therefore, to implement<br />
filters (simply initialize it with the result from, say, a map over<br />
hGetContents from another Stream object), codecs, and simple I/O<br />
testing. Because it is lazy, it needs not hold the entire string in<br />
memory. You can create a 'StringReader' with a call to<br />
'newStringReader'.<br />
<br />
'StringBuffer' is a similar type, but with a different purpose. It<br />
provides a full interface like Handle (it supports read, write and<br />
seek operations). However, it maintains an in-memory buffer with the<br />
contents of the file, rather than an actual on-disk file. You can<br />
access the entire contents of this buffer at any time. This can be<br />
quite useful for testing I/O code, or for cases where existing APIs<br />
use I/O, but you prefer a String representation. Note however that<br />
this stream type is very inefficient. You can create a 'StringBuffer'<br />
with a call to 'newStringBuffer'.<br />
<br />
One significant improvement over the original HVIO library is that<br />
'StringReader' and 'StringBuffer' can work not only in IO, but also in<br />
ST monad.<br />
<br />
=== Pipes (passing data between Haskell threads) ===<br />
<br />
Finally, there are pipes. These pipes are analogous to the Unix pipes<br />
that are available from System.Posix, but don't require Unix and work<br />
only in Haskell. When you create a pipe, you actually get two Stream<br />
objects: a 'PipeReader' and a 'PipeWriter'. You must use the<br />
'PipeWriter' in one thread and the 'PipeReader' in another thread.<br />
Data that's written to the 'PipeWriter' will then be available for<br />
reading with the 'PipeReader'. The pipes are implemented completely<br />
with existing Haskell threading primitives, and require no special<br />
operating system support. Unlike Unix pipes, these pipes cannot be<br />
used across a fork(). Also unlike Unix pipes, these pipes are<br />
portable and interact well with Haskell threads. A new pipe can be<br />
created with a call to 'newHVIOPipe'.<br />
<br />
<br />
<br />
== Additional details ==<br />
<br />
=== Support for [[GHC]], [[Hugs]] and other compilers ===<br />
<br />
The library is compatible with [[GHC]] 6.4<br />
<br />
<br />
The library fully supports [[Hugs]] 2003-2006, but<br />
<br />
1) support for FD and MMFile is temporarily disabled because I don't know how<br />
to build DLLs<br />
<br />
2) Hugs 2003 doesn't include support for "instance Bits Word" and vGetBuf/vPutBuf,<br />
so you need to add these implementations manually or delete the lines that use it<br />
(look for "2003" in the sources)<br />
<br />
3) WinHugs doesn't support preprocessing, so I included the MakeHugs.cmd script<br />
to preprocess source files using cpphs<br />
<br />
<br />
Main disadvantage of the library is that it supports only Hugs and GHC<br />
because of using extensions in type classe system (namely, MPTC+FD). I think that it<br />
can be made H98-compatible at the cost of excluding support for non-IO<br />
monads. I will try to make such a stripped version for other compilers<br />
if people are interested.<br />
<br />
=== Downloading and installation ===<br />
<br />
To get Streams 0.1.7, you can download one of<br />
http://files.pupeno.com/software/streams/Streams-0.1.7.tar.bz2<br />
http://files.pupeno.com/software/streams/Streams-0.1.7.tar.gz<br />
or you can get it from its repository by running:<br />
<br />
darcs get --tag=0.1.7 http://software.pupeno.com/Streams-0.1 Streams-0.1.7<br />
<br />
You can also download and keep track of the 0.1 branch, which is<br />
supposed to remain stable and only get bug-fixes by running <br />
<br />
darcs get http://software.pupeno.com/Streams-0.1/<br />
<br />
and then run 'darcs pull' inside it to get further changes.<br />
<br />
To get the latest unstable and fluctuating version, the development<br />
version, run:<br />
<br />
darcs get http://software.pupeno.com/Streams/<br />
<br />
Note: as of this moment, while the project is being darcsified you are<br />
not going to find anything useful there, but we expect that to change.<br />
<br />
Preferably, you should send patches to code to <br />
[mailto:Bulat.Ziganshin@gmail.com Bulat.Ziganshin@gmail.com]<br />
and to other parts of library to Pupeno. Documentation may<br />
be edited right at the project homepage, which remains<br />
http://haskell.org/haskellwiki/Library/Streams<br />
<br />
Thanks to Jeremy Shaw, the library is now cabalized. To install it, run command:<br />
<br />
make install<br />
<br />
Directory "Examples" contains examples of using the library.<br />
<br />
=== Stage of development ===<br />
<br />
The library is currently at the beta stage. It contains a number of<br />
known minor problems and an unknown number of yet-to-be-discovered bugs.<br />
It is not properly documented, doesn't include QuickCheck tests, is not<br />
cabalized, and not all "h*" operations have their "v*" equivalents yet.<br />
If anyone wants to join this effort in order to help fix these oddities<br />
and prepare the lib for inclusion in the standard libraries suite, I would<br />
be really happy. :) I will also be happy (although much less ;) to see<br />
bug reports and suggestions about its interface and internal<br />
organization. It's just a first public version, so we still can change<br />
everything here!<br />
<br />
In particular, this wiki page is an official library documentation.<br />
Please continue to improve it and add more information about using the library.<br />
Feel free to ask me about library usage via email:<br />
[mailto:Bulat.Ziganshin@gmail.com Bulat.Ziganshin@gmail.com]<br />
<br />
=== Changelog ===<br />
<br />
User-visible improvements made in Streams library since version 0.1 (6 Feb 2006)<br />
<br />
0.1a (6 Feb 2006)<br />
- Fixed bug: System.MMFile was uncompilable on non-Windows systems<br />
<br />
0.1b (9 Feb 2006)<br />
- Fixed bug: very slow WithLocking.vGetLine<br />
- Fixed bug: System.FD was also uncompilable on non-Windows systems<br />
<br />
0.1c (12 Feb 2006)<br />
- Fixed bug: System.FD modified one more time to reach Unix compatibility<br />
<br />
0.1d (13 Feb 2006)<br />
- Fixed bug: BufferedBlockStream.vGetLine caused exception<br />
* CharEncoding transformer was made faster, but vSetEncoding no more supported<br />
<br />
0.1e (8 Jun 2006)<br />
- Fixed bug: "openFD name WriteMode" didn't truncate files on unixes<br />
* Full library now released under BSD3 license, thanks to John Goerzen<br />
+ Now cabalized, thanks to Jeremy Shaw<br />
<br />
0.1.6 (Oct 14 2006)<br />
* Added compatibility with just released GHC 6.6<br />
<br />
0.1.7 (Nov 24 2006)<br />
* true support for GHC 6.6<br />
* support of files larger than 4 gb on windows (see FD5gb.hs example)<br />
* files are now open in shared mode on all systems<br />
* haddock'ized internal docs<br />
* ready to be included in any unix packaging system</div>Steve Chttps://wiki.haskell.org/index.php?title=Introduction_to_QuickCheck1&diff=29772Introduction to QuickCheck12009-08-28T05:03:02Z<p>Steve C: Order resources by date.</p>
<hr />
<div>A quick introduction to QuickCheck, and testing Haskell code.<br />
<br />
== Motivation ==<br />
<br />
In September 2006, Bruno MartÃnez<br />
[http://www.haskell.org/pipermail/haskell-cafe/2006-September/018302.html asked] <br />
the following question:<br />
<br />
<haskell><br />
-- I've written a function that looks similar to this one<br />
<br />
getList = find 5 where<br />
find 0 = return []<br />
find n = do<br />
ch <- getChar<br />
if ch `elem` ['a'..'e'] then do<br />
tl <- find (n-1)<br />
return (ch : tl) else<br />
find n<br />
<br />
-- I want to test this function, without hitting the filesystem. In C++ I<br />
-- would use a istringstream. I couldn't find a function that returns a<br />
-- Handle from a String. The closer thing that may work that I could find<br />
-- was making a pipe and convertind the file descriptor. Can I simplify<br />
-- that function to take it out of the IO monad?<br />
</haskell><br />
<br />
So the problem is: how to effectively test this function in Haskell? The<br />
solution we turn to is refactoring and QuickCheck.<br />
<br />
== Keeping things pure ==<br />
<br />
The reason your getList is hard to test, is that the side effecting monadic code <br />
is mixed in with the pure computation, making it difficult to test<br />
without moving entirely into a "black box" IO-based testing model.<br />
Such a mixture is not good for reasoning about code.<br />
<br />
Let's untangle that, and then test the referentially transparent<br />
parts simply with QuickCheck. We can take advantage of lazy IO firstly,<br />
to avoid all the unpleasant low-level IO handling. <br />
<br />
So the first step is to factor out the IO part of the function into a<br />
thin "skin" layer:<br />
<br />
<haskell><br />
-- A thin monadic skin layer<br />
getList :: IO [Char]<br />
getList = fmap take5 getContents<br />
<br />
-- The actual worker<br />
take5 :: [Char] -> [Char]<br />
take5 = take 5 . filter (`elem` ['a'..'e'])<br />
</haskell><br />
<br />
== Testing with QuickCheck ==<br />
<br />
Now we can test the 'guts' of the algorithm, the take5 function, in<br />
isolation. Let's use QuickCheck. First we need an Arbitrary instance for<br />
the Char type -- this takes care of generating random Chars for us to<br />
test with. I'll restrict it to a range of nice chars just for<br />
simplicity:<br />
<br />
<haskell><br />
import Data.Char<br />
import Test.QuickCheck<br />
<br />
instance Arbitrary Char where<br />
arbitrary = choose ('\32', '\128')<br />
coarbitrary c = variant (ord c `rem` 4)<br />
</haskell><br />
<br />
Let's fire up GHCi (or Hugs) and try some generic properties (its nice<br />
that we can use the QuickCheck testing framework directly from the<br />
Haskell prompt). An easy one first, a [Char] is equal to itself:<br />
<br />
<haskell><br />
*A> quickCheck ((\s -> s == s) :: [Char] -> Bool)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
What just happened? QuickCheck generated 100 random [Char] values, and<br />
applied our property, checking the result was True for all cases.<br />
QuickCheck ''generated the test sets for us''!<br />
<br />
A more interesting property now: reversing twice is the identity:<br />
<br />
<haskell><br />
*A> quickCheck ((\s -> (reverse.reverse) s == s) :: [Char] -> Bool)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Great!<br />
<br />
== Testing take5 ==<br />
<br />
The first step to testing with QuickCheck is to work out some properties<br />
that are true of the function, for all inputs. That is, we need to find<br />
''invariants''.<br />
<br />
A simple invariant might be:<br />
<math>\forall~s~.~length~(take5~s)~=~5</math><br />
<br />
So let's write that as a QuickCheck property:<br />
<haskell><br />
\s -> length (take5 s) == 5<br />
</haskell><br />
<br />
Which we can then run in QuickCheck as:<br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) == 5)<br />
Falsifiable, after 0 tests:<br />
""<br />
</haskell><br />
<br />
Ah! QuickCheck caught us out. If the input string contains less than 5<br />
filterable characters, the resulting string will be less than 5<br />
characters long. So let's weaken the property a bit:<br />
<math>\forall~s~.~length~(take5~s)~\le~5</math><br />
<br />
That is, take5 returns a string of at most 5 characters long. Let's test<br />
this: <br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) <= 5)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Good!<br />
<br />
== Another property ==<br />
<br />
Another thing to check would be that the correct characters are<br />
returned. That is, for all returned characters, those characters are<br />
members of the set ['a','b','c','d','e'].<br />
<br />
We can specify that as:<br />
<math>\forall~s~.~\forall~e~.~e~\in~take5~s~\to~e~\in~[abcde] </math><br />
<br />
And in QuickCheck:<br />
<haskell><br />
*A> quickCheck (\s -> all (`elem` ['a'..'e']) (take5 s))<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
Excellent. So we can have some confidence that the function neither<br />
returns strings that are too long, nor includes invalid characters.<br />
<br />
== Coverage ==<br />
<br />
One issue with the default QuickCheck configuration, when testing<br />
[Char], is that the standard 100 tests isn't enough for our situation.<br />
In fact, QuickCheck never generates a String greater than 5 characters<br />
long, when using the supplied Arbtrary instance for Char! We can confirm<br />
this:<br />
<br />
<haskell><br />
*A> quickCheck (\s -> length (take5 s) < 5)<br />
OK, passed 100 tests.<br />
</haskell><br />
<br />
QuickCheck wastes its time generating different Chars, when what we<br />
really need is longer strings. One solution to this is to modify<br />
QuickCheck's default configuration to test deeper:<br />
<br />
<haskell><br />
deepCheck p = check (defaultConfig { configMaxTest = 10000}) p<br />
</haskell><br />
<br />
This instructs the system to find at least 10000 test cases before<br />
concluding that all is well. Let's check that it is generating longer<br />
strings:<br />
<br />
<haskell><br />
*A> deepCheck (\s -> length (take5 s) < 5)<br />
Falsifiable, after 125 tests:<br />
";:iD^*NNi~Y\\RegMob\DEL@krsx/=dcf7kub|EQi\DELD*"<br />
</haskell><br />
<br />
We can check the test data QuickCheck is generating using the<br />
'verboseCheck' hook. Here, testing on integers lists:<br />
<br />
<haskell><br />
*A> verboseCheck (\s -> length s < 5)<br />
0: []<br />
1: [0]<br />
2: []<br />
3: []<br />
4: []<br />
5: [1,2,1,1]<br />
6: [2]<br />
7: [-2,4,-4,0,0]<br />
Falsifiable, after 7 tests:<br />
[-2,4,-4,0,0]<br />
</haskell><br />
<br />
== Going further ==<br />
<br />
QuickCheck is effectively an embedded domain specific language for<br />
testing Haskell code, and allows for much more complex properties than<br />
those you've seen here to be tested. Some sources for further reading<br />
are:<br />
* [http://www.cse.unsw.edu.au/~dons/data/QuickCheck.html The QuickCheck source]<br />
** [http://mathburritos.org/code/darcsweb/browse?r=ghcQC;a=summary QuickCheck GHC batch script]<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/QuickCheck/Test-QuickCheck.html Library documentation]<br />
* [http://www.cse.unsw.edu.au/~dons/code/fps/tests/Properties.hs A large testsuite of QuickCheck code]<br />
* [http://www.cs.chalmers.se/~rjmh/QuickCheck/manual.html QuickCheck Manual]<br />
* Tutorial: [[QuickCheck as a test set generator]]<br />
* Tutorial: [[QuickCheck / GADT]]<br />
* More [http://haskell.org/haskellwiki/Research_papers/Testing_and_correctness research on correctness and testing] in Haskell<br />
<br />
* 2009 Blog article: [http://koweycode.blogspot.com/2009/07/some-ideas-for-practical-quickcheck.html some ideas for practical QuickCheck]<br />
* 2004 Paper [http://www.math.chalmers.se/~koen/pubs/entry-tt04-quickcheck.html QuickCheck: Specification-based Random Testing], Koen Claessen. Presentation at Summer Institute on Trends in Testing: Theory, Techniques and Tools, August 2004.<br />
* 2003 Paper [http://www.math.chalmers.se/~koen/pubs/entry-fop-quickcheck.html Specification Based Testing with QuickCheck], Koen Claessen and John Hughes. In Jeremy Gibbons and Oege de Moor (eds.), The Fun of Programming, Cornerstones of Computing, pp. 17--40, Palgrave, 2003.<br />
* 2002 Paper [http://www.cs.chalmers.se/~rjmh/Papers/QuickCheckST.ps Testing Monadic Programs with QuickCheck], Koen Claessen, John Hughes. SIGPLAN Notices 37(12): 47-59 (2002):<br />
* 2000 Paper [http://www.cs.chalmers.se/~koen/pubs/icfp00-quickcheck.ps QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs], Koen Claessen and John Hughes. In Proc. of International Conference on Functional Programming (ICFP), ACM SIGPLAN, 2000.<br />
<br />
Note, QuickCheck doesn't need to just be an embedded domain specific language for testing ''Haskell'' code. By making instances of Arbitrary for FFI types you can use Haskell and QuickCheck to check code in other languages.<br />
<br />
[[Category:Tutorials]]</div>Steve Chttps://wiki.haskell.org/index.php?title=Applications_and_libraries/Concurrency_and_parallelism&diff=29561Applications and libraries/Concurrency and parallelism2009-08-15T07:45:58Z<p>Steve C: Remove unnecessary newline</p>
<hr />
<div>'''Concurrent and Parallel Programming'''<br />
<br />
Haskell has been designed for parallel and concurrent programming since<br />
its inception. In particular, Haskell's [http://haskell.org/haskellwiki/Why_Haskell_matters purity] greatly simplifies reasoning about<br />
parallel programs. This page lists libraries and extensions for programming<br />
concurrent and parallel applications in Haskell. See also the<br />
[[Research_papers/Parallelism_and_concurrency|research papers]] on<br />
parallel and concurrent Haskell.<br />
<br />
Collected tutorials and information on multicore programming with Haskell:<br />
<br />
* [http://haskell.org/haskellwiki/Haskell_for_multicores Haskell for multicores]<br />
<br />
== SMP Haskell ==<br />
<br />
;[http://haskell.org/haskellwiki/GHC/Concurrency Multiprocessor GHC]<br />
:As of GHC 6.5, [http://www.haskell.org/ghc/dist/current/docs/users_guide/sec-using-smp.html GHC supports] running programs in [http://www.haskell.org/ghc/dist/current/docs/users_guide/lang-parallel.html parallel] on an SMP or multicore machine, and has been used successfully on up to 40 cpus.<br />
<br />
== Concurrency==<br />
<br />
;[http://haskell.org/haskellwiki/GHC/Concurrency Concurrent Haskell]<br />
:GHC has supported concurrency via forkIO and MVars for more than a decade, and its threading support [http://shootout.alioth.debian.org/gp4/benchmark.php?test=threadring&lang=all is very fast].<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/base-3.0.0.0/Control-Concurrent.html Documentation]<br />
* [http://haskell.org/haskellwiki/Concurrency_demos Examples]<br />
<br />
;[[WrapConc | Wrapped Concurrency]]<br />
:A wrapper around Control.Concurrency and Control.Exception that provides versions of forkIO that have more guarantees.<br />
<br />
== Concurrent channels ==<br />
<br />
;Control.Concurrent.Chan: channels allow threads to pass messages to each other<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/base-3.0.0.0/Control-Concurrent-Chan.html Documentation]<br />
<br />
== Software transactional memory ==<br />
<br />
;[[Software transactional memory|Software Transactional Memory]]<br />
:GHC supports a sophisticated version of software transactional memory. Software Transactional Memory (STM) is a new way to coordinate concurrent threads.<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/stm/Control-Concurrent-STM.html Documentation]<br />
* The paper [http://research.microsoft.com/~simonpj/papers/stm/index.htm Composable memory transactions]. <br />
* The paper [http://research.microsoft.com/~simonpj/papers/stm/lock-free.htm Lock-free data structures using Software Transactional Memory in Haskell] gives further examples of concurrent programming using STM.<br />
<br />
== Parallel strategies == <br />
<br />
;Control.Parallel<br />
:Parallel evaluation stratagies may be hinted, which is a much lighter way to achieve paralellism than to use explicit concurrency primitives.<br />
* [http://haskell.org/ghc/docs/latest/html/libraries/base-3.0.0.0/Control-Parallel-Strategies.html Documentation]. ''-- Broken?''<br />
* The [http://www.haskell.org/ghc/dist/current/docs/parallel/Control-Parallel-Strategies.html HEAD version] of the same contains more information.<br />
* The [http://www.macs.hw.ac.uk/~dsg/gph/papers/html/Strategies/strategies.html original paper]<br />
<br />
== Data Parallel Haskell ==<br />
<br />
;[http://haskell.org/haskellwiki/GHC/Data_Parallel_Haskell Data Parallel Haskell]<br />
:Implicitly parallel, high performance (nested) arrays, supporting large multicore programming.<br />
<br />
== Communicating Haskell Processes ==<br />
<br />
;[http://www.cs.kent.ac.uk/projects/ofa/chp/ CHP: Communicating Haskell Processes]<br />
:CHP is built on the ideas of CSP (Communicating Sequential Processes), featuring encapsulated parallel processes (no shared data!) communicating over synchronous channels. This is a very composable mode that also allows choice on communications, so that a process may offer to either read on one channel or write on another, but will only take the first that is available.<br />
<br />
== Actors ==<br />
<br />
;[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/actor Actors with multi-headed receive clauses]<br />
:Actor-based concurrency for Haskell<br />
<br />
== Transactional events ==<br />
<br />
;[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/transactional-events<br />
Transactional events, based on Concurrent ML ]<br />
:Transactional events for Haskell.<br />
<br />
== Unified events and threads ==<br />
<br />
;[http://www.seas.upenn.edu/~lipeng/homepage/unify.html User-level events and threads]<br />
:Ultra lightweight, user level threads for GHC Haskell, layered over epoll. Supports up to 10 million lightweight threads. Experimental.<br />
<br />
== Feedback-directed implicit parallelism ==<br />
<br />
[http://research.microsoft.com/~tharris/papers/2007-fdip.pdf Implicit parallelism in Haskell, and a feedback-directed mechanism to increase its granularity]<br />
<br />
== Parallel Haskell ==<br />
<br />
;[http://www.macs.hw.ac.uk/~dsg/gph/ GpH: Glasgow Parallel Haskell]<br />
:A complete, GHC-based implementation of the parallel Haskell extension GpH and of evaluation strategies is available. Extensions of the runtime-system and language to improve performance and support new platforms are under development.<br />
<br />
== Distributed Haskell ==<br />
<br />
;[http://www.macs.hw.ac.uk/~dsg/gdh/ GdH: Glasgow Distributed Haskell]<br />
:GdH supports distributed stateful interactions on multiple locations. It is a conservative extension of both Concurrent Haskell and GpH, enabling the distribution of the stateful IO threads of the former on the multiple locations of the latter. The programming model includes forking stateful threads on remote locations, explicit communication over channels, and distributed exception handling.<br />
<br />
;[http://www.macs.hw.ac.uk/~dubois/mhaskell Mobile Haskell (mHaskell)]<br />
:Mobile Haskell supports both strong and weak mobility of computations across open networks. The mobility primitives are higher-order polymorphic channels. Mid-level abstractions like remote evaluation, analogous to Java RMI, are readily constructed. High-level mobility skeletons like mobile map and mobile fold encapsulate common patterns of mobile computation.<br />
<br />
;[http://www.mathematik.uni-marburg.de/~eden Eden]<br />
:Eden extends Haskell with a small set of syntactic constructs for explicit process specification and creation. While providing enough control to implement parallel algorithms efficiently, it frees the programmer from the tedious task of managing low-level details by introducing automatic communication (via head-strict lazy lists), synchronisation, and process handling.<br />
<br />
== MPI ==<br />
<br />
;[http://www.foldr.org/~michaelw/hmpi/ hMPI]<br />
:hMPI is an acronym for HaskellMPI. It is a Haskell binding conforming to MPI (Message Passing Interface) standard 1.1/1.2. The programmer is in full control over the communication between the nodes of a cluster.<br />
<br />
== Distributed Haskell: Ports ==<br />
<br />
;[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/ports-0.4.3.2 The Haskell Ports Library (HPL)]<br />
:Ports are an abstraction for modelling variables whose values evolve over time without the need to resort to mutable variable, such as IORefs. More precisely, a port represents all values that a time-dependent variable successively takes as a stream, where each element of the stream corresponds to a state change - we can also say that a port represents a time series. Moreover, a port supports concurrent construction of the time series, or stream of values.<br />
<br />
== Modelling concurrent and distributed systems ==<br />
<br />
;[http://www.cs.kent.ac.uk/~cr3/HCPN/ HCPN: Haskell-Coloured Petri Nets]<br />
:Haskell-Coloured Petri Nets (HCPN) are an instance of high-level Petri Nets, in which anonymous tokens are replaced by Haskell data objects (and transitions can operate on that data, in addition to moving it around). This gives us a hybrid graphical/textual modelling formalism for Haskell, especially suited for modelling concurrent and distributed systems.</div>Steve C