https://wiki.haskell.org/api.php?action=feedcontributions&user=AdamLangley&feedformat=atomHaskellWiki - User contributions [en]2020-09-24T21:32:16ZUser contributionsMediaWiki 1.27.4https://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18891Dealing with binary data2008-02-02T23:38:23Z<p>AdamLangley: Add documentation about incremental parsing</p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== Bytestrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit characters. This has a<br />
number of useful properties like coverage of the Unicode space and laziness,<br />
however when it comes to dealing with bytewise data, <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code>—although bytestrings know their length and don't allow overflows, etc.<br />
<br />
There are two major flavours of bytestrings: strict and lazy. Strict<br />
bytestrings are exactly what you would expect—a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings; often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict, the code will read the whole of <code>stdin</code> into<br />
memory and then write it out. If the input was too large this would overflow<br />
the available memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason—misuse<br />
can crash you program!<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package. You should read the instructions on<br />
[http://haskell.org/haskellwiki/Cabal/How_to_install_a_Cabal_package how to install a Cabal package] if you haven't done so already.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python—it<br />
has its own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings; this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory: they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of laziness, might not be thrown exactly where you expect. If this<br />
is a problem, you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes three big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF-terminated list of<br />
numbers just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty <- isEmpty<br />
if empty<br />
then return []<br />
else do v <- getWord64be<br />
rest <- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a strict one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hask>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed—at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
That's it; it's otherwise the same as the <tt>Get</tt> monad.<br />
<br />
====Incremental parsing====<br />
<br />
If you have to deal with a protocol which isn't length prefixed, or otherwise<br />
chunkable, from the network then you are faced with the problem of knowing when<br />
you have enough data to parse something semantically useful. You could run a<br />
strict <tt>Get</tt> over what you have and catch the truncation result, but<br />
that means that you're parsing the data multiple times etc.<br />
<br />
Instead, you can use an incremental parser. There's an incremental version of<br />
the <tt>Get</tt> monad in <tt>Data.Binary.Strict.IncrementalGet</tt> (you'll<br />
need the <tt>binary-strict</tt> package).<br />
<br />
You use it as normal, but rather than returning an <tt>Either</tt> value, you<br />
get a [http://hackage.haskell.org/packages/archive/binary-strict/0.2.4/doc/html/Data-Binary-Strict-IncrementalGet.html#t%3AResult Result]. You need to go follow that link and look at the documentation for <tt>Result</tt>.<br />
<br />
It reflects the three outcomes of parsing possibly truncated data. Either the<br />
data is invalid as is, or it's complete, or it's truncated. In the truncated<br />
case you are given a function (called a continuation), to which you can pass<br />
more data, when you get it, and continue the parse. The continuation, again,<br />
returns a <tt>Result</tt> depending on the result of parsing the additional<br />
data as well.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.<br />
<br />
<table><br />
<tr><th>Name</th><th>C operator</th><th>Haskell</th></tr><br />
<tr><td>AND</td><td><tt>&amp;</tt></td><td><hask>.&.</hask></td></tr><br />
<tr><td>OR</td><td><tt>|</tt></td><td><hask>.|.</hask></td></tr><br />
<tr><td>XOR</td><td><tt>^</tt></td><td><hask>`xor`</hask></td></tr><br />
<tr><td>NOT</td><td><tt>&not;</tt></td><td><hask>`complement`</hask></td></tr><br />
<tr><td>Left shift</td><td><tt>&lt;&lt;</tt></td><td><hask>`shiftL`</hask></td></tr><br />
<tr><td>Right shift</td><td><tt>&gt;&gt;</tt></td><td><hask>`shiftR`</hask></td></tr><br />
</table><br />
<br />
====The <tt>BitGet</tt> monad====<br />
<br />
As an alternative to bit twiddling, you can also use the <tt>BitGet</tt> monad.<br />
This is another state-like monad, like <tt>Get</tt>, but here the state<br />
includes the current bit-offset in the input. This means that you can easily pull out<br />
unaligned data. Sadly, haddock is currently breaking when trying to generate the<br />
documentation for <tt>BitGet</tt> so I'll start with an example. Again, you'll<br />
need the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict] package installed.<br />
<br />
Here's a description of the header of a DNS packet, direct from RFC 1035:<br />
<br />
<pre> 1 1 1 1 1 1<br />
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ID |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
|QR| Opcode |AA|TC|RD|RA| Z | RCODE |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| QDCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ANCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| NSCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ARCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+</pre><br />
<br />
The actual fields don't matter, but here's a function for parsing it:<br />
<br />
<haskell>parseHeader :: G.Get Header<br />
parseHeader = do<br />
id <- G.getWord16be<br />
flags <- G.getByteString 2<br />
qdcount <- G.getWord16be >>= return . fromIntegral<br />
ancount <- G.getWord16be >>= return . fromIntegral<br />
nscount <- G.getWord16be >>= return . fromIntegral<br />
arcount <- G.getWord16be >>= return . fromIntegral<br />
<br />
let r = BG.runBitGet flags (do<br />
isquery <- BG.getBit<br />
opcode <- BG.getAsWord8 4 >>= parseEnum<br />
aa <- BG.getBit<br />
tc <- BG.getBit<br />
rd <- BG.getBit<br />
ra <- BG.getBit<br />
<br />
BG.getAsWord8 3<br />
rcode <- BG.getAsWord8 4 >>= parseEnum<br />
<br />
return $ Header id isquery opcode aa tc rd ra rcode qdcount ancount nscount arcount)<br />
<br />
case r of<br />
Left error -> fail error<br />
Right x -> return x</haskell><br />
<br />
Here you can see that only the second line (from the ASCII-art diagram) is<br />
parsed using <tt>BitGet</tt>. An outer <tt>Get</tt> monad is used for<br />
everything else and the bit fields are pulled out with<br />
<hask>getByteString</hask>. Again, <tt>BitGet</tt> is a strict monad and<br />
returns an <tt>Either</tt>, but it doesn't return the remaining bytestring,<br />
just because there's no obvious way to represent a bytestring of a fractional<br />
number of bytes.<br />
<br />
You can see the list of <tt>BitGet</tt> functions and their comments in the<br />
[http://darcs.imperialviolet.org/darcsweb.cgi?r=binary-strict;a=headblob;f=/src/Data/Binary/Strict/BitGet.hs source code].<br />
<br />
===Binary generation===<br />
<br />
In contrast to parsing binary data, you might want to generate it. This is the<br />
job of the <tt>Put</tt> monad. Follow along with the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Put.html documentation]<br />
if you like.<br />
<br />
The <tt>Put</tt> monad is another state-like monad, but the state is an offset<br />
into a series of buffers where the generated data is placed. All the buffer<br />
creation and handling is done for you, so you can just forget about it. It<br />
results in a lazy bytestring (so you can generate outputs that are larger than memory).<br />
<br />
Here's the reverse of our simple <tt>Get</tt> example:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Put<br />
<br />
serialiseSomething :: Put<br />
serialiseSomething = do<br />
putWord32be 1<br />
putWord16be 2<br />
putWord8 3<br />
<br />
main :: IO ()<br />
main = BL.putStr $ runPut serialiseSomething</haskell><br />
<br />
And running it shows that it's generating the correct serialisation:<br />
<br />
<pre>% runhaskell /tmp/example.hs| hexdump -C<br />
00000000 00 00 00 01 00 02 03 |.......|</pre><br />
<br />
If you want the output of <tt>runPut</tt> to be a strict bytestring, you just<br />
need to convert it with <hask>B.concat $ BL.toChunks $ runPut xyz</hask>.<br />
<br />
One limitation of <tt>Put</tt>, due to the nature of the <tt>Builder</tt> monad<br />
which it works with, is that you can't get the current offset into the output.<br />
This can be an issue with some formats which require you to encode byte offsets<br />
into the file. You have to calculate these byte offsets yourself.<br />
<br />
=== Other useful packages ===<br />
<br />
There are other packages which you should know about, but which are mostly<br />
covered by their documentation:<br />
<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/network-bytestring-0.1.1 network-bytestring]: for reading and writing bytestring from the network<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/zlib-0.4.0.2 zlib] and [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bzlib-0.4.0.1 bzlib]: for compressed formats<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/encoding-0.3 encoding]: for dealing with character encodings<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]: as an example of lazy parsing/serialisation</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Tutorials&diff=18766Tutorials2008-01-29T20:56:08Z<p>AdamLangley: Add link to DealingWithBinaryData</p>
<hr />
<div>==Introductions to Haskell==<br />
<br />
These are the recommended places to start learning, short of buying a textbook.<br />
<br />
=== Best places to start ===<br />
<br />
;[http://darcs.haskell.org/yaht/yaht.pdf Yet Another Haskell Tutorial]<br />
:By Hal Daume III et al. A recommended tutorial for Haskell that is still under construction but covers already much ground. Also a classic text. Now available [http://en.wikibooks.org/wiki/Haskell/YAHT as a wikibook].<br />
<br />
;[http://en.wikibooks.org/wiki/Haskell Haskell Wikibook] <br />
:A communal effort by several authors to produce the definitive Haskell textbook. Its very much a work in progress at the moment, and contributions are welcome.<br />
<br />
;[http://halogen.note.amherst.edu/~jdtang/scheme_in_48/tutorial/overview.html Write Yourself a Scheme in 48 Hours in Haskell]<br />
:A Haskell Tutorial, by Jonathan Tang. Most Haskell tutorials on the web seem to take a language-reference-manual approach to teaching. They show you the syntax of the language, a few language constructs, and then have you construct a few simple functions at the interactive prompt. The "hard stuff" of how to write a functioning, useful program is left to the end, or sometimes omitted entirely. This tutorial takes a different tack. You'll start off with command-line arguments and parsing, and progress to writing a fully-functional Scheme interpreter that implements a good-sized subset of R5RS Scheme. Along the way, you'll learn Haskell's I/O, mutable state, dynamic typing, error handling, and parsing features. By the time you finish, you should be fairly fluent in both Haskell and Scheme.<br />
<br />
=== More tutorials ===<br />
<br />
;[http://www.haskell.org/tutorial/ A Gentle Introduction to Haskell] :By Paul Hudak, John Peterson, and Joseph H. Fasel. The title is misleading. Some knowledge of another functional programming language is expected. The emphasis is on the type system and those features which are really new in Haskell (compared to other functional programming languages). A classic, but not for the faint of heart (it's not so gentle). Also available in [http://gorgonite.developpez.com/livres/traductions/haskell/gentle-haskell/ French] and [http://www.rsdn.ru/article/haskell/haskell_part1.xml Russian].<br />
<br />
;[[H-99: Ninety-Nine Haskell Problems]]<br />
:A collection of programming puzzles, with Haskell solutions. Solving these is a great way to get into Haskell programming.<br />
<br />
;[http://www.haskell.org/~pairwise/intro/intro.html Haskell Tutorial for C Programmers]<br />
:By Eric Etheridge. From the intro: "This tutorial assumes that the reader is familiar with C/C++, Python, Java, or Pascal. I am writing for you because it seems that no other tutorial was written to help students overcome the difficulty of moving from C/C++, Java, and the like to Haskell."<br />
<br />
;[http://www-106.ibm.com/developerworks/edu/os-dw-linuxhask-i.html Beginning Haskell] <br />
:From IBM developerWorks. This tutorial targets programmers of imperative languages wanting to learn about functional programming in the language Haskell. If you have programmed in languages such as C, Pascal, Fortran, C++, Java, Cobol, Ada, Perl, TCL, REXX, JavaScript, Visual Basic, or many others, you have been using an imperative paradigm. This tutorial provides a gentle introduction to the paradigm of functional programming, with specific illustrations in the Haskell 98 language. (Free registration required.)<br />
<br />
;[http://www.informatik.uni-bonn.de/~ralf/teaching/Hskurs_toc.html Online Haskell Course] <br />
:By Ralf Hinze (in German).<br />
<br />
;[http://www.cs.chalmers.se/~rjmh/tutorials.html Tutorial Papers in Functional Programming].<br />
:A collection of links to other Haskell tutorials, from John Hughes.<br />
<br />
;[http://www.cs.ou.edu/cs1323h/textbook/haskell.shtml Two Dozen Short Lessons in Haskell] <br />
:By Rex Page. A draft of a textbook on functional programming, available by ftp. It calls for active participation from readers by omitting material at certain points and asking the reader to attempt to fill in the missing information based on knowledge they have already acquired. The missing information is then supplied on the reverse side of the page. <br />
<br />
;[ftp://ftp.geoinfo.tuwien.ac.at/navratil/HaskellTutorial.pdf Haskell-Tutorial] <br />
:By Damir Medak and Gerhard Navratil. The fundamentals of functional languages for beginners. <br />
<br />
;[http://video.s-inf.de/#FP.2005-SS-Giesl.(COt).HD_Videoaufzeichnung Video Lectures] <br />
:Lectures (in English) by Jürgen Giesl. About 30 hours in total, and great for learning Haskell. The lectures are 2005-SS-FP.V01 through 2005-SS-FP.V26. Videos 2005-SS-FP.U01 through 2005-SS-FP.U11 are exercise answer sessions, so you probably don't want those.<br />
<br />
;[http://www.cs.utoronto.ca/~trebla/fp/ Albert's Functional Programming Course] <br />
:A 15 lesson introduction to most aspects of Haskell.<br />
<br />
;[http://www.iceteks.com/articles.php/haskell/1 Introduction to Haskell]<br />
:By Chris Dutton, An "attempt to bring the ideas of functional programming to the masses here, and an experiment in finding ways to make it easy and interesting to follow".<br />
<br />
;[http://www.csc.depauw.edu/~bhoward/courses/0203Spring/csc122/haskintro/ An Introduction to Haskell]<br />
:A brief introduction, by Brian Howard.<br />
<br />
;[http://web.syntaxpolice.org/lectures/haskellTalk/slides/index.html Introduction to Haskell]<br />
:By Isaac Jones (2003).<br />
<br />
;[http://www.linuxjournal.com/article/9096 Translating Haskell into English]<br />
:By Shannon Behrens, a glimpse of the Zen of Haskell, without requiring that they already be Haskell converts.<br />
<br />
;[http://www.shlomifish.org/lecture/Perl/Haskell/slides/ Haskell for Perl Programmers]<br />
:Brief introduction to Haskell, with a view to what perl programmers are interested in<br />
<br />
;[http://lisperati.com/haskell/ How To Organize a Picnic on a Computer]<br />
:Fun introduction to Haskell, step by step building of a program to seat people at a planned picnic, based on their similarities using data from a survey and a map of the picnic location.<br />
<br />
;[http://cs.wwc.edu/KU/PR/Haskell.html Haskell Tutorial]<br />
<br />
<br />
== Motivation for using Haskell ==<br />
<br />
;[http://www.md.chalmers.se/~rjmh/Papers/whyfp.html Why Functional Programming Matters] <br />
:By [http://www.md.chalmers.se/~rjmh/ John Hughes], The Computer Journal, Vol. 32, No. 2, 1989, pp. 98 - 107. Also in: David A. Turner (ed.): Research Topics in Functional Programming, Addison-Wesley, 1990, pp. 17 - 42.<BR> Exposes the advantages of functional programming languages. Demonstrates how higher-order functions and lazy evaluation enable new forms of modularization of programs.<br />
<br />
;[[Why Haskell matters]] <br />
:Discussion of the advantages of using Haskell in particular. An excellent article.<br />
<br />
;[http://www.cs.ukc.ac.uk/pubs/1997/224/index.html Higher-order + Polymorphic = Reusable] <br />
:By [http://www.cs.ukc.ac.uk/people/staff/sjt/index.html Simon Thompson]. Unpublished, May 1997.<BR> <STRONG>Abstract:</STRONG> This paper explores how certain ideas in object oriented languages have their correspondents in functional languages. In particular we look at the analogue of the iterators of the C++ standard template library. We also give an example of the use of constructor classes which feature in Haskell 1.3 and Gofer.<br />
<br />
;[http://www-128.ibm.com/developerworks/java/library/j-cb07186.html Explore functional programming with Haskell]<br />
:Introduction to the benefits of functional programming in Haskell by Bruce Tate.<br />
<br />
== Blog articles ==<br />
<br />
There are a large number of tutorials covering diverse Haskell topics<br />
published as blogs. Some of the best of these articles are collected<br />
here:<br />
<br />
;[[Blog articles]]<br />
<br />
==Practical Haskell==<br />
<br />
These tutorials examine using Haskell to writing complex real-world applications<br />
<br />
;[http://research.microsoft.com/%7Esimonpj/Papers/marktoberdorf Tackling the awkward squad: monadic input/output, concurrency, exceptions, and foreign-language calls in Haskell]<br />
:Simon Peyton Jones. Presented at the 2000 Marktoberdorf Summer School. In "Engineering theories of software construction", ed Tony Hoare, Manfred Broy, Ralf Steinbruggen, IOS Press, ISBN 1-58603-1724, 2001, pp47-96. The standard reference for monadic IO in GHC/Haskell. <br><strong>Abstract:</strong>Functional programming may be beautiful, but to write real applications we must grapple with awkward real-world issues: input/output, robustness, concurrency, and interfacing to programs written in other languages.<br />
<br />
;[[Hitchhikers Guide to the Haskell]]<br />
: Tutorial for C/Java/OCaml/... programers by Dmitry Astapov. From the intro: "This text intends to introduce the reader to the practical aspects of Haskell from the very beginning (plans for the first chapters include: I/O, darcs, Parsec, QuickCheck, profiling and debugging, to mention a few)".<br />
<br />
;[http://haskell.org/haskellwiki/IO_inside Haskell I/O inside: Down the Rabbit's Hole]<br />
:By Bulat Ziganshin (2006), a comprehensive tutorial on using IO monad.<br />
<br />
;[http://web.archive.org/web/20060622030538/http://www.reid-consulting-uk.ltd.uk/docs/ffi.html A Guide to Haskell's Foreign Function Interface]<br />
:A guide to using the foreign function interface extension, using the rich set of functions in the Foreign libraries, design issues, and FFI preprocessors.<br />
<br />
;[http://blogs.nubgames.com/code/?p=22 Haskell IO for imperative programmers]<br />
:A short introduction to IO from the perspective of an imperative programmer.<br />
<br />
;[[A brief introduction to Haskell|A Brief Introduction to Haskell]]<br />
:A translation of the article, [http://www.cs.jhu.edu/~scott/pl/lectures/caml-intro.html Introduction to OCaml], to Haskell.<br />
<br />
;[[Roll your own IRC bot]]<br />
:This tutorial is designed as a practical guide to writing real world code in Haskell and hopes to intuitively motivate and introduce some of the advanced features of Haskell to the novice programmer, including monad transformers. Our goal is to write a concise, robust and elegant IRC bot in Haskell.<br />
<br />
;[http://j-van-thiel.speedlinq.nl/EddyAhmed/GladeGtk2Hs.html Developing Gnome Apps with Glade]<br />
:For the absolute beginner in both Glade and Gtk2Hs and covers the basics of Glade and how to access a .glade file and widgets in Gtk2Hs. Estimated learning time: 2 hours.<br />
<br />
;Applications of Functional Programming<br />
:Colin Runciman and David Wakeling (ed.), UCL Press, 1995, ISBN 1-85728-377-5 HB. From the cover:<blockquote>This book is unique in showcasing real, non-trivial applications of functional programming using the Haskell language. It presents state-of-the-art work from the FLARE project and will be an invaluable resource for advanced study, research and implementation.</blockquote><br />
<br />
;[[DealingWithBinaryData]] a guide to bytestrings, the various <tt>Get</tt> monads and the <tt>Put</tt> monad.<br />
<br />
===Testing===<br />
<br />
;[http://blog.moertel.com/articles/2006/10/31/introductory-haskell-solving-the-sorting-it-out-kata Small overview of QuickCheck]<br />
<br />
;[[Introduction to QuickCheck]]<br />
<br />
==Reference material==<br />
<br />
;[http://haskell.org/haskellwiki/Category:Tutorials A growing list of Haskell tutorials on a diverse range of topics]<br />
:Available on this wiki<br />
<br />
;[http://undergraduate.csse.uwa.edu.au/units/230.301/lectureNotes/tourofprelude.html A Tour of the Haskell Prelude (basic functions)] <br />
:By Bernie Pope and Arjan van IJzendoorn.<br />
<br />
;[http://cs.anu.edu.au/Student/comp1100/haskell/tourofsyntax.html Tour of the Haskell Syntax] <br />
:By Arjan van IJzendoorn.<br />
<br />
;[http://zvon.org/other/haskell/Outputglobal/index.html Haskell Reference] <br />
:By Miloslav Nic.<br />
<br />
;[http://members.chello.nl/hjgtuyl/tourdemonad.html A tour of the Haskell Monad functions]<br />
:By Henk-Jan van Tuyl.<br />
<br />
;[http://www.cse.unsw.edu.au/~en1000/haskell/inbuilt.html Useful Haskell functions]<br />
:An explanation for beginners of many Haskell functions that are predefined in the Haskell Prelude.<br />
<br />
;[http://www.cs.chalmers.se/Cs/Grundutb/Kurser/d1pt/d1pta/ListDoc/ Haskell's Standard List Functions]<br />
:A tour of the standard Haskell functions, directed by what you want to achieve<br />
<br />
;[http://haskell.org/ghc/docs/latest/html/libraries/ Documentation for the standard libraries]<br />
:Complete documentation of the standard Haskell libraries.<br />
<br />
;[http://www.haskell.org/haskellwiki/Category:Idioms Haskell idioms]<br />
:A collection of articles describing some common Haskell idioms. Often quite advanced.<br />
<br />
;[http://www.haskell.org/haskellwiki/Blow_your_mind Useful idioms]<br />
:A collection of short, useful Haskell idioms.<br />
<br />
;[http://www.haskell.org/haskellwiki/Programming_guidelines Programming guidelines]<br />
:Some Haskell programming and style conventions.<br />
<br />
;[http://www.md.chalmers.se/~rjmh/Combinators/LightningTour/index.htm Lightning Tour of Haskell]<br />
:By John Hughes, as part of a Chalmers programming course<br />
<br />
;[http://www.cs.chalmers.se/~augustss/AFP/manuals/haskeller.dvi.gz The Little Haskeller] <br />
:By Cordelia Hall and John Hughes. 9. November 1993, 26 pages. An introduction using the Chalmers Haskell B interpreter (hbi). Beware that it relies very much on the user interface of hbi which is quite different for other Haskell systems, and the tutorials cover Haskell 1.2 , not Haskell 98.<br />
<br />
;[http://www.cs.uu.nl/people/jeroen/courses/fp-eng.pdf Functional Programming]<br />
:By Jeroen Fokker, 1995. (153 pages, 600 KB). Textbook for learning functional programming with Gofer (an older implementation of Haskell). Here without Chapters&nbsp;6 and&nbsp;7.<br />
<br />
== Comparisons to other languages ==<br />
<br />
Articles constrasting feature of Haskell with other languages.<br />
<br />
;[http://programming.reddit.com/goto?id=nq1k Haskell versus Scheme]<br />
:Mark C. Chu-Carroll, Haskell and Scheme: Which One and Why?<br />
<br />
;[http://wiki.python.org/moin/PythonVsHaskell Comparing Haskell and Python]<br />
:A short overview of similarities and differences between Haskell and Python.<br />
<br />
;[http://programming.reddit.com/goto?id=nwm2 Monads in OCaml]<br />
:Syntax extension for monads in OCaml<br />
<br />
;[http://www.shlomifish.org/lecture/Perl/Haskell/slides/ Haskell for Perl programmers]<br />
:Short intro for perlers<br />
<br />
;[[A_brief_introduction_to_Haskell|Introduction to Haskell]] versus [http://www.cs.jhu.edu/~scott/pl/lectures/caml-intro.html Introduction to OCaml].<br />
<br />
;[http://www.thaiopensource.com/relaxng/derivative.html An algorithm for RELAX NG validation]<br />
:by James Clark (of RELAX NG fame). Describes an algorithm for validating an XML document against a RELAX NG schema, uses Haskell to describe the algorithm. The algorithm in Haskell and Java is then [http://www.donhopkins.com/drupal/node/117 discussed here].<br />
<br />
;[http://mult.ifario.us/articles/2006/10/11/first-steps-with-haskell-for-web-applications Haskell + FastCGI versus Ruby on Rails]<br />
:A short blog entry documenting performance results with ruby on rails and Haskell with fastcgi<br />
<br />
;[http://haskell.org/papers/NSWC/jfp.ps Haskell vs. Ada vs. C++ vs. Awk vs. ..., An Experiment in Software Prototyping Productivity] (postscript)<br />
:Paul Hudak and Mark P. Jones, 16 pages.<blockquote>Description of the results of an experiment in which several conventional programming languages, together with the functional language Haskell, were used to prototype a Naval Surface Warfare Center requirement for Geometric Region Servers. The resulting programs and development metrics were reviewed by a committee chosen by the US Navy. The results indicate that the Haskell prototype took significantly less time to develop and was considerably more concise and easier to understand than the corresponding prototypes written in several different imperative languages, including Ada and C++. </blockquote> <br />
<br />
;[http://www.osl.iu.edu/publications/prints/2003/comparing_generic_programming03.pdf A Comparative Study of Language Support for Generic Programming] (pdf)<br />
:Ronald Garcia, Jaakko Jrvi, Andrew Lumsdaine, Jeremy G. Siek, and Jeremiah Willcock. In Proceedings of the 2003 ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications (OOPSLA'03), October 2003.<blockquote>An interesting comparison of generic programming support across languages, including: Haskell, SML, C++, Java, C#. Haskell supports all constructs described in the paper -- the only language to do so. </blockquote><br />
<br />
;[http://homepages.inf.ed.ac.uk/wadler/realworld/index.html Functional Programming in the Real World]<br />
:A list of functional programs applied to real-world tasks. The main criterion for being real-world is that the program was written primarily to perform some task, not primarily to experiment with functional programming. Functional is used in the broad sense that includes both `pure' programs (no side effects) and `impure' (some use of side effects). Languages covered include CAML, Clean, Erlang, Haskell, Miranda, Scheme, SML, and others.<br />
<br />
;[http://www.defmacro.org/ramblings/lisp-in-haskell.html Lisp in Haskell]<br />
:Writing A Lisp Interpreter In Haskell, a tutorial<br />
<br />
== Teaching Haskell ==<br />
<br />
;[http://www.cs.ukc.ac.uk/pubs/1997/208/index.html Where do I begin? A problem solving approach to teaching functional programming]<br />
:By [http://www.cs.ukc.ac.uk/people/staff/sjt/index.html Simon Thompson]. In Krzysztof Apt, Pieter Hartel, and Paul Klint, editors, First International Conference on Declarative Programming Languages in Education. Springer-Verlag, September 1997. <br> <STRONG>Abstract:</STRONG> This paper introduces a problem solving method for teaching functional programming, based on Polya's `How To Solve It', an introductory investigation of mathematical method. We first present the language independent version, and then show in particular how it applies to the development of programs in Haskell. The method is illustrated by a sequence of examples and a larger case study. <br />
<br />
;[http://www.cs.ukc.ac.uk/pubs/1995/214/index.html Functional programming through the curriculum]<br />
:By [http://www.cs.ukc.ac.uk/people/staff/sjt/index.html Simon Thompson] and Steve Hill. In Pieter H. Hartel and Rinus Plasmeijer, editors, Functional Programming Languages in Education, LNCS 1022, pages 85-102. Springer-Verlag, December 1995. <br> <STRONG>Abstract:</STRONG> This paper discusses our experience in using a functional language in topics across the computer science curriculum. After examining the arguments for taking a functional approach, we look in detail at four case studies from different areas: programming language semantics, machine architectures, graphics and formal languages. <br />
<br />
;[http://www.cse.unsw.edu.au/~chak/papers/CK02a.html The Risks and Benefits of Teaching Purely Functional Programming in First Year]<br />
:By [http://www.cse.unsw.edu.au/~chak Manuel M. T. Chakravarty] and [http://www.cse.unsw.edu.au/~keller Gabriele Keller]. Journal of Functional Programming 14(1), pp 113-123, 2004. An earlier version of this paper was presented at Functional and Declarative Programming in Education (FDPE02). <br> <strong>Abstract</strong> We argue that teaching purely functional programming as such in freshman courses is detrimental to both the curriculum as well as to promoting the paradigm. Instead, we need to focus on the more general aims of teaching elementary techniques of programming and essential concepts of computing. We support this viewpoint with experience gained during several semesters of teaching large first-year classes (up to 600 students) in Haskell. These classes consisted of computer science students as well as students from other disciplines. We have systematically gathered student feedback by conducting surveys after each semester. This article contributes an approach to the use of modern functional languages in first year courses and, based on this, advocates the use of functional languages in this setting.<br />
<br />
<br />
==Using monads==<br />
<br />
See also the [[Monad]] HaskellWiki page.<br />
<br />
<br />
===Recommended tutorials===<br />
<br />
;[http://www.haskell.org/all_about_monads/html/index.html All About Monads] <br />
:By Jeff Newbern. This tutorial aims to explain the concept of a monad and its application to functional programming in a way that is easy to understand and useful to beginning and intermediate Haskell programmers. Familiarity with the Haskell language is assumed, but no prior experience with monads is required. <br />
<br />
;[[Monads as computation]]<br />
:A tutorial which gives a broad overview to motivate the use of monads as an abstraction in functional programming and describe their basic features. It makes an attempt at showing why they arise naturally from some basic premises about the design of a library.<br />
<br />
;[[Monads as containers]]<br />
:A tutorial describing monads from a rather different perspective: as an abstraction of container-types, rather than an abstraction of types of computation.<br />
<br />
;[http://uebb.cs.tu-berlin.de/~magr/pub/Transformers.en.html Monad Transformers Step by Step]<br />
:By Martin Grabm&uuml;ller. A small tutorial on using monad transformers. In contrast to others found on the web, it concentrates on using them, not on their implementation.<br />
<br />
===Parser===<br />
<br />
;[http://www.haskell.org/sitewiki/images/c/c6/ICMI45-paper-en.pdf The Parser monad and other monad (i.e. a monad with state and I/O string)]. <br />
:The parser monad is used to build modular, flexible, parsers. <br />
<br />
;[http://www.haskell.org/sitewiki/images/c/c6/ICMI45-paper-en.pdf How to build a monadic interpreter in one day] (pdf)<br />
:By Dan Popa. A small tutorial on how to build a language in one day, using the Parser Monad in the front end and a monad with state and I/O string in the back end. Read it if you are interested in learning: <br />
:# language construction and <br />
:# interpreter construction<br />
<br />
===More tutorials===<br />
<br />
;[http://stefan-klinger.de/files/monadGuide.pdf The Haskell Programmer's Guide to the IO Monad - Don't Panic.] <br />
:By Stefan Klinger. This report scratches the surface of category theory, an abstract branch of algebra, just deep enough to find the monad structure. It seems well written.<br />
<br />
;[http://www.prairienet.org/~dsb/monads.htm A (hopefully) painless introduction to monads] <br />
:By Dan Bensen. A straightforward beginner's guide with intuitive explanations and examples.<br />
<br />
;[http://www-users.mat.uni.torun.pl/~fly/materialy/fp/haskell-doc/Monads.html What the hell are Monads?] <br />
:By Noel Winstanley. A basic introduction to monads, monadic programming and IO. This introduction is presented by means of examples rather than theory, and assumes a little knowledge of Haskell. <br />
<br />
;[http://www.engr.mun.ca/~theo/Misc/haskell_and_monads.htm Monads for the Working Haskell Programmer -- a short tutorial]<br />
:By Theodore Norvell. <br />
<br />
;[http://sigfpe.blogspot.com/2006/08/you-could-have-invented-monads-and.html You Could Have Invented Monads! (And Maybe You Already Have.)]<br />
:A short tutorial on monads, introduced from a pragmatic approach, with less category theory references <br />
<br />
;[http://www.cs.chalmers.se/~augustss/AFP/monads.html Systematic Design of Monads]<br />
:By John Hughes and Magnus Carlsson. Many useful monads can be designed in a systematic way, by successively adding facilities to a trivial monad. The capabilities that can be added in this way include state, exceptions, backtracking, and output. Here we give a brief description of the trivial monad, each kind of extension, and sketches of some interesting operations that each monad supports.<br />
<br />
;[[Meet Bob The Monadic Lover]]<br />
:By Andrea Rossato. A by-the-author-supposed-to-be funny and short introduction to Monads, with code but without any reference to category theory: what monads look like and what they are useful for, from the perspective of a ... lover. (There is also the slightly more serious [[The Monadic Way]] by the same author.)<br />
<br />
;[http://www.haskell.org/pipermail/haskell-cafe/2006-November/019190.html Monstrous Monads]<br />
:Andrew Pimlott's humourous introduction to monads, using the metaphor of "monsters".<br />
<br />
;Computational monads [http://programming.reddit.com/info/ox6s/comments/coxiv part 1] and [http://programming.reddit.com/info/ox6s/comments/coxoh part 2].<br />
<br />
;[http://www.loria.fr/~kow/monads/index.html Of monads and space suits]<br />
:By Eric Kow.<br />
<br />
;[[The Monadic Way]]<br />
<br />
;Computational monads [http://programming.reddit.com/info/ox6s/comments/coxiv part 1] and [http://programming.reddit.com/info/ox6s/comments/coxoh part 2].<br />
<br />
;[http://www.alpheccar.org/fr/posts/show/60 Three kind of monads] : sequencing, side effects or containers<br />
<br />
;[[Simple monad examples]]<br />
<br />
;[http://en.wikipedia.org/wiki/Monads_in_functional_programming Article on monads on Wikipedia]<br />
<br />
;[[IO inside]] page<br />
:Explains why I/O in Haskell is implemented with a monad.<br />
<br />
;[http://haskell.org/haskellwiki/Blog_articles#Monads Blog articles]<br />
<br />
See also [[Research papers/Monads and arrows]]<br />
<br />
==Workshops on advanced functional programming==<br />
<br />
;[http://compilers.iecc.com/comparch/article/95-04-024 Advanced Functional Programming: 1st International Spring School on Advanced Functional Programming Techniques], Bastad, Sweden, May 24 - 30, 1995. Tutorial Text (Lecture Notes in Computer Science) <br />
<br />
;[http://www.cse.ogi.edu/PacSoft/conf/summerschool96.html Advanced Functional Programming: 2nd International School], Olympia, Wa, Usa, August 26-30, 1996 Tutorial Text (Lecture Notes in Computer Science) <br />
<br />
;[http://alfa.di.uminho.pt/~afp98/ Advanced Functional Programming: 3rd International School], AFP'98, Braga, Portugal, September 12-19, 1998, Revised Lectures (Lecture Notes in Computer Science) <br />
<br />
;[http://www.cs.uu.nl/~johanj/afp/afp4/ Advanced Functional Programming: 4th International School], AFP 2002, Oxford, UK, August 19-24, 2002, Revised Lectures (Lecture Notes in Computer Science) <br />
<br />
;[http://www.cs.ut.ee/afp04/ Advanced Functional Programming: 5th International School], AFP 2004, Tartu, Estonia, August 14-21, 2004, Revised Lectures (Lecture Notes in Computer Science) <br />
<br />
More advanced materials available from the [[Conferences|conference proceedings]], and the [[Research papers]] collection.<br />
<br />
<br />
[[Category:Tutorials]]</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18756Dealing with binary data2008-01-29T06:31:07Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data, <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has its own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF terminated list of<br />
numbers list just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty <- isEmpty<br />
if empty<br />
then return []<br />
else do v <- getWord64be<br />
rest <- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a strict one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hask>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed - at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
That's it; it's otherwise the same as the <tt>Get</tt> monad.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.<br />
<br />
<table><br />
<tr><th>Name</th><th>C operator</th><th>Haskell</th></tr><br />
<tr><td>AND</td><td><tt>&amp;</tt></td><td><hask>.&.</hask></td></tr><br />
<tr><td>OR</td><td><tt>|</tt></td><td><hask>.|.</hask></td></tr><br />
<tr><td>XOR</td><td><tt>^</tt></td><td><hask>`xor`</hask></td></tr><br />
<tr><td>NOT</td><td><tt>&not;</tt></td><td><hask>`complement`</hask></td></tr><br />
<tr><td>Left shift</td><td><tt>&lt;&lt;</tt></td><td><hask>`shiftL`</hask></td></tr><br />
<tr><td>Right shift</td><td><tt>&gt;&gt;</tt></td><td><hask>`shiftR`</hask></td></tr><br />
</table><br />
<br />
====The <tt>BitGet</tt> monad====<br />
<br />
As an alternative to bit twiddling, you can also use the <tt>BitGet</tt> monad.<br />
This is another state-like monad, like <tt>Get</tt>, but here the state<br />
includes the current bit-offest in the input. This means that you can easily pull out<br />
unaligned data. Sadly, haddock is currently breaking when trying to generate the<br />
documentation for <tt>BitGet</tt> so I'll start with an example. Again, you'll<br />
need the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict] package installed.<br />
<br />
Here's a description of the header of a DNS packet, direct from RFC 1035:<br />
<br />
<pre> 1 1 1 1 1 1<br />
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ID |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
|QR| Opcode |AA|TC|RD|RA| Z | RCODE |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| QDCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ANCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| NSCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ARCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+</pre><br />
<br />
The actual fields don't matter, but here's a function for parsing it:<br />
<br />
<haskell>parseHeader :: G.Get Header<br />
parseHeader = do<br />
id <- G.getWord16be<br />
flags <- G.getByteString 2<br />
qdcount <- G.getWord16be >>= return . fromIntegral<br />
ancount <- G.getWord16be >>= return . fromIntegral<br />
nscount <- G.getWord16be >>= return . fromIntegral<br />
arcount <- G.getWord16be >>= return . fromIntegral<br />
<br />
let r = BG.runBitGet flags (do<br />
isquery <- BG.getBit<br />
opcode <- BG.getAsWord8 4 >>= parseEnum<br />
aa <- BG.getBit<br />
tc <- BG.getBit<br />
rd <- BG.getBit<br />
ra <- BG.getBit<br />
<br />
BG.getAsWord8 3<br />
rcode <- BG.getAsWord8 4 >>= parseEnum<br />
<br />
return $ Header id isquery opcode aa tc rd ra rcode qdcount ancount nscount arcount)<br />
<br />
case r of<br />
Left error -> fail error<br />
Right x -> return x</haskell><br />
<br />
Here you can see that only the second line (from the ASCII-art diagram) is<br />
parsed using <tt>BitGet</tt>. An outer <tt>Get</tt> monad is used for<br />
everythign else and the bit fields are pulled out with<br />
<hask>getByteString</hask>. Again, <tt>BitGet</tt> is a strict monad and<br />
returns an <tt>Either</tt>, but it doesn't return the remaining bytestring,<br />
just because there's no obvious way to represent a bytestring of a fractional<br />
number of bytes.<br />
<br />
You can see the list of <tt>BitGet</tt> functions and their comments in the<br />
[http://darcs.imperialviolet.org/darcsweb.cgi?r=binary-strict;a=headblob;f=/src/Data/Binary/Strict/BitGet.hs source code].<br />
<br />
===Binary generation===<br />
<br />
In contrast to parsing binary data, you might want to generate it. This is the<br />
job of the <tt>Put</tt> monad. Follow along with the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Put.html documentation]<br />
if you like.<br />
<br />
The <tt>Put</tt> monad is another state-like monad, but the state is an offset<br />
into a series of buffers where the generated data is placed. All the buffer<br />
creation and handling is done for you, so you can just forget about it. It<br />
results in a lazy bytestring (so you can generate outputs that are larger than memory).<br />
<br />
Here's the reverse of our simple <tt>Get</tt> example:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Put<br />
<br />
serialiseSomething :: Put<br />
serialiseSomething = do<br />
putWord32be 1<br />
putWord16be 2<br />
putWord8 3<br />
<br />
main :: IO ()<br />
main = BL.putStr $ runPut serialiseSomething</haskell><br />
<br />
And running it shows that it's generating the correct serialisation:<br />
<br />
<pre>% runhaskell /tmp/example.hs| hexdump -C<br />
00000000 00 00 00 01 00 02 03 |.......|</pre><br />
<br />
If you want the output of <tt>runPut</tt> to be a strict bytestring, you just<br />
need to convert it with <hask>B.concat $ BL.toChunks $ runPut xyz</hask>.<br />
<br />
One limitation of <tt>Put</tt>, due to the nature of the <tt>Builder</tt> monad<br />
which it works with, is that you can't get the current offset into the output.<br />
This can be an issue with some formats which require you to encode byte offsets<br />
into the file. You have to calculate these byte offsets yourself.<br />
<br />
=== Other useful packages ===<br />
<br />
There are other packages which you should know about, but which are mostly<br />
covered by their documentation:<br />
<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/network-bytestring-0.1.1 network-bytestring]: for reading and writing bytestring from the network<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/zlib-0.4.0.2 zlib] and [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bzlib-0.4.0.1 bzlib]: for compressed formats<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/encoding-0.3 encoding]: for dealing with charactor encodings<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]: as an example of lazy parsing/serialisation</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18755Dealing with binary data2008-01-29T06:21:57Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF terminated list of<br />
numbers list just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty <- isEmpty<br />
if empty<br />
then return []<br />
else do v <- getWord64be<br />
rest <- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a string one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hask>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed - at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.<br />
<br />
<table><br />
<tr><th>Name</th><th>C operator</th><th>Haskell</th></tr><br />
<tr><td>AND</td><td><tt>&amp;</tt></td><td><hask>.&.</hask></td></tr><br />
<tr><td>OR</td><td><tt>|</tt></td><td><hask>.|.</hask></td></tr><br />
<tr><td>XOR</td><td><tt>^</tt></td><td><hask>`xor`</hask></td></tr><br />
<tr><td>NOT</td><td><tt>&not;</tt></td><td><hask>`complement`</hask></td></tr><br />
<tr><td>Left shift</td><td><tt>&lt;&lt;</tt></td><td><hask>`shiftL`</hask></td></tr><br />
<tr><td>Right shift</td><td><tt>&gt;&gt;</tt></td><td><hask>`shiftR`</hask></td></tr><br />
</table><br />
<br />
====The <tt>BitGet</tt> monad====<br />
<br />
As an alternative to bit twiddling, you can also use the <tt>BitGet</tt> monad.<br />
This is another state-like monad, like <tt>Get</tt>, but here the state<br />
includes the current bit-offest in the input. This means that you can easily pull out<br />
unaligned data. Sadly, haddock is current breaking when trying to generate the<br />
documentation for <tt>BitGet</tt> so I'll start with an example.<br />
<br />
Here's a description of the header of a DNS packet, direct from RFC 1035:<br />
<br />
<pre> 1 1 1 1 1 1<br />
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ID |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
|QR| Opcode |AA|TC|RD|RA| Z | RCODE |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| QDCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ANCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| NSCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ARCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+</pre><br />
<br />
The actual fields don't matter, but here's a function for parsing it:<br />
<br />
<haskell>parseHeader :: G.Get Header<br />
parseHeader = do<br />
id <- G.getWord16be<br />
flags <- G.getByteString 2<br />
qdcount <- G.getWord16be >>= return . fromIntegral<br />
ancount <- G.getWord16be >>= return . fromIntegral<br />
nscount <- G.getWord16be >>= return . fromIntegral<br />
arcount <- G.getWord16be >>= return . fromIntegral<br />
<br />
let r = BG.runBitGet flags (do<br />
isquery <- BG.getBit<br />
opcode <- BG.getAsWord8 4 >>= parseEnum<br />
aa <- BG.getBit<br />
tc <- BG.getBit<br />
rd <- BG.getBit<br />
ra <- BG.getBit<br />
<br />
BG.getAsWord8 3<br />
rcode <- BG.getAsWord8 4 >>= parseEnum<br />
<br />
return $ Header id isquery opcode aa tc rd ra rcode qdcount ancount nscount arcount)<br />
<br />
case r of<br />
Left error -> fail error<br />
Right x -> return x</haskell><br />
<br />
Here you can see that only the second line (from the ASCII-art diagram) is<br />
parsed using <tt>BitGet</tt>. An outer <tt>Get</tt> monad is used for<br />
everythign else and the bit fields are pulled out with<br />
<hask>getByteString</hask>. Again, <tt>BitGet</tt> is a strict monad and<br />
returns an <tt>Either</tt>, but it doesn't return the remaining bytestring,<br />
just because there's no obvious way to represent a bytestring of a fractional<br />
number of bytes.<br />
<br />
You can see the list of <tt>BitGet</tt> functions and their comments in the<br />
[http://darcs.imperialviolet.org/darcsweb.cgi?r=binary-strict;a=headblob;f=/src/Data/Binary/Strict/BitGet.hs source code].<br />
<br />
===Binary generation===<br />
<br />
In contrast to parsing binary data, you might want to generate it. This is the<br />
job of the <tt>Put</tt> monad. Follow along with the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Put.html documentation]<br />
if you like.<br />
<br />
The <tt>Put</tt> monad is another state-like monad, but the state is an offset<br />
into a series of buffers where the generated data is placed. All the buffer<br />
creation and handling is done for you, so you can just forget about it. It<br />
results in a lazy bytestring (so you can generate output's larger than memory).<br />
<br />
Here's the reverse of our simple <tt>Get</tt> example:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Put<br />
<br />
serialiseSomething :: Put<br />
serialiseSomething = do<br />
putWord32be 1<br />
putWord16be 2<br />
putWord8 3<br />
<br />
main :: IO ()<br />
main = BL.putStr $ runPut serialiseSomething</haskell><br />
<br />
And running it shows that it's generating the correct serialisation:<br />
<br />
<pre>% runhaskell /tmp/example.hs| hexdump -C<br />
00000000 00 00 00 01 00 02 03 |.......|</pre><br />
<br />
If you want the output of <tt>runPut</tt> to be a strict bytestring, you just<br />
need to convert it with <hask>B.concat $ BL.toChunks $ runPut xyz</hask>.<br />
<br />
One limitation of <tt>Put</tt>, due to the nature of the <tt>Builder</tt> monad<br />
which it works with, is that you can't get the current offset into the output.<br />
This can be an issue with some formats which require you to encode byte offsets<br />
into the file. You have to calculate these byte offsets yourself.<br />
<br />
=== Other useful packages ===<br />
<br />
There are other packages which you should know about, but which are mostly<br />
covered by their documentation:<br />
<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/network-bytestring-0.1.1 network-bytestring]: for reading and writing bytestring from the network<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/zlib-0.4.0.2 * zlib] and [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/bzlib-0.4.0.1 bzlib]: for compressed formats<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/encoding-0.3 encoding]: for dealing with charactor encodings<br />
* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]: as an example of lazy parsing/serialisation</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18754Dealing with binary data2008-01-29T06:08:42Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF terminated list of<br />
numbers list just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty <- isEmpty<br />
if empty<br />
then return []<br />
else do v <- getWord64be<br />
rest <- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a string one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hask>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed - at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.<br />
<br />
<table><br />
<tr><th>Name</th><th>C operator</th><th>Haskell</th></tr><br />
<tr><td>AND</td><td><tt>&amp;</tt></td><td><hask>.&.</hask></td></tr><br />
<tr><td>OR</td><td><tt>|</tt></td><td><hask>.|.</hask></td></tr><br />
<tr><td>XOR</td><td><tt>^</tt></td><td><hask>`xor`</hask></td></tr><br />
<tr><td>NOT</td><td><tt>&not;</tt></td><td><hask>`complement`</hask></td></tr><br />
<tr><td>Left shift</td><td><tt>&lt;&lt;</tt></td><td><hask>`shiftL`</hask></td></tr><br />
<tr><td>Right shift</td><td><tt>&gt;&gt;</tt></td><td><hask>`shiftR`</hask></td></tr><br />
</table><br />
<br />
====The <tt>BitGet</tt> monad====<br />
<br />
As an alternative to bit twiddling, you can also use the <tt>BitGet</tt> monad.<br />
This is another state-like monad, like <tt>Get</tt>, but here the state<br />
includes the current bit-offest in the input. This means that you can easily pull out<br />
unaligned data. Sadly, haddock is current breaking when trying to generate the<br />
documentation for <tt>BitGet</tt> so I'll start with an example.<br />
<br />
Here's a description of the header of a DNS packet, direct from RFC 1035:<br />
<br />
<pre> 1 1 1 1 1 1<br />
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ID |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
|QR| Opcode |AA|TC|RD|RA| Z | RCODE |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| QDCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ANCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| NSCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+<br />
| ARCOUNT |<br />
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+</pre><br />
<br />
The actual fields don't matter, but here's a function for parsing it:<br />
<br />
<haskell>parseHeader :: G.Get Header<br />
parseHeader = do<br />
id <- G.getWord16be<br />
flags <- G.getByteString 2<br />
qdcount <- G.getWord16be >>= return . fromIntegral<br />
ancount <- G.getWord16be >>= return . fromIntegral<br />
nscount <- G.getWord16be >>= return . fromIntegral<br />
arcount <- G.getWord16be >>= return . fromIntegral<br />
<br />
let r = BG.runBitGet flags (do<br />
isquery <- BG.getBit<br />
opcode <- BG.getAsWord8 4 >>= parseEnum<br />
aa <- BG.getBit<br />
tc <- BG.getBit<br />
rd <- BG.getBit<br />
ra <- BG.getBit<br />
<br />
BG.getAsWord8 3<br />
rcode <- BG.getAsWord8 4 >>= parseEnum<br />
<br />
return $ Header id isquery opcode aa tc rd ra rcode qdcount ancount nscount arcount)<br />
<br />
case r of<br />
Left error -> fail error<br />
Right x -> return x</haskell><br />
<br />
Here you can see that only the second line (from the ASCII-art diagram) is<br />
parsed using <tt>BitGet</tt>. An outer <tt>Get</tt> monad is used for<br />
everythign else and the bit fields are pulled out with<br />
<hask>getByteString</hask>. Again, <tt>BitGet</tt> is a strict monad and<br />
returns an <tt>Either</tt>, but it doesn't return the remaining bytestring,<br />
just because there's no obvious way to represent a bytestring of a fractional<br />
number of bytes.<br />
<br />
You can see the list of <tt>BitGet</tt> functions and their comments in the<br />
[http://darcs.imperialviolet.org/darcsweb.cgi?r=binary-strict;a=headblob;f=/src/Data/Binary/Strict/BitGet.hs source code].</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18753Dealing with binary data2008-01-29T05:54:49Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen &lt;- getWord32be<br />
plen &lt;- getWord32be<br />
chksum &lt;- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input &lt;- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF terminated list of<br />
numbers list just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty &lt;- isEmpty<br />
if empty<br />
then return []<br />
else do v &lt;- getWord64be<br />
rest &lt;- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a string one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hask>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen &lt;- getWord32be<br />
plen &lt;- getWord32be<br />
chksum &lt;- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input &lt;- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed - at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18752Dealing with binary data2008-01-29T05:53:54Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen &lt;- getWord32be<br />
plen &lt;- getWord32be<br />
chksum &lt;- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input &lt;- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF terminated list of<br />
numbers list just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty &lt;- isEmpty<br />
if empty<br />
then return []<br />
else do v &lt;- getWord64be<br />
rest &lt;- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a string one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hash>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen &lt;- getWord32be<br />
plen &lt;- getWord32be<br />
chksum &lt;- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input &lt;- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed - at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18751Dealing with binary data2008-01-29T05:52:30Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.<br />
<br />
So the <tt>Get</tt> monad consists of a set of operations like<br />
<hask>getWord32be</hask> which walk over the input and return some type of<br />
data. You can see the full list of those functions in the<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary-Get.html documentation].<br />
<br />
Here's another example; decoding an EOF terminated list of<br />
numbers list just involves recursion:<br />
<br />
<haskell>listOfWord16 = do<br />
empty <- isEmpty<br />
if empty<br />
then return []<br />
else do v <- getWord64be<br />
rest <- listOfWord16<br />
return (v : rest)</haskell><br />
<br />
==== Strict <tt>Get</tt> monad ====<br />
<br />
If you're parsing small messages then, firstly your input isn't going to be a<br />
lazy bytestring but a string one. That's not reallly a problem because you can<br />
easilly convert between them. However, if you want to handle parse failures you<br />
either have to write your parser very carefully, or you have to deal with the<br />
fact that you can only catch exceptions in the IO monad.<br />
<br />
If this is your dilemma, then you need a strict version of the <tt>Get</tt><br />
monad. It's almost exactly the same, but a parser of type <hask>Get a</hask><br />
results in <hask>(Either String a, ByteString)</hask> as the result of<br />
<hask>runGet</hash>. That type is a tuple where the first value is ''either'' a<br />
string (an error string from the parse) or the result, and the second value is<br />
the remaining bytestring when the parser finished.<br />
<br />
Let's update the first example with this strict version of <tt>Get</tt>. You'll<br />
have to install the<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict-0.2.2 binary-strict]<br />
package for it to work.<br />
<br />
<haskell>import qualified Data.ByteString as B<br />
import Data.Binary.Strict.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- B.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
Note that all we're done is change from lazy bytestrings to strict bytestrings<br />
and change change to importing <tt>Data.Binary.Strict.Get</tt>. Now we'll run<br />
it again:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(Right (825373492,825373492,825373493),"\n")</pre><br />
<br />
Now we can see that the parser was successful (we got a <tt>Right</tt>) and we<br />
can see that our shell actually added an extra newline on the input (correctly)<br />
and the parser didn't consume that, so it's also returned to us. Now we try it<br />
with a truncated input:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> tooshort<br />
heredoc> EOF<br />
(Left "too few bytes","\n")</pre><br />
<br />
This time we didn't get an exception, but a <tt>Left</tt> value, which can be<br />
handled in pure code. The remaining bytestring is the same because our<br />
truncated input is 9 bytes long, parsing the first two <tt>Word32</tt>'s<br />
consumed 8 bytes and parsing the third failed - at which point we had the last<br />
byte still in the input.<br />
<br />
In your parser, you can also call <hask>fail</hask>, with an error string,<br />
which will result in a <tt>Left</tt> value.<br />
<br />
====Bit twiddling====<br />
<br />
Even with all this monadic goodness, sometimes you just need to move some bits<br />
around. That's perfectly possible in Haskell too. Just import<br />
<tt>Data.Bits</tt> and use the following table.</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18748Dealing with binary data2008-01-29T03:37:20Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that sometimes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you can have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial,<br />
but should be fairly easy to find.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====<br />
<br />
The <tt>Get</tt> monad is a state monad; it keeps some state and each action<br />
updates that state. The state in this case is an offset into the bytestring<br />
which is getting parsed. <tt>Get</tt> parses lazy bytestrings, this is how<br />
packages like<br />
[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar-0.1.1.1 tar]<br />
can parse files several gigabytes long in constant memory - they are using a<br />
pipeline of lazy bytestrings. However, this also has a downside. When parsing a<br />
lazy bytestring a parse failure (such as running off the end of the bytestring)<br />
is signified by an exception. Exceptions can only be caught in the IO monad<br />
and, because of lazyness, might not be thrown exactly where you expect. If this<br />
is a problem you probably want a strict version of <tt>Get</tt>, which is<br />
covered below.<br />
<br />
Here's an example of using the <tt>Get</tt> monad:<br />
<br />
<haskell>import qualified Data.ByteString.Lazy as BL<br />
import Data.Binary.Get<br />
import Data.Word<br />
<br />
deserialiseHeader :: Get (Word32, Word32, Word32)<br />
deserialiseHeader = do<br />
alen <- getWord32be<br />
plen <- getWord32be<br />
chksum <- getWord32be<br />
return (alen, plen, chksum)<br />
<br />
main :: IO ()<br />
main = do<br />
input <- BL.getContents<br />
print $ runGet deserialiseHeader input</haskell><br />
<br />
This code takes 3, big-endian, 32-bit unsigned numbers from the input string<br />
and returns them as a tuple. Let's try running it:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
heredoc> 123412341235<br />
heredoc> EOF<br />
(825373492,825373492,825373493)</pre><br />
<br />
Makes sense, right? Look what happens if the input is too short:<br />
<br />
<pre>% runhaskell /tmp/example.hs << EOF<br />
tooshort<br />
EOF<br />
(1953460083,1752134260,example.hs: too few bytes. Failed reading at byte position 12</pre><br />
<br />
Here an exception was thrown because we ran out of bytes.</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18747Dealing with binary data2008-01-29T03:15:03Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that somes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you have have a look at the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions] and the<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]<br />
<br />
==== The <tt>Get</tt> monad ====</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18743Dealing with binary data2008-01-28T23:08:37Z<p>AdamLangley: </p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings] and<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- B.getContents<br />
B.putStr contents</haskell><br />
<br />
Note that we are using strict bytestrings here. (It's quite common to import the<br />
<code>ByteString</code> module under the names <code>B</code> or <code>BS</code>.)<br />
Since the bytestrings are strict the code will read the whole of stdin into<br />
memory and then write it out. If the input was too large this would overflow<br />
the availble memory and fail.<br />
<br />
Let's see the same program using lazy bytestrings. We are just changing the<br />
imported ByteString module to be the lazy one and calling the exact same<br />
functions from the new module:<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString.Lazy as BL<br />
<br />
main :: IO ()<br />
main = do<br />
contents <- BL.getContents<br />
BL.putStr contents</haskell><br />
<br />
This code, because of the lazy bytestrings, will cope with any sized input and<br />
will start producing output before all the input has been read. You can think<br />
of the code as setting up a pipeline, rather than executing in-order, as you<br />
might expect. As <hask>putStr</hask> needs more data, it will cause the lazy<br />
bytestring <hask>contents</hask> to read more until the end of the input is<br />
found.<br />
<br />
You should review the [[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html documentation]]<br />
which lists all the functions which operate on ByteStrings. The documentation<br />
for the various types (lazy Word8, strict Char8, ...) are all very similar. You<br />
generally find the same functions in each, with the same names. Remember to<br />
import the modules as <code>qualified</code> and give them different names.<br />
<br />
==== The Guts of ByteStrings ====<br />
<br />
I'll just mention in passing that somes you need to do something which would<br />
endanger the referential transparency of ByteStrings. Generally you only need<br />
to do this when using the FFI to interface with C libraries. Should such a need<br />
arise, you have have a look at the<br />
[[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Internal.html internal functions]] and the<br />
[[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Unsafe.html unsafe functions]].<br />
Remember that the last set of functions are called unsafe for a reason - misuse<br />
can crash you program!.<br />
<br />
=== Binary parsing ===<br />
<br />
Once you have your data as a bytestring you'll be wanting to parse something<br />
from it. Here you need to install the<br />
<tt>[[http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-0.4.1 binary]]</tt> package.<br />
Instructions for installing Cabal packages are out of scope for this tutorial.<br />
<br />
The <tt>binary</tt> package has three major parts: the <code>Get</code> monad,<br />
the <code>Put</code> monad and a general serialisation for Haskell types. The<br />
latter is like the <tt>pickle</tt> module that you may know from Python - it<br />
has it's own serialisation format and I won't be covering it any more here.<br />
However, if you just need to persist some Haskell data structures, it might be<br />
exactly what you want: the documentation is<br />
[[http://hackage.haskell.org/packages/archive/binary/0.4.1/doc/html/Data-Binary.html here]]<br />
<br />
==== The <tt>Get</tt> monad ====</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=Dealing_with_binary_data&diff=18734Dealing with binary data2008-01-28T20:18:03Z<p>AdamLangley: Incremental saving: page not ready</p>
<hr />
<div>== Handling Binary Data with Haskell ==<br />
<br />
Many programming problems call for the use of binary formats for compactness,<br />
ease-of-use, compatibility or speed. This page quickly covers some common<br />
libraries for handling binary data in Haskell.<br />
<br />
=== ByteStrings ===<br />
<br />
Everything else in this tutorial will be based on bytestrings. Normal Haskell<br />
<hask>String</hask> types are linked lists of 32-bit charactors. This has a<br />
number of useful properties like coverage of the Unicode space and lazyness,<br />
however when it comes to dealing with byte-wise data the <hask>String</hask><br />
involves a space-inflation of about 24x and a large reduction in speed.<br />
<br />
Bytestrings are packed arrays of bytes or 8-bit chars. If you have experience<br />
in C, their memory representation would be the same as a <code>uint8_t[]</code><br />
- although bytestrings know their length and don't allow overflows etc.<br />
<br />
Their are two major flavours of bytestrings, strict and lazy. Strict<br />
bytestrings are exactly what you would expect - a linear array of bytes in<br />
memory. Lazy bytestrings are a list of strict bytestrings, often this is called<br />
a cord in other languages. When reading a lazy bytestring from a file, the data<br />
will be read chunk by chunk and the file can be larger than the size of memory.<br />
The default chunk size is currently 32K.<br />
<br />
Within each flavour of bytestring comes the Word8 and Char8 versions. These are<br />
mostly an aid to the type system since they are fundamentally the same size of<br />
element. The Word8 unpacks as a list of <hask>Word8</hask> elements (bytes),<br />
the Char8 unpacks as a list of <hask>Char</hask>, which may be useful if you<br />
want to convert them to <hask>Strings</hask><br />
<br />
You might want to open the documentation for<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString.html strict bytestrings]<br />
[http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Lazy.html lazy bytestrings]<br />
in another tab so that you can follow along.<br />
<br />
==== Simple file IO ====<br />
<br />
Here's a very simple program which copies a file from standard input to<br />
standard output<br />
<br />
<haskell>module Main where<br />
<br />
import qualified Data.ByteString as B<br />
import System.IO (stdin, stdout)<br />
<br />
main = do<br />
contents <- B.hGet stdin<br />
B.hPut stdout contents</haskell></div>AdamLangleyhttps://wiki.haskell.org/index.php?title=User:AdamLangley&diff=18733User:AdamLangley2008-01-28T20:17:06Z<p>AdamLangley: </p>
<hr />
<div>[[DealingWithBinaryData]]</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=User:AdamLangley&diff=18732User:AdamLangley2008-01-28T20:16:49Z<p>AdamLangley: </p>
<hr />
<div>DealingWithBinaryData</div>AdamLangleyhttps://wiki.haskell.org/index.php?title=User:AdamLangley&diff=18731User:AdamLangley2008-01-28T20:16:26Z<p>AdamLangley: </p>
<hr />
<div>[DealingWithBinaryData]</div>AdamLangley