Difference between revisions of "UnicodeByteString"

From HaskellWiki
Jump to: navigation, search
(Deleting page that hasn't been updated for over 10 years)
Line 1: Line 1:
== Motivation ==
<hask>ByteString</hask> provides a faster and more memory efficient data type than <hask>[Word8]</hask> for processing raw bytes.  By creating a Unicode data type similar to <hask>ByteString</hask> that deals in units of characters instead of units of bytes we can achieve similar performance improvements over <hask>String</hask> for text processing.  A Unicode data type also removes the error prone process of keeping track of strings encoded as raw bytes stored in <hask>ByteString</hask>s.  Using functions such as <hask>length</hask> on a Unicode string just works even though different encodings use different numbers of bytes to represent a character.
== Specification ==
A new module, <hask>Text.Unicode</hask>, defines the efficient Unicode string data type:
<haskell>data UnicodeString</haskell>
Functions to encode and decode Unicode strings to and from <hask>ByteString</hask>s are provided together with <hask>Data.List</hask> like functions.
data Encoding = Ascii | Utf8 | Utf16 | Iso88591
decode :: Encoding -> ByteString -> UnicodeString
encode :: Encoding -> UnicodeString -> ByteString
=== Error handling ===
When a <hask>ByteString</hask> is decoded using the wrong codec several error handling strategies are possible:
* An exception is raised using <hask>error</hask>.  This may be fine for many cases.
* Unknown byte sequences are replaced with some character (e.g. <hask>'?'</hask>).  This is useful for debugging, etc. where some input/output is better than none.
* The decode function returns values of type <hask>Either CodecError UnicodeString</hask> where <hask>CodecError</hask> contains some useful error information.
The final API should provide at least a few error handling strategies of different sophistication.
One example in this design space for error handling is this iconv library:
It provides a most general conversion function with type:
:: EncodingName  -- ^ Name of input string encoding
-> EncodingName  -- ^ Name of output string encoding
-> L.ByteString  -- ^ Input text
-> [Span]
data Span =
    -- | An ordinary output span in the target encoding
    Span !S.ByteString
    -- | An error in the conversion process. If this occurs it will be the
    -- last span.
  | ConversionError !ConversionError
Then the other simpler error handling strategies are wrappers over this interface. One converts strictly and returns Either L.ByteString ConversionError, the other converts lazily and uses exceptions. There is also a fuzzy mode where conversion errors are ignored or transliterated, using similar replacement characters or <hask>'?'</hask>.
=== I/O ===
Several I/O functions that deal with <hask>UnicodeString</hask>s might be needed. All text based I/O should require an explicit encoding or use the default encoding (as set by the user's locale).  Example:
readFile :: Encoding -> FilePath -> UnicodeString
== Open Issues ==
=== API duplication ===
The <hask>Data.List</hask> API is already duplicated in large parts in <hask>Data.ByteString</hask>.  It will be duplicated again here.  Will keeping the APIs in sync be a huge pain in the future?
=== New I/O library ===
How many new I/O functions are needed?  Would it be enough to use <hask>ByteString</hask>'s I/O interface:
import qualified Data.ByteString as B
echo = do
  content <- decode Utf8 <$> B.readFile "myfile.txt"
  B.putStrLn $ encode Utf8 content
=== Different representations ===
Should the encoding used to represent Unicode code points be included in the type?
data Encoding e => UnicodeString e
This might save some recoding as opposed to always using the same internal encoding for <hask>UnicodeString</hask>.  It's necessary that UnicodeString can be used between different text processing libraries.  Is this possible or will any library end up specifying a particular value for <hask>Encoding e</hask> and thus make it harder to interact with that library?
This approach is used by <hask>CompactString</hask>
== References ==
Python 3000 will see an overhaul of their Unicode approach, including a new <code>bytes</code> type, a merge of <code>str</code> and <code>unicode</code> and a new I/O library.  This proposals takes many ideas from that overhaul.  The relevant PEPs are:
# http://www.python.org/dev/peps/pep-0358/ - PEP 3116 -- New I/O
# http://python.org/dev/peps/pep-3116/ - PEP 358 -- The "bytes" Object
# http://www.python.org/dev/peps/pep-3137/ - PEP 3137 -- Immutable Bytes and Mutable Buffer

Revision as of 15:04, 6 February 2021