Personal tools

UTF-8

From HaskellWiki

(Difference between revisions)
Jump to: navigation, search
Line 1: Line 1:
 
[[Category:Code]]
 
[[Category:Code]]
A small example showing how to read and write UTF-8 in Haskell.
 
   
Do whatever you want; it's going in the public domain (Eric Kow on 2007-02-02 says so, anyway)
+
The simplest solution seems to be to use the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/utf8-string utf8-string package] from Galois. It provides a drop-in replacement for System.IO
   
 
<haskell>
 
<haskell>
> module Main where
+
import System.IO.UTF8
  +
import Prelude hiding (getContents, putStrLn)
   
> import Control.Monad (mapM_)
+
main =
> import Data.Word (Word8)
+
do putStrLn "what is your name? "
> import Foreign.Marshal.Array (allocaArray, peekArray, pokeArray)
+
name <- getContents
> import System.Environment (getArgs)
+
putStrLn $ "hello, " ++ name ++ "!"
> import System.IO (hFileSize, Handle, hGetBuf, hPutBuf, openBinaryFile, hClose,
 
> IOMode(ReadMode, WriteMode))
 
</haskell>
 
 
We're going to be using the 2002 UTF-8 implementation by Sven Moritz Hallberg. It happens to be the one that darcs uses ( http://abridgegame.org/repos/darcs/UTF8.lhs ). Note that Pugs also has a UTF-8 library of its own, which if I believe to handle ByteStrings, but I'm sticking with this one because it's what I know.
 
 
<haskell>
 
> import UTF8
 
</haskell>
 
 
What we want to show is that we can both read and write UTF-8. We do this by reading a file in, reversing every one of its lines, and writing it back out with the extra extension '.rev'. We'll do this for every filename that is passed in on the command line.
 
 
<haskell>
 
> main :: IO ()
 
> main =
 
> do args <- getArgs
 
> mapM_ reverseUTF8File args
 
 
> reverseUTF8File :: FilePath -> IO ()
 
> reverseUTF8File f =
 
> do fb <- readFileBytes f
 
> case decode fb of
 
> (cs, []) -> writeFileBytes (f ++ ".rev") $ encode $ reverseLines cs
 
> (_, xs) -> fail $ show xs
 
> where
 
> reverseLines = unlines . map reverse . lines
 
</haskell>
 
 
For this to work, we need to have some helper functions for reading and
 
writing [Word8]. It would be nice is if there were some standard functions for reading and writing [Word8] in files. (Note: I grabbed half of this off a post on one of the Haskell mailing lists)
 
 
<haskell>
 
> readFileBytes :: FilePath -> IO [Word8]
 
> readFileBytes f =
 
> do h <- openBinaryFile f ReadMode
 
> hsize <- fromIntegral `fmap` hFileSize h
 
> hGetBytes h hsize
 
>
 
> writeFileBytes :: FilePath -> [Word8] -> IO ()
 
> writeFileBytes f ws =
 
> do h <- openBinaryFile f WriteMode
 
> hPutBytes h (length ws) ws
 
> hClose h
 
 
> hGetBytes :: Handle -> Int -> IO [Word8]
 
> hGetBytes h c = allocaArray c $ \p ->
 
> do c' <- hGetBuf h p c
 
> peekArray c' p
 
>
 
> hPutBytes :: Handle -> Int -> [Word8] -> IO ()
 
> hPutBytes h c ws = allocaArray c $ \p ->
 
> do pokeArray p ws
 
> hPutBuf h p c
 
 
</haskell>
 
</haskell>

Revision as of 09:24, 21 July 2008


The simplest solution seems to be to use the utf8-string package from Galois. It provides a drop-in replacement for System.IO

import System.IO.UTF8
import Prelude hiding (getContents, putStrLn)
 
main =
 do putStrLn "what is your name? "
    name <- getContents
    putStrLn $ "hello, " ++ name ++ "!"