UTF-8

Categories: Code

A small example showing how to read and write UTF-8 in Haskell.

Do whatever you want; it's going in the public domain (Eric Kow on 2007-02-02 says so, anyway)

> module Main where
 
> import Control.Monad (mapM_)
> import Data.Word (Word8)
> import Foreign.Marshal.Array (allocaArray, peekArray, pokeArray)
> import System.Environment (getArgs)
> import System.IO (hFileSize, Handle, hGetBuf, hPutBuf, openBinaryFile, hClose,
>                   IOMode(ReadMode, WriteMode))

We're going to be using the 2002 UTF-8 implementation by Sven Moritz Hallberg. It happens to be the one that darcs uses ( http://abridgegame.org/repos/darcs/UTF8.lhs ). Note that Pugs also has a UTF-8 library of its own, which if I believe to handle ByteStrings, but I'm sticking with this one because it's what I know.

> import UTF8

What we want to show is that we can both read and write UTF-8. We do this by reading a file in, reversing every one of its lines, and writing it back out with the extra extension '.rev'. We'll do this for every filename that is passed in on the command line.

> main :: IO ()
> main =
>  do args <- getArgs
>     mapM_ reverseUTF8File args
 
> reverseUTF8File :: FilePath -> IO ()
> reverseUTF8File f =
>   do fb <- readFileBytes f
>      case decode fb of
>        (cs, []) -> writeFileBytes (f ++ ".rev") $ encode $ reverseLines cs
>        (_,  xs) -> fail $ show xs
>   where
>     reverseLines = unlines . map reverse . lines

For this to work, we need to have some helper functions for reading and writing [Word8]. It would be nice is if there were some standard functions for reading and writing [Word8] in files. (Note: I grabbed half of this off a post on one of the Haskell mailing lists)

> readFileBytes :: FilePath -> IO [Word8]
> readFileBytes f =
>   do h <- openBinaryFile f ReadMode
>      hsize <- fromIntegral `fmap` hFileSize h
>      hGetBytes h hsize
>
> writeFileBytes :: FilePath -> [Word8] -> IO ()
> writeFileBytes f ws =
>  do h <- openBinaryFile f WriteMode
>     hPutBytes h (length ws) ws
>     hClose h
 
> hGetBytes :: Handle -> Int -> IO [Word8]
> hGetBytes h c = allocaArray c $ \p ->
>                   do c' <- hGetBuf h p c
>                      peekArray c' p
>
> hPutBytes :: Handle -> Int -> [Word8] -> IO ()
> hPutBytes h c ws = allocaArray c $ \p ->
>                      do pokeArray p ws
>                         hPutBuf h p c

Recent content is available under a simple permissive license.