Difference between revisions of "UTF-8"
Jump to navigation
Jump to search
m |
m (Reverted edits by Tomjaguarpaw (talk) to last revision by PhilipNeustrom) |
||
| (10 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Code]] |
[[Category:Code]] |
||
| − | A small example showing how to read and write UTF-8 in Haskell. |
||
| + | The simplest solution seems to be to use the [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/utf8-string utf8-string package] from Galois. It |
||
| − | Do whatever you want; it's going in the public domain (Eric Kow on 2007-02-02 says so, anyway) |
||
| + | provides a drop-in replacement for System.IO |
||
| + | ''What about other string encodings?'' |
||
| − | <haskell> |
||
| − | > module Main where |
||
| − | |||
| − | > import Control.Monad (mapM_) |
||
| − | > import Data.Word (Word8) |
||
| − | > import Foreign.Marshal.Array (allocaArray, peekArray, pokeArray) |
||
| ⚫ | |||
| − | > import System.IO (hFileSize, Handle, hGetBuf, hPutBuf, openBinaryFile, |
||
| − | > IOMode(ReadMode, WriteMode)) |
||
| − | </haskell> |
||
| + | == Example == |
||
| − | We're going to be using the 2002 UTF-8 implementation by Sven Moritz Hallberg. It happens to be the one that darcs uses ( http://abridgegame.org/repos/darcs/UTF8.lhs ). Note that Pugs also has a UTF-8 library of its own, which if I believe to handle ByteStrings, but I'm sticking with this one because it's what I know. |
||
| + | If we use a function from System.IO.UTF8, we should also hide the equivalent one from the Prelude. (Alternatively, we could import the UTF8 module qualified) |
||
<haskell> |
<haskell> |
||
| − | > import UTF8 |
+ | > import System.IO.UTF8 |
| + | > import Prelude hiding (readFile, writeFile) |
||
| ⚫ | |||
</haskell> |
</haskell> |
||
| + | The readFile and writeFile functions are the same as before... |
||
| − | We perform the demonstration on a list of files, specified as command line arguments. What we want to show is that we can both read and write UTF-8, so the demonstration will be of reading a file in, reverse every one of its |
||
| − | lines, and writing it back out with the extension '.reversed' |
||
<haskell> |
<haskell> |
||
| Line 32: | Line 25: | ||
> reverseUTF8File :: FilePath -> IO () |
> reverseUTF8File :: FilePath -> IO () |
||
> reverseUTF8File f = |
> reverseUTF8File f = |
||
| − | > do |
+ | > do c <- readFile f |
| ⚫ | |||
| − | > case decode fb of |
||
| ⚫ | |||
| − | > (_, xs) -> fail $ show xs |
||
> where |
> where |
||
> reverseLines = unlines . map reverse . lines |
> reverseLines = unlines . map reverse . lines |
||
| − | </haskell> |
||
| − | |||
| − | For this to work, we need to have some helper functions for reading and |
||
| − | writing [Word8]. It would be nice is if there were some standard functions for reading and writing [Word8] in files. |
||
| − | |||
| − | <haskell> |
||
| − | > readFileBytes :: FilePath -> IO [Word8] |
||
| − | > readFileBytes f = |
||
| − | > do h <- openBinaryFile f ReadMode |
||
| − | > hsize <- fromIntegral `fmap` hFileSize h |
||
| − | > hGetBytes h hsize |
||
| − | > |
||
| − | > writeFileBytes :: FilePath -> [Word8] -> IO () |
||
| − | > writeFileBytes f ws = |
||
| − | > do h <- openBinaryFile f WriteMode |
||
| − | > hPutBytes h (length ws) ws |
||
| − | |||
| − | > hGetBytes :: Handle -> Int -> IO [Word8] |
||
| − | > hGetBytes h c = allocaArray c $ \p -> |
||
| − | > do c' <- hGetBuf h p c |
||
| − | > peekArray c' p |
||
| − | > |
||
| − | > hPutBytes :: Handle -> Int -> [Word8] -> IO () |
||
| − | > hPutBytes h c ws = allocaArray c $ \p -> |
||
| − | > do pokeArray p ws |
||
| − | > hPutBuf h p c |
||
</haskell> |
</haskell> |
||
Latest revision as of 15:20, 6 February 2021
The simplest solution seems to be to use the utf8-string package from Galois. It
provides a drop-in replacement for System.IO
What about other string encodings?
Example
If we use a function from System.IO.UTF8, we should also hide the equivalent one from the Prelude. (Alternatively, we could import the UTF8 module qualified)
> import System.IO.UTF8
> import Prelude hiding (readFile, writeFile)
> import System.Environment (getArgs)
The readFile and writeFile functions are the same as before...
> main :: IO ()
> main =
> do args <- getArgs
> mapM_ reverseUTF8File args
> reverseUTF8File :: FilePath -> IO ()
> reverseUTF8File f =
> do c <- readFile f
> writeFile (f ++ ".rev") $ reverseLines c
> where
> reverseLines = unlines . map reverse . lines