On Fri, Aug 31, 2012 at 7:59 AM, jeff p <span dir="ltr"><<a href="mailto:mutjida@gmail.com" target="_blank">mutjida@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
I have a sample file (attached) which I cannot read into Text:<br>
<br>
Prelude Control.Applicative> Data.Text.IO.readFile "foo"<br>
*** Exception: utf8.txt: hGetContents: invalid argument (invalid<br>
byte sequence)<br>
<br>
Prelude Control.Applicative> Data.Text.Encoding.decodeUtf8 <$><br>
Data.ByteString.Char8.readFile "foo"<br>
"*** Exception: Cannot decode byte '\x6e':<br>
Data.Text.Encoding.decodeUtf8: Invalid UTF-8 stream<br>
<br>
So it seems that foo doesn't contain valid UTF-8. However,<br>
System.IO.UTF8 has no problem reading the data:<br>
<br>
Prelude Control.Applicative> System.IO.UTF8.readFile "foo"<br>
"3591,,,dihigma99h,1905,5,25,CUBA,,Matanzas,1971,5,20,CUBA,,Cienfuegos,Martin,Dihigo,,Mart\65533n<br>
Magdaleno Dihigo<br>
(Llanos),,190,74,R,R,,,,dihigma99,dihigma99,dihim001,dihigma99,dihigma99\r\n"<br>
<br>
Shouldn't these all have the same behavior?<br></blockquote><div><br></div><div>\65533 is the unicode replacement character U+FFFD. This means that the source text is not valid UTF-8; the parser in System.IO.UTF8 is silently replacing the bad characters while the others are throwing an exception. If you want the same behaviour with the Text parser, use Data.Text.Encoding.decodeUtf8With which allows you to replicate this. It's likely, however, that your input text is in some other encoding like ISO-8859-1. Use the text-icu package (<a href="http://hackage.haskell.org/package/text-icu">http://hackage.haskell.org/package/text-icu</a>) to decode these.</div>
<div><br></div><div>G</div></div>-- <br>Gregory Collins <<a href="mailto:greg@gregorycollins.net" target="_blank">greg@gregorycollins.net</a>><br>