Sounds to me like we need a lazy Data.Text variation that allows UTF-8 and UTF-16 "segments" in it list of strict text elements :) Then big chunks of western text will be encoded efficiently, and same with CJK! Not sure what to do about strict Data.Text though :)<br>
<br><div class="gmail_quote">On Tue, Aug 17, 2010 at 1:40 PM, Ketil Malde <span dir="ltr"><<a href="mailto:ketil@malde.org">ketil@malde.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">Michael Snoyman <<a href="mailto:michael@snoyman.com">michael@snoyman.com</a>> writes:<br>
<br>
</div><div class="im">> As far as space usage, you are correct that CJK data will take up more<br>
> memory in UTF-8 than UTF-16.<br>
<br>
</div>With the danger of sounding ... alphabetist? as well as belaboring a<br>
point I agree is irrelevant (the storage format):<br>
<br>
I'd point out that it seems at least as unfair to optimize for CJK at<br>
the cost of Western languages. UTF-16 uses two bytes for (most) CJK<br>
ideograms, and (all, I think) characters in Western and other phonetic<br>
scripts. UTF-8 uses one to two bytes for a lot of Western alphabets,<br>
but three for CJK ideograms.<br>
<br>
Now, CJK has about 20K ideograms, which is almost 15 bits per ideogram,<br>
while an ASCII letter is about six bits. Thus, the information density<br>
of CJK and ASCII is about equal for UTF-8, 5/8 vs 6/8 - compared to<br>
15/16 vs 6/16 for UTF-16. In other words a given document translated<br>
between Chinese and English should occupy roughly the same space in<br>
UTF-8, but be 2.5 times longer in English for UTF-16.<br>
<div class="im"><br>
-k<br>
--<br>
If I haven't seen further, it is by standing in the footprints of giants<br>
_______________________________________________<br>
</div><div><div></div><div class="h5">Haskell-Cafe mailing list<br>
<a href="mailto:Haskell-Cafe@haskell.org">Haskell-Cafe@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/haskell-cafe" target="_blank">http://www.haskell.org/mailman/listinfo/haskell-cafe</a><br>
</div></div></blockquote></div><br>