<div dir="ltr">Alright, here's the results for the first three in the list (please forgive me for being lazy- I am a Haskell programmer after all):<div><br></div><div><a href="http://ifeng.com">ifeng.com</a>:</div><div>
UTF8: 299949</div><div>UTF16: 566610</div><div><br></div><div><a href="http://dzh.mop.com">dzh.mop.com</a>:</div><div>GBK: 1866</div><div>UTF8: 1891</div><div>UTF16: 3684</div><div><br></div><div><a href="http://www.csdn.net">www.csdn.net</a>:</div>
<div>UTF8: 122870</div><div>UTF16: 217420</div><div><br></div><div>Seems like UTF8 is a consistent winner versus UTF16, and not much of a loser to the native formats.</div><div><br></div><div>Michael</div><div><br><div class="gmail_quote">
On Wed, Aug 18, 2010 at 11:01 AM, anderson leo <span dir="ltr"><<a href="mailto:fireman119@gmail.com">fireman119@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
More typical Chinese web sites:<br> <a href="http://www.ifeng.com" target="_blank">www.ifeng.com</a> (web site likes nytimes)<br> <a href="http://dzh.mop.com" target="_blank">dzh.mop.com</a> (community for fun)<br>
<a href="http://www.csdn.net" target="_blank">www.csdn.net</a> (web site for IT) <br>
<a href="http://www.sohu.com" target="_blank">www.sohu.com</a> (web site like yahoo)<br> <a href="http://www.sina.com" target="_blank">www.sina.com</a> (web site like yahoo)<br><br>-- Andrew<div><div>
</div><div class="h5"><br><br><div class="gmail_quote">On Wed, Aug 18, 2010 at 11:40 AM, Michael Snoyman <span dir="ltr"><<a href="mailto:michael@snoyman.com" target="_blank">michael@snoyman.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex"><div dir="ltr">Well, I'm not certain if it counts as a typical Chinese website, but here are the stats;<div>
<br></div><div>UTF8: 64,198</div><div>UTF16: 113,160</div><div><br></div><div>And just for fun, after gziping:</div>
<div><br></div><div>UTF8: 17,708</div><div>UTF16: 19,367<div><div></div><div><br><br><div class="gmail_quote">On Wed, Aug 18, 2010 at 2:59 AM, anderson leo <span dir="ltr"><<a href="mailto:fireman119@gmail.com" target="_blank">fireman119@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">Hi michael, here is a web site <a href="http://zh.wikipedia.org/zh-cn/" target="_blank">http://zh.wikipedia.org/zh-cn/</a>. It is the wikipedia for Chinese.<br>
<br>-Andrew<br><br><div class="gmail_quote"><div><div></div><div>On Tue, Aug 17, 2010 at 7:00 PM, Michael Snoyman <span dir="ltr"><<a href="mailto:michael@snoyman.com" target="_blank">michael@snoyman.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex"><div><div></div><div><div dir="ltr"><br><br><div class="gmail_quote"><div>
On Tue, Aug 17, 2010 at 1:50 PM, Yitzchak Gale <span dir="ltr"><<a href="mailto:gale@sefer.org" target="_blank">gale@sefer.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div>Ketil Malde wrote:<br>
> I haven't benchmarked it, but I'm fairly sure that, if you try to fit a<br>
> 3Gbyte file (the Human genome, say¹), into a computer with 4Gbytes of<br>
</div>> RAM, UTF-16 will be slower than UTF-8...<br>
<br>
I don't think the genome is typical text. And<br>
I doubt that is true if that text is in a CJK language.<br>
<div><br>
> I think that *IF* we are aiming for a single, grand, unified text<br>
> library to Rule Them All, it needs to use UTF-8.<br>
<br>
</div>Given the growth rate of China's economy, if CJK isn't<br>
already the majority of text being processed in the world,<br>
it will be soon. I have seen media reports claiming CJK is<br>
now a majority of text data going over the wire on the web,<br>
though I haven't seen anything scientific backing up those claims.<br>
It certainly seems reasonable. I believe Google's measurements<br>
based on their own web index showing wide adoption of UTF-8<br>
are very badly skewed due to a strong Western bias.<br>
<br>
In that case, if we have to pick one encoding for Data.Text,<br>
UTF-16 is likely to be a better choice than UTF-8, especially<br>
if the cost is fairly low even for the special case of Western<br>
languages. Also, UTF-16 has become by far the dominant internal<br>
text format for most software and for most user platforms.<br>
Except on desktop Linux - and whether we like it or not, Linux<br>
desktops will remain a tiny minority for the foreseeable future.<br>
<div><br></div></blockquote></div><div> I think you are conflating two points here, and ignoring some important data. Regarding the data: you haven't actually quoted any statistics about the prevalence of CJK data, but even if the majority of web pages served are in those three languages, a fairly high percentage of the content will *still* be ASCII, due simply to the HTML, CSS and Javascript overhead. I'd hate to make up statistics on the spot, especially when I don't have any numbers from you to compare them with.</div>
<div><br></div><div>As far as the conflation, there are two questions with regard to the encoding choice: encoding/decoding time and space usage. I don't think *anyone* is asserting that UTF-16 is a common encoding for files anywhere, so by using UTF-16 we are simply incurring an overhead in every case. We can't consider a CJK encoding for text, so its prevalence is irrelevant to this topic. What *is* relevant is that a very large percentage of web pages *are*, in fact, standardizing on UTF-8, and that all 7-bit text files are by default UTF-8.</div>
<div><br></div><div>As far as space usage, you are correct that CJK data will take up more memory in UTF-8 than UTF-16. The question still remains whether the overall document size will be larger: I'd be interested in taking a random sampling of CJK-encoded pages and comparing their UTF-8 and UTF-16 file sizes. I think simply talking about this in the vacuum of data is pointless. If anyone can recommend a CJK website which would be considered representative (or a few), I'll do the test myself.</div>
<div><br></div><font color="#888888"><div>Michael</div></font></div></div>
<br></div></div><div>_______________________________________________<br>
Haskell-Cafe mailing list<br>
<a href="mailto:Haskell-Cafe@haskell.org" target="_blank">Haskell-Cafe@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/haskell-cafe" target="_blank">http://www.haskell.org/mailman/listinfo/haskell-cafe</a><br>
<br></div></blockquote></div><br>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br>
</div></div></blockquote></div><br></div></div>