<br><br><div class="gmail_quote">On Tue, Aug 17, 2010 at 13:40, Ketil Malde <span dir="ltr"><<a href="mailto:ketil@malde.org">ketil@malde.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im">Michael Snoyman <<a href="mailto:michael@snoyman.com">michael@snoyman.com</a>> writes:<br>
<br>
</div><div class="im">> As far as space usage, you are correct that CJK data will take up more<br>
> memory in UTF-8 than UTF-16.<br>
<br>
</div>With the danger of sounding ... alphabetist? as well as belaboring a<br>
point I agree is irrelevant (the storage format):<br>
<br>
I'd point out that it seems at least as unfair to optimize for CJK at<br>
the cost of Western languages.<br></blockquote><div><br>Thing is that here you're only talking about size optimizations, for somebody having to handle a lot of international texts (and I'm not necessarily talking about Chinese or Japanese here) it would be important that this is handled in the most efficient way possible, because in the end storing and retrieving you only do once each while maybe doing a lot of processing in between. And the on-disk storage or the over-the-wire format might very well be different than the in-memory format. Each can be selected for what it's best at.<br>
<br>I'll repeat here that in my opinion a Text package should be good at handling text, human text, from whatever country. If I need to handle large streams of ASCII I'll use something else.<br><br>:)<br><br>Cheers,<br>
-Tako<br><br></div></div>