Argh. Email fail.<div><br></div><div>Hopefully this time I have managed to reply-all to the list *and* keep the unicode properly intact.</div><div><br></div><div>Sorry about any duplicates you may have received.</div><div>
<br></div><div><div class="im" style><div class="gmail_quote">On Sun, Mar 11, 2012 at 1:33 PM, Jason Dusek <span dir="ltr"><<a href="mailto:jason.dusek@gmail.com" target="_blank" style="color:rgb(17,85,204)">jason.dusek@gmail.com</a>></span> wrote:</div>
</div><div class="gmail_quote" style><div class="im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
2012/3/11 Jeremy Shaw <<a href="mailto:jeremy@n-heptane.com" target="_blank" style="color:rgb(17,85,204)">jeremy@n-heptane.com</a>>:<br><div>> Also, URIs are not defined in terms of octets.. but in terms<br>> of characters. If you write a URI down on a piece of paper --<br>
> what octets are you using? None.. it's some scribbles on a<br>> paper. It is the characters that are important, not the bit<br>> representation.<br></div></blockquote><div><br></div><div><br></div></div><div>
To quote RFC1738:</div><div><br></div><div><pre style="white-space:pre-wrap;font-size:1em;margin-top:0px;margin-bottom:0px"> URLs are sequences of characters, i.e., letters, digits, and special
characters. A URLs may be represented in a variety of ways: e.g., ink
on paper, or a sequence of octets in a coded character set. The
interpretation of a URL depends only on the identity of the
characters used.</pre></div><div class="im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div></div>Well, to quote one example from RFC 3986:<br><br> 2.1. Percent-Encoding<br><br> A percent-encoding mechanism is used to represent a data octet in a<br> component when that octet's corresponding character is outside the<br>
allowed set or is being used as a delimiter of, or within, the<br> component.<br></blockquote><div><br></div></div><div>Right. This describes how to convert an octet into a sequence of characters, since the only thing that can appear in a URI is sequences of characters.</div>
<div class="im" style="color:rgb(80,0,80)"><div> </div><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The syntax of URIs is a mechanism for describing data octets,<br>not Unicode code points. It is at variance to describe URIs in<br>terms of Unicode code points.</blockquote><div><br></div></div><div>Not sure what you mean by this. As the RFC says, a URI is defined entirely by the identity of the characters that are used. There is definitely no single, correct byte sequence for representing a URI. If I give you a sequence of bytes and tell you it is a URI, the only way to decode it is to first know what encoding the byte sequence represents.. ascii, utf-16, etc. Once you have decoded the byte sequence into a sequence of characters, only then can you parse the URI.</div>
<div> </div><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>
<div class="im" style="color:rgb(80,0,80)">> If you render a URI in a utf-8 encoded document versus a<br></div>> utf-16 encoded document.. the octets will be diffiFor example, let's say that we have a unicode string and we want to use it in the URI path.</div>
<div class="im" style="color:rgb(80,0,80)"><div><br>> the meaning will be the same. Because it is the characters<br>> that are important. For a URI Text would be a more compact<br>> representation than String.. but ByteString is a bit dodgy<br>
> since it is not well defined what those bytes represent.<br>> (though if you use a newtype wrapper around ByteString to<br>> declare that it is Ascii, then that would be fine).<br><br></div>This is all fine well and good for what a URI is parsed from<br>
and what it is serialized too; but once parsed, the major<br>components of a URI are all octets, pure and simple.</div></blockquote><div><br></div><div>Not quite. We can not, for example, change uriPath to be a ByteString and decode any percent encoded characters for the user, because that would change the meaning of the path and break applications.</div>
<div><br></div><div>For example, let's say we have the path segments ["foo", "bar/baz"] and we wish to use them in the path info of a URI. Because / is a special character it must be percent encoded as %2F. So, the path info for the url would be:</div>
<div><br></div><div> foo/bar%2Fbaz</div><div><br></div><div>If we had the path segments, ["foo","bar","baz"], however that would be encoded as:</div><div><br></div><div> foo/bar/baz</div><div>
<br></div><div>Now let's look at decoding the path. If we simple decode the percent encoded characters and give the user a ByteString then both urls will decode to:</div><div><br></div><div> pack "foo/bar/baz"</div>
<div><br></div><div>Which is incorrect. ["foo", "bar/baz"] and ["foo","bar","baz"] represent different paths. The percent encoding there is required to distinguish between to two unique paths.</div>
<div><br></div><div>Let's look at another example, Let's say we want to encode the path segments:</div><div><br></div><div> ["I❤λ"]</div><div><br></div><div>How do we do that?</div><div><br></div><div>Well.. the RFCs do not mandate a specific way. While a URL is a sequence of characters -- the set of allow characters in pretty restricted. So, we must use some application specific way to transform that string into something that is allowed in a uri path. We could do it by converting all characters to their unicode character numbers like:</div>
<div><br></div><div> "u73u2764u03BB"</div><div><br></div><div>Since the string now only contains acceptable characters, we can easily convert it to a valid uri path. Later when someone requests that url, our application can convert it back to a unicode character sequence. </div>
<div><br></div><div>Of course, no one actually uses that method. The commonly used (and I believe, officially endorsed, but not required) method is a bit more complicated.</div><div><br></div><div> 1. first we take the string "I❤λ" and utf-8 encoded it to get a octet sequence:</div>
<div><br></div><div> 49 e2 9d a4 ce bb </div><div><br></div><div> 2. next we percent encode the bytes to get *back* to a character sequence (such as a String, Text, or Ascii)</div><div><br></div><div> "I%E2%9D%A4%CE%BB"</div>
<div><br></div><div>So, that is character sequence that would appear in the URI. *But* we do not yet have octets that we can transmit over the internet. We only have a sequence of characters. We must now convert those characters into octets. For example, let's say we put the url as an 'href' in an <a> tag in a web page that is UTF-16 encoded.</div>
<div><br></div><div> 3. Now we must convert the character sequence to a (big endian) utf-16 octet sequence:</div><div><br></div><div> 00 49 00 25 00 45 00 32 00 25 00 39 00 44 00 25 00 41 00 34 00 25 00 43 00 45 00 25 00 42 00 42</div>
<div><br></div><div> So those are the octets that actually get embedded in the utf-16 encoded .html document and transmitted over the net.</div><div><br></div><div> 4. the browser then decodes the utf-16 web page and gets back the sequence of characters:</div>
<div><br></div><div> "I%E2%9D%A4%CE%BB"</div><div><br></div><div> Note that here the browser has a sequence of characters -- we know nothing about how those bytes are represented internally by the browser. If the browser was written in Haskell it might be String or Text.</div>
<div><br></div><div> Now let's say the browser wants to request the URL. It *must* encode the url as ASCII (as per the spec). </div><div><br></div><div> 5. So, the browser encodes the string as the octet sequence</div>
<div><br></div><div> 49 25 45 32 25 39 44 25 41 34 25 43 45 25 42 42</div><div><br></div><div> 6. The server can now decode that sequence of octets back into a sequence of characters:</div><div><br></div><div> "I%E2%9D%A4%CE%BB"</div>
<div><br></div><div> Now, the low-level Network.URI library can not really do much more than that, because it does not know what those octets are really supposed to mean (see the / example above).</div><div><br></div><div>
7. the application specific code, however, knows that it should now first split the path on any / characters to get </div><div><br></div><div> ["I%E2%9D%A4%CE%BB"]</div><div><br></div><div> 8. next it should percent decode each path segment to get a ByteString sequence:</div>
<div><br></div><div> 49 e2 9d a4 ce bb </div><div><br></div><div> 9. And now it can utf-8 decode that octet sequence get a unicode character sequence:</div><div><br></div><div> I❤λ</div><div><br></div><div>So... the basic gist is that if you unicode characters embedded in an html document, they will generally be encoded *three* different times. (First the unicode characters are converted to a utf-8 byte sequence, then the byte sequence is percent encoded, and then the percent encoded character sequence is encoded as another byte sequence). But, applications can choose to use other methods as well.</div>
<div><br></div><div>In terms of applicability to the URI type.. uriPath :: ByteString definitely does not work. It is possible that uriPath :: [ByteString] might work... assuming / is the only special character we need to worry about in the uriPath. But, doing all the breaking on '/' and the percent decoding may not be required for many applications. So, choosing to always do the extra work raises some concerns.</div>
<div><br></div><div>Also, even with, uriPath :: [ByteString], we are losing some information. The browser is free to percent encode characters -- even if it is not required. For example the browser could request:</div><div>
<br></div><div> "hello"</div><div><br></div><div>Or it could request:</div><div><br></div><div> "%68%65%6c%6c%6f"</div><div><br></div><div>In this case the *meaning* is the same. So, doing the decoding is less problematic. But I wonder if there might still be cases where we still want to distinguished between those two requests?</div>
<div><br></div><div>hope this helps.</div></div></div>