Replies are inline. Thanks for the quick and thoughtful response!<br><br><div class="gmail_quote">On Sat, Jan 21, 2012 at 8:56 AM, Michael Snoyman <span dir="ltr"><<a href="mailto:michael@snoyman.com">michael@snoyman.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Myles,<div><br></div><div>These sound like two solid features, and I'd be happy to merge in code to support it. Some comments below.<br>
<br><div class="gmail_quote"><div class="im">On Sat, Jan 21, 2012 at 8:38 AM, Myles C. Maxfield <span dir="ltr"><<a href="mailto:myles.maxfield@gmail.com" target="_blank">myles.maxfield@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">To: <span style="line-height:18px;text-align:left;font-size:13px;font-family:sans-serif">Michael Snoyman, author and maintainer of http-conduit</span><div>
CC: haskell-cafe</div><div><br></div>
<div>Hello!<div><br></div><div>I am interested in contributing to the http-conduit library. I've been using it for a little while and reading through its source, but have felt that it could be improved with two features:</div>
<div><ul><li>Allowing the caller to know the final URL that ultimately resulted in the HTTP Source. Because httpRaw is not exported, the caller can't even re-implement the redirect-following code themselves. Ideally, the caller would be able to know not only the final URL, but also the entire chain of URLs that led to the final request. I was thinking that it would be even cooler if the caller could be notified of these redirects as they happen in another thread. There are a couple ways to implement this that I have been thinking about:</li>
<ul><li>A straightforward way would be to add a [W.Ascii] to the type of Response, and getResponse can fill in this extra field. getResponse already knows about the Request so it can tell if the response should be gunzipped.</li>
</ul></ul></div></div></blockquote></div><div>What would be in the [W.Ascii], a list of all paths redirected to? Also, I'm not sure what gunzipping has to do with here, can you clarify?</div><div class="im"><div> </div>
</div></div></div></div></blockquote><div>Yes; my idea was to make the [W.Ascii] represent the list of all URLs redirected to, in order.</div><div><br></div><div>My comment about gunzipping is only tangentially related. I meant that in the latest version of the code on GitHub, the getResponse function already takes a Request as an argument. This means that the getResponse function already knows what URL its data is coming from, so modifying the getResponse function to return that URL is simple. (I mentioned gunzip because, as far as I can tell, the reason that getResponse <i>already</i> takes a Request is so that the function can tell if the request should be gunzipped.)</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div><ul><ul>
<li>It would be nice for the caller to be able to know in real time what URLs the request is being redirected to. A possible way to do this would be for the 'http' function to take an extra argument of type (Maybe (Control.Concurrent.Chan W.Ascii)) which httpRaw can push URLs into. If the caller doesn't want to use this variable, they can simply pass Nothing. Otherwise, the caller can create an IO thread which reads the Chan until some termination condition is met (Perhaps this will change the type of the extra argument to (Maybe (Chan (Maybe W.Ascii)))). I like this solution, though I can see how it could be considered too heavyweight.</li>
</ul></ul></div></div></blockquote><div><br></div></div><div>I do think it's too heavyweight. I think if people really want lower-level control of the redirects, they should turn off automatic redirect and allow 3xx responses.</div>
</div></div></div></blockquote><div>Yeah, that totally makes more sense. As it stands, however, httpRaw isn't exported, so a caller has no way of knowing about each individual HTTP transaction. Exporting httpRaw solves the problem I'm trying to solve. If we export httpRaw, should we <i>also</i> make 'http' return the URL chain? Doing both is probably the best solution, IMHO.</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div class="im">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><ul><ul>
</ul><li>Making the redirection aware of cookies. There are redirects around the web where the first URL returns a Set-Cookie header and a 3xx code which redirects to another site that expects the cookie that the first HTTP transaction set. I propose to add an (IORef to a Data.Set of Cookies) to the Manager datatype, letting the Manager act as a cookie store as well as a repository of available TCP connections. httpRaw could deal with the cookie store. Network.HTTP.Types does not declare a Cookie datatype, so I would probably be adding one. I would probably take it directly from Network.HTTP.Cookie.</li>
</ul></div></div></blockquote></div><div>Actually, we already have the cookie package for this. I'm not sure if putting the cookie store in the manager is necessarily the right approach, since I can imagine wanting to have separate sessions while reusing the same connections. A different approach could be adding a list of Cookies to both the Request and Response. </div>
</div></div></div></blockquote><div>Ah, looks like you're the maintainer of that package as well! I didn't realize it existed. I should have, though; Yesod must need to know about cookies somehow.</div><div><br></div>
<div>As the http-conduit package stands, the headers of the original Request can be set, and the headers of the last Response can be read. Because cookies are implemented on top of headers, the caller knows about the cookies before and after the redirection chain. I'm more interested in the preservation of cookies <i>within</i> the redirection chain. As discussed earlier, exposing the httpRaw function allows the entire redirection chain to be handled by the caller, which alleviates the problem.</div>
<div><br></div><div>That being said, however, the simpleHttp function (and all functions built upon 'http' inside of http-conduit) should probably respect cookies inside redirection chains. Under the hood, Network.Browser does this by having the State monad keep track of these cookies (as well as the connection pool) and making HTTP requests mutate that State, but that's a pretty different architecture than Network.HTTP.Conduit.</div>
<div><br></div><div>One way I can think to do this would be to let the user supply a CookieStore (probably implemented as a (Data.Set Web.Cookie.SetCookie)) and receive a (different) CookieStore from the 'http' function. That way, the caller can manage the CookieStores independently from the connection pool. The downside is that it's one more bit of ugliness the caller has to deal with. How do you feel about this? You probably have a better idea :-)<br>
<div dir="ltr"><div><div class="gmail_quote"><div class="im">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><ul>
</ul>I'd be happy to do both of these things, but I'm hoping for your input on how to go about this endeavor. Are these features even good to be pursuing? Should I be going about this entirely differently?</div><div>
<br></div><div>Thanks,</div><span><font color="#888888"><div>Myles C. Maxfield</div></font></span><div><br></div><div>P.S. I'm curious about the lack of Network.URI throughout Network.HTTP.Conduit. Is there a particular design decision that led you to use raw ascii strings?</div>
</div>
</blockquote></div></div><br></div><div>Because there are plenty of URIs that are valid that we don't handle at all, e.g., ftp.</div><div><br></div><div>I'm a little surprised by this, since you can easily test for unhandled URIs because they're already parsed. Whatever; It doesn't really matter to me, I was just surprised by it.</div>
<span class="HOEnZb"><font color="#888888"><div><br></div><div>Michael</div></font></span></div>
</div></div><br><div>Thanks again for the feedback! I'm hoping to make a difference :]</div><div><br></div><div>--Myles</div>