From dominic.steinitz at blueyonder.co.uk Sun Feb 1 08:47:59 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sun Feb 1 08:49:42 2009 Subject: Bug in Cabal? In-Reply-To: <1233148429.26754.105.camel@localhost> References: <1232529549.8432.229.camel@localhost> <1233148429.26754.105.camel@localhost> Message-ID: <4985A80F.7090809@blueyonder.co.uk> Duncan Coutts wrote: > Hia Dominic, > > I've looked into this a bit. I think I may now be less confused. :-) > > On Wed, 2009-01-21 at 18:01 +0000, Dominic Steinitz wrote: > >>> ls -ltr haskell_packages\doc\zlib-0.5.0.0 >> total 2 >> -r--r--r-- 1 user group 1327 Jan 21 11:37 LICENSE > > Ah! Is that the only file that is read-only? That makes sense since it's > an original source file from your read-only source control system. The > System.Directory.copyFile function does not copy permissions (though it > does try to) but it does (unhelpfully) copy the read-only attribute. It may not be but a lot of packages only have a LICENSE file. I'll have a look tomorrow. > >> When I rebuild the package *without* having cleaned out haskell_packages then I >> get the access error. I think this is because cabal (copyFile) doesn't want to >> overwrite a read only file even though the directory permissions should allow >> it to do so. > > Yes. The windows MoveFile function does not let us move a file over a > file that has the read-only attribute. > > It's an annoying interaction between the semantics of old DOS/FAT > read-only attribute and new windows file permissions (ACLs). > > The solution I think, is never to copy the read-only attribute when > installing files. It's thoroughly unhelpful. I've got a patch to do this > which I shall push to Cabal HEAD shortly. If you have a moment to help > me test that it would be much appreciated. Thanks very much. I'll try it out tomorrow. From duncan.coutts at worc.ox.ac.uk Sun Feb 1 12:57:15 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Sun Feb 1 12:56:22 2009 Subject: Bug in Cabal? In-Reply-To: <4985A80F.7090809@blueyonder.co.uk> References: <1232529549.8432.229.camel@localhost> <1233148429.26754.105.camel@localhost> <4985A80F.7090809@blueyonder.co.uk> Message-ID: <1233511035.26754.400.camel@localhost> On Sun, 2009-02-01 at 13:47 +0000, Dominic Steinitz wrote: > Duncan Coutts wrote: > > Hia Dominic, > > > > I've looked into this a bit. I think I may now be less confused. :-) > > > > On Wed, 2009-01-21 at 18:01 +0000, Dominic Steinitz wrote: > > > >>> ls -ltr haskell_packages\doc\zlib-0.5.0.0 > >> total 2 > >> -r--r--r-- 1 user group 1327 Jan 21 11:37 LICENSE > > > > Ah! Is that the only file that is read-only? That makes sense since it's > > an original source file from your read-only source control system. The > > System.Directory.copyFile function does not copy permissions (though it > > does try to) but it does (unhelpfully) copy the read-only attribute. > > It may not be but a lot of packages only have a LICENSE file. I'll have > a look tomorrow. Sure, I didn't mean that because it's only a license file it's ok. It's just that it makes sense given that it's one of the only files copied direct from the sources directory (rather than generated during the build). It took me a while to work out what was actually going on (and to realise it was different from the other permissions problems people have reported on Windows). > > The solution I think, is never to copy the read-only attribute when > > installing files. It's thoroughly unhelpful. I've got a patch to do this > > which I shall push to Cabal HEAD shortly. If you have a moment to help > > me test that it would be much appreciated. > > Thanks very much. I'll try it out tomorrow. Thanks. The fixes are now in Cabal HEAD and the Cabal-1.6 branch so use whichever is easier for you. Duncan From johan.tibell at gmail.com Sun Feb 1 18:19:39 2009 From: johan.tibell at gmail.com (Johan Tibell) Date: Sun Feb 1 18:09:54 2009 Subject: Proposal #2998: Expose the internals of the network library to make it more extensible Message-ID: <90889fe70902011519x2b6c209s81b1056a9f9b19da@mail.gmail.com> Hi all, In short, I propose that we export some of the internals of the network library in a new module to make it easier to write portable libraries on top of the library. I've added the full proposal, copied from Trac, below. The motivating example is network-bytestring which needs to marshall different socket related data types to support e.g. `sendmsg`. http://hackage.haskell.org/trac/ghc/ticket/2998 Discussion deadline: 2 weeks (non disruptive change) --- When creating new libraries on top of the network library it would be convenient if the library exposed some functions that hide platform variations so that the new libraries can be written in a portable manner. Take for example the currently un-exported pokeSockAddr function which marshalls a SockAddr into a sockaddr struct. This function would be useful when e.g. writing a binding for the sendmsg system call. However, it's not exported so libraries are forced to define their own version. This is unfortunate. I propose we add a new module, Network.Socket.Internal, to the network library, that exposes these portable marshalling functions. It might also be a good idea to expose some internal data representations in the same way bytestring currently does. People who write code that uses this internal do so knowing that there is a risk that the interface and data representations might change. I've written a patch that exposes some of the library's internal functions. There are still some documentation to write and perhaps a few more functions to expose but I think we should reach consensus on whether this is a good idea to begin with before continuing. Since this change isn't disruptive I propose a two week consideration deadline. Cabal package: http://johantibell.com/network-2.2.1.tar.gz Haddock: http://johantibell.com/network/doc/html/network/ --- Cheers, Johan From dons at galois.com Sun Feb 1 18:24:10 2009 From: dons at galois.com (Don Stewart) Date: Sun Feb 1 18:14:21 2009 Subject: Proposal #2998: Expose the internals of the network library to make it more extensible In-Reply-To: <90889fe70902011519x2b6c209s81b1056a9f9b19da@mail.gmail.com> References: <90889fe70902011519x2b6c209s81b1056a9f9b19da@mail.gmail.com> Message-ID: <20090201232410.GK24687@scytale.galois.com> Let's do it. The FFI layer should in general be exposed imo, via some .Internals module. -- Don johan.tibell: > Hi all, > > In short, I propose that we export some of the internals of the > network library in a new module to make it easier to write portable > libraries on top of the library. I've added the full proposal, copied > from Trac, below. The motivating example is network-bytestring which > needs to marshall different socket related data types to support e.g. > `sendmsg`. > > http://hackage.haskell.org/trac/ghc/ticket/2998 > > Discussion deadline: 2 weeks (non disruptive change) > > --- > > When creating new libraries on top of the network library it would be > convenient if the library exposed some functions that hide platform > variations so that the new libraries can be written in a portable > manner. > > Take for example the currently un-exported pokeSockAddr function which > marshalls a SockAddr into a sockaddr struct. This function would be > useful when e.g. writing a binding for the sendmsg system call. > However, it's not exported so libraries are forced to define their own > version. > > This is unfortunate. I propose we add a new module, > Network.Socket.Internal, to the network library, that exposes these > portable marshalling functions. It might also be a good idea to expose > some internal data representations in the same way bytestring > currently does. People who write code that uses this internal do so > knowing that there is a risk that the interface and data > representations might change. > > I've written a patch that exposes some of the library's internal > functions. There are still some documentation to write and perhaps a > few more functions to expose but I think we should reach consensus on > whether this is a good idea to begin with before continuing. > > Since this change isn't disruptive I propose a two week consideration deadline. > > Cabal package: > > http://johantibell.com/network-2.2.1.tar.gz > > Haddock: > > http://johantibell.com/network/doc/html/network/ > > --- > > Cheers, > > Johan > _______________________________________________ > Libraries mailing list > Libraries@haskell.org > http://www.haskell.org/mailman/listinfo/libraries From haskell at list.mightyreason.com Mon Feb 2 15:27:47 2009 From: haskell at list.mightyreason.com (ChrisK) Date: Mon Feb 2 15:18:05 2009 Subject: ANN: regex-posix-unittest-1.0 AND regex-posix-0.94.1 AND regex-tdfa-0.97.1 Message-ID: <49875743.2090502@list.mightyreason.com> I have three announcements to make about regex-* related packages. The regex-posix-0.94.1 package update provides better semantics for multiple matches. Below version 0.94, if any match was empty the matching would stop. Now the empty match is returned and the position is incremented and the searching continues. The regex-tdfa-0.71.1 package update provides the same new multiple match semantics. It also fixes a bug I found. I know of no outstanding bugs in regex-tdfa, and version 0.71.1 now passes all the tests used in regex-posix-unittest-1.0 announced below. We should care about the correctness of our operating system libraries. To help with this, I have a NEW package to announce: regex-posix-unittest-1.0 The accompanying wiki page is http://www.haskell.org/haskellwiki/Regex_Posix This new package provides an executable called regex-posix-unittest which you can install as --user or --global. The regex-posix-unittest executable with no arguments runs a suite of unit tests, all of which are described by text files in the package, the format is documented in the wiki page. By editing the text files in the package you can add to or delete from the unit tests being run. With two arguments the program expects the text first and the pattern second and will run just that match and print all the results. How does regex-posix-unittest help us care about the OS libraries? The regex-posix distributed in the GHC bundle uses the OS C library's "regex.h" API. The regex-posix-unittest package will quite likely show you that your OS C library "regex.h" API is full of bugs. If you are on Linux, it will show you a plethora of GLIBC bugs in Posix conformance. If you are on OS X, FreeBSD, or NetBSD, it will show you many bugs including a critical bug where it fail to find a match where one actually exists. These bugs in the OS library are inherited by your "sed" program as well as regex-posix and Haskell. If you are on Windows, or OpenBSD, or Solaris, or anything else, then please update the wiki page at http://www.haskell.org/haskellwiki/Regex_Posix or email me with your results so I can update the wiki. You may have evil and ingenious tests of Posix extended regular expressions to add to the test suite. Adding them is easy and if you send them to me I will put them in an updated version of regex-posix-unittest. Cheers, Chris From jnf at arcor.de Mon Feb 2 17:56:06 2009 From: jnf at arcor.de (=?ISO-8859-1?Q?J=FCrgen?= Nicklisch-Franken) Date: Mon Feb 2 17:46:28 2009 Subject: cabal install problems Message-ID: <1233615366.6329.39.camel@jutaro-laptop> I recently had some (well actually a lot of) troubles with cabal install which are not over. I installed ghc-6.10 on Windows with Cygwin and the fetched cabal-install 0.6.0. When I then tried to install a package it had regex-posix as dependency it fails to install the newest version package. This was because it doesn't find the c header files. So I tried to use the --extra-include-dirs=PATH flag, but this didn't help, and I guess it was because it was the c2hs preprocessor which didn't get it. So I did a manual install of the package and somehow did it. But although ghc-pkg listed the package, when I next time called cabal install it started with installing the package that was already installed. When called with a verbose flag I saw that cabal-install detected that the package was not installed by cabal install. However why does it try to reinstall an already installed package? How does it know, which packages are installed with cabal install? I then changed the version of the required regex-posix package to the exact version of the lib pre-installed with ghc on Windows. Yet I can never use cabal upgrade again, because it will always fail. Is their any way to exclude a package from the cabal install process? I realised as well that there is no way to ever un-install a package installed with cabal-install, as far as I see. This is not satisfactory, right? I regret as well that no option exist to have the source code expanded (I mean from tar.gz), which can be useful if you want to look up some library code later, which I happen to do frequently. I regret as well that there is no way to adopt a cabal file to my needs when using cabal install, which I find sometimes necessary. As cabal install is supposed to become a standard way of installing Haskell packages, at least for developers, it would be necessary to have some documentation on it, other then just explaining command line parameters. E.g. if you install some packages as user, which it does by default, it will later not be possible to install other packages global, which anyone using cabal install should know. (Or am I wrong) So please help us from spending our time with stupid install problems, and make the road for new Haskell users as smooth as possible, and installation is a big issue for them. J?rgen From duncan.coutts at worc.ox.ac.uk Mon Feb 2 20:03:06 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Feb 2 19:53:05 2009 Subject: cabal install problems In-Reply-To: <1233615366.6329.39.camel@jutaro-laptop> References: <1233615366.6329.39.camel@jutaro-laptop> Message-ID: <1233622986.26754.508.camel@localhost> On Mon, 2009-02-02 at 23:56 +0100, J?rgen Nicklisch-Franken wrote: > I recently had some (well actually a lot of) troubles with cabal install > which are not over. I installed ghc-6.10 on Windows with Cygwin and the > fetched cabal-install 0.6.0. > > When I then tried to install a package it had regex-posix as dependency > it fails to install the newest version package. This was because it > doesn't find the c header files. So I tried to use the > --extra-include-dirs=PATH flag, but this didn't help, and I guess it > was because it was the c2hs preprocessor which didn't get it. Do you have any more details on that? It's the 'hsc2hs' pre-processor btw. Confusingly there is a similar pre-processor called 'c2hs'. > So I did a manual install of the package and somehow did it. That's interesting. It should be the same either way. More details would be interesting. > But although ghc-pkg listed the package, when I next time called cabal > install it started with installing the package that was already > installed. When called with a verbose flag I saw that cabal-install > detected that the package was not installed by cabal install. However > why does it try to reinstall an already installed package? How does it > know, which packages are installed with cabal install? Actually it does not track which agent installed the package. If you run with --dry-run -v it gives some explanation for what it is doing and why it might think it needs to reinstall the package. If you paste that output we might be able to see what is going on. There is one specific problem with these symptoms that's been fixed in the development version, but there are some other reasons why the solver might choose to reinstall a package. > I then changed the version of the required regex-posix package to the > exact version of the lib pre-installed with ghc on Windows. Yet I can > never use cabal upgrade again, because it will always fail. cabal upgrade isn't such a good idea anyway. I'd stick to the install command. The only difference is that upgrade tries to install the latest version all dependencies too, and that tends to lead to problems. I'm considering disabling the feature for now. > Is their any way to exclude a package from the cabal install process? There is a flag --constraint="foo == 1.0" which is quite handy. The next version may or may not have a way to specify a constraint to always pick an installed version of a particular package. It depends if there is enough time. > I realised as well that there is no way to ever un-install a package > installed with cabal-install, as far as I see. This is not satisfactory, > right? Right. Currently only by using ghc-pkg to unregister and the file manager to delete the files. For some reason uninstall never seems to be as high on users priorities as other bugs and features. At least that is my impression. Of course if people help out we'll get round to it sooner. > I regret as well that no option exist to have the source code expanded > (I mean from tar.gz), which can be useful if you want to look up some > library code later, which I happen to do frequently. There is in the development version cabal unpack foo > I regret as well that there is no way to adopt a cabal file to my needs > when using cabal install, which I find sometimes necessary. unpack is great for that. > As cabal install is supposed to become a standard way of installing > Haskell packages, at least for developers, it would be necessary to have > some documentation on it, other then just explaining command line > parameters. I would very much appreciate some help with the documentation. We already have a Cabal user guide and there are not that many extra things in the command line interface of cabal-install. > E.g. if you install some packages as user, which it does by default, it > will later not be possible to install other packages global, which > anyone using cabal install should know. (Or am I wrong) Global packages cannot depend on user ones. So installing something globally will also install all of its dependencies globally. > So please help us from spending our time with stupid install problems, > and make the road for new Haskell users as smooth as possible, and > installation is a big issue for them. That is what we're trying to do :-). That is what this whole project is about. I can assure you it used to be much harder than it is now. We're making progress but with more people helping we could make progress quicker :-) Duncan From lemming at henning-thielemann.de Tue Feb 3 08:10:14 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Tue Feb 3 08:00:22 2009 Subject: MaybeT to transformers Message-ID: What about adding the Maybe monad transformer as provided in MaybeT package to the transformers package? The module name could be Control.Monad.Trans.Maybe which would also avoid conflicts with the MaybeT package. From marlowsd at gmail.com Tue Feb 3 10:01:10 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Tue Feb 3 09:51:26 2009 Subject: Missing headers from SharedMem.hsc In-Reply-To: <20090112235142.GU3582@scytale.galois.com> References: <20090112235142.GU3582@scytale.galois.com> Message-ID: <49885C36.1060002@gmail.com> Don Stewart wrote: > It was also missing HsUnix.h, so the #defines weren't propagating > anyway. > > > Mon Jan 12 15:07:58 PST 2009 Don Stewart > * Add check for -lrt to get the shm* functions. Subst. in buildinfo > > Mon Jan 12 15:47:17 PST 2009 Don Stewart > * SharedMem.hsc wasn't including HsUnixConfig.h, so no #defines were propagating Pushed, eventually. Cheers, Simon From marlowsd at gmail.com Tue Feb 3 11:42:44 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Tue Feb 3 11:33:04 2009 Subject: Ready for testing: Unicode support for Handle I/O Message-ID: <49887404.4070504@gmail.com> I've been working on adding proper Unicode support to Handle I/O in GHC, and I finally have something that's ready for testing. I've put a patchset here: http://www.haskell.org/~simonmar/base-unicode.tar.gz That is a set of patches against a GHC repo tree: unpack the tarball, and say 'sh apply /path/to/ghc/repo' to apply all the patches. Then clean your tree and build it from scratch (or if you're using the new GHC build system, just say 'make' ;-). It should validate, bar one or two minor failures. Oh, it doesn't work on Windows yet. That's the major thing left to do. If anyone else felt like tackling this I'd be delighted: all you have to do is implement a Win32 equivalent of the module GHC.IO.Encoding.Iconv (see below), everything else should work unchanged. Depending on whether any further changes are required, I may amend-record some of these patches, so treat them as temporary patches for testing only. Below is what will be the patch description in the patch for libraries/base. Comments/discussion please! Cheers, Simon Unicode-aware Handles ~~~~~~~~~~~~~~~~~~~~~ This is a significant restructuring of the Handle implementation with the primary goal of supporting Unicode character encodings. The only change to the existing behaviour is that by default, text IO is done in the prevailing encoding of the system. Handles created by openBinaryFile use the Latin-1 encoding, as do Handles placed in binary mode using hSetBinaryMode. We provide a way to change the encoding for an existing Handle: hSetEncoding :: Handle -> TextEncoding -> IO () and various encodings: latin1, utf8, utf16, utf16le, utf16be, utf32, utf32le, utf32be, localeEncoding, and a way to lookup other encodings: mkTextEncoding :: String -> IO TextEncoding (it's system-dependent whether the requested encoding will be available). Currently hSetEncoding is availble from GHC.IO.Handle, and the encodings are available from GHC.IO.Encoding. We may want to export these from somewhere more permanent; that's something for a library proposal. Thanks to suggestions from Duncan Coutts, it's possible to call hSetEncoding even on buffered read Handles, and the right thing happens. So we can read from text streams that include multiple encodings, such as an HTTP response or email message, without having to turn buffering off (though there is a penalty for switching encodings on a buffered Handle, as the IO system has to do some re-decoding to figure out where it should start reading from again). If there is a decoding error, it is reported when an attempt is made to read the offending character from the Handle, as you would expect. Performance is about 30% slower on "hGetContents >>= putStr" than before. I've profiled it, and about 25% of this is in doing the actual encoding/decoding, the rest is accounted for by the fact that we're shuffling around 32-bit chars rather than bytes in the Handle buffer, so there's not much we can do to improve this. IO library restructuring ~~~~~~~~~~~~~~~~~~~~~~~~ The major change here is that the implementation of the Handle operations is separated from the underlying IO device, using type classes. File descriptors are just one IO provider; I have also implemented memory-mapped files (good for random-access read/write) and a Handle that pipes output to a Chan (useful for testing code that writes to a Handle). New kinds of Handle can be implemented outside the base package, for instance someone could write bytestringToHandle. A Handle is made using mkFileHandle: -- | makes a new 'Handle' mkFileHandle :: (IODevice dev, BufferedIO dev, Typeable dev) => dev -- ^ the underlying IO device, which must support -- 'IODevice', 'BufferedIO' and 'Typeable' -> FilePath -- ^ a string describing the 'Handle', e.g. the file -- path for a file. Used in error messages. -> IOMode -- ^ The mode in which the 'Handle' is to be used -> Maybe TextEncoding -- ^ text encoding to use, if any -> IO Handle This also means that someone can write a completely new IO implementation on Windows based on native Win32 HANDLEs, and distribute it as a separate package (I really hope somebody does this!). This restructuring isn't as radical as previous designs. I haven't made any attempt to make a separate binary I/O layer, for example (although hGetBuf/hPutBuf do bypass the text encoding). The main goal here was to get Unicode support in, and to allow others to experiment with making new kinds of Handle. We could split up the layers further later. API changes and Module structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NB. GHC.IOBase and GHC.Handle are now DEPRECATED (they are still present, but are just re-exporting things from other modules now). For 6.12 we'll want to bump base to version 5 and add a base4-compat. For now I'm using #if __GLASGOW_HASKEL__ >= 611 to avoid deprecated warnings. I split modules into smaller parts in many places. For example, we now have GHC.IORef, GHC.MVar and GHC.IOArray containing the implementations of IORef, MVar and IOArray respectively. This was necessary for untangling dependencies, but it also makes things easier to follow. The new module structurue for the IO-relatied parts of the base package is: GHC.IO Implementation of the IO monad; unsafe*; throw/catch GHC.IO.IOMode The IOMode type GHC.IO.Buffer Buffers and operations on them GHC.IO.Device The IODevice and RawIO classes. GHC.IO.BufferedIO The BufferedIO class. GHC.IO.FD The FD type, with instances of IODevice, RawIO and BufferedIO. GHC.IO.Exception IO-related Exceptions GHC.IO.Encoding The TextEncoding type; built-in TextEncodings; mkTextEncoding GHC.IO.Encoding.Types GHC.IO.Encoding.Iconv Implementation internals for GHC.IO.Encoding GHC.IO.Handle The main API for GHC's Handle implementation, provides all the Handle operations + mkFileHandle + hSetEncoding. GHC.IO.Handle.Types GHC.IO.Handle.Internals GHC.IO.Handle.Text Implementation of Handles and operations. GHC.IO.Handle.FD Parts of the Handle API implemented by file-descriptors: openFile, stdin, stdout, stderr, fdToHandle etc. From jgoerzen at complete.org Tue Feb 3 12:03:58 2009 From: jgoerzen at complete.org (John Goerzen) Date: Tue Feb 3 11:54:09 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <49887404.4070504@gmail.com> References: <49887404.4070504@gmail.com> Message-ID: <498878FE.2040502@complete.org> Simon Marlow wrote: > I've been working on adding proper Unicode support to Handle I/O in GHC, > and I finally have something that's ready for testing. I've put a patchset > here: Yay! Comments below. > Comments/discussion please! Do you expect Hugs will be able to pick up all of this? > The only change to the existing behaviour is that by default, text IO > is done in the prevailing encoding of the system. Handles created by > openBinaryFile use the Latin-1 encoding, as do Handles placed in > binary mode using hSetBinaryMode. Sounds very good and reasonable. > We provide a way to change the encoding for an existing Handle: > > hSetEncoding :: Handle -> TextEncoding -> IO () > > and various encodings: > > latin1, > utf8, > utf16, utf16le, utf16be, > utf32, utf32le, utf32be, > localeEncoding, Will there also be something to handle the UTF-16 BOM marker? I'm not sure what the best API for that is, since it may or may not be present, but it should be considered -- and could perhaps help autodetect encoding. > Thanks to suggestions from Duncan Coutts, it's possible to call > hSetEncoding even on buffered read Handles, and the right thing > happens. So we can read from text streams that include multiple > encodings, such as an HTTP response or email message, without having > to turn buffering off (though there is a penalty for switching > encodings on a buffered Handle, as the IO system has to do some > re-decoding to figure out where it should start reading from again). Sounds useful, but is this the bit that causes the 30% performance hit? > Performance is about 30% slower on "hGetContents >>= putStr" than > before. I've profiled it, and about 25% of this is in doing the > actual encoding/decoding, the rest is accounted for by the fact that > we're shuffling around 32-bit chars rather than bytes in the Handle > buffer, so there's not much we can do to improve this. Does this mean that if we set the encoding to latin1, tat we should see performance 5% worse than present? 30% slower is a big deal, especially since we're not all that speedy now. > IO library restructuring > ~~~~~~~~~~~~~~~~~~~~~~~~ > > The major change here is that the implementation of the Handle > operations is separated from the underlying IO device, using type > classes. File descriptors are just one IO provider; I have also > implemented memory-mapped files (good for random-access read/write) > and a Handle that pipes output to a Chan (useful for testing code that > writes to a Handle). New kinds of Handle can be implemented outside > the base package, for instance someone could write bytestringToHandle. > A Handle is made using mkFileHandle: Very nice. That means I can eliminate all the HVIO stuff I have in MissingH, which does roughly the same thing. > with making new kinds of Handle. We could split up the layers further > later. Would it now be possible to make the Socket an instance of this typeclass, so we can work with it directly rather than having to convert it to a Handle first? Thanks, -- John From duncan.coutts at worc.ox.ac.uk Tue Feb 3 17:56:13 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 3 17:46:16 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <498878FE.2040502@complete.org> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> Message-ID: <1233701773.26754.590.camel@localhost> On Tue, 2009-02-03 at 11:03 -0600, John Goerzen wrote: > Will there also be something to handle the UTF-16 BOM marker? I'm not > sure what the best API for that is, since it may or may not be present, > but it should be considered -- and could perhaps help autodetect encoding. I think someone else mentioned this already, but utf16 (as opposed to utf16be/le) will use the BOM if its present. I'm not quite sure what happens when you switch encoding, presumably it'll accept and consider a BOM at that point. > > Thanks to suggestions from Duncan Coutts, it's possible to call > > hSetEncoding even on buffered read Handles, and the right thing > > happens. So we can read from text streams that include multiple > > encodings, such as an HTTP response or email message, without having > > to turn buffering off (though there is a penalty for switching > > encodings on a buffered Handle, as the IO system has to do some > > re-decoding to figure out where it should start reading from again). > > Sounds useful, but is this the bit that causes the 30% performance hit? No. You only pay that penalty if you switch encoding. The standard case has no extra cost. > > Performance is about 30% slower on "hGetContents >>= putStr" than > > before. I've profiled it, and about 25% of this is in doing the > > actual encoding/decoding, the rest is accounted for by the fact that > > we're shuffling around 32-bit chars rather than bytes in the Handle > > buffer, so there's not much we can do to improve this. > > Does this mean that if we set the encoding to latin1, tat we should see > performance 5% worse than present? No, I think that's 30% for latin1. The cost is not really the character conversion but the copying from a byte buffer via iconv to a char buffer. > 30% slower is a big deal, especially since we're not all that speedy now. Bear in mind that's talking about the [Char] interface, and nobody using that is expecting great performance. We already have an API for getting big chunks of bytes out of a Handle, with the new Handle we'll also want something equivalent for a packed text representation. Hopefully we can get something nice with the new text package. Duncan From jgoerzen at complete.org Tue Feb 3 18:39:43 2009 From: jgoerzen at complete.org (John Goerzen) Date: Tue Feb 3 18:29:58 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <1233701773.26754.590.camel@localhost> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> Message-ID: <20090203233943.GA10645@complete.org> On Tue, Feb 03, 2009 at 10:56:13PM +0000, Duncan Coutts wrote: > > > Thanks to suggestions from Duncan Coutts, it's possible to call > > > hSetEncoding even on buffered read Handles, and the right thing > > > happens. So we can read from text streams that include multiple > > > encodings, such as an HTTP response or email message, without having > > > to turn buffering off (though there is a penalty for switching > > > encodings on a buffered Handle, as the IO system has to do some > > > re-decoding to figure out where it should start reading from again). > > > > Sounds useful, but is this the bit that causes the 30% performance hit? > > No. You only pay that penalty if you switch encoding. The standard case > has no extra cost. I'm confused. I thought the standard case was conversion to the system's local encoding? How is that different than selecting the same encoding manually? There always has to be *some* conversion from a 32-bit Char to the system's selection, right? What exactly do we have to do to avoid the penalty? > No, I think that's 30% for latin1. The cost is not really the character > conversion but the copying from a byte buffer via iconv to a char > buffer. Don't we already have to copy between a byte buffer and a char buffer, since read() and write() use a byte buffer? > > 30% slower is a big deal, especially since we're not all that speedy now. > > Bear in mind that's talking about the [Char] interface, and nobody using > that is expecting great performance. We already have an API for getting Yes, I know, but it's still the most convenient interface, and making it suck more isn't cool -- though there are certainly big wins here. -- John From ross at soi.city.ac.uk Tue Feb 3 18:50:53 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Tue Feb 3 18:41:04 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <49887404.4070504@gmail.com> References: <49887404.4070504@gmail.com> Message-ID: <20090203235053.GA18429@soi.city.ac.uk> On Tue, Feb 03, 2009 at 04:42:44PM +0000, Simon Marlow wrote: > Unicode-aware Handles > ~~~~~~~~~~~~~~~~~~~~~ > > This is a significant restructuring of the Handle implementation with > the primary goal of supporting Unicode character encodings. > > The only change to the existing behaviour is that by default, text IO > is done in the prevailing encoding of the system. Handles created by > openBinaryFile use the Latin-1 encoding, as do Handles placed in > binary mode using hSetBinaryMode. I would like to see the API make a type distinction between text handles with Char-based operations, and binary handles with Word8-based operations. (And phase out openBinaryFile, hSetBinaryMode etc.) It's a vital distinction that often trips people up. Treating binary data as Chars with a Latin-1 encoding just feels like a kludge. From duncan.coutts at worc.ox.ac.uk Tue Feb 3 20:32:59 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 3 20:22:52 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <20090203233943.GA10645@complete.org> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> <20090203233943.GA10645@complete.org> Message-ID: <1233711179.26754.616.camel@localhost> On Tue, 2009-02-03 at 17:39 -0600, John Goerzen wrote: > On Tue, Feb 03, 2009 at 10:56:13PM +0000, Duncan Coutts wrote: > > > > Thanks to suggestions from Duncan Coutts, it's possible to call > > > > hSetEncoding even on buffered read Handles, and the right thing > > > > happens. So we can read from text streams that include multiple > > > > encodings, such as an HTTP response or email message, without having > > > > to turn buffering off (though there is a penalty for switching > > > > encodings on a buffered Handle, as the IO system has to do some > > > > re-decoding to figure out where it should start reading from again). > > > > > > Sounds useful, but is this the bit that causes the 30% performance hit? > > > > No. You only pay that penalty if you switch encoding. The standard case > > has no extra cost. > > I'm confused. I thought the standard case was conversion to the > system's local encoding? How is that different than selecting the > same encoding manually? Sorry, I think we've been talking at cross purposes. > There always has to be *some* conversion from a 32-bit Char to the > system's selection, right? Yes. In text mode there is always some conversion going on. Internally there is a byte buffer and a char buffer (ie UTF32). > What exactly do we have to do to avoid the penalty? The penalty we're talking about here is not the cost of converting bytes to characters, it's in switching which encoding the Handle is using. For example you might read some HTTP headers in ASCII and then switch the Handle encoding to UTF8 to read some XML. Switching the Handle encoding has a penalty. We have to discard the characters that we pre-decoded and re-decode the byte buffer in the new encoding. It's actually slightly more complicated because we do not track exactly how the byte and character buffers relate to each other (it'd be too expensive in the normal cases) so to work out the relationship when switching encoding we have to re-decode all the way from the beginning of the current byte buffer. The point is, in terms of performance we get the ability to switch handle encoding more or less for free. It has a cost in terms of code complexity. The simpler alternative design was that you would not be able to switch encoding on a read handle that used any buffering at the character level without loosing bytes. The performance penalty when switching encoding is the downside to the ordinary code path being fast. > > No, I think that's 30% for latin1. The cost is not really the character > > conversion but the copying from a byte buffer via iconv to a char > > buffer. > > Don't we already have to copy between a byte buffer and a char buffer, > since read() and write() use a byte buffer? In the existing Handle mechanism we read() into a byte buffer and then when doing say getLine or getContents we allocate [Char]'s in a loop reading bytes directly from the byte buffer. There is no separate character buffer. Duncan From jgoerzen at complete.org Tue Feb 3 22:49:10 2009 From: jgoerzen at complete.org (John Goerzen) Date: Tue Feb 3 22:39:27 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <1233711179.26754.616.camel@localhost> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> <20090203233943.GA10645@complete.org> <1233711179.26754.616.camel@localhost> Message-ID: <49891036.1060100@complete.org> Duncan Coutts wrote: > Sorry, I think we've been talking at cross purposes. I think so. >> There always has to be *some* conversion from a 32-bit Char to the >> system's selection, right? > > Yes. In text mode there is always some conversion going on. Internally > there is a byte buffer and a char buffer (ie UTF32). > >> What exactly do we have to do to avoid the penalty? > > The penalty we're talking about here is not the cost of converting bytes > to characters, it's in switching which encoding the Handle is using. For > example you might read some HTTP headers in ASCII and then switch the > Handle encoding to UTF8 to read some XML. Simon referenced a 30% penalty. Are you saying that if we read from a Handle using the same encoding that we used when we first opened it, that we won't see any slowdown vs. the system in 6.10? > Switching the Handle encoding has a penalty. We have to discard the > characters that we pre-decoded and re-decode the byte buffer in the new > encoding. It's actually slightly more complicated because we do not Got it. That makes sense, as does the decision to optimize for the more common (not switching the encoding) case. -- John From marlowsd at gmail.com Wed Feb 4 04:16:57 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Feb 4 04:07:12 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <20090203235053.GA18429@soi.city.ac.uk> References: <49887404.4070504@gmail.com> <20090203235053.GA18429@soi.city.ac.uk> Message-ID: <49895D09.5010900@gmail.com> Ross Paterson wrote: > On Tue, Feb 03, 2009 at 04:42:44PM +0000, Simon Marlow wrote: >> Unicode-aware Handles >> ~~~~~~~~~~~~~~~~~~~~~ >> >> This is a significant restructuring of the Handle implementation with >> the primary goal of supporting Unicode character encodings. >> >> The only change to the existing behaviour is that by default, text IO >> is done in the prevailing encoding of the system. Handles created by >> openBinaryFile use the Latin-1 encoding, as do Handles placed in >> binary mode using hSetBinaryMode. > > I would like to see the API make a type distinction between text > handles with Char-based operations, and binary handles with Word8-based > operations. (And phase out openBinaryFile, hSetBinaryMode etc.) It's a > vital distinction that often trips people up. Treating binary data as > Chars with a Latin-1 encoding just feels like a kludge. Yes I agree, but I think we should tackle that separately. The important thing right now is that we can do I/O in the local encoding, and that existing programs that depend on reading/writing binary data using openBinaryFile will continue to work. Let's get this in now, and then talk about a binary I/O layer later. The restructuring I've done in the IO library should make it easier to separate the binary and text layers now too. Cheers, Simon From marlowsd at gmail.com Wed Feb 4 04:26:06 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Feb 4 04:16:18 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <49891036.1060100@complete.org> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> <20090203233943.GA10645@complete.org> <1233711179.26754.616.camel@localhost> <49891036.1060100@complete.org> Message-ID: <49895F2E.9000807@gmail.com> John Goerzen wrote: > Duncan Coutts wrote: >> Sorry, I think we've been talking at cross purposes. > > I think so. > >>> There always has to be *some* conversion from a 32-bit Char to the >>> system's selection, right? >> Yes. In text mode there is always some conversion going on. Internally >> there is a byte buffer and a char buffer (ie UTF32). >> >>> What exactly do we have to do to avoid the penalty? >> The penalty we're talking about here is not the cost of converting bytes >> to characters, it's in switching which encoding the Handle is using. For >> example you might read some HTTP headers in ASCII and then switch the >> Handle encoding to UTF8 to read some XML. > > Simon referenced a 30% penalty. Are you saying that if we read from a > Handle using the same encoding that we used when we first opened it, > that we won't see any slowdown vs. the system in 6.10? No, there's a fixed 30% penalty for hGetContents/readFile/hPutStr in the new library, regardless of what encoding you're using. Presumably if you're using a complex encoding there will be an extra penalty imposed by iconv. The cost is mostly in decoding (or copying, in the case of latin1) bytes from the byte buffer into characters in the character buffer. Previously there was only a byte buffer. I was surprised at the slowdown too, so I looked into it. As it turns out, hGetContents and hPutStr are actually quite well optimised already: virtually all the allocation is accounted for by the [Char], and we have good tight inner loops, so the cost of shuffling between the byte buffer and the char buffer is quite noticeable. We could add a special-case for the latin1 encoding and eliminate the intermediate char buffer, but that would add significant complexity to the code, and it's not the right way to go about it. If you want binary data and speed, then hGetBuf/hPutBuf (perhaps via bytestring) should be as fast as before. Cheers, Simon From paolo at hypersonic.it Wed Feb 4 05:33:21 2009 From: paolo at hypersonic.it (Paolo Losi) Date: Wed Feb 4 05:35:08 2009 Subject: Ready for testing: Unicode support for Handle I/O In-Reply-To: <49887404.4070504@gmail.com> References: <49887404.4070504@gmail.com> Message-ID: <49896EF1.3080700@hypersonic.it> Simon Marlow wrote: > The only change to the existing behaviour is that by default, text IO > is done in the prevailing encoding of the system. Handles created by > openBinaryFile use the Latin-1 encoding, as do Handles placed in > binary mode using hSetBinaryMode. wouldn't be semantically correct for a "binary handle" to "return" [Word8]? also switching from text to binary (hSetBinaryMode) doesn't seem "natural" I understand that this has "heavy" consequences... Pao From max.vasin at gmail.com Wed Feb 4 06:32:51 2009 From: max.vasin at gmail.com (Max Vasin) Date: Wed Feb 4 06:30:06 2009 Subject: Ready for testing: Unicode support for Handle I/O In-Reply-To: <49896EF1.3080700@hypersonic.it> References: <49887404.4070504@gmail.com> <49896EF1.3080700@hypersonic.it> Message-ID: Paolo Losi ?????: > wouldn't be semantically correct for a "binary handle" > to "return" [Word8]? Wouldn't it be more correct to separate binary IO, which return [Word8] (or ByteString) and text IO which return [Char] and deal with text encoding? IIRC that was done in Bulat Ziganshin's streams library. -- WBR, Max Vasin. From paolo at hypersonic.it Wed Feb 4 06:53:18 2009 From: paolo at hypersonic.it (Paolo Losi) Date: Wed Feb 4 06:50:09 2009 Subject: Ready for testing: Unicode support for Handle I/O In-Reply-To: References: <49887404.4070504@gmail.com> <49896EF1.3080700@hypersonic.it> Message-ID: <498981AE.4040006@hypersonic.it> Max Vasin wrote: > Wouldn't it be more correct to separate binary IO, which > return [Word8] (or ByteString) and text IO which return > [Char] and deal with text encoding? IIRC that was done in > Bulat Ziganshin's streams library. That's exactly what I meant. Text IO could be then implemented on top of binary IO. Would it be possible to envision a solution that enables composition of IO low level strategies vs binary <-> text conversion strategies? Pao From marlowsd at gmail.com Wed Feb 4 08:25:11 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Feb 4 08:15:26 2009 Subject: Ready for testing: Unicode support for Handle I/O In-Reply-To: <49896EF1.3080700@hypersonic.it> References: <49887404.4070504@gmail.com> <49896EF1.3080700@hypersonic.it> Message-ID: <49899737.5020106@gmail.com> Paolo Losi wrote: > Simon Marlow wrote: > >> The only change to the existing behaviour is that by default, text IO >> is done in the prevailing encoding of the system. Handles created by >> openBinaryFile use the Latin-1 encoding, as do Handles placed in >> binary mode using hSetBinaryMode. > > wouldn't be semantically correct for a "binary handle" > to "return" [Word8]? > > also switching from text to binary (hSetBinaryMode) > doesn't seem "natural" Yes, and as I said in the original message I haven't done the binary/text separation (yet). I agree it's something that should be done, and the current API leaves a lot to be desired, but the main goal was to get Unicode text I/O working without breaking any existing code (or at least without breaking any code that isn't already morally broken :-). As a side-effect I managed to do some useful refactoring which should make further separation of layers much easier. So you should think of this as a step in the right direction, with further steps to come in the future. A while back there was a lot of activity on developing new IO library designs. There are a bunch of these: Bulat's streams library, a variant of Bulat's done by Takano Akio, John Goerzen's HVIO, and I had a prototype streams library too. The problem is, it's a lot of work to make a complete IO library implementation, agree on the API, and migrate over from the old one. And while we're on the subject of redesigning IO libraries, it's not at all clear that the imperative approach is the right one either. Cheers, Simon From marlowsd at gmail.com Wed Feb 4 08:31:20 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Feb 4 08:21:38 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <1233701773.26754.590.camel@localhost> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> Message-ID: <498998A8.1060909@gmail.com> Duncan Coutts wrote: > On Tue, 2009-02-03 at 11:03 -0600, John Goerzen wrote: > >> Will there also be something to handle the UTF-16 BOM marker? I'm not >> sure what the best API for that is, since it may or may not be present, >> but it should be considered -- and could perhaps help autodetect encoding. > > I think someone else mentioned this already, but utf16 (as opposed to > utf16be/le) will use the BOM if its present. > > I'm not quite sure what happens when you switch encoding, presumably > it'll accept and consider a BOM at that point. Yes; the utf16 and utf32 encodings accept a BOM (and generate a BOM in write mode). This caused interesting bugs when doing re-decoding after switching encodings, because the BOM constitutes state in the decoder, which means that decoding is not necessarily repeatable unless you save the state (which iconv doesn't provide a way to do). Are there other encodings that have this kind of state? If so, then they might be restricted to NoBuffering at least when switching encodings. Cheers, Simon From duncan.coutts at worc.ox.ac.uk Wed Feb 4 08:46:16 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Wed Feb 4 08:36:08 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <498998A8.1060909@gmail.com> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> <498998A8.1060909@gmail.com> Message-ID: <1233755176.26754.617.camel@localhost> On Wed, 2009-02-04 at 13:31 +0000, Simon Marlow wrote: > Duncan Coutts wrote: > > On Tue, 2009-02-03 at 11:03 -0600, John Goerzen wrote: > > > >> Will there also be something to handle the UTF-16 BOM marker? I'm not > >> sure what the best API for that is, since it may or may not be present, > >> but it should be considered -- and could perhaps help autodetect encoding. > > > > I think someone else mentioned this already, but utf16 (as opposed to > > utf16be/le) will use the BOM if its present. > > > > I'm not quite sure what happens when you switch encoding, presumably > > it'll accept and consider a BOM at that point. > > Yes; the utf16 and utf32 encodings accept a BOM (and generate a BOM in > write mode). This caused interesting bugs when doing re-decoding after > switching encodings, because the BOM constitutes state in the decoder, > which means that decoding is not necessarily repeatable unless you save the > state (which iconv doesn't provide a way to do). > > Are there other encodings that have this kind of state? If so, then they > might be restricted to NoBuffering at least when switching encodings. Yes, I believe there are some Asian encodings that are stateful. Duncan From ross at soi.city.ac.uk Wed Feb 4 08:52:36 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Wed Feb 4 08:42:43 2009 Subject: [Haskell-cafe] Ready for testing: Unicode support for Handle I/O In-Reply-To: <1233755176.26754.617.camel@localhost> References: <49887404.4070504@gmail.com> <498878FE.2040502@complete.org> <1233701773.26754.590.camel@localhost> <498998A8.1060909@gmail.com> <1233755176.26754.617.camel@localhost> Message-ID: <20090204135236.GA21097@soi.city.ac.uk> On Wed, Feb 04, 2009 at 01:46:16PM +0000, Duncan Coutts wrote: > On Wed, 2009-02-04 at 13:31 +0000, Simon Marlow wrote: > > Yes; the utf16 and utf32 encodings accept a BOM (and generate a BOM in > > write mode). This caused interesting bugs when doing re-decoding after > > switching encodings, because the BOM constitutes state in the decoder, > > which means that decoding is not necessarily repeatable unless you save the > > state (which iconv doesn't provide a way to do). > > > > Are there other encodings that have this kind of state? If so, then they > > might be restricted to NoBuffering at least when switching encodings. > > Yes, I believe there are some Asian encodings that are stateful. Yes, ISO-2022 encodings, like ISO-2022-JP, have a number of states and switch between them with escape sequences. They're a nightmare. From dave.a.tapley at gmail.com Wed Feb 4 17:52:32 2009 From: dave.a.tapley at gmail.com (Dave Tapley) Date: Wed Feb 4 17:42:35 2009 Subject: Broken dependencies on cabal install haddock Message-ID: Hi everyone, So I just grabbed the cabal and cabal-install tarballs and got them build without trouble. I thought I'd try it out by installing haddock but I get the following rather nonsensical output: ------------------------------------------------------------------------------------------------------ dave@eliza:~$ cabal --version cabal-install version 0.6.0 using version 1.6.0.1 of the Cabal library dave@eliza:~$ cabal install haddock Resolving dependencies... cabal: cannot configure haddock-2.4.1. It requires ghc >=6.8.2 && <6.13 There is no available version of ghc that satisfies >=6.8.2 && <6.13 dave@eliza:~$ ghc --version The Glorious Glasgow Haskell Compilation System, version 6.8.2 ------------------------------------------------------------------------------------------------------ I assume I'm not alone in thinking that 6.8.2 >= 6.8.2 is true? :) Cheers, Dave From ross at soi.city.ac.uk Wed Feb 4 17:59:31 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Wed Feb 4 17:49:37 2009 Subject: Broken dependencies on cabal install haddock In-Reply-To: References: Message-ID: <20090204225931.GA2813@soi.city.ac.uk> On Wed, Feb 04, 2009 at 10:52:32PM +0000, Dave Tapley wrote: > dave@eliza:~$ cabal install haddock > Resolving dependencies... > cabal: cannot configure haddock-2.4.1. It requires ghc >=6.8.2 && <6.13 > There is no available version of ghc that satisfies >=6.8.2 && <6.13 > dave@eliza:~$ ghc --version > The Glorious Glasgow Haskell Compilation System, version 6.8.2 There is also a library package called ghc. From david.waern at gmail.com Wed Feb 4 18:03:30 2009 From: david.waern at gmail.com (David Waern) Date: Wed Feb 4 17:53:34 2009 Subject: Broken dependencies on cabal install haddock In-Reply-To: <20090204225931.GA2813@soi.city.ac.uk> References: <20090204225931.GA2813@soi.city.ac.uk> Message-ID: 2009/2/4 Ross Paterson : > On Wed, Feb 04, 2009 at 10:52:32PM +0000, Dave Tapley wrote: >> dave@eliza:~$ cabal install haddock >> Resolving dependencies... >> cabal: cannot configure haddock-2.4.1. It requires ghc >=6.8.2 && <6.13 >> There is no available version of ghc that satisfies >=6.8.2 && <6.13 >> dave@eliza:~$ ghc --version >> The Glorious Glasgow Haskell Compilation System, version 6.8.2 > > There is also a library package called ghc. Which should be installed if you have the compiler. Hmm. David From duncan.coutts at worc.ox.ac.uk Wed Feb 4 18:54:55 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Wed Feb 4 18:44:42 2009 Subject: Broken dependencies on cabal install haddock In-Reply-To: References: <20090204225931.GA2813@soi.city.ac.uk> Message-ID: <1233791695.26754.648.camel@localhost> On Thu, 2009-02-05 at 00:03 +0100, David Waern wrote: > 2009/2/4 Ross Paterson : > > On Wed, Feb 04, 2009 at 10:52:32PM +0000, Dave Tapley wrote: > >> dave@eliza:~$ cabal install haddock > >> Resolving dependencies... > >> cabal: cannot configure haddock-2.4.1. It requires ghc >=6.8.2 && <6.13 > >> There is no available version of ghc that satisfies >=6.8.2 && <6.13 > >> dave@eliza:~$ ghc --version > >> The Glorious Glasgow Haskell Compilation System, version 6.8.2 > > > > There is also a library package called ghc. > > Which should be installed if you have the compiler. Hmm. If the package is broken due to missing dependencies then cabal-install ignores it. It's on my TODO list to make it report more clearly the fact that it needed a package that is broken (ie instead of removing the broken packages from the set of installed packages, add a constraint not to use that version and when we get a constraint conflict with we can report a more helpful message). Dave, you can confirm if the ghc package is broken by running ghc-pkg check. Duncan From claudiusmaximus at goto10.org Thu Feb 5 07:44:13 2009 From: claudiusmaximus at goto10.org (Claude Heiland-Allen) Date: Thu Feb 5 07:34:16 2009 Subject: piping with System.Process.createProcess In-Reply-To: <20090106112214.GA4225@zombie.inf.tu-dresden.de> References: <4961A65C.5050106@goto10.org> <20090106112214.GA4225@zombie.inf.tu-dresden.de> Message-ID: <498ADF1D.9070309@goto10.org> Bertram Felgenhauer wrote: > Claude Heiland-Allen wrote: >> Greetings, >> >> I think I found a bug in the process package. >> >> GNU/Linux Debian Lenny >> ghc-6.8.2 >> process-1.0.1.1 > > Hmm, I can't reproduce it in ghc 6.8.3 or 6.10.1. IIRC, there were some > changes with regards to pipes and broken pipe errors between 6.8.2 and > 6.8.3, but details elude me. > > Can you try a newer ghc? http://hackage.haskell.org/trac/ghc/ticket/2233 mentions this: """" - Don't use O_NONBLOCK for pipes, as it can confuse the process attached to the pipe (requires a fix to GHC.Handle in the base package). """" Thus I assume the issue is known, and will be fixed if it hasn't already been fixed (I haven't had time to check with newer GHC...). Thanks, Claude From ross at soi.city.ac.uk Thu Feb 5 08:53:12 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Thu Feb 5 08:43:21 2009 Subject: MaybeT to transformers In-Reply-To: References: Message-ID: <20090205135312.GB3405@soi.city.ac.uk> On Tue, Feb 03, 2009 at 02:10:14PM +0100, Henning Thielemann wrote: > What about adding the Maybe monad transformer as provided in MaybeT > package to the transformers package? > > The module name could be > Control.Monad.Trans.Maybe > which would also avoid conflicts with the MaybeT package. Sounds reasonable. From dominic.steinitz at blueyonder.co.uk Thu Feb 5 08:45:42 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Thu Feb 5 09:12:40 2009 Subject: Bug in Cabal? References: <1232529549.8432.229.camel@localhost> <1233148429.26754.105.camel@localhost> <4985A80F.7090809@blueyonder.co.uk> <1233511035.26754.400.camel@localhost> Message-ID: Duncan Coutts worc.ox.ac.uk> writes: > > On Sun, 2009-02-01 at 13:47 +0000, Dominic Steinitz wrote: > > Duncan Coutts wrote: > > > Hia Dominic, > > > > > > I've looked into this a bit. I think I may now be less confused. > > > > > > On Wed, 2009-01-21 at 18:01 +0000, Dominic Steinitz wrote: > > > > > >>> ls -ltr haskell_packages\doc\zlib-0.5.0.0 > > >> total 2 > > >> -r--r--r-- 1 user group 1327 Jan 21 11:37 LICENSE > > > Having installed darcs and beaten our proxy to allow darcs to "get", I installed the patched 1.6 version and all seems to work. >ls -ltr haskell_packages\doc\zlib-0.5.0.0 total 2 -rw-rw-rw- 1 user group 1327 Feb 5 13:29 LICENSE I had to make the following change in CopyFile.hs before it would compile so maybe it needs a further slight tweak? setFileOrdinary = return () setFileExecutable = return () to setFileOrdinary _ = return () setFileExecutable _ = return () Many thanks for your help, Dominic. From byorgey at seas.upenn.edu Thu Feb 5 17:05:22 2009 From: byorgey at seas.upenn.edu (Brent Yorgey) Date: Thu Feb 5 16:55:22 2009 Subject: MaybeT to transformers In-Reply-To: References: Message-ID: <20090205220522.GA22856@seas.upenn.edu> On Tue, Feb 03, 2009 at 02:10:14PM +0100, Henning Thielemann wrote: > > What about adding the Maybe monad transformer as provided in MaybeT package > to the transformers package? > > The module name could be > Control.Monad.Trans.Maybe > which would also avoid conflicts with the MaybeT package. +1 I have been surprised on several occasions that MaybeT is not in the mtl. -Brent From duncan.coutts at worc.ox.ac.uk Thu Feb 5 17:44:05 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Thu Feb 5 17:33:47 2009 Subject: Bug in Cabal? In-Reply-To: References: <1232529549.8432.229.camel@localhost> <1233148429.26754.105.camel@localhost> <4985A80F.7090809@blueyonder.co.uk> <1233511035.26754.400.camel@localhost> Message-ID: <1233873845.26754.685.camel@localhost> On Thu, 2009-02-05 at 13:45 +0000, Dominic Steinitz wrote: > Having installed darcs and beaten our proxy to allow darcs to "get", I > installed the patched 1.6 version and all seems to work. > > >ls -ltr haskell_packages\doc\zlib-0.5.0.0 > total 2 > -rw-rw-rw- 1 user group 1327 Feb 5 13:29 LICENSE Great. > I had to make the following change in CopyFile.hs before it would compile so > maybe it needs a further slight tweak? > > setFileOrdinary = return () > setFileExecutable = return () > > to > > setFileOrdinary _ = return () > setFileExecutable _ = return () Oops, thanks. Duncan From ganesh.sittampalam at credit-suisse.com Fri Feb 6 03:20:51 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Fri Feb 6 03:13:11 2009 Subject: MaybeT to transformers In-Reply-To: References: Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B4DCBAA@ELON17P32001A.csfb.cs-group.com> Henning Thielemann wrote: > What about adding the Maybe monad transformer as provided in MaybeT > package to the transformers package? > > The module name could be > Control.Monad.Trans.Maybe > which would also avoid conflicts with the MaybeT package. What MonadPlus instance will it get, if any? Ganesh ============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ============================================================================== From Christian.Maeder at dfki.de Fri Feb 6 11:20:55 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Fri Feb 6 11:10:44 2009 Subject: Proposal for Data.List.splitBy In-Reply-To: <49730C53.7080009@gabriel.name> References: <4961265C.2090709@gabriel.name> <1231620312.4794.10.camel@localhost> <496A4531.9050902@gabriel.name> <20090111235612.GA9642@seas.upenn.edu> <49730C53.7080009@gabriel.name> Message-ID: <498C6367.9020708@dfki.de> Marcus D. Gabriel wrote: > If I were to write > > organizeBy :: ([a] -> Bool) -> [a] -> [([a], [a])] I quite like your idea, but I think the input predicate "([a] -> Bool)" is too ambitious, although it would nicely unify Brent's data Delimiter a where DelimEltPred :: (a -> Bool) -> Delimiter a DelimSublist :: Eq a => [a] -> Delimiter a With the predicate ([a] -> Bool) you have to check all inits of your input to detect a delimiter and only if all inits are no delimiter you know the head element is not part of a delimiter and repeat checking the inits of the tail. I think this is too inefficient in general. In order to keep things simple I would vote for a split function that only takes the simple predicate (a -> Bool) and leaves the more complicated splitting to _additional_ functions (following http://haskell.org/haskellwiki/Simple_to_complex) For instance replace :: [a] -> a -> [a] -> [a] could replace a sublist (first argument) with a single element (second argument) This would help in simple cases, but leaves it to the user to choose a suitable delimiter element. But a general splitting on sublists could be implemented via splitBy isNothing (replace (map Just sl) Nothing (map Just l)) (For a fixed sublist sl, "Nothing" is enough to represent the delimiter) For a simple predicate "(a -> Bool)" it remains to discuss the output of splitBy. You've proposed "[([a], [a])]", but for the simpler case "[(a, [a])]" or "[([a], a)]" may do, but in order to capture the full information of lists before and after delimiters, something like "([a], [(a, [a])])" is needed. Since a tuple "(a, [a])" can be viewed as non-empty list of type "[a]", "([a], [(a, [a])])" collapses to a non-empty list of type "[[a]]" with a _tail_ of non-empty element lists. Therefore I propose the following splitBy as a "work horse". splitBy :: (a -> Bool) -> [a] -> [[a]] The implementation is simply using "break". The basic property is: concat (splitBy p l) == l (This is almost in the spirit of Data.List, since groupBy or words also produces non-empty element lists.) Getting rid of a final delimiter or all of them can be implemented in O(n) (see below). And finally: wordsBy p = filter (not . null) . dropDelims . splitBy p linesBy p = dropDelims . dropFinalDelim . splitBy p with: words s == wordsBy isSpace s lines s == linesBy (== '\n') s Surely, one might prefer a more type-safe version wrt non-empty lists, but the main idea is to use function composition rather than composing a (non-haskell98) splitter input data type as Brent does in his split package. Cheers Christian import Data.Char (isSpace) splitBy :: (a -> Bool) -> [a] -> [[a]] splitBy p l = let (fr, rt) = break p l in case rt of [] -> [fr] d : tl -> let hd : tll = splitBy p tl in fr : (d : hd) : tll dropFinalDelim :: [[a]] -> [[a]] dropFinalDelim ll@(l : ls) = if null ls then if null l then [] else ll else if null (tail (last ls)) then init ll else ll dropDelims :: [[a]] -> [[a]] dropDelims ll = case ll of [] -> [] l : ls -> l : map tail ls wordsBy :: (a -> Bool) -> [a] -> [[a]] wordsBy p = filter (not . null) . dropDelims . splitBy p linesBy :: (a -> Bool) -> [a] -> [[a]] linesBy p = dropDelims . dropFinalDelim . splitBy p prop_wordsBy :: String -> Bool prop_wordsBy s = words s == wordsBy isSpace s prop_linesBy :: String -> Bool prop_linesBy s = lines s == linesBy (== '\n') s replace :: Eq a => [a] -> a -> [a] -> [a] replace sl@(_ : _) r l = case l of [] -> l x : xs -> case stripPrefix sl l of Nothing -> x : replace sl r xs Just rt -> r : replace sl r rt subListSplit :: Eq a => [a] -> [a] -> [[Maybe a]] subListSplit sl@(_ : _) l = splitBy isNothing (replace (map Just sl) Nothing (map Just l)) unintercalate :: Eq a => [a] -> [a] -> [[a]] unintercalate sl@(_ : _) = map (map fromJust) . dropDelims . subListSplit sl From ganesh.sittampalam at credit-suisse.com Fri Feb 6 11:56:27 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Fri Feb 6 11:46:57 2009 Subject: MaybeT to transformers In-Reply-To: <498C6ACB.70605@henning-thielemann.de> References: <16442B752A06A74AB4D9F9A5FF076E4B4DCBAA@ELON17P32001A.csfb.cs-group.com> <498C6ACB.70605@henning-thielemann.de> Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B4DCBB6@ELON17P32001A.csfb.cs-group.com> Henning Thielemann wrote: > Sittampalam, Ganesh wrote: >> Henning Thielemann wrote: >>> What about adding the Maybe monad transformer as provided in MaybeT >>> package to the transformers package? >>> >>> The module name could be >>> Control.Monad.Trans.Maybe >>> which would also avoid conflicts with the MaybeT package. >> >> What MonadPlus instance will it get, if any? > > The one which is currently used by MaybeT: > > instance (Monad m) => MonadPlus (MaybeT m) where > mzero = MaybeT (return Nothing) > mplus x y = MaybeT $ do v <- runMaybeT x > case v of > Nothing -> runMaybeT y > Just _ -> return v > > > It is also the one I needed recently. I tend to agree that this is the right one to use, but I think care should be taken when adding this to what is/is expected to be a widely used package, as it could make it difficult for people who want the alternate behaviour, and it might imply a degree of standardisation/ canonicity that is not really justified. On the other hand, perhaps there are no real uses for the other one. Cheers, Ganesh ============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ============================================================================== From benja.fallenstein at gmail.com Fri Feb 6 12:40:13 2009 From: benja.fallenstein at gmail.com (Benja Fallenstein) Date: Fri Feb 6 12:30:11 2009 Subject: MaybeT to transformers In-Reply-To: <16442B752A06A74AB4D9F9A5FF076E4B4DCBB6@ELON17P32001A.csfb.cs-group.com> References: <16442B752A06A74AB4D9F9A5FF076E4B4DCBAA@ELON17P32001A.csfb.cs-group.com> <498C6ACB.70605@henning-thielemann.de> <16442B752A06A74AB4D9F9A5FF076E4B4DCBB6@ELON17P32001A.csfb.cs-group.com> Message-ID: On Fri, Feb 6, 2009 at 5:56 PM, Sittampalam, Ganesh wrote: > I tend to agree that this is the right one to use, but I think care > should be taken when adding this to what is/is expected to be a widely > used package, as it could make it difficult for people who want the > alternate behaviour, and it might imply a degree of standardisation/ > canonicity that is not really justified. > > On the other hand, perhaps there are no real uses for the other one. ErrorT has had the equivalent MonadPlus instance for a while, hasn't it? Anybody run into problems with that? All the best, - Benja From dave at zednenem.com Fri Feb 6 14:24:09 2009 From: dave at zednenem.com (David Menendez) Date: Fri Feb 6 14:14:06 2009 Subject: MaybeT to transformers In-Reply-To: References: <16442B752A06A74AB4D9F9A5FF076E4B4DCBAA@ELON17P32001A.csfb.cs-group.com> <498C6ACB.70605@henning-thielemann.de> <16442B752A06A74AB4D9F9A5FF076E4B4DCBB6@ELON17P32001A.csfb.cs-group.com> Message-ID: <49a77b7a0902061124p348d7b96lda27778fb98bf4c4@mail.gmail.com> On Fri, Feb 6, 2009 at 12:40 PM, Benja Fallenstein wrote: > On Fri, Feb 6, 2009 at 5:56 PM, Sittampalam, Ganesh > wrote: >> I tend to agree that this is the right one to use, but I think care >> should be taken when adding this to what is/is expected to be a widely >> used package, as it could make it difficult for people who want the >> alternate behaviour, and it might imply a degree of standardisation/ >> canonicity that is not really justified. >> >> On the other hand, perhaps there are no real uses for the other one. > > ErrorT has had the equivalent MonadPlus instance for a while, hasn't > it? Anybody run into problems with that? In my own monad implementations, I prefer to keep exception handling distinct from nondeterminism. From that perspective, Maybe and MaybeT are better suited to MonadError than MonadPlus. There are also times where it would be nice if you could assume that >>= distributed over mplus, as it does in a nondeterminism monad. Let's say I have a state monad implemented using CPS. newtype StateT s m a = C { unC :: forall b. (a -> s -> m b) -> s -> m b } The simple way to promote mplus is to do: mplus a b = C (\k s -> unC a k s `mplus` unC b k s) But that version always distributes, regardless of what the underlying mplus does. In particular, it does not satisfy this equation: lift (mplus a b) = mplus (lift a) (lift b) So intead, you have to do something like mplus a b = C (\k s -> unC a (uncurry return) s `mplus` unC b (uncurry return) s >>= curry k) which is more complicated and involves four operations in the underlying monad, instead of just one. Here's an easy way to test whether a monad's mplus distributes: test = mplus (return 0) (return 1) >>= \x -> guard (odd x) >> return x If the monad distributes over mplus, then test = return 1. Otherwise, test = mzero. -- Dave Menendez From dons at galois.com Sat Feb 7 16:22:47 2009 From: dons at galois.com (Don Stewart) Date: Sat Feb 7 16:13:17 2009 Subject: Updates for Jan 24 - Feb 7 Message-ID: <20090207212247.GP27081@whirlpool.galois.com> http://archhaskell.wordpress.com/2009/02/07/arch-haskell-news-feb-07-2009/ Due to a technical error, I lost the .xml for which packages were updated, however, we can establish that there are now 899 Haskell packages in Arch. Some noteworthy new packages: Full SDL bindings. http://archhaskell.wordpress.com/2009/02/07/sdl-bindings-for-haskell/ Happstack http://archhaskell.wordpress.com/2009/02/07/happstack/ LLVM http://archhaskell.wordpress.com/2009/02/07/llvm-bindings-for-haskell/ I also updated HDBC. -- Don From dominic.steinitz at blueyonder.co.uk Sun Feb 8 02:41:36 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sun Feb 8 02:43:15 2009 Subject: Djinn Doesn't Build Message-ID: <498E8CB0.1080801@blueyonder.co.uk> > dom@linux-6ofq:~/djinn-2009.2.1> runghc Setup.lhs build > Preprocessing executables for djinn-2009.2.1... > Building djinn-2009.2.1... > [1 of 7] Compiling REPL ( src/REPL.hs, dist/build/djinn/djinn-tmp/REPL.o ) > > src/REPL.hs:18:17: > Ambiguous type variable `e' in the constraint: > `Control.Exception.Exception e' > arising from a use of `Control.Exception.catch' > at src/REPL.hs:(18,17)-(33,12) > Probable fix: add a type signature that fixes these type variable(s) > dom@linux-6ofq:~/djinn-2009.2.1> ghc --version > The Glorious Glasgow Haskell Compilation System, version 6.10.1 From duncan.coutts at worc.ox.ac.uk Sun Feb 8 05:34:00 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Sun Feb 8 05:24:30 2009 Subject: Djinn Doesn't Build In-Reply-To: <498E8CB0.1080801@blueyonder.co.uk> References: <498E8CB0.1080801@blueyonder.co.uk> Message-ID: <1234089240.5238.15.camel@localhost> On Sun, 2009-02-08 at 07:41 +0000, Dominic Steinitz wrote: > > dom@linux-6ofq:~/djinn-2009.2.1> runghc Setup.lhs build > > Preprocessing executables for djinn-2009.2.1... > > Building djinn-2009.2.1... > > [1 of 7] Compiling REPL ( src/REPL.hs, dist/build/djinn/djinn-tmp/REPL.o ) > > > > src/REPL.hs:18:17: > > Ambiguous type variable `e' in the constraint: > > `Control.Exception.Exception e' > > arising from a use of `Control.Exception.catch' > > at src/REPL.hs:(18,17)-(33,12) > > Probable fix: add a type signature that fixes these type variable(s) > > > dom@linux-6ofq:~/djinn-2009.2.1> ghc --version > > The Glorious Glasgow Haskell Compilation System, version 6.10.1 Build it against base-3 then it works. This is what cabal-install does. $ cabal install djinn This applies to a load of packages written before base-4 came out (which changed the exceptions api). Many such packages did not list build-depends: base <4 in their .cabal files. There is what's basically a compatibility hack in cabal-install to make this continue to work for old packages. Duncan From dominic.steinitz at blueyonder.co.uk Sun Feb 8 13:08:33 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sun Feb 8 13:10:11 2009 Subject: Djinn Doesn't Build In-Reply-To: <1234089240.5238.15.camel@localhost> References: <498E8CB0.1080801@blueyonder.co.uk> <1234089240.5238.15.camel@localhost> Message-ID: <498F1FA1.3000200@blueyonder.co.uk> Duncan Coutts wrote: > On Sun, 2009-02-08 at 07:41 +0000, Dominic Steinitz wrote: >>> dom@linux-6ofq:~/djinn-2009.2.1> runghc Setup.lhs build >>> Preprocessing executables for djinn-2009.2.1... >>> Building djinn-2009.2.1... >>> [1 of 7] Compiling REPL ( src/REPL.hs, dist/build/djinn/djinn-tmp/REPL.o ) >>> >>> src/REPL.hs:18:17: >>> Ambiguous type variable `e' in the constraint: >>> `Control.Exception.Exception e' >>> arising from a use of `Control.Exception.catch' >>> at src/REPL.hs:(18,17)-(33,12) >>> Probable fix: add a type signature that fixes these type variable(s) >>> dom@linux-6ofq:~/djinn-2009.2.1> ghc --version >>> The Glorious Glasgow Haskell Compilation System, version 6.10.1 > > Build it against base-3 then it works. This is what cabal-install does. > > $ cabal install djinn > > This applies to a load of packages written before base-4 came out (which > changed the exceptions api). Many such packages did not list > build-depends: base <4 in their .cabal files. There is what's basically > a compatibility hack in cabal-install to make this continue to work for > old packages. > > Duncan > > > Duncan, Thanks. I thought it was probably something like that but I thought I had better report it. Dominic. From marcus at gabriel.name Sun Feb 8 16:18:42 2009 From: marcus at gabriel.name (Marcus D. Gabriel) Date: Sun Feb 8 16:08:35 2009 Subject: Proposal for Data.List.splitBy In-Reply-To: <498C6367.9020708@dfki.de> References: <4961265C.2090709@gabriel.name> <1231620312.4794.10.camel@localhost> <496A4531.9050902@gabriel.name> <20090111235612.GA9642@seas.upenn.edu> <49730C53.7080009@gabriel.name> <498C6367.9020708@dfki.de> Message-ID: <498F4C32.3040908@gabriel.name> Christian Maeder wrote: > Marcus D. Gabriel wrote: > >> If I were to write >> >> organizeBy :: ([a] -> Bool) -> [a] -> [([a], [a])] >> > > I quite like your idea, Thanks. > but I think the input predicate "([a] -> Bool)" is too ambitious, although it would nicely unify Brent's > > data Delimiter a where > DelimEltPred :: (a -> Bool) -> Delimiter a > DelimSublist :: Eq a => [a] -> Delimiter a > > With the predicate ([a] -> Bool) you have to check all inits of your > input to detect a delimiter and only if all inits are no delimiter you > know the head element is not part of a delimiter and repeat checking the > inits of the tail. I think this is too inefficient in general. > > The name "organizeBy" was a metaphor. I was trying to distance my language from words such as split, delimiter, and separator to something more general. As for the use of inits, yes, this is a performance hit. If you use something like > splitOnAList :: Eq a => [a] -> [a] -> to split on a list such as "\r\n", then you can use isPrefixOf whereupon the performance is good enough (actually, its not bad at all). () In retrospect, the idea of splitting, delimiters, separators, and fields is what this is all about anyway. > In order to keep things simple I would vote for a split function that > only takes the simple predicate (a -> Bool) and leaves the more > complicated splitting to _additional_ functions > (following http://haskell.org/haskellwiki/Simple_to_complex) > > For instance > > replace :: [a] -> a -> [a] -> [a] > > could replace a sublist (first argument) with a single element (second > argument) > > This would help in simple cases, but leaves it to the user to choose a > suitable delimiter element. > > But a general splitting on sublists could be implemented via > > splitBy isNothing (replace (map Just sl) Nothing (map Just l)) > > (For a fixed sublist sl, "Nothing" is enough to represent the delimiter) > Nice. It took me a moment, but nice. > For a simple predicate "(a -> Bool)" it remains to discuss the output of > splitBy. > > You've proposed "[([a], [a])]", but for the simpler case > "[(a, [a])]" or "[([a], a)]" may do, Actually, although enticing, I do not believe that [(a, [a])] is possible due to the corner cases when there is no beginning non-delimiter or ending delimiter, that is, one needs [(Maybe a, [a])]. (Please check me on this.) > but in order to capture the full > information of lists before and after delimiters, something like > "([a], [(a, [a])])" is needed. > > Since a tuple "(a, [a])" can be viewed as non-empty list of type "[a]", > "([a], [(a, [a])])" collapses to a non-empty list of type "[[a]]" with a > _tail_ of non-empty element lists. I unfortunately do not follow you here, sorry. Be that as it may, I have come to appreciate the output [[a]]. > Therefore I propose the following > splitBy as a "work horse". > > splitBy :: (a -> Bool) -> [a] -> [[a]] > This works for me. > The implementation is simply using "break". The basic property is: > > concat (splitBy p l) == l > I would consider this a requirement, so this works for me. > (This is almost in the spirit of Data.List, since groupBy or words also > produces non-empty element lists.) > > Getting rid of a final delimiter or all of them can be implemented in > O(n) (see below). > > And finally: > > wordsBy p = filter (not . null) . dropDelims . splitBy p > linesBy p = dropDelims . dropFinalDelim . splitBy p > > with: > > words s == wordsBy isSpace s > lines s == linesBy (== '\n') s > > Surely, one might prefer a more type-safe version wrt non-empty lists, > but the main idea is to use function composition rather than composing a > (non-haskell98) splitter input data type as Brent does in his split package. > I am aligned with your point of view above in the sense of additions to Data.List. The more that I think about and play with your definition of the function replace, the more I like it. It captures an idiom that I have used on the UNIX command line when hacking away on text, that is, replace sequences with an unambiguous marker for later use. So, in summary, your idea would be to introduce two functions into Data.List: > splitBy :: (a -> Bool) -> [a] -> [[a]] > replace :: Eq a => [a] -> a -> [a] -> [a] Is this correct? If so, how would you define > splitOnAList :: Eq a => [a] -> [a] -> [[a]] using splitBy and replace. For example, > splitOnAList "\r\n" "abc\r\nxyz\r\n" == ["abc","\r\n","xyz","\r\n"] Given this, maybe we should look at all of the many ways that software makes CSV (TSV) files and see if splitBy and replace can reasonably handle the task. Actually, can even Data.List.Split reasonably handle the task? (I only just recalled this common little problem that is almost trivial but never really so.) Cheers, - Marcus > Cheers Christian > > import Data.Char (isSpace) > > splitBy :: (a -> Bool) -> [a] -> [[a]] > splitBy p l = let (fr, rt) = break p l in case rt of > [] -> [fr] > d : tl -> let hd : tll = splitBy p tl in fr : (d : hd) : tll > > dropFinalDelim :: [[a]] -> [[a]] > dropFinalDelim ll@(l : ls) = > if null ls then > if null l then [] else ll > else > if null (tail (last ls)) then init ll else ll > > dropDelims :: [[a]] -> [[a]] > dropDelims ll = case ll of > [] -> [] > l : ls -> l : map tail ls > > wordsBy :: (a -> Bool) -> [a] -> [[a]] > wordsBy p = filter (not . null) . dropDelims . splitBy p > > linesBy :: (a -> Bool) -> [a] -> [[a]] > linesBy p = dropDelims . dropFinalDelim . splitBy p > > prop_wordsBy :: String -> Bool > prop_wordsBy s = words s == wordsBy isSpace s > > prop_linesBy :: String -> Bool > prop_linesBy s = lines s == linesBy (== '\n') s > > replace :: Eq a => [a] -> a -> [a] -> [a] > replace sl@(_ : _) r l = case l of > [] -> l > x : xs -> case stripPrefix sl l of > Nothing -> x : replace sl r xs > Just rt -> r : replace sl r rt > > subListSplit :: Eq a => [a] -> [a] -> [[Maybe a]] > subListSplit sl@(_ : _) l = > splitBy isNothing (replace (map Just sl) Nothing (map Just l)) > > unintercalate :: Eq a => [a] -> [a] -> [[a]] > unintercalate sl@(_ : _) = > map (map fromJust) . dropDelims . subListSplit sl > -- Marcus D. Gabriel, Ph.D. Saint Louis, FRANCE http://www.marcus.gabriel.name mailto:marcus@gabriel.name From johan.tibell at gmail.com Mon Feb 9 05:28:20 2009 From: johan.tibell at gmail.com (Johan Tibell) Date: Mon Feb 9 05:18:13 2009 Subject: Proposal #2998: Expose the internals of the network library to make it more extensible In-Reply-To: <90889fe70902011519x2b6c209s81b1056a9f9b19da@mail.gmail.com> References: <90889fe70902011519x2b6c209s81b1056a9f9b19da@mail.gmail.com> Message-ID: <90889fe70902090228p2b3d944fp9975fdaa1089b1a4@mail.gmail.com> One week remaining until discussion deadline. Cheers, Johan On Mon, Feb 2, 2009 at 12:19 AM, Johan Tibell wrote: > Hi all, > > In short, I propose that we export some of the internals of the > network library in a new module to make it easier to write portable > libraries on top of the library. I've added the full proposal, copied > from Trac, below. The motivating example is network-bytestring which > needs to marshall different socket related data types to support e.g. > `sendmsg`. > > http://hackage.haskell.org/trac/ghc/ticket/2998 > > Discussion deadline: 2 weeks (non disruptive change) > > --- > > When creating new libraries on top of the network library it would be > convenient if the library exposed some functions that hide platform > variations so that the new libraries can be written in a portable > manner. > > Take for example the currently un-exported pokeSockAddr function which > marshalls a SockAddr into a sockaddr struct. This function would be > useful when e.g. writing a binding for the sendmsg system call. > However, it's not exported so libraries are forced to define their own > version. > > This is unfortunate. I propose we add a new module, > Network.Socket.Internal, to the network library, that exposes these > portable marshalling functions. It might also be a good idea to expose > some internal data representations in the same way bytestring > currently does. People who write code that uses this internal do so > knowing that there is a risk that the interface and data > representations might change. > > I've written a patch that exposes some of the library's internal > functions. There are still some documentation to write and perhaps a > few more functions to expose but I think we should reach consensus on > whether this is a good idea to begin with before continuing. > > Since this change isn't disruptive I propose a two week consideration deadline. > > Cabal package: > > http://johantibell.com/network-2.2.1.tar.gz > > Haddock: > > http://johantibell.com/network/doc/html/network/ From Christian.Maeder at dfki.de Mon Feb 9 11:43:30 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Mon Feb 9 11:33:19 2009 Subject: Proposal for Data.List.splitBy In-Reply-To: <498F4C32.3040908@gabriel.name> References: <4961265C.2090709@gabriel.name> <1231620312.4794.10.camel@localhost> <496A4531.9050902@gabriel.name> <20090111235612.GA9642@seas.upenn.edu> <49730C53.7080009@gabriel.name> <498C6367.9020708@dfki.de> <498F4C32.3040908@gabriel.name> Message-ID: <49905D32.1000905@dfki.de> Marcus D. Gabriel wrote: > Christian Maeder wrote: [...] >> splitOnAList :: Eq a => [a] -> [a] -> > > to split on a list such as "\r\n", then you can use isPrefixOf > whereupon the performance is good enough (actually, its not bad > at all). The special case for "\r\n" is actually trivial, because "\r" can simply be filtered out first. >> But a general splitting on sublists could be implemented via >> >> splitBy isNothing (replace (map Just sl) Nothing (map Just l)) >> >> (For a fixed sublist sl, "Nothing" is enough to represent the delimiter) > > Nice. It took me a moment, but nice. Good, see below my definition of subListSplit and unintercalate. >> For a simple predicate "(a -> Bool)" it remains to discuss the output of >> splitBy. >> >> You've proposed "[([a], [a])]", but for the simpler case >> "[(a, [a])]" or "[([a], a)]" may do, > > Actually, although enticing, I do not believe that [(a, [a])] is > possible due to the corner cases when there is no beginning > non-delimiter or ending delimiter, that is, one needs > [(Maybe a, [a])]. (Please check me on this.) You're right here, there are several ways to accommodate all delimiters and non-delimiters. I've done it as below. >> but in order to capture the full >> information of lists before and after delimiters, something like >> "([a], [(a, [a])])" is needed. >> >> Since a tuple "(a, [a])" can be viewed as non-empty list of type "[a]", >> "([a], [(a, [a])])" collapses to a non-empty list of type "[[a]]" with a >> _tail_ of non-empty element lists. > > I unfortunately do not follow you here, sorry. Be that as it may, > I have come to appreciate the output [[a]]. In "([a], [(a, [a])])" the first component takes the (possibly empty) part before the first delimiter. The second part of type [(a, [a])] are the remaining delimiter with (possibly empty or longer) non-delimiter pairs. Such pairs are then viewed as non-empty lists. After changing the second component to [[a]] the resulting pair ([a], [[a]]) is then also changed to a non-empty list of lists [[a]]. >> wordsBy p = filter (not . null) . dropDelims . splitBy p >> linesBy p = dropDelims . dropFinalDelim . splitBy p I'ld like to improve linesBy as follows: linesBy p = dropFinalNil . dropDelims . splitBy p (dropFinalNil is simpler than dropFinalDelim and dropDelims can assume a non-empty list from splitBy.) > So, in summary, your idea would be to introduce two functions into > Data.List: > >> splitBy :: (a -> Bool) -> [a] -> [[a]] >> replace :: Eq a => [a] -> a -> [a] -> [a] > > Is this correct? Personally, I'ld be content with wordsBy only, but adding linesBy, splitBy and the combination of "dropDelims . splitBy p" (under some suitable names) would make sense for me with or without replace. (In fact, replace should be generalized further.) > If so, how would you define > >> splitOnAList :: Eq a => [a] -> [a] -> [[a]] > > using splitBy and replace. For example, > >> splitOnAList "\r\n" "abc\r\nxyz\r\n" == ["abc","\r\n","xyz","\r\n"] Again, for "\r\n" this is simply: concatMap (: ["\r\n"]) . linesBy (== '\n') . filter (/= '\r') For the general case take my unintercalate below: splitOnAList sl = intercalate [sl] . map (: []) . unintercalate sl This code only puts back identical delimiters that nobody needs because the delimiter is fixed and known via the input argument. For sublist matching, keeping delimiters is unnecessary in general! > reasonably handle the task. Actually, can even Data.List.Split > reasonably handle the task? (I only just recalled this common > little problem that is almost trivial but never really so.) Data.List.Split can handle all these tasks (and more), too, only less elegant (I think). >> import Data.Char (isSpace) >> >> splitBy :: (a -> Bool) -> [a] -> [[a]] >> splitBy p l = let (fr, rt) = break p l in case rt of >> [] -> [fr] >> d : tl -> let hd : tll = splitBy p tl in fr : (d : hd) : tll >> >> dropFinalDelim :: [[a]] -> [[a]] forget dropFinalDelim one and take: dropFinalNil :: [[a]] -> [[a]] dropFinalNil ll@(_ : _) = if null (last ll) then init ll else ll >> dropDelims :: [[a]] -> [[a]] dropDelims does no longer need to work for empty inputs: >> dropDelims ll = case ll of >> [] -> [] >> l : ls -> l : map tail ls dropDelims (l : ls) = l : map tail ls but a total variant makes also sense: dropDelims ll = let (ft, rt) = splitAt 1 ll in ft ++ map (drop 1) rt >> wordsBy :: (a -> Bool) -> [a] -> [[a]] >> wordsBy p = filter (not . null) . dropDelims . splitBy p >> >> linesBy :: (a -> Bool) -> [a] -> [[a]] change linesBy to: linesBy p = dropFinalNil . dropDelims . splitBy p >> replace :: Eq a => [a] -> a -> [a] -> [a] >> replace sl@(_ : _) r l = case l of >> [] -> l >> x : xs -> case stripPrefix sl l of >> Nothing -> x : replace sl r xs >> Just rt -> r : replace sl r rt >> >> subListSplit :: Eq a => [a] -> [a] -> [[Maybe a]] >> subListSplit sl@(_ : _) l = >> splitBy isNothing (replace (map Just sl) Nothing (map Just l)) >> >> unintercalate :: Eq a => [a] -> [a] -> [[a]] >> unintercalate sl@(_ : _) = >> map (map fromJust) . dropDelims . subListSplit sl unintercalate can also be simplified to: unintercalate sl@(_ : _) = map catMaybes . subListSplit sl Cheers Christian From marcus at gabriel.name Mon Feb 9 16:14:16 2009 From: marcus at gabriel.name (Marcus D. Gabriel) Date: Mon Feb 9 16:04:12 2009 Subject: Proposal for Data.List.splitBy In-Reply-To: <49905D32.1000905@dfki.de> References: <4961265C.2090709@gabriel.name> <1231620312.4794.10.camel@localhost> <496A4531.9050902@gabriel.name> <20090111235612.GA9642@seas.upenn.edu> <49730C53.7080009@gabriel.name> <498C6367.9020708@dfki.de> <498F4C32.3040908@gabriel.name> <49905D32.1000905@dfki.de> Message-ID: <49909CA8.5090801@gabriel.name> Thanks for the clarification. I would not have guessed wordsBy and linesBy as your preference over splitBy for additions to Data.List. I do not have an opinion about whether or not functional composition (pipelining) is more elegant than Brent's Splitter data type. I just know that I am very comfortable with constructs such as (f . g . h) or longer. By the way, out or curiosity, how would you generalize your current function replace? Cheers, - Marcus Christian Maeder wrote: > Marcus D. Gabriel wrote: > >> Christian Maeder wrote: >> > [...] > >>> splitOnAList :: Eq a => [a] -> [a] -> >>> >> to split on a list such as "\r\n", then you can use isPrefixOf >> whereupon the performance is good enough (actually, its not bad >> at all). >> > > The special case for "\r\n" is actually trivial, because "\r" can simply > be filtered out first. > > >>> But a general splitting on sublists could be implemented via >>> >>> splitBy isNothing (replace (map Just sl) Nothing (map Just l)) >>> >>> (For a fixed sublist sl, "Nothing" is enough to represent the delimiter) >>> >> Nice. It took me a moment, but nice. >> > > Good, see below my definition of subListSplit and unintercalate. > > >>> For a simple predicate "(a -> Bool)" it remains to discuss the output of >>> splitBy. >>> >>> You've proposed "[([a], [a])]", but for the simpler case >>> "[(a, [a])]" or "[([a], a)]" may do, >>> >> Actually, although enticing, I do not believe that [(a, [a])] is >> possible due to the corner cases when there is no beginning >> non-delimiter or ending delimiter, that is, one needs >> [(Maybe a, [a])]. (Please check me on this.) >> > > You're right here, there are several ways to accommodate all delimiters > and non-delimiters. I've done it as below. > > >>> but in order to capture the full >>> information of lists before and after delimiters, something like >>> "([a], [(a, [a])])" is needed. >>> >>> Since a tuple "(a, [a])" can be viewed as non-empty list of type "[a]", >>> "([a], [(a, [a])])" collapses to a non-empty list of type "[[a]]" with a >>> _tail_ of non-empty element lists. >>> >> I unfortunately do not follow you here, sorry. Be that as it may, >> I have come to appreciate the output [[a]]. >> > > In "([a], [(a, [a])])" the first component takes the (possibly empty) > part before the first delimiter. The second part of type [(a, [a])] are > the remaining delimiter with (possibly empty or longer) non-delimiter > pairs. Such pairs are then viewed as non-empty lists. After changing the > second component to [[a]] the resulting pair ([a], [[a]]) is then also > changed to a non-empty list of lists [[a]]. > > >>> wordsBy p = filter (not . null) . dropDelims . splitBy p >>> linesBy p = dropDelims . dropFinalDelim . splitBy p >>> > > I'ld like to improve linesBy as follows: > > linesBy p = dropFinalNil . dropDelims . splitBy p > > (dropFinalNil is simpler than dropFinalDelim and dropDelims can assume a > non-empty list from splitBy.) > > >> So, in summary, your idea would be to introduce two functions into >> Data.List: >> >> >>> splitBy :: (a -> Bool) -> [a] -> [[a]] >>> replace :: Eq a => [a] -> a -> [a] -> [a] >>> >> Is this correct? >> > > Personally, I'ld be content with wordsBy only, but adding linesBy, > splitBy and the combination of "dropDelims . splitBy p" (under some > suitable names) would make sense for me with or without replace. (In > fact, replace should be generalized further.) > > >> If so, how would you define >> >> >>> splitOnAList :: Eq a => [a] -> [a] -> [[a]] >>> >> using splitBy and replace. For example, >> >> >>> splitOnAList "\r\n" "abc\r\nxyz\r\n" == ["abc","\r\n","xyz","\r\n"] >>> > > Again, for "\r\n" this is simply: > > concatMap (: ["\r\n"]) . linesBy (== '\n') . filter (/= '\r') > > For the general case take my unintercalate below: > > splitOnAList sl = intercalate [sl] . map (: []) . unintercalate sl > > This code only puts back identical delimiters that nobody needs because > the delimiter is fixed and known via the input argument. > > For sublist matching, keeping delimiters is unnecessary in general! > > >> reasonably handle the task. Actually, can even Data.List.Split >> reasonably handle the task? (I only just recalled this common >> little problem that is almost trivial but never really so.) >> > > Data.List.Split can handle all these tasks (and more), too, only less > elegant (I think). > > >>> import Data.Char (isSpace) >>> >>> splitBy :: (a -> Bool) -> [a] -> [[a]] >>> splitBy p l = let (fr, rt) = break p l in case rt of >>> [] -> [fr] >>> d : tl -> let hd : tll = splitBy p tl in fr : (d : hd) : tll >>> >>> dropFinalDelim :: [[a]] -> [[a]] >>> > > forget dropFinalDelim one and take: > > dropFinalNil :: [[a]] -> [[a]] > dropFinalNil ll@(_ : _) = > if null (last ll) then init ll else ll > > >>> dropDelims :: [[a]] -> [[a]] >>> > > dropDelims does no longer need to work for empty inputs: > > >>> dropDelims ll = case ll of >>> [] -> [] >>> l : ls -> l : map tail ls >>> > > dropDelims (l : ls) = l : map tail ls > > but a total variant makes also sense: > > dropDelims ll = let (ft, rt) = splitAt 1 ll in > ft ++ map (drop 1) rt > > >>> wordsBy :: (a -> Bool) -> [a] -> [[a]] >>> wordsBy p = filter (not . null) . dropDelims . splitBy p >>> >>> linesBy :: (a -> Bool) -> [a] -> [[a]] >>> > > change linesBy to: > > linesBy p = dropFinalNil . dropDelims . splitBy p > > >>> replace :: Eq a => [a] -> a -> [a] -> [a] >>> replace sl@(_ : _) r l = case l of >>> [] -> l >>> x : xs -> case stripPrefix sl l of >>> Nothing -> x : replace sl r xs >>> Just rt -> r : replace sl r rt >>> >>> subListSplit :: Eq a => [a] -> [a] -> [[Maybe a]] >>> subListSplit sl@(_ : _) l = >>> splitBy isNothing (replace (map Just sl) Nothing (map Just l)) >>> >>> unintercalate :: Eq a => [a] -> [a] -> [[a]] >>> unintercalate sl@(_ : _) = >>> map (map fromJust) . dropDelims . subListSplit sl >>> > > unintercalate can also be simplified to: > > unintercalate sl@(_ : _) = map catMaybes . subListSplit sl > > Cheers Christian > > > -- Marcus D. Gabriel, Ph.D. Saint Louis, FRANCE http://www.marcus.gabriel.name mailto:marcus@gabriel.name Tel: +33.3.89.69.05.06 Portable: +33.6.34.56.07.75 From Christian.Maeder at dfki.de Tue Feb 10 05:31:31 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Tue Feb 10 05:21:21 2009 Subject: Proposal for Data.List.splitBy In-Reply-To: <49909CA8.5090801@gabriel.name> References: <4961265C.2090709@gabriel.name> <1231620312.4794.10.camel@localhost> <496A4531.9050902@gabriel.name> <20090111235612.GA9642@seas.upenn.edu> <49730C53.7080009@gabriel.name> <498C6367.9020708@dfki.de> <498F4C32.3040908@gabriel.name> <49905D32.1000905@dfki.de> <49909CA8.5090801@gabriel.name> Message-ID: <49915783.2000503@dfki.de> Marcus D. Gabriel wrote: > By the way, out or curiosity, how would you generalize your current > function replace? I would define: replaceBy :: ([a] -> ([b], [a])) -> [a] -> [b] replaceBy splt l = case l of [] -> [] _ -> let (ft, rt) = splt l in ft ++ replaceBy splt rt The first argument is a kind of splitting function that expects a non-empty input list to cut of a first part and change this part to the a replacement list of type [b]. The second part must be the rest of the input to be processed. The original replace function can then be defined via: replace :: Eq a => [a] -> a -> [a] -> [a] replace sl@(_ : _) r = replaceBy $ \ l@(hd : tl) -> case stripPrefix sl l of Nothing -> ([hd], tl) Just rt -> ([r], rt) Other replacements are also possible, i.e. replaceDelim :: ([a] -> ([a], [a])) -> [a] -> [Either [a] a] replaceDelim splt = replaceBy $ \ l@(hd : tl) -> let (ft, rt) = splt l in if null ft then ([Right hd], tl) else ([Left ft], rt) For replaceDelim consider an input function like "span isSpace". A sequence of spaces is grouped to a "Left String", whereas other characters become "Right". The resulting list is suitable for my original splitBy function yielding a list of type "[[Either [a] a]]". This result can be further flattened to [[a]] (by the function flatEitherSplits below). splitByDelim :: ([a] -> ([a], [a])) -> [a] -> [[a]] splitByDelim split = flatEitherSplits . splitBy (either (const True) (const False)) . replaceDelim split As a concrete example consider: splitBySpaces :: String -> [String] splitBySpaces = splitByDelim (span isSpace) splitBySpaces "ab c def" == ["ab"," ","c"," ","def"] For the sake of completeness I add the definition of flatEitherSplits that exploits the result structure of splitBy and uses auxiliary functions: flatRight :: [Either [a] a] -> [a] flatRight = map (either (error "flatRight") id) flatEither :: [Either [a] a] -> [[a]] flatEither (Left d : rt) = d : [flatRight rt] flatEitherSplits :: [[Either [a] a]] -> [[a]] flatEitherSplits (h : tl) = flatRight h : concatMap flatEither tl Cheers Christian P.S. >> For the general case take my unintercalate below: >> >> splitOnAList sl = intercalate [sl] . map (: []) . unintercalate sl By the way, this can be expressed simpler: splitOnAList sl = intersperse sl . unintercalate sl From simonpj at microsoft.com Thu Feb 12 04:22:31 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Thu Feb 12 04:12:17 2009 Subject: Newtype unwrapping in the FFI Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C33399201B40@EA-EXMSG-C334.europe.corp.microsoft.com> [This email concerns an infelicity in the FFI spec. I'm directing it primarily to the Haskell Prime mailing list, but ccing the libraries list so that people there know about the thread. I suggest that replies go to Haskell Prime only.] Consider this program (see http://hackage.haskell.org/trac/ghc/ticket/3008) module NT( N ) where newtype N = MkN Int module FFI where foreign import "f" :: N -> IO () Is module FFI OK? It would definitely be OK if N was defined in the module FFI: the FFI spec says that the compiler must automatically unwrap any newtype arguments. http://www.cse.unsw.edu.au/~chak/haskell/ffi/ffi/ffise3.html#x6-120003.2 But it's less clear when N is defined in another module *and* its representation isn't exported. The author of NT might believe that because N's representation is hidden, she can change it to, say data N = MkN Int without affecting the rest of the world. But she can't. This is a nasty failure of abstraction. It is, I believe the only way in which a client of a NT could be affected by N's representation, even though that representation is allegedly hidden. (ToDo: check on generalised newtype deriving.) This seems Bad to me. Indeed, the cause of the above bug report is that GHC's implementation assumes that the representation is fully hidden if the constructor is not exported, and does not expose the representation of N even to separately-compiled modules (at least when you are not using -O). But the point here is not what GHC stumbles on but what is supposed to happen. Maybe we should fix it. Proposal: * Clarify the spec to say that a newtype can only be automatically unwrapped if the newtype constructor (MkN in this case) is in scope It happens that a large set of types from Foreign.C.Types, such as CInt and friends, are exported without their constructors, so adopting this new rule would require us to change Foreign.C.Types to expose the representation of CInt and friends. (As it happens, nhc requires this already, so there's some #ifdeffery there already.) Simon From lemming at henning-thielemann.de Thu Feb 12 09:17:04 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 12 09:06:47 2009 Subject: MaybeT to transformers In-Reply-To: <16442B752A06A74AB4D9F9A5FF076E4B4DCBB6@ELON17P32001A.csfb.cs-group.com> References: <16442B752A06A74AB4D9F9A5FF076E4B4DCBAA@ELON17P32001A.csfb.cs-group.com> <498C6ACB.70605@henning-thielemann.de> <16442B752A06A74AB4D9F9A5FF076E4B4DCBB6@ELON17P32001A.csfb.cs-group.com> Message-ID: On Fri, 6 Feb 2009, Sittampalam, Ganesh wrote: > I tend to agree that this is the right one to use, but I think care > should be taken when adding this to what is/is expected to be a widely > used package, as it could make it difficult for people who want the > alternate behaviour, and it might imply a degree of standardisation/ > canonicity that is not really justified. > > On the other hand, perhaps there are no real uses for the other one. I guess the other one is simply liftM2 mplus on (m (Maybe a)) and thus there would not be much advantage of using (MaybeT m) instead of (m (Maybe a)). From ross at soi.city.ac.uk Thu Feb 12 09:40:27 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Thu Feb 12 09:30:11 2009 Subject: Binary: Put -> Builder In-Reply-To: References: Message-ID: <20090212144027.GA5680@soi.city.ac.uk> Here's a concrete suggestion: have Data.Binary.Put export the existing function tell :: Builder -> Put and also execPut :: PutM a -> Builder execPut = sndS . unPut From dons at galois.com Thu Feb 12 10:28:07 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 12 10:18:22 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212144027.GA5680@soi.city.ac.uk> References: <20090212144027.GA5680@soi.city.ac.uk> Message-ID: <20090212152807.GB9850@whirlpool.galois.com> ross: > Here's a concrete suggestion: have Data.Binary.Put export the existing > function > > tell :: Builder -> Put > > and also > > execPut :: PutM a -> Builder > execPut = sndS . unPut What's the use case? From ross at soi.city.ac.uk Thu Feb 12 10:38:32 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Thu Feb 12 10:28:24 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212152807.GB9850@whirlpool.galois.com> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> Message-ID: <20090212153832.GB5680@soi.city.ac.uk> On Thu, Feb 12, 2009 at 07:28:07AM -0800, Don Stewart wrote: > ross: > > Here's a concrete suggestion: have Data.Binary.Put export the existing > > function > > > > tell :: Builder -> Put > > > > and also > > > > execPut :: PutM a -> Builder > > execPut = sndS . unPut > > What's the use case? Quoting a previous message: > I imagine one might want to make a Builder for a composite object, > using the Binary instances of some types. And one might want to go > the other way too, defining a Binary instance using an existing Builder. From aslatter at gmail.com Thu Feb 12 10:40:19 2009 From: aslatter at gmail.com (Antoine Latter) Date: Thu Feb 12 10:29:57 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212152807.GB9850@whirlpool.galois.com> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> Message-ID: <694519c50902120740s66d536f3ge356ff635cd15bd2@mail.gmail.com> On Thu, Feb 12, 2009 at 9:28 AM, Don Stewart wrote: > ross: >> Here's a concrete suggestion: have Data.Binary.Put export the existing >> function >> >> tell :: Builder -> Put >> >> and also >> >> execPut :: PutM a -> Builder >> execPut = sndS . unPut > > What's the use case? Maybe 'tell' could be renamed to 'putBuilder', thereby not exposing the fact that the function is reaching into the guts of the abstraction. -Antoine From dons at galois.com Thu Feb 12 10:40:21 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 12 10:30:36 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212153832.GB5680@soi.city.ac.uk> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> <20090212153832.GB5680@soi.city.ac.uk> Message-ID: <20090212154021.GD9850@whirlpool.galois.com> ross: > On Thu, Feb 12, 2009 at 07:28:07AM -0800, Don Stewart wrote: > > ross: > > > Here's a concrete suggestion: have Data.Binary.Put export the existing > > > function > > > > > > tell :: Builder -> Put > > > > > > and also > > > > > > execPut :: PutM a -> Builder > > > execPut = sndS . unPut > > > > What's the use case? > > Quoting a previous message: > > I imagine one might want to make a Builder for a composite object, > > using the Binary instances of some types. And one might want to go > > the other way too, defining a Binary instance using an existing Builder. Sounds good. Do you just want these exposed as is? -- Don From lemming at henning-thielemann.de Thu Feb 12 10:44:20 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 12 10:34:02 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212144027.GA5680@soi.city.ac.uk> References: <20090212144027.GA5680@soi.city.ac.uk> Message-ID: On Thu, 12 Feb 2009, Ross Paterson wrote: > Here's a concrete suggestion: have Data.Binary.Put export the existing > function > > tell :: Builder -> Put > > and also > > execPut :: PutM a -> Builder > execPut = sndS . unPut This would be nice. From ross at soi.city.ac.uk Thu Feb 12 10:48:44 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Thu Feb 12 10:38:26 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212154021.GD9850@whirlpool.galois.com> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> <20090212153832.GB5680@soi.city.ac.uk> <20090212154021.GD9850@whirlpool.galois.com> Message-ID: <20090212154844.GC5680@soi.city.ac.uk> On Thu, Feb 12, 2009 at 07:40:21AM -0800, Don Stewart wrote: > ross: > > On Thu, Feb 12, 2009 at 07:28:07AM -0800, Don Stewart wrote: > > > ross: > > > > Here's a concrete suggestion: have Data.Binary.Put export the existing > > > > function > > > > > > > > tell :: Builder -> Put > > > > > > > > and also > > > > > > > > execPut :: PutM a -> Builder > > > > execPut = sndS . unPut > > > > > > What's the use case? > > > > Quoting a previous message: > > > I imagine one might want to make a Builder for a composite object, > > > using the Binary instances of some types. And one might want to go > > > the other way too, defining a Binary instance using an existing Builder. > > Sounds good. Do you just want these exposed as is? Yes, though Antoine's suggestion of exporting tell as putBuilder may be better. From dons at galois.com Thu Feb 12 11:20:20 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 12 11:10:41 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212154844.GC5680@soi.city.ac.uk> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> <20090212153832.GB5680@soi.city.ac.uk> <20090212154021.GD9850@whirlpool.galois.com> <20090212154844.GC5680@soi.city.ac.uk> Message-ID: <20090212162020.GF9850@whirlpool.galois.com> ross: > On Thu, Feb 12, 2009 at 07:40:21AM -0800, Don Stewart wrote: > > ross: > > > On Thu, Feb 12, 2009 at 07:28:07AM -0800, Don Stewart wrote: > > > > ross: > > > > > Here's a concrete suggestion: have Data.Binary.Put export the existing > > > > > function > > > > > > > > > > tell :: Builder -> Put > > > > > > > > > > and also > > > > > > > > > > execPut :: PutM a -> Builder > > > > > execPut = sndS . unPut > > > > > > > > What's the use case? > > > > > > Quoting a previous message: > > > > I imagine one might want to make a Builder for a composite object, > > > > using the Binary instances of some types. And one might want to go > > > > the other way too, defining a Binary instance using an existing Builder. > > > > Sounds good. Do you just want these exposed as is? > > Yes, though Antoine's suggestion of exporting tell as putBuilder may > be better. OK. I'm preparing the 0.5 release of binary atm, so will expose these. -- Don From dons at galois.com Thu Feb 12 12:51:27 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 12 12:41:44 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212154844.GC5680@soi.city.ac.uk> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> <20090212153832.GB5680@soi.city.ac.uk> <20090212154021.GD9850@whirlpool.galois.com> <20090212154844.GC5680@soi.city.ac.uk> Message-ID: <20090212175127.GC10195@whirlpool.galois.com> ross: > On Thu, Feb 12, 2009 at 07:40:21AM -0800, Don Stewart wrote: > > ross: > > > On Thu, Feb 12, 2009 at 07:28:07AM -0800, Don Stewart wrote: > > > > ross: > > > > > Here's a concrete suggestion: have Data.Binary.Put export the existing > > > > > function > > > > > > > > > > tell :: Builder -> Put > > > > > > > > > > and also > > > > > > > > > > execPut :: PutM a -> Builder > > > > > execPut = sndS . unPut > > > > > > > > What's the use case? > > > > > > Quoting a previous message: > > > > I imagine one might want to make a Builder for a composite object, > > > > using the Binary instances of some types. And one might want to go > > > > the other way too, defining a Binary instance using an existing Builder. > > > > Sounds good. Do you just want these exposed as is? > > Yes, though Antoine's suggestion of exporting tell as putBuilder may > be better. Is this what you're looking for? Thu Feb 12 09:47:34 PST 2009 Don Stewart * Expose Builder/Put relationship, courtesy Ross New patches: [Expose Builder/Put relationship, courtesy Ross Don Stewart **20090212174734] { hunk ./src/Data/Binary/Put.hs 22 , PutM(..) , runPut , runPutM + , putBuilder + , execPut -- * Flushing the implicit parse state , flush hunk ./src/Data/Binary/Put.hs 112 tell b = Put $ PairS () b {-# INLINE tell #-} +putBuilder :: Builder -> Put +putBuilder = tell +{-# INLINE putBuilder #-} + +-- | Run the 'Put' monad +execPut :: PutM a -> Builder +execPut = sndS . unPut +{-# INLINE execPut #-} + -- | Run the 'Put' monad with a serialiser runPut :: Put -> L.ByteString runPut = toLazyByteString . sndS . unPut } From lemming at henning-thielemann.de Thu Feb 12 12:54:13 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 12 12:43:53 2009 Subject: IdentityT to transformers Message-ID: Ross, thanks a lot for adding MaybeT! Unfortunately I have just another wish. In a parser I like to show by types whether the parser can fail or not. To this end I have a type parameter for a monad transformer that can be instantiated either by MaybeT or by IdentityT. The only problem is, that there is no IdentityT in transformers package, yet. From lemming at henning-thielemann.de Thu Feb 12 14:06:30 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 12 13:56:10 2009 Subject: IdentityT to transformers In-Reply-To: References: Message-ID: On Thu, 12 Feb 2009, Henning Thielemann wrote: > Unfortunately I have just another wish. In a parser I like to show by types > whether the parser can fail or not. To this end I have a type parameter for a > monad transformer that can be instantiated either by MaybeT or by IdentityT. > The only problem is, that there is no IdentityT in transformers package, yet. Does the attached module makes sense? -------------- next part -------------- A non-text attachment was scrubbed... Name: Identity.hs Type: text/x-haskell Size: 2481 bytes Desc: Url : http://www.haskell.org/pipermail/libraries/attachments/20090212/01b9f275/Identity.bin From ross at soi.city.ac.uk Thu Feb 12 14:59:02 2009 From: ross at soi.city.ac.uk (Ross Paterson) Date: Thu Feb 12 14:48:42 2009 Subject: IdentityT to transformers In-Reply-To: References: Message-ID: <20090212195902.GA10617@soi.city.ac.uk> On Thu, Feb 12, 2009 at 08:06:30PM +0100, Henning Thielemann wrote: > > On Thu, 12 Feb 2009, Henning Thielemann wrote: > >> Unfortunately I have just another wish. In a parser I like to show by >> types whether the parser can fail or not. To this end I have a type >> parameter for a monad transformer that can be instantiated either by >> MaybeT or by IdentityT. The only problem is, that there is no IdentityT >> in transformers package, yet. > > Does the attached module makes sense? I've already got Magnus Therning's from his blog. From haskell at list.mightyreason.com Thu Feb 12 18:45:03 2009 From: haskell at list.mightyreason.com (Chris Kuklewicz) Date: Thu Feb 12 18:34:44 2009 Subject: Binary: Put -> Builder In-Reply-To: <20090212162020.GF9850@whirlpool.galois.com> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> <20090212153832.GB5680@soi.city.ac.uk> <20090212154021.GD9850@whirlpool.galois.com> <20090212154844.GC5680@soi.city.ac.uk> <20090212162020.GF9850@whirlpool.galois.com> Message-ID: <4994B47F.8050504@list.mightyreason.com> Don Stewart wrote: > OK. I'm preparing the 0.5 release of binary atm, so will expose these. > > -- Don Don: I had proposed (with a patch) to let the caller of runPut give a size hint so the first chunk allocated for the Lazy ByteString would not have to always be the default value (which I remember being about 32 kilo bytes). If the first allocation filled up the subsequent chunks are still the default size. My use case was for the protocol-buffers package where the serialized message sizes are both known in advance and often much much smaller than this. I have no benchmarks, but it feels wrong to have my library constantly over-allocating. Is such a feature going to be accepted into the binary package? If not then I may just copy over and patch the Put monad. (I do not use an external Get monad, the protocol-buffers package includes its own Get monad). Cheers, Chris From dons at galois.com Thu Feb 12 19:14:41 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 12 19:04:57 2009 Subject: Binary: Put -> Builder In-Reply-To: <4994B47F.8050504@list.mightyreason.com> References: <20090212144027.GA5680@soi.city.ac.uk> <20090212152807.GB9850@whirlpool.galois.com> <20090212153832.GB5680@soi.city.ac.uk> <20090212154021.GD9850@whirlpool.galois.com> <20090212154844.GC5680@soi.city.ac.uk> <20090212162020.GF9850@whirlpool.galois.com> <4994B47F.8050504@list.mightyreason.com> Message-ID: <20090213001441.GN10706@whirlpool.galois.com> haskell: > Don Stewart wrote: >> OK. I'm preparing the 0.5 release of binary atm, so will expose these. >> >> -- Don > > Don: I had proposed (with a patch) to let the caller of runPut give a > size hint so the first chunk allocated for the Lazy ByteString would not > have to always be the default value (which I remember being about 32 kilo > bytes). If the first allocation filled up the subsequent chunks are > still the default size. > > My use case was for the protocol-buffers package where the serialized > message sizes are both known in advance and often much much smaller than > this. I have no benchmarks, but it feels wrong to have my library > constantly over-allocating. > > Is such a feature going to be accepted into the binary package? If not > then I may just copy over and patch the Put monad. (I do not use an > external Get monad, the protocol-buffers package includes its own Get > monad). Could you dig up the patch again? From dave at zednenem.com Thu Feb 12 23:12:54 2009 From: dave at zednenem.com (David Menendez) Date: Thu Feb 12 23:03:01 2009 Subject: Control.Concurrent.MVar documentation broken Message-ID: <49a77b7a0902122012r337d7683n5e44719a5e8a5479@mail.gmail.com> I was just looking at and I noticed that the documentation for withMVar, modifyMVar, and modifyMVar_ is missing. This appears to be caused by the INLINE pragmas. I've done some experiments, and I've found that Haddock 2.4.1 will not display the documentation if an INLINE, NOINLINE, or SPECIALIZE pragma appears between the documentation and the signature. I don't know whether this is a bug in Haddock or Control.Concurrent.MVar. -- Dave Menendez From dave at zednenem.com Thu Feb 12 23:12:54 2009 From: dave at zednenem.com (David Menendez) Date: Thu Feb 12 23:07:18 2009 Subject: Control.Concurrent.MVar documentation broken Message-ID: <49a77b7a0902122012r337d7683n5e44719a5e8a5479@mail.gmail.com> I was just looking at and I noticed that the documentation for withMVar, modifyMVar, and modifyMVar_ is missing. This appears to be caused by the INLINE pragmas. I've done some experiments, and I've found that Haddock 2.4.1 will not display the documentation if an INLINE, NOINLINE, or SPECIALIZE pragma appears between the documentation and the signature. I don't know whether this is a bug in Haddock or Control.Concurrent.MVar. -- Dave Menendez From david.waern at gmail.com Fri Feb 13 04:00:04 2009 From: david.waern at gmail.com (David Waern) Date: Fri Feb 13 03:49:42 2009 Subject: Control.Concurrent.MVar documentation broken In-Reply-To: <49a77b7a0902122012r337d7683n5e44719a5e8a5479@mail.gmail.com> References: <49a77b7a0902122012r337d7683n5e44719a5e8a5479@mail.gmail.com> Message-ID: 2009/2/13 David Menendez : > I was just looking at > > and I noticed that the documentation for withMVar, modifyMVar, and > modifyMVar_ is missing. > > This appears to be caused by the INLINE pragmas. I've done some > experiments, and I've found that Haddock 2.4.1 will not display the > documentation if an INLINE, NOINLINE, or SPECIALIZE pragma appears > between the documentation and the signature. > > I don't know whether this is a bug in Haddock or Control.Concurrent.MVar. It's a bug in Haddock which has been fixed. The fix will be in 2.4.2. David From dominic.steinitz at blueyonder.co.uk Sun Feb 15 06:41:01 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sun Feb 15 06:42:33 2009 Subject: Crypto Package Ownership Message-ID: <4997FF4D.1040507@blueyonder.co.uk> Creighton Hogg has very kindly volunteered to take over its maintenance. The code base is kept on code.haskell.org: > dom@nun:/srv/code/crypto$ ls -ltr > total 92 > -rw-rw-r-- 1 dom crypto-p 6711 Dec 6 09:29 SymmetricTest.hs > -rw-rw-r-- 1 dom crypto-p 79 Dec 6 09:29 Setup.hs > -rw-rw-r-- 1 dom crypto-p 8779 Dec 6 09:29 SHA1Test.hs > -rw-rw-r-- 1 dom crypto-p 4910 Dec 6 09:29 RSATest.hs > -rw-rw-r-- 1 dom crypto-p 2245 Dec 6 09:29 QuickTest.hs > -rw-rw-r-- 1 dom crypto-p 4918 Dec 6 09:29 HMACTest.hs > drwxrwsr-x 3 dom crypto-p 4096 Dec 6 09:29 Data > -rw-rw-r-- 1 dom crypto-p 1975 Dec 6 09:29 CryptoHomePage.html~ > drwxrwsr-x 5 dom crypto-p 4096 Dec 6 09:35 oldrepo > drwxrwsr-x 2 dom crypto-p 4096 Jan 4 03:56 doc > -rw-rw-r-- 1 dom crypto-p 2534 Jan 4 03:56 WordListTest.hs > drwxrwsr-x 5 dom crypto-p 4096 Jan 4 03:56 Codec > -rw-rw-r-- 1 dom crypto-p 3644 Jan 4 03:58 crypto.cabal > -rw-rw-r-- 1 dom crypto-p 10839 Jan 4 03:58 ReadMe.tex > drwxrwsr-x 6 dom crypto-p 4096 Jan 4 03:58 _darcs > -rw-rw-r-- 1 dom crypto-p 1973 Jan 4 04:01 CryptoHomePage.html Is there a way of changing the owner? Creighton's account is wchogg. Many thanks, Dominic. From duncan.coutts at worc.ox.ac.uk Sun Feb 15 08:46:29 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Sun Feb 15 08:36:14 2009 Subject: Crypto Package Ownership In-Reply-To: <4997FF4D.1040507@blueyonder.co.uk> References: <4997FF4D.1040507@blueyonder.co.uk> Message-ID: <1234705589.10457.26.camel@localhost> On Sun, 2009-02-15 at 11:41 +0000, Dominic Steinitz wrote: > Creighton Hogg has very kindly volunteered to take over its maintenance. > > The code base is kept on code.haskell.org: > > > dom@nun:/srv/code/crypto$ ls -ltr > > total 92 > > -rw-rw-r-- 1 dom crypto-p 6711 Dec 6 09:29 SymmetricTest.hs [...] > Is there a way of changing the owner? Creighton's account is wchogg. The standard procedure would be to email the support address and ask. (as documented on http://community.haskell.org/admin/ ) However I don't think you actually need to do anything. Assuming you've added wchogg to the project group then that's it, both of you can push patches and who ends up owning each individual file does not really matter since they are all group writable. If it's just adding Creighton to the project group that you need to do then see the instructions here: http://community.haskell.org/admin/project_admin.html Duncan From dominic.steinitz at blueyonder.co.uk Sun Feb 15 11:35:41 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sun Feb 15 11:37:09 2009 Subject: Crypto Package Ownership In-Reply-To: <1234705589.10457.26.camel@localhost> References: <4997FF4D.1040507@blueyonder.co.uk> <1234705589.10457.26.camel@localhost> Message-ID: <4998445D.3020806@blueyonder.co.uk> Duncan Coutts wrote: > On Sun, 2009-02-15 at 11:41 +0000, Dominic Steinitz wrote: >> Creighton Hogg has very kindly volunteered to take over its maintenance. >> >> The code base is kept on code.haskell.org: >> >>> dom@nun:/srv/code/crypto$ ls -ltr >>> total 92 >>> -rw-rw-r-- 1 dom crypto-p 6711 Dec 6 09:29 SymmetricTest.hs > [...] > >> Is there a way of changing the owner? Creighton's account is wchogg. > > The standard procedure would be to email the support address and ask. > (as documented on http://community.haskell.org/admin/ ) > > However I don't think you actually need to do anything. Assuming you've > added wchogg to the project group then that's it, both of you can push > patches and who ends up owning each individual file does not really > matter since they are all group writable. > > If it's just adding Creighton to the project group that you need to do > then see the instructions here: > http://community.haskell.org/admin/project_admin.html > > Duncan > > > Duncan, My apologies for not having read the docs. It is indeed simple and I have just added Creighton to the group. Many thanks, Dominic. From duncan.coutts at worc.ox.ac.uk Mon Feb 16 19:24:55 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Feb 16 19:14:35 2009 Subject: advance warning of Cabal-1.6.0.2 and package breakages Message-ID: <1234830295.10457.72.camel@localhost> All, I'll be releasing Cabal-1.6.0.2 soon. I've been testing with packages on hackage for regressions. However even with the regressions fixed there will be a small handful of packages that will break with the new release. This is because they are already incorrect it's just that this was previously hidden. All these errors get picked up now because the new Cabal version checks that header files and C libs can be found at configure time. This check is generally a great help to users. Unfortunately in the case of these packages it picks up issues that did not always previously cause build failure (though could cause failure in some configurations). So I'm sorry that these 6 package now break. They are all genuine pre-existing errors though and fixing them seems a small price to pay for a better user experience for the other 1000 packages on hackage. I'm cc'ing the maintainers of the 6 packages. There is no need to release immediately, in fact you may like to wait for the release and test against it yourself to confirm whatever fixes you make. These are the packages and their errors: digest-0.0.0.2: This lists zutil.u as an include file. This is just a spelling error. It should be zutil.h of course. This does break with ghc-6.8 using -fvia-C. The behaviour of ghc-6.10 masks this error. hopenssl-1.0: This lists as an include file. There is of course no such file. It should be openssl/evp.h instead. This does break with ghc-6.8 using -fvia-C. The behaviour of ghc-6.10 masks this error. libxml-0.1.1: This lists libxml/xmlIO.h as an include file. On my system at least this is found under /usr/include/libxml2 not /usr/include. The best solution here is probably to use pkg-config to find the right include dir. Cabal supports this directly via: pkgconfig-depends: libxml-2.0 Again this happens to work with ghc-6.10 because it does not really use headers when compiling, but it breaks with older ghc or non-ghc. plugins-1.4.0: This specifies Linker.h as an include file. This is from the ghc rts include files. However this file cannot be included on its own. It does not compile. It needs some other rts headers to be included first. This would have failed with ghc-6.8, only the package does not work with 6.8. It does not fail wit 6.10 because 6.10 doesn't really #include headers at all. riot-1.20080618: This specifies an include file that does not exist at configure time. It specifies a file under dist/ that would be generated by running hsc2hs. This is already incorrect, but happens to work with the default dist location and current layout. If this package were uploaded today it would be rejected. sqlite-0.4.2: This specifies a include file that is not a file, but is actually a directory. It looks like confusion. Thanks Duncan From dave.a.tapley at gmail.com Tue Feb 17 11:03:17 2009 From: dave.a.tapley at gmail.com (Dave Tapley) Date: Tue Feb 17 10:52:42 2009 Subject: Broken dependencies on cabal install haddock In-Reply-To: <1233791695.26754.648.camel@localhost> References: <20090204225931.GA2813@soi.city.ac.uk> <1233791695.26754.648.camel@localhost> Message-ID: Hey Duncan, Thanks for the fast reply / sorry about my slow one! The next instalment: dave@eliza:~$ ghc-pkg check package ghc-6.8.2 has missing dependencies: Cabal-1.2.3.0 Which is interesting, no? Dave, 2009/2/4 Duncan Coutts : > On Thu, 2009-02-05 at 00:03 +0100, David Waern wrote: >> 2009/2/4 Ross Paterson : >> > On Wed, Feb 04, 2009 at 10:52:32PM +0000, Dave Tapley wrote: >> >> dave@eliza:~$ cabal install haddock >> >> Resolving dependencies... >> >> cabal: cannot configure haddock-2.4.1. It requires ghc >=6.8.2 && <6.13 >> >> There is no available version of ghc that satisfies >=6.8.2 && <6.13 >> >> dave@eliza:~$ ghc --version >> >> The Glorious Glasgow Haskell Compilation System, version 6.8.2 >> > >> > There is also a library package called ghc. >> >> Which should be installed if you have the compiler. Hmm. > > If the package is broken due to missing dependencies then cabal-install > ignores it. It's on my TODO list to make it report more clearly the fact > that it needed a package that is broken (ie instead of removing the > broken packages from the set of installed packages, add a constraint not > to use that version and when we get a constraint conflict with we can > report a more helpful message). > > Dave, you can confirm if the ghc package is broken by running ghc-pkg > check. > > Duncan > > From duncan.coutts at worc.ox.ac.uk Tue Feb 17 17:26:42 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 17 17:16:19 2009 Subject: Broken dependencies on cabal install haddock In-Reply-To: References: <20090204225931.GA2813@soi.city.ac.uk> <1233791695.26754.648.camel@localhost> Message-ID: <1234909602.10457.105.camel@localhost> On Tue, 2009-02-17 at 16:03 +0000, Dave Tapley wrote: > Hey Duncan, > > Thanks for the fast reply / sorry about my slow one! > The next instalment: > > dave@eliza:~$ ghc-pkg check > package ghc-6.8.2 has missing dependencies: > Cabal-1.2.3.0 > > Which is interesting, no? Indeed. It looks like you accidentally unregistered Cabal-1.2.3.0, which breaks dependent packages (ghc-6.8.2). Unfortunately you cannot fix this by re-installing Cabal-1.2.3.0. You cannot guarantee that you'll get the exact same ABI. You either need to re-register the original version (the actual files are almost certainly still installed) or re-install ghc. If you're on debian or gentoo then you're in luck as they both provide a backup of the original package.conf as package.conf.shipped. This will let you re-register the original by: $ ghc-pkg describe Cabal-1.2.3.0 -f package.conf.shipped > pkg $ ghc-pkg register pkg In ghc-6.10 it's be a bit harder to break packages by unregistering their dependencies. ghc-pkg unregister now complains and makes you use --force if you really do want to break dependencies. Duncan > Dave, > > > 2009/2/4 Duncan Coutts : > > On Thu, 2009-02-05 at 00:03 +0100, David Waern wrote: > >> 2009/2/4 Ross Paterson : > >> > On Wed, Feb 04, 2009 at 10:52:32PM +0000, Dave Tapley wrote: > >> >> dave@eliza:~$ cabal install haddock > >> >> Resolving dependencies... > >> >> cabal: cannot configure haddock-2.4.1. It requires ghc >=6.8.2 && <6.13 > >> >> There is no available version of ghc that satisfies >=6.8.2 && <6.13 > >> >> dave@eliza:~$ ghc --version > >> >> The Glorious Glasgow Haskell Compilation System, version 6.8.2 > >> > > >> > There is also a library package called ghc. > >> > >> Which should be installed if you have the compiler. Hmm. > > > > If the package is broken due to missing dependencies then cabal-install > > ignores it. It's on my TODO list to make it report more clearly the fact > > that it needed a package that is broken (ie instead of removing the > > broken packages from the set of installed packages, add a constraint not > > to use that version and when we get a constraint conflict with we can > > report a more helpful message). > > > > Dave, you can confirm if the ghc package is broken by running ghc-pkg > > check. > > > > Duncan > > > > > _______________________________________________ > Libraries mailing list > Libraries@haskell.org > http://www.haskell.org/mailman/listinfo/libraries From duncan.coutts at worc.ox.ac.uk Wed Feb 18 06:18:52 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Wed Feb 18 06:08:26 2009 Subject: advance warning of Cabal-1.6.0.2 and package breakages In-Reply-To: <1234830295.10457.72.camel@localhost> References: <1234830295.10457.72.camel@localhost> Message-ID: <1234955932.10457.120.camel@localhost> On Tue, 2009-02-17 at 00:24 +0000, Duncan Coutts wrote: > All, > > I'll be releasing Cabal-1.6.0.2 soon. I've been testing with packages on > hackage for regressions. However even with the regressions fixed there > will be a small handful of packages that will break with the new > release. This is because they are already incorrect it's just that this > was previously hidden. > > All these errors get picked up now because the new Cabal version checks > that header files and C libs can be found at configure time. This check > is generally a great help to users. Unfortunately in the case of these > packages it picks up issues that did not always previously cause build > failure (though could cause failure in some configurations). > > So I'm sorry that these 6 package now break. They are all genuine > pre-existing errors though and fixing them seems a small price to pay > for a better user experience for the other 1000 packages on hackage. > > I'm cc'ing the maintainers of the 6 packages. There is no need to > release immediately, in fact you may like to wait for the release and > test against it yourself to confirm whatever fixes you make. > > These are the packages and their errors: Oh, one more: cedict-0.2.5 The tarball is missing c/data.h. It happens not to fail with ghc-6.10 but almost certainly will fail with 6.8. sdist also fails. Duncan From Christian.Maeder at dfki.de Wed Feb 18 06:20:10 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Wed Feb 18 06:09:30 2009 Subject: modules of cabal-install Message-ID: <499BEEEA.7080000@dfki.de> Hi, I would like to reuse the module Distribution.Client.Tar from the cabal-install package. Are you willing to split up your package? Is it worth to have single module packages? Currently, Distribution.Client.Tar depends on Distribution.Client.Utils.writeFileAtomic, but this final call could be left to the user. Maybe writeFileAtomic should be in another package, too? Trying to avoid copying source code, Christian From duncan.coutts at worc.ox.ac.uk Wed Feb 18 08:23:45 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Wed Feb 18 08:13:18 2009 Subject: advance warning of Cabal-1.6.0.2 and package breakages In-Reply-To: <1234830295.10457.72.camel@localhost> References: <1234830295.10457.72.camel@localhost> Message-ID: <1234963425.10457.162.camel@localhost> On Tue, 2009-02-17 at 00:24 +0000, Duncan Coutts wrote: > All, > > I'll be releasing Cabal-1.6.0.2 soon. I've been testing with packages on > hackage for regressions. However even with the regressions fixed there > will be a small handful of packages that will break with the new > release. This is because they are already incorrect it's just that this > was previously hidden. I should like to point out that the fact that Cabal now does the checks for C header files and libs means we can generally do a better job now if the packages themselves do not try to do anything clever. We get more consistent error messages but we also take into account extra flags passed in by the user which a custom check may omit (which means the user may not be able to install at all if they have the C libs in a non-standard location). For example: jack-0.5: Searching for jack/jack.h...setup: user error (ERROR: jack/jack.h not found) And it ignores the --extra-include-dirs flag. If you maintain a FFI binding package, take a look to see if it has redundant checks in the ./configure or Setup.hs for C headers and libs. While we're on the topic of good practise in configure scripts. Try to avoid if possible grabbing random environment variables and putting them into a .buildinfo file. For example $CPPFLAGS or $LDFLAGS. The current way for users to specify non-standard install locations for C libs is via --extra-include-dirs and --extra-lib-dirs. If you want to argue that env vars are the better user interface then make the case and we people agree then we can do it in Cabal itself. Having it different in each package just confuses users. The other problem with putting the env vars into .buildinfo files is that the cc-options, ld-options don't just get used to build your package but they get put into the package registration info and used when building every dependent package. More advice: * Don't make configure set buildable: False. Just fail. * Do declare the key header files your package needs in the .cabal file rather than just in .hs or .hsc files. That way Cabal can check for them and tools that convert to native distro packages will notice the dependencies on foreign packages. Looking at the failing packages on hackage at the moment we've got about 100 failing at build time and about 50 failing at configure time. A bunch more fail because they depend on other things that failed. The biggest cause for build failure seems to be incorrectly specified dependencies. That is they probably worked against one version and now fail to compile against some different version. Quite often that is the base package, but it's not the majority. Duncan From aslatter at gmail.com Wed Feb 18 08:47:04 2009 From: aslatter at gmail.com (Antoine Latter) Date: Wed Feb 18 08:36:25 2009 Subject: modules of cabal-install In-Reply-To: <499BEEEA.7080000@dfki.de> References: <499BEEEA.7080000@dfki.de> Message-ID: <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> On Wed, Feb 18, 2009 at 5:20 AM, Christian Maeder wrote: > Hi, > > I would like to reuse the module Distribution.Client.Tar > from the cabal-install package. > > Are you willing to split up your package? Is it worth to have single > module packages? > > Currently, Distribution.Client.Tar depends on > Distribution.Client.Utils.writeFileAtomic, but this final call could be > left to the user. Maybe writeFileAtomic should be in another package, too? > There's a 'tar' package on Hackage: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar I've had good lusck using it before, but I'm not sure how it's different from what cabal-install uses. Antoine From patperry at stanford.edu Wed Feb 18 11:54:13 2009 From: patperry at stanford.edu (Patrick Perry) Date: Wed Feb 18 11:43:36 2009 Subject: advance warning of Cabal-1.6.0.2 and package breakages In-Reply-To: <1234963425.10457.162.camel@localhost> References: <1234830295.10457.72.camel@localhost> <1234963425.10457.162.camel@localhost> Message-ID: Thanks for the heads-up, Duncan. For the BLAS package, the situation is slightly complicated, and I will still be sticking with autoconf for now. The reasons are twofold: 1. The functions I am binding to are written in Fortran, and have different calling conventions on different architectures. On some architectures, the equivalent C function names have a trailing underscore (e.g. "dgemm_"), and on some architectures they don't (e.g. "dgemm"). 2. There is no standard name for a BLAS shared library. The autoconf macro I use checks in libraries named atlas, blas, mkl, cxml, dxml, sunperf, essl, and a few others. For me to switch to git rid of autoconf, I would need Cabal to provide the following: 1. A way to find out the C calling convention for Fortran functions. 2. A way to check if a specific function exists in a library. 3. An option to try a different library name if a check for a library fails. For reference, here is the autoconf macro I am using: http://github.com/patperry/blas/blob/master/m4/ax_blas.m4 Patrick On Feb 18, 2009, at 5:23 AM, Duncan Coutts wrote: > On Tue, 2009-02-17 at 00:24 +0000, Duncan Coutts wrote: >> All, >> >> I'll be releasing Cabal-1.6.0.2 soon. I've been testing with >> packages on >> hackage for regressions. However even with the regressions fixed >> there >> will be a small handful of packages that will break with the new >> release. This is because they are already incorrect it's just that >> this >> was previously hidden. > > I should like to point out that the fact that Cabal now does the > checks > for C header files and libs means we can generally do a better job now > if the packages themselves do not try to do anything clever. We get > more > consistent error messages but we also take into account extra flags > passed in by the user which a custom check may omit (which means the > user may not be able to install at all if they have the C libs in a > non-standard location). > > For example: > > jack-0.5: > Searching for jack/jack.h...setup: user error (ERROR: > jack/jack.h not found) > > And it ignores the --extra-include-dirs flag. > > > If you maintain a FFI binding package, take a look to see if it has > redundant checks in the ./configure or Setup.hs for C headers and > libs. > > While we're on the topic of good practise in configure scripts. Try to > avoid if possible grabbing random environment variables and putting > them > into a .buildinfo file. For example $CPPFLAGS or $LDFLAGS. The current > way for users to specify non-standard install locations for C libs is > via --extra-include-dirs and --extra-lib-dirs. > > If you want to argue that env vars are the better user interface then > make the case and we people agree then we can do it in Cabal itself. > Having it different in each package just confuses users. > > The other problem with putting the env vars into .buildinfo files is > that the cc-options, ld-options don't just get used to build your > package but they get put into the package registration info and used > when building every dependent package. > > More advice: > > * Don't make configure set buildable: False. Just fail. > * Do declare the key header files your package needs in > the .cabal > file rather than just in .hs or .hsc files. That way Cabal can > check for them and tools that convert to native distro packages > will notice the dependencies on foreign packages. > > Looking at the failing packages on hackage at the moment we've got > about > 100 failing at build time and about 50 failing at configure time. A > bunch more fail because they depend on other things that failed. The > biggest cause for build failure seems to be incorrectly specified > dependencies. That is they probably worked against one version and now > fail to compile against some different version. Quite often that is > the > base package, but it's not the majority. > > Duncan > From duncan.coutts at worc.ox.ac.uk Wed Feb 18 14:45:14 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Wed Feb 18 14:34:49 2009 Subject: advance warning of Cabal-1.6.0.2 and package breakages In-Reply-To: References: <1234830295.10457.72.camel@localhost> <1234963425.10457.162.camel@localhost> Message-ID: <1234986314.10457.181.camel@localhost> On Wed, 2009-02-18 at 08:54 -0800, Patrick Perry wrote: > Thanks for the heads-up, Duncan. For the BLAS package, the situation > is slightly complicated, and I will still be sticking with autoconf > for now. The reasons are twofold: Sorry, I was not actually suggesting that your blas package can drop its configure script yet. The only suggestion that applied to your packages (glas and gsl-random) was: > > More advice: > > > > * Don't make configure set buildable: False. Just fail. I should have made that clearer when I added you to the cc list. The reason behind this advice is that it pinpoints the location and cause of failure more precisely. If the configure script discovers the prerequisites are not met it should print a helpful error and fail there and then. > For me to switch to git rid of autoconf, I would need Cabal to provide > the following: > > 1. A way to find out the C calling convention for Fortran functions. > 2. A way to check if a specific function exists in a library. > 3. An option to try a different library name if a check for a library > fails. I think we'll be able to do 3. automatically in the future. For 2. we should probably supply a library function that Setup.hs scripts can use. As for 1. that probably just needs a custom test in the Setup.hs. Don't worry about it for now though. Duncan From wil.pearson at gmail.com Thu Feb 19 05:32:44 2009 From: wil.pearson at gmail.com (William Pearson) Date: Thu Feb 19 05:22:01 2009 Subject: Proposal for Max/Min Monoids Trac ticket 3036 Message-ID: Here is the ticket information. Will ---------- Forwarded message ---------- From: GHC Date: 2009/2/19 Subject: [GHC] #3036: Max/Min Monoids To: Cc: glasgow-haskell-bugs@haskell.org #3036: Max/Min Monoids -----------------------------+---------------------------------------------- Reporter: whpearson | Owner: Type: proposal | Status: new Priority: normal | Component: libraries/base Version: 6.10.1 | Severity: normal Keywords: | Testcase: Os: Unknown/Multiple | Architecture: Unknown/Multiple -----------------------------+---------------------------------------------- A small patch to Data.Monoid to give Max a and Min a monoids for (Ord a, Bounded a) = > a. Very similar to Sum and Product. Some alternatives here for unbounded Max/Mins http://hpaste.org/fastcgi/hpaste.fcgi/view?id=1528 My current preferred one is to not to bother and perhaps put Data.AddBounds in the main branch and people can wrap Integers in that if they want. It seems conceptually purer. -- Ticket URL: GHC The Glasgow Haskell Compiler From dons at galois.com Thu Feb 19 13:12:10 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 19 13:02:09 2009 Subject: A typeclass for byte operations? Message-ID: <20090219181210.GA13137@whirlpool.galois.com> Hey guys, At work we have many and varied needs to do bytewise operations on Haskell data. Things like: toBigEndian rotateBytes etc. Has anyone thought about: * A "Bytes" class, like the Bits class? Providing standard operations on values treated as bytes? * An Endian class, a la fromIntegral, for flipping bytes into a particular endianness. -- Don From lemming at henning-thielemann.de Thu Feb 19 19:01:18 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 19 18:42:42 2009 Subject: A typeclass for byte operations? In-Reply-To: <20090219181210.GA13137@whirlpool.galois.com> References: <20090219181210.GA13137@whirlpool.galois.com> Message-ID: <499DF2CE.1050809@henning-thielemann.de> Don Stewart schrieb: > Hey guys, > > At work we have many and varied needs to do bytewise operations on > Haskell data. Things like: > > toBigEndian > rotateBytes > > etc. > > Has anyone thought about: > > * A "Bytes" class, like the Bits class? Providing standard > operations on values treated as bytes? > > * An Endian class, a la fromIntegral, for flipping bytes into a > particular endianness. In contrast to Data.Bits it should use the proper parameter order, e.g. rotateBytes :: Int -> a -> a From dons at galois.com Thu Feb 19 18:58:25 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 19 18:48:22 2009 Subject: A typeclass for byte operations? In-Reply-To: <499DF2CE.1050809@henning-thielemann.de> References: <20090219181210.GA13137@whirlpool.galois.com> <499DF2CE.1050809@henning-thielemann.de> Message-ID: <20090219235825.GY13137@whirlpool.galois.com> lemming: > Don Stewart schrieb: > > Hey guys, > > > > At work we have many and varied needs to do bytewise operations on > > Haskell data. Things like: > > > > toBigEndian > > rotateBytes > > > > etc. > > > > Has anyone thought about: > > > > * A "Bytes" class, like the Bits class? Providing standard > > operations on values treated as bytes? > > > > * An Endian class, a la fromIntegral, for flipping bytes into a > > particular endianness. > > In contrast to Data.Bits it should use the proper parameter order, e.g. > rotateBytes :: Int -> a -> a Proper in the Okasaki sense? From lemming at henning-thielemann.de Thu Feb 19 19:03:15 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 19 18:52:31 2009 Subject: A typeclass for byte operations? In-Reply-To: <20090219235825.GY13137@whirlpool.galois.com> References: <20090219181210.GA13137@whirlpool.galois.com> <499DF2CE.1050809@henning-thielemann.de> <20090219235825.GY13137@whirlpool.galois.com> Message-ID: On Thu, 19 Feb 2009, Don Stewart wrote: > lemming: >> >> In contrast to Data.Bits it should use the proper parameter order, e.g. >> rotateBytes :: Int -> a -> a > > Proper in the Okasaki sense? Unfortunately I don't know what Okasaki's sense is, but I refered to this one: http://haskell.org/haskellwiki/Parameter_order From dons at galois.com Thu Feb 19 19:12:01 2009 From: dons at galois.com (Don Stewart) Date: Thu Feb 19 19:01:56 2009 Subject: A typeclass for byte operations? In-Reply-To: References: <20090219181210.GA13137@whirlpool.galois.com> <499DF2CE.1050809@henning-thielemann.de> <20090219235825.GY13137@whirlpool.galois.com> Message-ID: <20090220001201.GZ13137@whirlpool.galois.com> lemming: > > On Thu, 19 Feb 2009, Don Stewart wrote: > >> lemming: >>> >>> In contrast to Data.Bits it should use the proper parameter order, e.g. >>> rotateBytes :: Int -> a -> a >> >> Proper in the Okasaki sense? > > Unfortunately I don't know what Okasaki's sense is, but I refered to this > one: > http://haskell.org/haskellwiki/Parameter_order You made me go and dig through the archives! http://article.gmane.org/gmane.comp.lang.haskell.libraries/3650/ http://article.gmane.org/gmane.comp.lang.haskell.libraries/3654/ In response to questions about why the convention of putting the data structure as the last argument... This style supports multiple operations nicely, especially with combined with the $ operator. For example, to insert 3 elements into a set, you can say insert 1 $ insert 2 $ insert 3 $ someSet (the last $ is optional). With the other argument ordering, you would say insert (insert (insert someSet 3) 2) 1 See the entire thread: http://thread.gmane.org/gmane.comp.lang.haskell.libraries/3622/focus=3650 from 2005. -- Don From lennart at augustsson.net Thu Feb 19 19:21:20 2009 From: lennart at augustsson.net (Lennart Augustsson) Date: Thu Feb 19 19:10:37 2009 Subject: A typeclass for byte operations? In-Reply-To: <20090219181210.GA13137@whirlpool.galois.com> References: <20090219181210.GA13137@whirlpool.galois.com> Message-ID: You might want to make it more general than byte and byte-vector (or whatever it is you imagine rotating). The endian problem occurs anytime a sequence of smaller units are aggregated to a larger unit. So it could be bits in a word, 8-bit bytes in a vector, nybbles in a byte, etc. On Thu, Feb 19, 2009 at 7:12 PM, Don Stewart wrote: > Hey guys, > > At work we have many and varied needs to do bytewise operations on > Haskell data. Things like: > > toBigEndian > rotateBytes > > etc. > > Has anyone thought about: > > * A "Bytes" class, like the Bits class? Providing standard > operations on values treated as bytes? > > * An Endian class, a la fromIntegral, for flipping bytes into a > particular endianness. > > -- Don > _______________________________________________ > Libraries mailing list > Libraries@haskell.org > http://www.haskell.org/mailman/listinfo/libraries > From dave at zednenem.com Fri Feb 20 00:00:38 2009 From: dave at zednenem.com (David Menendez) Date: Thu Feb 19 23:49:54 2009 Subject: Proposal for Max/Min Monoids Trac ticket 3036 In-Reply-To: References: Message-ID: <49a77b7a0902192100h1fece0b4rbd9298eaec398eeb@mail.gmail.com> I like the Min/Max monoids, and the possibility of adding something like AddBounds (although I'd prefer a name like "Bounded"; "Bounded Integer" reads better than "AddBounds Integer"). One possible snag is that, so far as I know, there's no explicit requirement that a type that instantiates Ord and Bounded should satisfy the laws, min maxBound a = a = min a maxBound max minBound a = a = max a minBound We're just assuming (reasonably) that instances will always work out that way. (Although, I guess we're making the same assumption with Sum and Prod.) -- Dave Menendez From allbery at ece.cmu.edu Fri Feb 20 01:16:39 2009 From: allbery at ece.cmu.edu (Brandon S. Allbery KF8NH) Date: Fri Feb 20 01:06:07 2009 Subject: A typeclass for byte operations? In-Reply-To: <20090219181210.GA13137@whirlpool.galois.com> References: <20090219181210.GA13137@whirlpool.galois.com> Message-ID: <0F6ACC54-D4B6-4C39-9BA4-BF90ADFB5AA0@ece.cmu.edu> On 2009 Feb 19, at 13:12, Don Stewart wrote: > At work we have many and varied needs to do bytewise operations on > Haskell data. Things like: > > toBigEndian > rotateBytes > > etc. > > Has anyone thought about: > > * A "Bytes" class, like the Bits class? Providing standard > operations on values treated as bytes? > > * An Endian class, a la fromIntegral, for flipping bytes into a > particular endianness. I'd think these would be additions to the Word* types, reorganized under a Word class. -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 195 bytes Desc: This is a digitally signed message part Url : http://www.haskell.org/pipermail/libraries/attachments/20090220/6c1ef38b/PGP-0001.bin From duncan.coutts at worc.ox.ac.uk Fri Feb 20 06:05:55 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Fri Feb 20 05:55:38 2009 Subject: file permissions and copying files Message-ID: <1235127955.10457.301.camel@localhost> All, I have complained before about the System.Directory file permissions API, and how it relates to copyFile. Let me give an example of where the existing copyFile API is too limited. Perhaps this will help us think about what an improved API might be. The example of course is build system component of Cabal. It turns out that we actually need *three* different kinds of copyFile. The only difference in each case is the treatment of file permissions / attributes. To achieve the worst case consider a source tree that is completely read only and a user umask that makes new files not world readable. The cases are: 1. Copying files within the build tree, eg as a dummy pre-processor. In this case it should behave like "cat src > dest" and not like cp. That is, it is essential that we create the new files with default permissions and not copy the source permissions. Otherwise we would end up making read-only files which would cause problems later (especially on windows). 2. Copying source files into a temp directory to prepare a source tarball. In this case we really do want to copy file permissions. In particular we want to copy executable permissions on files like ./configure. In this case it does not matter if we also copy read-only permissions because we will only delete these files, not try to overwrite them. Also, the tarball creation code will only copy executable permissions. 3. Installing files. In this case it is essential that we specify exact permissions and ignore the umask of the installing user or the permissions of the source files. The source files could be read-only and that would mess things up on windows as we would not be able to re-install over the read-only files. We also do not want to copy permissions generally because if the umask of the user that built the files was such that they're not globally readable then installing globally (eg via sudo) will install files only readable by root. Instead we have to specify that the files are read/write by their owner and readable by everyone else (and executable for binaries). If we were to generalise from these examples we'd have some copyFile function that took an extra function to generate the permissions from a combination of the source and default permissions (or it could ignore both and use specific ones). Note that in the case where I had to set specific permissions I didn't really need full control over unix permission bits, I just wanted a couple properties. I needed executable or not, readable/writable by the file owner and readable/writable by others. Is that sufficiently abstract to map onto both Windows and Unix file permissions? The thing that works least well between unix and windows file permissions is group ownership and permissions but can these be ignored in most portable situations? So at the moment we have: data Permissions = Permissions { readable :: Bool, writable :: Bool, executable :: Bool, searchable :: Bool } The main problem with this is that it is not abstract so it looses information when we do: getPermissions src >>= setPermissions dst However suppose it was an abstract type such that this worked and had the same query functions as above. Then for setting permissions we'd also have setReadable, etc :: Permissions -> Permissions. However as I mentioned above for Cabal we'd need to distinguish setting readable/writable for the file owner vs for all other users. So how about this: data Permissions -- abstract getPermissions :: FilePath -> IO Permissions setPermissions :: FilePath -> Permissions -> IO () -- | only tells us effective permissions for the current process readable :: Permissions -> Bool writable :: Permissions -> Bool executable :: Permissions -> Bool searchable :: Permissions -> Bool setUserReadable, setOtherReadable :: Bool -> Permissions -> Permissions setUserWritable, setOtherWritable :: Bool -> Permissions -> Permissions setExecutable :: Bool -> Permissions -> Permissions setSearchable :: Bool -> Permissions -> Permissions So for Unix the mapping is: setUserReadable True/False chmod u+r / u-r setUserWritable True/False chmod u+w / u-w setOtherReadable True/False chmod go+r / go-r setOtherWritable True/False chmod go+w / go-w setExecutable True/False chmod +x / -x (for files) setSearchable True/False chmod +x / -x (for dirs) So I'm aligning group with other and not distinguishing executable/searchable for user vs others. Note also that what is set by set* is not necessarily the same as what we get back from the query functions because those take all permissions into account including group permissions and other ACLs that we cannot manipulate via this API. Comments? Duncan From wil.pearson at gmail.com Fri Feb 20 08:57:22 2009 From: wil.pearson at gmail.com (William Pearson) Date: Fri Feb 20 08:46:36 2009 Subject: Proposal for Max/Min Monoids Trac ticket 3036 In-Reply-To: <49a77b7a0902192100h1fece0b4rbd9298eaec398eeb@mail.gmail.com> References: <49a77b7a0902192100h1fece0b4rbd9298eaec398eeb@mail.gmail.com> Message-ID: 2009/2/20 David Menendez : > I like the Min/Max monoids, and the possibility of adding something > like AddBounds (although I'd prefer a name like "Bounded"; "Bounded > Integer" reads better than "AddBounds Integer"). We already have a Bounded typeclass, so things would get confusing. How about BoundsAdded Integer? > One possible snag is that, so far as I know, there's no explicit > requirement that a type that instantiates Ord and Bounded should > satisfy the laws, > > ? ?min maxBound a = a = min a maxBound > ? ?max minBound a = a = max a minBound > > We're just assuming (reasonably) that instances will always work out > that way. (Although, I guess we're making the same assumption with Sum > and Prod.) > You could do something like class (Bounded a, Ord a) => BoundedOrd a where minB :: a -> a -> a minB a b | b == maxBound = a | a == maxBound = b | (a == minBound) || (b == minBound) = minBound | otherwise = min a b maxB :: a -> a -> a maxB a b | b == minBound = a | a == minBound = b | (a == maxBound) || (b == maxBound) = maxBound | otherwise = max a b And have the types inside Min and Max monoids need to be instances of this type. It depends how much we are trusting the implementors of min and max... It also doesn't stop people reimplementing them, perhaps we'll have to wait for conal's type class morphisms (I haven't fully digested these yet)? Will Pearson From sof at galois.com Fri Feb 20 13:55:32 2009 From: sof at galois.com (Sigbjorn Finne) Date: Fri Feb 20 13:44:40 2009 Subject: parsec:Text.ParserCombinators.Parsec.Token wibble Message-ID: <499EFCA4.5060201@galois.com> Hi, the ASCII caret escape handling in the above module's makeTokenParser isn't quite right -- off by one, and only supporting A-Z carets. Could someone with privs (and maintainership badge) commit the following tweak? foo$ darcs whatsnew hunk ./Text/ParserCombinators/Parsec/Token.hs 196 - ; code <- upper - ; return (toEnum (fromEnum code - fromEnum 'A')) + ; code <- (oneOf ['@'..'_']) <|> char '?' + ; return (if code == '?' then '\DEL' else toEnum (fromEnum code - fromEnum '@')) cheers --sigbjorn From benja.fallenstein at gmail.com Fri Feb 20 19:45:10 2009 From: benja.fallenstein at gmail.com (Benja Fallenstein) Date: Fri Feb 20 19:34:26 2009 Subject: Proposal for Max/Min Monoids Trac ticket 3036 In-Reply-To: References: <49a77b7a0902192100h1fece0b4rbd9298eaec398eeb@mail.gmail.com> Message-ID: On Fri, Feb 20, 2009 at 2:57 PM, William Pearson wrote: > 2009/2/20 David Menendez : >> I like the Min/Max monoids, and the possibility of adding something >> like AddBounds (although I'd prefer a name like "Bounded"; "Bounded >> Integer" reads better than "AddBounds Integer"). > > We already have a Bounded typeclass, so things would get confusing. > How about BoundsAdded Integer? OrInfinity? :-) I suppose Extended -- as in "extended real number" -- would not make the intent clear enough... -b From dons at galois.com Sat Feb 21 15:55:07 2009 From: dons at galois.com (Don Stewart) Date: Sat Feb 21 15:44:59 2009 Subject: A typeclass for byte operations? In-Reply-To: <20090219181210.GA13137@whirlpool.galois.com> References: <20090219181210.GA13137@whirlpool.galois.com> Message-ID: <20090221205507.GA22794@whirlpool.galois.com> dons: > Hey guys, > > At work we have many and varied needs to do bytewise operations on > Haskell data. Things like: > > toBigEndian > rotateBytes > > etc. > > Has anyone thought about: > > * A "Bytes" class, like the Bits class? Providing standard > operations on values treated as bytes? > > * An Endian class, a la fromIntegral, for flipping bytes into a > particular endianness. > I found http://hackage.haskell.org/packages/archive/byteorder/1.0.0/doc/html/System-ByteOrder.html http://hackage.haskell.org/cgi-bin/hackage-scripts/package/byteorder-1.0.0 On Hackage. which does a few things in this area. From dons at galois.com Sat Feb 21 16:16:35 2009 From: dons at galois.com (Don Stewart) Date: Sat Feb 21 16:06:29 2009 Subject: Arch Haskell News: Feb 21 2009 Message-ID: <20090221211635.GK19753@whirlpool.galois.com> http://archhaskell.wordpress.com/2009/02/21/arch-haskell-news-feb-21-2009/ Arch now has 934 Haskell packages in AUR. Packages were updated or released in the following areas this week: Concurrency Control Cryptography Data Database Devel Gui Graphics Network System Web -- Don From wnoise at ofb.net Sun Feb 22 14:19:11 2009 From: wnoise at ofb.net (Aaron Denney) Date: Sun Feb 22 14:08:27 2009 Subject: Proposal for Max/Min Monoids Trac ticket 3036 References: <49a77b7a0902192100h1fece0b4rbd9298eaec398eeb@mail.gmail.com> Message-ID: On 2009-02-21, Benja Fallenstein wrote: > On Fri, Feb 20, 2009 at 2:57 PM, William Pearson wrote: >> 2009/2/20 David Menendez : >>> I like the Min/Max monoids, and the possibility of adding something >>> like AddBounds (although I'd prefer a name like "Bounded"; "Bounded >>> Integer" reads better than "AddBounds Integer"). >> >> We already have a Bounded typeclass, so things would get confusing. >> How about BoundsAdded Integer? > > OrInfinity? :-) > > I suppose Extended -- as in "extended real number" -- would not make > the intent clear enough... Borrowing from topology, "completion", or "compactified" suggest themselves, but I don't think they're clear enough in this context. WithBounds, BoundsAdded, or Bounded seem clear enough. I personally am not too concerned with the overlap with the Bounded typeclass. In fact, I think that agreeing is a nice concurrence of terminology. -- Aaron Denney -><- From ml at isaac.cedarswampstudios.org Sun Feb 22 17:15:05 2009 From: ml at isaac.cedarswampstudios.org (Isaac Dupree) Date: Sun Feb 22 17:04:20 2009 Subject: Proposal for Max/Min Monoids Trac ticket 3036 In-Reply-To: References: Message-ID: <200902221715.05796.ml@isaac.cedarswampstudios.org> Aaron Denney wrote: > Borrowing from topology, "completion", or "compactified" suggest > themselves, but I don't think they're clear enough in this context. > WithBounds, BoundsAdded, or Bounded seem clear enough. I personally > am not too concerned with the overlap with the Bounded typeclass. In > fact, I think that agreeing is a nice concurrence of terminology. I think "Bounded" is outright misleading here. It *adds* an upper bound and a lower bound, even if they already exist. Bounded Integer -- well, okay Bounded Int -- hey, Int is already bounded, but this adds more bounds. or for a type that's conceptually more bounded to start with, Bounded Natural -- MinBound then is not 0, it's less than 0. Bounded (Bounded X) -- adds two layers of bounds; it's not idempotent. -Isaac From lemming at henning-thielemann.de Sun Feb 22 18:46:50 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Sun Feb 22 18:35:58 2009 Subject: file permissions and copying files In-Reply-To: <1235127955.10457.301.camel@localhost> References: <1235127955.10457.301.camel@localhost> Message-ID: On Fri, 20 Feb 2009, Duncan Coutts wrote: > I have complained before about the System.Directory file permissions > API, and how it relates to copyFile. > > Let me give an example of where the existing copyFile API is too > limited. Perhaps this will help us think about what an improved API > might be. > > ... > > So for Unix the mapping is: > > setUserReadable True/False chmod u+r / u-r > setUserWritable True/False chmod u+w / u-w > > setOtherReadable True/False chmod go+r / go-r > setOtherWritable True/False chmod go+w / go-w It's not hard to predict that 'Other' will be too coarse for some applications, although I haven't an example. However, maybe we should have a new package which provides common functions for Unix and Windows as needed by Cabal. I expect that the 3 cases of permission transfer that you described for Cabal will also occur in other applications. From this package a general API for System.Directory may evolve. From duncan.coutts at worc.ox.ac.uk Mon Feb 23 04:34:09 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Feb 23 04:23:27 2009 Subject: file permissions and copying files In-Reply-To: References: <1235127955.10457.301.camel@localhost> Message-ID: <1235381649.10457.339.camel@localhost> On Mon, 2009-02-23 at 00:46 +0100, Henning Thielemann wrote: > On Fri, 20 Feb 2009, Duncan Coutts wrote: > > > I have complained before about the System.Directory file permissions > > API, and how it relates to copyFile. > > > > Let me give an example of where the existing copyFile API is too > > limited. Perhaps this will help us think about what an improved API > > might be. > > > > ... > > > > So for Unix the mapping is: > > > > setUserReadable True/False chmod u+r / u-r > > setUserWritable True/False chmod u+w / u-w > > > > setOtherReadable True/False chmod go+r / go-r > > setOtherWritable True/False chmod go+w / go-w > > It's not hard to predict that 'Other' will be too coarse for some > applications, although I haven't an example. However, maybe we should have > a new package which provides common functions for Unix and Windows as > needed by Cabal. I expect that the 3 cases of permission transfer that you > described for Cabal will also occur in other applications. From this > package a general API for System.Directory may evolve. Certainly 'Other' is too coarse in general, but I fear there is not much that is more fine grained that is also portable and useful. Is there a useful abstraction that covers Windows ACLs (and legacy file attributes) and POSIX ACLs and classic Unix permission bits? There might be but I'm not hopeful. POSIX ACLs subsume the Unix permission bits but the Windows ACL model is rather different. My guess is that something around the granularity of owner/other is what we can get in a portable API and more detailed things just have to be platform-specific. We need both. Duncan From Christian.Maeder at dfki.de Mon Feb 23 06:03:00 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Mon Feb 23 05:52:07 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> Message-ID: <49A28264.7050209@dfki.de> Antoine Latter wrote: > On Wed, Feb 18, 2009 at 5:20 AM, Christian Maeder > wrote: >> Hi, >> >> I would like to reuse the module Distribution.Client.Tar >> from the cabal-install package. [...] > > There's a 'tar' package on Hackage: > http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar Thanks, for pointing this out. This package depends (in contrast to cabal-install) on additional packages binary and unix-compat, which isn't a real problem, but only seems unnecessary. The tar package does also not support compression (via zlib), but maybe this could/should be a separate package. (There's also an older package "htar" with "compression".) The sources in cabal-install seem most up-to-date (because of cabal-install-0.6.2) and it would make sense to take this sources and replace those in the tar-package. Does the tar-package have any advantage (i.e. speed or portability) over Distribution.Client.Tar? The module structure Codec.Archive.Tar looks a bit nicer, but re-exporting the internal data structures seems unnecessary to me. Where are the actual repositories with the most recent sources? Cheers Christian From duncan.coutts at worc.ox.ac.uk Mon Feb 23 06:34:44 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Feb 23 06:24:01 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <49A28264.7050209@dfki.de> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> Message-ID: <1235388884.10457.358.camel@localhost> On Mon, 2009-02-23 at 12:03 +0100, Christian Maeder wrote: > Antoine Latter wrote: > > On Wed, Feb 18, 2009 at 5:20 AM, Christian Maeder > > wrote: > >> Hi, > >> > >> I would like to reuse the module Distribution.Client.Tar > >> from the cabal-install package. > [...] > > > > There's a 'tar' package on Hackage: > > http://hackage.haskell.org/cgi-bin/hackage-scripts/package/tar > > Thanks, for pointing this out. This package depends (in contrast to > cabal-install) on additional packages binary and unix-compat, which > isn't a real problem, but only seems unnecessary. Yes, I agree, I've already removed them. > The tar package does also not support compression (via zlib), but maybe > this could/should be a separate package. Yes it's easy to compose. The docs in the new code give examples. I don't think it's right to have the package depend on zlib and bzlib and lzma and ... etc when it is trivial to just compose (de)compression in the pipeline: Tar.unpack dir . Tar.read . GZip.decompress =<< BS.readFile tar or BS.writeFile tar . GZip.compress . Tar.write =<< Tar.pack base dir > (There's also an older package "htar" with "compression".) Oh, that's really just a test program for the tar package. The current one uses the zlib and bzlib packages for compression. > The sources in cabal-install seem most up-to-date (because of > cabal-install-0.6.2) and it would make sense to take this sources and > replace those in the tar-package. Yes, that's what I was doing over the weekend. > Does the tar-package have any advantage (i.e. speed or portability) over > Distribution.Client.Tar? It does now! :-) It's now better structured, has better error checking and is better documented. > The module structure Codec.Archive.Tar looks a bit nicer, but > re-exporting the internal data structures seems unnecessary to me. > Where are the actual repositories with the most recent sources? darcs get http://code.haskell.org/tar/ Let me know what you think about the API and documentation. You mention above about exporting internal data structures. As far as I can see everything that is exported in the current code is needed. Let me know if you think it is too much or too little. Currently I get round-trip byte-for-byte compatibility with about 50% of the .tar.gz packages on my system (I'm on gentoo so there's lots of those). The ones that are not byte-for-byte equal after reading/writing are still readable by other tools (and probably normalised and closer to standard compliant) but it needs investigating in more detail. The checking API is incomplete (security, tarbombs, portability) and there are no tests for the lazy streaming properties yet (ie that we can process arbitrary large archives in constant space). Duncan From Christian.Maeder at dfki.de Tue Feb 24 04:34:23 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Tue Feb 24 04:23:27 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <1235388884.10457.358.camel@localhost> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> Message-ID: <49A3BF1F.3030308@dfki.de> Duncan Coutts wrote: [...] > Tar.unpack dir . Tar.read . GZip.decompress =<< BS.readFile tar > > or > > BS.writeFile tar . GZip.compress . Tar.write =<< Tar.pack base dir [...] >> The sources in cabal-install seem most up-to-date (because of >> cabal-install-0.6.2) and it would make sense to take this sources and >> replace those in the tar-package. > > Yes, that's what I was doing over the weekend. thanks a lot! > darcs get http://code.haskell.org/tar/ > > Let me know what you think about the API and documentation. You mention > above about exporting internal data structures. As far as I can see > everything that is exported in the current code is needed. Let me know > if you think it is too much or too little. Ok, I think the api is too big (for a casual user). I don't want to know anything about the internals of an "Entry" or about a "TarPath". For refactoring cabal-install (using your tar package) the following interface was enough: create :: FilePath -> FilePath -> FilePath -> IO () extract :: FilePath -> FilePath -> IO () read :: ByteString -> Entries write :: [Entry] -> ByteString pack :: FilePath -> FilePath -> IO [Entry] unpack :: FilePath -> Entries -> IO () data Entry fileName :: Entry -> FilePath fileContent :: Entry -> ByteString data Entries = Next Entry Entries | Done | Fail String Maybe only a "isNormalFile" test-function for an Entry is missing. checkSecurity is not needed in the API, because it is done by unpack. (checkTarBomb does nothing currently). Tar entries should (usually will) not be constructed by the user. getDirectoryContentsRecursive does not really belong into this tar package. I would be happy, if the existence of TarPath (and all the other funny entry fields) could be hidden from the user. Manipulating Entries is also not a typical user task. (Maybe the type Entries should just be "[Either String Entry]", but the given type is fine, as it only allows a final failure string) So rather than re-exporting almost everything from the other modules in the top module, I suggest my API above and simply expose all other modules in case some wants the internals. > Currently I get round-trip byte-for-byte compatibility with about 50% of > the .tar.gz packages on my system (I'm on gentoo so there's lots of > those). The ones that are not byte-for-byte equal after reading/writing > are still readable by other tools (and probably normalised and closer to > standard compliant) but it needs investigating in more detail. > > The checking API is incomplete (security, tarbombs, portability) and > there are no tests for the lazy streaming properties yet (ie that we can > process arbitrary large archives in constant space). I can only suggest to release it soon, use it for cabal-install and make a new release of cabal-install for ghc-6.10.2 Thank you, Duncan Christian P.S. I could (darcs) send you my (humble) changes to cabal-install and tar From Christian.Maeder at dfki.de Tue Feb 24 06:04:50 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Tue Feb 24 05:53:53 2009 Subject: Pickling a finite map (Binary + zlib) [was: Data.Binary poor read performance] In-Reply-To: References: <404396ef0902231043x78abed07lc969909a2526f6a1@mail.gmail.com> <49A34307.7010008@freegeek.org> <20090224010314.GG32563@whirlpool.galois.com> <20090224075916.GA1941@whirlpool.galois.com> Message-ID: <49A3D452.8080103@dfki.de> Felipe Lessa wrote: > The builds seem fine, but we spot a (length xs) on the beginning. > Maybe this is the culprit? We already know the size of the map (it was > serialized), so it is just a matter of exporting > > fromDistinctAscSizedList :: Int -> [(k, a)] -> Map k a Excellent idea, what does stop you? fromDistinctAscList is already a function with a precondition that is not checked. > Too bad 'Map' is exported as an abstract data type and it's not > straighforward to test this conjecture. Any ideas? One really doesn't want to see the actual implementation (except for debugging or tuning purposes), but maybe conversions to and from a fully (and uniquely) balanced binary tree are desirable. (a basic binary tree data type is still missing on hackage). I think, equal maps should not yield different serialization results just because the internal tree structure happens to be different, but that's of course debatable. Cheers Christian Maybe the discussion should be continued on libraries@haskell.org? From haskell at list.mightyreason.com Tue Feb 24 06:57:34 2009 From: haskell at list.mightyreason.com (ChrisK) Date: Tue Feb 24 06:46:39 2009 Subject: ANN: bug fix for regex-tdfa, version 0.97.4 (and "regex-ast") Message-ID: <49A3E0AE.404@list.mightyreason.com> Hello, The regex-tdfa package has had a series of bug fix releases (0.97.1 and 2 and 3 and now 4). This 0.97.4 releases finishes fixing the bug that was only mostly fixed in the 0.97.1 release. An example of the fixed bug: Apply the regex pattern (BB(B?))+(B?) to the text BBBB. The "BB" in the pattern should be used twice and both "B?" should match nothing. My code grouped the "+" wrong and matched the "BB" once and then both the "B?" matched a "B". The case fixed here was not initially caught because of how I search for unknown bugs. I use "Arbitrary" from QuickCheck to generate random patterns and strings to search, and compare regex-tdfa to another POSIX engine. Because I am on OS X, I am limited by the the native POSIX libraries bugs: this bug in regex-tdfa was triggered only when the native POSIX was also buggy. But the source of most of my unit tests is AT&T research [1], and they have a "libast" with a POSIX implementation. I have adapted my regex-* wrapper packages to make a "regex-ast" Haskell interface, but the difficulties with the AT&T headers prevent me from releasing this on hackage. This "regex-ast" has given me access to a less buggy POSIX back-end, and randomized testing has led to catching the bug fixed here (as well as a few bug reports back to AT&T). So while regex-tdfa will not win many speed contests, it is the only POSIX regular expression library I have running that passes all the unit tests. [1] http://www.research.att.com/sw/download/ http://www.research.att.com/~gsf/testregex/ http://www.research.att.com/~gsf/testregex/re-interpretation.html From duncan.coutts at worc.ox.ac.uk Tue Feb 24 07:00:04 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 24 06:49:17 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <49A3BF1F.3030308@dfki.de> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> Message-ID: <1235476804.10457.385.camel@localhost> On Tue, 2009-02-24 at 10:34 +0100, Christian Maeder wrote: > > Let me know what you think about the API and documentation. You mention > > above about exporting internal data structures. As far as I can see > > everything that is exported in the current code is needed. Let me know > > if you think it is too much or too little. > > Ok, I think the api is too big (for a casual user). I don't want to know > anything about the internals of an "Entry" or about a "TarPath". For > refactoring cabal-install (using your tar package) the following > interface was enough: > > create :: FilePath -> FilePath -> FilePath -> IO () > extract :: FilePath -> FilePath -> IO () > read :: ByteString -> Entries > write :: [Entry] -> ByteString > pack :: FilePath -> FilePath -> IO [Entry] > unpack :: FilePath -> Entries -> IO () > data Entry > fileName :: Entry -> FilePath > fileContent :: Entry -> ByteString > data Entries > = Next Entry Entries > | Done > | Fail String > > Maybe only a "isNormalFile" test-function for an Entry is missing. isNormalFile entry = fileType entry == NormalFile No we really want one of these for every file type? isDirectory, isHardLink. I hope not. So perhaps that means you just want to expose fileType and FileType too in the top level API. > checkSecurity is not needed in the API, because it is done by unpack. > (checkTarBomb does nothing currently). It's needed if you're checking a tar file now because you expect to unpack it later, eg on hackage. > Tar entries should (usually will) not be constructed by the user. I've got a use case where we do. > getDirectoryContentsRecursive does not really belong into this tar package. True but it's so useful you'd have to implement it yourself if it was not provided. We can't get it into the System.Directory package very quickly. > I would be happy, if the existence of TarPath (and all the other funny > entry fields) could be hidden from the user. You'll have to trade that for some cases where you get error instead. Translating file paths to tar paths does not work in all cases. We can hide that in an IO error for functions that are in IO, like pack. > Manipulating Entries is also not a typical user task. (Maybe the type > Entries should just be "[Either String Entry]", but the given type is > fine, as it only allows a final failure string) Yeah I think only one failure is good. Manipulating entries is more common than you think, if you are working with a tar file in memory and never extracting files to disk. For example the hackage-server both constructs and dissects tar files but never packs or unpacks them. > So rather than re-exporting almost everything from the other modules in > the top module, I suggest my API above and simply expose all other > modules in case some wants the internals. Ok, so perhaps we should split the API into two, in the way you suggest above. I'll try that and see how it looks. > > Currently I get round-trip byte-for-byte compatibility with about 50% of > > the .tar.gz packages on my system (I'm on gentoo so there's lots of > > those). The ones that are not byte-for-byte equal after reading/writing > > are still readable by other tools (and probably normalised and closer to > > standard compliant) but it needs investigating in more detail. > > > > The checking API is incomplete (security, tarbombs, portability) and > > there are no tests for the lazy streaming properties yet (ie that we can > > process arbitrary large archives in constant space). > > I can only suggest to release it soon, use it for cabal-install and make > a new release of cabal-install for ghc-6.10.2 I'll not be using the tar package for cabal-install immediately. Keeping the number of dependencies low while we work out installation and packaging issues is fairly important. People already complain that cabal-install has any dependencies at all. Once it's bundled in the platform then that restriction will be lifted. > P.S. I could (darcs) send you my (humble) changes to cabal-install and tar Please. Duncan From duncan.coutts at worc.ox.ac.uk Tue Feb 24 07:20:33 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 24 07:09:47 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <1235476804.10457.385.camel@localhost> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> <1235476804.10457.385.camel@localhost> Message-ID: <1235478033.10457.395.camel@localhost> On Tue, 2009-02-24 at 12:00 +0000, Duncan Coutts wrote: > On Tue, 2009-02-24 at 10:34 +0100, Christian Maeder wrote: > > checkSecurity is not needed in the API, because it is done by unpack. > > (checkTarBomb does nothing currently). > > It's needed if you're checking a tar file now because you expect to > unpack it later, eg on hackage. > > > Tar entries should (usually will) not be constructed by the user. > > I've got a use case where we do. So I should say what these use cases are. One is cabal-install of course. That's fairly easy it just needs to create and extract tar files. The other case is hackage. For that we want to upload and check the contents of tar files, without ever unpacking them to local files. We want to check the tar file itself to make sure it is a portable format (ie not containing any funky extensions that not all tar readers will grok) or things that are not portable between platforms, like file names that would be invalid on Windows. We also want to extract a single file in memory (the .cabal file). Another case within hackage is constructing the 00-index.tar.gz file. That is built in memory from an another internal representation of the package index. For that we really are constructing each entry ourselves, supplying all the appropriate info including file modification time, ownership etc. A final case in hackage is serving the contents of .tar files. This is to let users browse the contents of packages, eg to read the README without having to download the whole .tar.gz. Should also make all the code more easily googleable. It's also the method we want to use for serving haddock docs. Bots or package owners will upload .tar.gz bundles of documentation and we'll serve the contents directly without unpacking them. We'll do that by gunziping and storing the .tar file on disk, scanning it once to generate a file name -> (offset, length) index and then when we service a request we open the .tar file seek to the offset and return that length. I think that's it. Duncan From Christian.Maeder at dfki.de Tue Feb 24 07:42:00 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Tue Feb 24 07:31:02 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <1235476804.10457.385.camel@localhost> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> <1235476804.10457.385.camel@localhost> Message-ID: <49A3EB18.5050301@dfki.de> Duncan Coutts wrote: [...] >> data Entry >> fileName :: Entry -> FilePath >> fileContent :: Entry -> ByteString [...] >> Maybe only a "isNormalFile" test-function for an Entry is missing. > > isNormalFile entry = fileType entry == NormalFile > > No we really want one of these for every file type? isDirectory, > isHardLink. I hope not. So perhaps that means you just want to expose > fileType and FileType too in the top level API. I only suggested isNormalFile in order to test, if fileContent can be reasonably used. Directly working with FileType and Entry should be for experts only. Surely the file types Directory and SymbolicLink/HardLink (a single "isLink" test) may be of some user interest, too, but for these types fileContent has no meaning and additionally the linkTarget should be exported. For directories the entry does not reveal the contents. Therefore, I think, fileName, fileContent, and isNormalFile is just enough for a top level API. >> checkSecurity is not needed in the API, because it is done by unpack. >> (checkTarBomb does nothing currently). > > It's needed if you're checking a tar file now because you expect to > unpack it later, eg on hackage. It's perfect in Codec.Archive.Tar.Check >> Tar entries should (usually will) not be constructed by the user. > > I've got a use case where we do. Ok, importing Codec.Archive.Tar.Types and Codec.Archive.Tar.Pack should be fine then. >> getDirectoryContentsRecursive does not really belong into this tar package. > > True but it's so useful you'd have to implement it yourself if it was > not provided. We can't get it into the System.Directory package very > quickly. Keeping it for a while in Codec.Archive.Tar.Pack only, should be fine. (A separate module is overkill.) [...] >> So rather than re-exporting almost everything from the other modules in >> the top module, I suggest my API above and simply expose all other >> modules in case some wants the internals. > > Ok, so perhaps we should split the API into two, in the way you suggest > above. I'll try that and see how it looks. [...] Thanks again, Christian From Christian.Maeder at dfki.de Tue Feb 24 07:57:03 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Tue Feb 24 07:46:06 2009 Subject: Entries type: Re: tar-package was: Re: modules of cabal-install In-Reply-To: <1235476804.10457.385.camel@localhost> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> <1235476804.10457.385.camel@localhost> Message-ID: <49A3EE9F.70202@dfki.de> Duncan Coutts wrote: >> Manipulating Entries is also not a typical user task. (Maybe the type >> Entries should just be "[Either String Entry]", but the given type is >> fine, as it only allows a final failure string) > > Yeah I think only one failure is good. data Entries = Entries [Entry] (Maybe String) or directly using "([Entry], Maybe String)" is an alternative, where "Nothing" indicates no failure. This would avoid the extra folding and mapping on Entries and maybe "unpack" could then work on [Entry] only. Cheers Christian From duncan.coutts at worc.ox.ac.uk Tue Feb 24 08:19:49 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 24 08:09:00 2009 Subject: Entries type: Re: tar-package was: Re: modules of cabal-install In-Reply-To: <49A3EE9F.70202@dfki.de> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> <1235476804.10457.385.camel@localhost> <49A3EE9F.70202@dfki.de> Message-ID: <1235481589.10457.403.camel@localhost> On Tue, 2009-02-24 at 13:57 +0100, Christian Maeder wrote: > Duncan Coutts wrote: > >> Manipulating Entries is also not a typical user task. (Maybe the type > >> Entries should just be "[Either String Entry]", but the given type is > >> fine, as it only allows a final failure string) > > > > Yeah I think only one failure is good. > > data Entries = Entries [Entry] (Maybe String) > or > directly using "([Entry], Maybe String)" Sadly that does not account for streaming. We may not discover the error until we've processed 100 megabytes of tar file. So we cannot give top level access to an error that might happen at the end. > is an alternative, where "Nothing" indicates no failure. This would > avoid the extra folding and mapping on Entries The nice thing about the current Entries is that it models the failure modes and works lazily. > and maybe "unpack" could then work on [Entry] only. True, though it's also convenient that unpack just raises any entry errors as IO errors. Converting Entries to [Entry] requires reading the whole thing (to check there are no errors), again defeating lazyness/streaming which would mean we could not work with large tar files in constant space. Duncan From duncan.coutts at worc.ox.ac.uk Tue Feb 24 09:31:56 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Tue Feb 24 09:21:12 2009 Subject: tar-package was: Re: modules of cabal-install In-Reply-To: <49A3EB18.5050301@dfki.de> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> <1235476804.10457.385.camel@localhost> <49A3EB18.5050301@dfki.de> Message-ID: <1235485916.10457.412.camel@localhost> On Tue, 2009-02-24 at 13:42 +0100, Christian Maeder wrote: > > Ok, so perhaps we should split the API into two, in the way you suggest > > above. I'll try that and see how it looks. > [...] Ok, I've applied your patch with a slight variation. See the haddocks: http://haskell.org/~duncan/tar/docs/ Basically it's split into Codec.Archive.Tar for the high level bits of the api you suggested and Codec.Archive.Tar.Entry for the details of what an Entry is, how to construct one etc. Hopefully the documentation makes that split clear. I've exported fileType :: Entry -> FileType data FileType = ... from the high level api, rather than adding isNormalFile :: Entry -> Bool Thanks again for the feedback. Duncan From rturk at science.uva.nl Tue Feb 24 20:29:06 2009 From: rturk at science.uva.nl (Remi Turk) Date: Tue Feb 24 20:18:07 2009 Subject: Entries type: Re: tar-package was: Re: modules of cabal-install In-Reply-To: <1235481589.10457.403.camel@localhost> References: <499BEEEA.7080000@dfki.de> <694519c50902180547k58c71693hf8f29a6af740f0ac@mail.gmail.com> <49A28264.7050209@dfki.de> <1235388884.10457.358.camel@localhost> <49A3BF1F.3030308@dfki.de> <1235476804.10457.385.camel@localhost> <49A3EE9F.70202@dfki.de> <1235481589.10457.403.camel@localhost> Message-ID: <20090225012906.GA453@frogstar.lan> On Tue, Feb 24, 2009 at 01:19:49PM +0000, Duncan Coutts wrote: > On Tue, 2009-02-24 at 13:57 +0100, Christian Maeder wrote: > > Duncan Coutts wrote: > > >> Manipulating Entries is also not a typical user task. (Maybe the type > > >> Entries should just be "[Either String Entry]", but the given type is > > >> fine, as it only allows a final failure string) > > > > > > Yeah I think only one failure is good. > > > > data Entries = Entries [Entry] (Maybe String) > > or > > directly using "([Entry], Maybe String)" > > Sadly that does not account for streaming. We may not discover the error > until we've processed 100 megabytes of tar file. So we cannot give top > level access to an error that might happen at the end. Something that does account for streaming is MList (Either String) Entry, where MList is from "ListT done right". [1] This would have the advantage of being able to use all the standard Monad/MList machinery. It would also have the disadvantage that MList isn't in any standard library, and adds some extra constructor (un)packing. Cheers, Remi [1] http://www.haskell.org/haskellwiki/ListT_done_right From Christian.Maeder at dfki.de Thu Feb 26 05:15:47 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Thu Feb 26 05:04:44 2009 Subject: documentation of Data.List.intersect(By) Message-ID: <49A66BD3.2080708@dfki.de> Hi, for Data.List.intersect I'm missing the documentation that the element order in the result list is taken from the first list. The comment "If the first list contains duplicates, so will the result." indicates that, but also the example could be improved to show this: > [1,2,2,3,4] `intersect` [6,4,4,2] == [2,2,4] Cheers Christian From Christian.Maeder at dfki.de Thu Feb 26 05:36:05 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Thu Feb 26 05:24:59 2009 Subject: Data.List.select Message-ID: <49A67095.1010800@dfki.de> Libraries-List, browsing through the sources of Data.List I discovered "select", which is only used for the implementation of partition and not exported. Can someone explain, why the irrefutable pattern "~" is used? Both components are used in both branches. select :: (a -> Bool) -> a -> ([a], [a]) -> ([a], [a]) select p x ~(ts,fs) | p x = (x:ts,fs) | otherwise = (ts, x:fs) Why is it not just in where-clause? Doesn't "INLINE partition" work then? Cheers Christian From dan.doel at gmail.com Thu Feb 26 09:47:10 2009 From: dan.doel at gmail.com (Dan Doel) Date: Thu Feb 26 09:36:09 2009 Subject: Data.List.select In-Reply-To: <49A67095.1010800@dfki.de> References: <49A67095.1010800@dfki.de> Message-ID: <200902260947.11458.dan.doel@gmail.com> On Thursday 26 February 2009 5:36:05 am Christian Maeder wrote: > Libraries-List, > > browsing through the sources of Data.List I discovered "select", which > is only used for the implementation of partition and not exported. > > Can someone explain, why the irrefutable pattern "~" is used? Both > components are used in both branches. To see the difference, go into ghci, and define select without the irrefutable pattern. Then run the following: foldr (select even) ([],[]) [1..] From Christian.Maeder at dfki.de Thu Feb 26 10:46:12 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Thu Feb 26 10:35:09 2009 Subject: Data.List.select In-Reply-To: <200902260947.11458.dan.doel@gmail.com> References: <49A67095.1010800@dfki.de> <200902260947.11458.dan.doel@gmail.com> Message-ID: <49A6B944.7080608@dfki.de> Dan Doel wrote: > On Thursday 26 February 2009 5:36:05 am Christian Maeder wrote: >> Libraries-List, >> >> browsing through the sources of Data.List I discovered "select", which >> is only used for the implementation of partition and not exported. >> >> Can someone explain, why the irrefutable pattern "~" is used? Both >> components are used in both branches. > > To see the difference, go into ghci, and define select without the irrefutable > pattern. Then run the following: > > foldr (select even) ([],[]) [1..] Cool! Thanks, Christian From midfield at gmail.com Thu Feb 26 10:56:54 2009 From: midfield at gmail.com (Ben) Date: Thu Feb 26 10:45:51 2009 Subject: cabal install profiling and documentation Message-ID: <9157df230902260756s7197829ajca7c1e6429e878f5@mail.gmail.com> hi -- i asked this on haskell cafe but got no response. here it is again: i've gone and cabal installed a lot of packages, but now i want to go back and install their profiling libraries and documentation. is there an easy way of doing this, short of reinstalling all of them (in the proper dependency order)? thanks, ben From lemming at henning-thielemann.de Thu Feb 26 17:53:35 2009 From: lemming at henning-thielemann.de (Henning Thielemann) Date: Thu Feb 26 17:42:32 2009 Subject: Foldable and Traversable vs. FunctorM Message-ID: After writing some FunctorM instance for GHC-6.4, then converting them to Traversable instances for GHC-6.8, I found that most of the time, if not always, I actually needed a FunctorM instance (i.e. fmapM method), not a Traversable instance. Defining a Traversable instance requires me to also define a Foldable instance, i.e. a foldMap function. I had trees, lists and singletons extended by additional information, e.g. data PositionList a = PositionList [(Position, a)] data AsyncException e a = AsyncException (Maybe e) a However 'foldMap' cannot utilize the additional information, namely Position and (Maybe e), respectively. In the first case this is sad, in the second case I consider this bad, since it is too easy to throw away and thus ignore the exception (Maybe e). Some kind of fold is always useful on these structures, but 'foldMap' is too limited. For Traversable there is always both sequence and mapM (and their Applicative counterparts), although you need to define only one of them. However 'sequence' works on interim data structures that are not very natural. Imagine a data structure with a 'string' parameter that is intended for instantiation by String or ByteString. Now 'sequence' requires instantiations like (IO String) or (State s ByteString) which do not share common properties of String and ByteString, e.g. having a 'length' function. Although possible, those interim types look inappropriate to me. Even more, consider a data structure like StorableVector which requires certain properties of its elements. StorableVector is intended for Int, Double and other Storable types, but you cannot store (IO Double) values. This example is not practical, because the Storable restriction disallows to define both Traversable and FunctorM instances. However you see, that one cannot even imagine 'sequence' on StorableVector, whereas 'mapM' is possible, yet very inefficient (needs copying the entire vector for each added element). My conclusion is, that FunctorM was better than its reputation, and that Traversable as it is, is not what I need most of the time. What are the experiences of other Traversable users? From ganesh.sittampalam at credit-suisse.com Fri Feb 27 04:43:38 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Fri Feb 27 04:33:03 2009 Subject: NFData Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B01ABA89E@ELON17P32001A.csfb.cs-group.com> Hi, NFData seems to be the "standard" deepseq-style class these days, but it lives in Control.Parallel.Strategies in the parallel package whereas it obviously has uses beyond this. Is there a case for moving it into its own package and module? I'd also like to add something like strictly :: (Monad m, NFData a) => m a -> m a strictly ma = do a <- ma rnf a `seq` return a The most important use for this would be to provide a standard way to do strict IO: strictly $ readFile "..." strictly $ hGetContents ... Thoughts? Cheers, Ganesh =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html =============================================================================== From duncan.coutts at worc.ox.ac.uk Fri Feb 27 05:53:20 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Fri Feb 27 05:42:29 2009 Subject: cabal install profiling and documentation In-Reply-To: <9157df230902260756s7197829ajca7c1e6429e878f5@mail.gmail.com> References: <9157df230902260756s7197829ajca7c1e6429e878f5@mail.gmail.com> Message-ID: <1235732000.10457.659.camel@localhost> On Thu, 2009-02-26 at 07:56 -0800, Ben wrote: > hi -- > > i asked this on haskell cafe but got no response. here it is again: > > i've gone and cabal installed a lot of packages, but now i want to go > back and install their profiling libraries and documentation. is > there an easy way of doing this, short of reinstalling all of them (in > the proper dependency order)? Not at the moment, sorry. Unlike native package managers, cabal-install only uses the information from ghc-pkg. Unfortunately profiling libs are not tracked separately from ordinary ones so we simply do not know if they are installed or not. The plan for ghc-6.12 is to track each "way" that a package is built separately. It's not impossible that we could hack things for the special case of profiling libs within the current scheme. (It would involve probing lots of disk files.) It is however a task amongst many. It would need a volunteer or it's not likely to get done soon. http://hackage.haskell.org/trac/hackage/ticket/282 Duncan From ml at isaac.cedarswampstudios.org Fri Feb 27 08:37:31 2009 From: ml at isaac.cedarswampstudios.org (Isaac Dupree) Date: Fri Feb 27 08:26:40 2009 Subject: Foldable and Traversable vs. FunctorM In-Reply-To: References: Message-ID: <200902270837.36426.ml@isaac.cedarswampstudios.org> Henning Thielemann wrote: > ? However 'foldMap' cannot utilize the additional information, namely > Position and (Maybe e), respectively. In the first case this is sad, in > the second case I consider this bad, since it is too easy to throw away > and thus ignore the exception (Maybe e). Some kind of fold is always > useful on these structures, but 'foldMap' is too limited. I'm not sure what you mean, but if you define Traversable in a sensible way, then you can define Foldable with Data.Traversable.foldMapDefault and get a sensible definition. > Even more, consider a data structure like StorableVector which requires > certain properties of its elements. StorableVector is intended for Int, > Double and other Storable types, but you cannot store (IO Double) values. > This example is not practical, because the Storable restriction disallows > to define both Traversable and FunctorM instances. However you see, that > one cannot even imagine 'sequence' on StorableVector, whereas 'mapM' is > possible, yet very inefficient (needs copying the entire vector for each > added element). Sure you can define 'sequence'... because you cannot construct a StorableVector with elements that are not Storable. (You could save this information inside the StorableVector using existentials/GADTs.) Except... > My conclusion is, that FunctorM was better than its reputation, and that > Traversable as it is, is not what I need most of the time. What are the > experiences of other Traversable users? that the fundamental complaint is that Traversables must be in Functor which requires to accomodate all types of kind *. On the other hand, as you note, either FunctorM or Traversable instance requires that property as well. (cf. Data.Traversable.fmapDefault, which uses "traverse" to define it) Basically, if you can make a type Traversable by defining (Traversable t) : traverse :: Applicative f => (a -> f b) -> t a -> f (t b) then this is almost always just as easy as defining (FunctorM f) : fmapM :: Monad m => (a -> m b) -> f a -> m (f b) for that type. (note that the type signatures are almost the same, and have almost equal power. Assuming you didn't want to do weird monadic more-complicated-than- Applicative things, that is.) In other words, my experience is that Traversable is more beautiful and no less usable. -Isaac From dons at galois.com Fri Feb 27 12:18:25 2009 From: dons at galois.com (Don Stewart) Date: Fri Feb 27 12:08:04 2009 Subject: NFData In-Reply-To: <16442B752A06A74AB4D9F9A5FF076E4B01ABA89E@ELON17P32001A.csfb.cs-group.com> References: <16442B752A06A74AB4D9F9A5FF076E4B01ABA89E@ELON17P32001A.csfb.cs-group.com> Message-ID: <20090227171825.GB18853@whirlpool.galois.com> Should NFData class and instances be in base? Data.Strict ? ganesh.sittampalam: > Hi, > > NFData seems to be the "standard" deepseq-style class these days, but it > lives in Control.Parallel.Strategies in the parallel package whereas it > obviously has uses beyond this. Is there a case for moving it into its > own package and module? > > I'd also like to add something like > > strictly :: (Monad m, NFData a) => m a -> m a > strictly ma = do a <- ma > rnf a `seq` return a > > The most important use for this would be to provide a standard way to do > strict IO: > > strictly $ readFile "..." > strictly $ hGetContents ... > > Thoughts? > > Cheers, > > Ganesh > > =============================================================================== > Please access the attached hyperlink for an important electronic communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > =============================================================================== > > _______________________________________________ > Libraries mailing list > Libraries@haskell.org > http://www.haskell.org/mailman/listinfo/libraries From ganesh.sittampalam at credit-suisse.com Fri Feb 27 12:20:39 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Fri Feb 27 12:10:03 2009 Subject: NFData In-Reply-To: <20090227171825.GB18853@whirlpool.galois.com> References: <16442B752A06A74AB4D9F9A5FF076E4B01ABA89E@ELON17P32001A.csfb.cs-group.com> <20090227171825.GB18853@whirlpool.galois.com> Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B01ABA8A3@ELON17P32001A.csfb.cs-group.com> I think that would make most sense given how important deep strictness can be in certain situations. A separate, very small package would also be reasonable. Don Stewart wrote: > Should NFData class and instances be in base? > > Data.Strict ? > > ganesh.sittampalam: >> Hi, >> >> NFData seems to be the "standard" deepseq-style class these days, but >> it lives in Control.Parallel.Strategies in the parallel package >> whereas it obviously has uses beyond this. Is there a case for moving >> it into its own package and module? >> >> I'd also like to add something like >> >> strictly :: (Monad m, NFData a) => m a -> m a strictly ma = do a <- >> ma rnf a `seq` return a >> >> The most important use for this would be to provide a standard way >> to do strict IO: >> >> strictly $ readFile "..." >> strictly $ hGetContents ... >> >> Thoughts? >> >> Cheers, >> >> Ganesh >> >> ====================================================================== >> ========= Please access the attached hyperlink for an important >> electronic communications disclaimer: >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html >> >> ====================================================================== >> ========= >> >> _______________________________________________ >> Libraries mailing list >> Libraries@haskell.org >> http://www.haskell.org/mailman/listinfo/libraries =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html =============================================================================== From bos at serpentine.com Fri Feb 27 12:56:20 2009 From: bos at serpentine.com (Bryan O'Sullivan) Date: Fri Feb 27 12:45:13 2009 Subject: NFData In-Reply-To: <20090227171825.GB18853@whirlpool.galois.com> References: <16442B752A06A74AB4D9F9A5FF076E4B01ABA89E@ELON17P32001A.csfb.cs-group.com> <20090227171825.GB18853@whirlpool.galois.com> Message-ID: On Fri, Feb 27, 2009 at 9:18 AM, Don Stewart wrote: > Should NFData class and instances be in base? > > Data.Strict ? > That would be great. Right now, there's a strong disincentive for library authors to use NFData, as it drags in a spurious dependency on another package. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.haskell.org/pipermail/libraries/attachments/20090227/a4d73fcb/attachment.htm From dominic.steinitz at blueyonder.co.uk Sat Feb 28 03:24:56 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sat Feb 28 03:25:49 2009 Subject: lhs2TeX Build Failure Message-ID: <49A8F4D8.2030801@blueyonder.co.uk> The latest version (1.14) on hackage fails under ghc --version The Glorious Glasgow Haskell Compilation System, version 6.10.1 Curiously, I think I built it at work under windows yesterday but maybe I was using ghc 6.8. I can't check until Monday. FileNameUtils.lhs:81:35: Ambiguous type variable `b' in the constraint: `GHC.Exception.Exception b' arising from a use of `catch' at FileNameUtils.lhs:(81,35)-(89,60) Probable fix: add a type signature that fixes these type variable(s) FileNameUtils.lhs:98:44: Ambiguous type variable `e2' in the constraint: `GHC.Exception.Exception e2' arising from a use of `try' at FileNameUtils.lhs:98:44-57 Probable fix: add a type signature that fixes these type variable(s) FileNameUtils.lhs:118:35: Ambiguous type variable `e1' in the constraint: `GHC.Exception.Exception e1' arising from a use of `catch' at FileNameUtils.lhs:(118,35)-(119,104) Probable fix: add a type signature that fixes these type variable(s) FileNameUtils.lhs:122:35: Ambiguous type variable `e' in the constraint: `GHC.Exception.Exception e' arising from a use of `catch' at FileNameUtils.lhs:122:35-54 Probable fix: add a type signature that fixes these type variable(s) From bulat.ziganshin at gmail.com Sat Feb 28 03:42:30 2009 From: bulat.ziganshin at gmail.com (Bulat Ziganshin) Date: Sat Feb 28 03:31:33 2009 Subject: lhs2TeX Build Failure In-Reply-To: <49A8F4D8.2030801@blueyonder.co.uk> References: <49A8F4D8.2030801@blueyonder.co.uk> Message-ID: <1716231193.20090228114230@gmail.com> Hello Dominic, Saturday, February 28, 2009, 11:24:56 AM, you wrote: > Curiously, I think I built it at work under windows yesterday but maybe > I was using ghc 6.8. I can't check until Monday. definitely, these are 6.8 -> 6.10 incompatibility messages. it needs to add base<=3 to libraries clause of .cabal file > FileNameUtils.lhs:81:35: > Ambiguous type variable `b' in the constraint: > `GHC.Exception.Exception b' > arising from a use of `catch' at FileNameUtils.lhs:(81,35)-(89,60) > Probable fix: add a type signature that fixes these type variable(s) > FileNameUtils.lhs:98:44: > Ambiguous type variable `e2' in the constraint: > `GHC.Exception.Exception e2' > arising from a use of `try' at FileNameUtils.lhs:98:44-57 > Probable fix: add a type signature that fixes these type variable(s) > FileNameUtils.lhs:118:35: > Ambiguous type variable `e1' in the constraint: > `GHC.Exception.Exception e1' > arising from a use of `catch' > at FileNameUtils.lhs:(118,35)-(119,104) > Probable fix: add a type signature that fixes these type variable(s) > FileNameUtils.lhs:122:35: > Ambiguous type variable `e' in the constraint: > `GHC.Exception.Exception e' > arising from a use of `catch' at FileNameUtils.lhs:122:35-54 > Probable fix: add a type signature that fixes these type variable(s) > _______________________________________________ > Libraries mailing list > Libraries@haskell.org > http://www.haskell.org/mailman/listinfo/libraries -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com From duncan.coutts at worc.ox.ac.uk Sat Feb 28 06:53:47 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Sat Feb 28 06:42:54 2009 Subject: lhs2TeX Build Failure In-Reply-To: <1716231193.20090228114230@gmail.com> References: <49A8F4D8.2030801@blueyonder.co.uk> <1716231193.20090228114230@gmail.com> Message-ID: <1235822027.7915.17.camel@localhost> On Sat, 2009-02-28 at 11:42 +0300, Bulat Ziganshin wrote: > Hello Dominic, > > Saturday, February 28, 2009, 11:24:56 AM, you wrote: > > > Curiously, I think I built it at work under windows yesterday but maybe > > I was using ghc 6.8. I can't check until Monday. > > definitely, these are 6.8 -> 6.10 incompatibility messages. it needs > to add base<=3 to libraries clause of .cabal file You mean base < 4. It's also possible for users to build it using cabal-install which will automatically pick base 3 if the package does not declare it needs base 4. Or if you're building manually step by step then use configure --constraint='base < 4'. Duncan From dominic.steinitz at blueyonder.co.uk Sat Feb 28 08:46:57 2009 From: dominic.steinitz at blueyonder.co.uk (Dominic Steinitz) Date: Sat Feb 28 08:47:52 2009 Subject: lhs2TeX Build Failure In-Reply-To: <1235822027.7915.17.camel@localhost> References: <49A8F4D8.2030801@blueyonder.co.uk> <1716231193.20090228114230@gmail.com> <1235822027.7915.17.camel@localhost> Message-ID: <49A94051.1040405@blueyonder.co.uk> Duncan Coutts wrote: > On Sat, 2009-02-28 at 11:42 +0300, Bulat Ziganshin wrote: >> Hello Dominic, >> >> Saturday, February 28, 2009, 11:24:56 AM, you wrote: >> >>> Curiously, I think I built it at work under windows yesterday but maybe >>> I was using ghc 6.8. I can't check until Monday. >> definitely, these are 6.8 -> 6.10 incompatibility messages. it needs >> to add base<=3 to libraries clause of .cabal file > > You mean base < 4. > > It's also possible for users to build it using cabal-install which will > automatically pick base 3 if the package does not declare it needs base > 4. Or if you're building manually step by step then use configure > --constraint='base < 4'. > > Duncan > > > That's done the trick. Thanks, Dominic. From igloo at earth.li Sat Feb 28 09:04:03 2009 From: igloo at earth.li (Ian Lynagh) Date: Sat Feb 28 08:52:52 2009 Subject: darcs patch: Expose socket internals in a new module. (and 3 more) Message-ID: Sun Feb 1 21:55:08 GMT 2009 Johan Tibell * Expose socket internals in a new module. Expose some of the internals in a new module, Network.Socket.Internal, to make it easier to extend the functionality of the library. Sat Feb 28 08:59:05 GMT 2009 Johan Tibell * Expose protocol family data type and helpers in internals module. Ignore-this: 23fdb3b21bac47c98dcb2a0212f9b0d7 Sat Feb 28 09:05:39 GMT 2009 Johan Tibell * Whitespace changes only (remove tabs). Ignore-this: 3a695c355c6122ea13360046512b58f9 Sat Feb 28 10:00:24 GMT 2009 Johan Tibell * Added more documentation. Ignore-this: daae026cbffc50163316e5d7815be87e -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/x-darcs-patch Size: 39210 bytes Desc: A darcs patch for your repository! Url : http://www.haskell.org/pipermail/libraries/attachments/20090228/41af3ba0/attachment-0001.bin From ben.franksen at online.de Sat Feb 28 09:14:01 2009 From: ben.franksen at online.de (Ben Franksen) Date: Sat Feb 28 09:03:00 2009 Subject: Deadlock when using Control.Concurrent.Chan Message-ID: This code demonstrates the problem: import Control.Concurrent import Control.Concurrent.Chan import Control.Monad test = do c <- newChan forkIO $ forever $ do i <- readChan c print i writeChan c 1 threadDelay 1000000 isEmptyChan c >>= print Test session: ben@sarun[1]: .../hca/current > ghci Bug5.hs GHCi, version 6.10.1: http://www.haskell.org/ghc/ :? for help Loading package ghc-prim ... linking ... done. Loading package integer ... linking ... done. Loading package base ... linking ... done. [1 of 1] Compiling Main ( Bug5.hs, interpreted ) Ok, modules loaded: Main. *Main> test 1 The program hangs here. This is a deadlock caused by the calls to isEmptyChan and readChan both waiting on the same MVar, as a quick look at the sources reveals. I don't have an immediate fix but is shouldn't be too hard to find one. I can try to, if it is consensus that this is a bug. Incidentally, if I replace Chan with TChan (and wrap all xxxTChan calls in an atomically), then this code terminates just fine. Cheers Ben From daveroundy at gmail.com Sat Feb 28 11:59:12 2009 From: daveroundy at gmail.com (David Roundy) Date: Sat Feb 28 11:48:00 2009 Subject: bug in System.Info in ghc 6.8.2 and ghc 6.10.1 Message-ID: <117f2cc80902280859t7fe2be8fm7950290fd6e2e7c2@mail.gmail.com> It seems that System.Info.compilerVersion doesn't know the minor release number of ghc. This looks like a bug to me. -- David Roundy From ben.franksen at online.de Sat Feb 28 14:15:42 2009 From: ben.franksen at online.de (Ben Franksen) Date: Sat Feb 28 14:04:45 2009 Subject: Deadlock when using Control.Concurrent.Chan References: Message-ID: Ben Franksen wrote: > import Control.Concurrent > import Control.Concurrent.Chan > import Control.Monad > > test = do > c <- newChan > forkIO $ forever $ do > i <- readChan c > print i > writeChan c 1 > threadDelay 1000000 > isEmptyChan c >>= print > > Test session: > > ben@sarun[1]: .../hca/current > ghci Bug5.hs > GHCi, version 6.10.1: http://www.haskell.org/ghc/ :? for help > Loading package ghc-prim ... linking ... done. > Loading package integer ... linking ... done. > Loading package base ... linking ... done. > [1 of 1] Compiling Main ( Bug5.hs, interpreted ) > Ok, modules loaded: Main. > *Main> test > 1 Correction: This is not a deadlock, technically speaking. I still think isEmptyChan should not block indefinitely only because some reader is blocked on an empty chan. Cheers Ben From duncan.coutts at worc.ox.ac.uk Sat Feb 28 14:24:05 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Sat Feb 28 14:13:02 2009 Subject: bug in System.Info in ghc 6.8.2 and ghc 6.10.1 In-Reply-To: <117f2cc80902280859t7fe2be8fm7950290fd6e2e7c2@mail.gmail.com> References: <117f2cc80902280859t7fe2be8fm7950290fd6e2e7c2@mail.gmail.com> Message-ID: <1235849045.7915.92.camel@localhost> On Sat, 2009-02-28 at 08:59 -0800, David Roundy wrote: > It seems that System.Info.compilerVersion doesn't know the minor > release number of ghc. This looks like a bug to me. Last time I asked I was told it was by design. Similarly the __GLASGOW_HASKELL__ cpp macro does not tell you the patch level. I believe the rationale is that minor ghc versions are compatible (though because they're not really 100% in all cases they did add a patch level cpp macro). So you can try opening a ghc ticket, see if you can persuade them to reconsider. Duncan From timtoorop at quicknet.nl Sat Feb 28 18:34:08 2009 From: timtoorop at quicknet.nl (timtoorop@quicknet.nl) Date: Sat Feb 28 18:23:00 2009 Subject: randomIO causing stack overflow ? Message-ID: Hi, I was just messing with randomIO for no particular reason, and it suddenly caused a stack overflow which would only go away after restarting ghci. Here's what I see in ghci 6.8.2 http://moonpatio.com/fastcgi/hpaste.fcgi/view?id=1642 mmorrow posted the same using ghci 6.10.1 which gave the same stack overflow Anyone got any idea what causes this funny behavior? - Tim