From joshua at reverberate.org Sun Aug 2 02:50:40 2009 From: joshua at reverberate.org (Joshua Haberman) Date: Sun Aug 2 02:36:01 2009 Subject: equivalent of EXPLAIN PLAN with GHC? Message-ID: Hello, I'm quite new to Haskell, but experienced in other languages (C, Python, Ruby, SQL, etc). I am interested in Haskell because I've heard that the language is capable of lots of optimizations based on laziness, and I want to learn more about that. I dug in with Project Euler problem #1, and wrote: main = print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || x `mod` 5 == 0])) So far so good, but I want to have some way of observing what optimizations GHC has performed. Most notably, in this example I want to know if the list was ever actually constructed in memory. The "sum" function only needs the elements one at a time, in order, so if Haskell and GHC are everything I've heard about them, I would fully expect the list construction to be optimized out. :) Unfortunately I was not able to see any way of examining ghc's output to determine whether it had performed this optimization. The C it produced with '-fvia-C -C' was totally unreadable -- it looked like something from the IOCCC. :( I couldn't find any way to match up any of the code it had generated with code I had written. My attempts to objdump the binaries was similarly unproductive. Is there any kind of intermediate form that a person can examine to see how their code is being optimized? Anything like EXPLAIN PLAN in SQL? It would make it much easier to understand the kinds of optimizations Haskell can perform. I'm not looking so much for profiling -- obvious this program is trivial and takes no time. I just want to better understand what kind of optimizations are possible given Haskell's language model. Thanks! Josh From thomas.dubuisson at gmail.com Sun Aug 2 03:12:03 2009 From: thomas.dubuisson at gmail.com (Thomas DuBuisson) Date: Sun Aug 2 02:52:59 2009 Subject: equivalent of EXPLAIN PLAN with GHC? In-Reply-To: References: Message-ID: <4c44d90b0908020012p71a6e02cqd6766c39fb207988@mail.gmail.com> Josh, In general you'll find the haskell-cafe (haskell-cafe@haskell.org) to be a more lively place for this type of discussion, but since we're here I might as well mention that memory use of a Haskell function is one of the hardest things to gain an understanding about. > main = print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || x `mod` 5 == 0])) > I want > to know if the list was ever actually constructed in memory. For such a simple program I suggest you test with lists of various lengths: ---------------- main with n == 999 ------------------- [tommd@Mavlo Test]$ ghc --make -O2 opt.hs [1 of 1] Compiling Main ( opt.hs, opt.o ) Linking opt ... [tommd@Mavlo Test]$ ./opt +RTS -sstderr ./opt +RTS -sstderr "233168" 94,604 bytes allocated in the heap 700 bytes copied during GC 17,144 bytes maximum residency (1 sample(s)) 23,816 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 0 collections, 0 parallel, 0.00s, 0.00s elapsed Generation 1: 1 collections, 0 parallel, 0.00s, 0.00s elapsed INIT time 0.00s ( 0.00s elapsed) MUT time 0.00s ( 0.00s elapsed) GC time 0.00s ( 0.00s elapsed) EXIT time 0.00s ( 0.00s elapsed) Total time 0.00s ( 0.00s elapsed) %GC time 0.0% (14.0% elapsed) Alloc rate 47,325,662 bytes per MUT second Productivity 0.0% of total user, 0.0% of total elapsed ---------------------------------------------------------------- ---------------- main with n == 10000000 ------------------- [tommd@Mavlo Test]$ ./opt +RTS -sstderr ./opt +RTS -sstderr "23333341666668" 906,451,272 bytes allocated in the heap 128,992 bytes copied during GC 18,104 bytes maximum residency (1 sample(s)) 18,760 bytes maximum slop 1 MB total memory in use (0 MB lost due to fragmentation) Generation 0: 1742 collections, 0 parallel, 0.01s, 0.01s elapsed Generation 1: 1 collections, 0 parallel, 0.00s, 0.00s elapsed INIT time 0.00s ( 0.00s elapsed) MUT time 0.99s ( 1.02s elapsed) GC time 0.01s ( 0.01s elapsed) EXIT time 0.00s ( 0.00s elapsed) Total time 1.00s ( 1.03s elapsed) %GC time 0.6% (1.0% elapsed) Alloc rate 912,980,911 bytes per MUT second Productivity 99.4% of total user, 96.7% of total elapsed ----------------------------------------------------------- So obviously there was a lot more data moving around but no significant difference in maximum memory use - a good sign! >?The C it produced > with '-fvia-C -C' was totally unreadable Deprecating the via-C compilation path has been discussed - hopefully no one ever suggested it as a useful path for human inspection. Thomas From joshua at reverberate.org Sun Aug 2 03:45:57 2009 From: joshua at reverberate.org (Joshua Haberman) Date: Sun Aug 2 03:27:05 2009 Subject: equivalent of EXPLAIN PLAN with GHC? References: <4c44d90b0908020012p71a6e02cqd6766c39fb207988@mail.gmail.com> Message-ID: Hi Thomas, thanks for the reply! Thomas DuBuisson gmail.com> writes: > Josh, > > In general you'll find the haskell-cafe (haskell-cafe haskell.org) to > be a more lively place for this type of discussion Good to know, I just wasn't sure if it was appropriate for GHC-specific questions. > but since we're > here I might as well mention that memory use of a Haskell function is > one of the hardest things to gain an understanding about. Hmm, that's unfortunate. :( > > main = print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || > > x `mod` 5 == 0])) > > I want > > to know if the list was ever actually constructed in memory. > > For such a simple program I suggest you test with lists of various lengths: Cool, thanks for the example. > 94,604 bytes allocated in the heap Is there any way I could find out what these 94kb of RAM were allocated for? This seems high -- my entire program's working set is <6kb. That's if you construct all my lists in full, including both the [3..999] list (which has 997 elements) and the list comprehension list (which has 466). This assumes 32-bit integers. Allow for some memory to format the string, and you still aren't within an order of magnitude of 94kb. I guess I'm holding Haskell to the standard of a compiled language rather than interpreted one, and assuming it doesn't need to allocate heap space for the code. Any idea how I could dig into this? > >?The C it produced > > with '-fvia-C -C' was totally unreadable > > Deprecating the via-C compilation path has been discussed - hopefully > no one ever suggested it as a useful path for human inspection. I do still wonder if there isn't an intermediate form that *is* suitable for human inspection. Thanks, Josh From dons at galois.com Sun Aug 2 03:50:13 2009 From: dons at galois.com (Don Stewart) Date: Sun Aug 2 03:33:12 2009 Subject: equivalent of EXPLAIN PLAN with GHC? In-Reply-To: References: Message-ID: <20090802075013.GA9740@whirlpool.galois.com> joshua: > Hello, I'm quite new to Haskell, but experienced in other languages (C, > Python, Ruby, SQL, etc). I am interested in Haskell because I've heard > that the language is capable of lots of optimizations based on laziness, > and I want to learn more about that. > > I dug in with Project Euler problem #1, and wrote: > > main = print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || x `mod` 5 == 0])) > > So far so good, but I want to have some way of observing what > optimizations GHC has performed. Most notably, in this example I want > to know if the list was ever actually constructed in memory. The "sum" > function only needs the elements one at a time, in order, so if Haskell > and GHC are everything I've heard about them, I would fully expect the > list construction to be optimized out. :) > > Unfortunately I was not able to see any way of examining ghc's output to > determine whether it had performed this optimization. The C it produced > with '-fvia-C -C' was totally unreadable -- it looked like something > from the IOCCC. :( I couldn't find any way to match up any of the code > it had generated with code I had written. My attempts to objdump the > binaries was similarly unproductive. > > Is there any kind of intermediate form that a person can examine to see > how their code is being optimized? Anything like EXPLAIN PLAN in SQL? > It would make it much easier to understand the kinds of optimizations > Haskell can perform. > > I'm not looking so much for profiling -- obvious this program is trivial > and takes no time. I just want to better understand what kind of > optimizations are possible given Haskell's language model. > So the optimization you're looking for here is fusion of some kind. GHC ships with build/foldr fusion, and there are libraries for an alternative system, stream fusion. GHC uses an intermediate representation called 'Core', which is a mini-Haskell, essentially, that is optimized repeatedly via type-preserving transformations. You can inspect this with a number of tools, including "ghc-core" (available on Hackage). Now, you're example uses a list comprehension (which is translated into an enumFromTo call, and a call to filter. It also uses a call to sum, which is a non-fusing left-fold under build/foldr fusion, but fuses under stream fusion. I'll desugar your code explicitly, and translate the calls from build/foldr to stream-fusible functions: $ cabal install uvector import Data.Array.Vector main = print . sumU . filterU (\x -> x `mod` 3 == 0 || x `mod` 5 == 0) $ enumFromToU 3 (999 :: Int) Running through ghc-core we see: 146 PreInlineUnconditionally 320 PostInlineUnconditionally 84 UnfoldingDone 18 RuleFired 7 +# 1 -# 1 <=# 2 ==#->case 1 ># 3 SC:$wfold0 1 fromIntegral/Int->Int 2 streamU/unstreamU 12 LetFloatFromLet 1 EtaReduction 210 BetaReduction 8 CaseOfCase 94 KnownBranch 4 CaseMerge 5 CaseElim 6 CaseIdentity 1 FillInCaseDefault 18 SimplifierDone Showing what transformations happened. Notably, 2 occurences of the "streamU/unstreamU" transformation, to remove intermediate structures. The final code looks like: $s$wfold :: Int# -> Int# $s$wfold = \ (sc_s19l :: Int#) -> case modInt# (-9223372036854775807) 3 of wild21_a14L { __DEFAULT -> case modInt# (-9223372036854775807) 5 of wild211_X159 { __DEFAULT -> $wfold sc_s19l (-9223372036854775806); 0 -> $wfold (+# sc_s19l (-9223372036854775807)) (-9223372036854775806) }; 0 -> $wfold (+# sc_s19l (-9223372036854775807)) (-9223372036854775806) } $wfold :: Int# -> Int# -> Int# $wfold = \ (ww_s189 :: Int#) (ww1_s18d :: Int#) -> case ># ww1_s18d 999 of wild_a15N { False -> case ww1_s18d of wild2_a14K { __DEFAULT -> case modInt# wild2_a14K 3 of wild21_a14L { __DEFAULT -> case modInt# wild2_a14K 5 of wild211_X159 { __DEFAULT -> $wfold ww_s189 (+# wild2_a14K 1); 0 -> $wfold (+# ww_s189 wild2_a14K) (+# wild2_a14K 1) }; 0 -> $wfold (+# ww_s189 wild2_a14K) (+# wild2_a14K 1) }; (-9223372036854775808) -> case modInt# (-9223372036854775808) 3 of wild21_a14N { __DEFAULT -> case lvl_r19G of wild211_a14x { False -> $s$wfold ww_s189; True -> $s$wfold (+# ww_s189 (-9223372036854775808)) }; 0 -> $s$wfold (+# ww_s189 (-9223372036854775808)) } }; True -> ww_s189 } Which might take a while to understand, but the key thing is the types at the top level are Int# -- they don't allocate [Int], but instead use unboxed, machine-level int values. Optimization succesful. There are papers and manuals describing "GHC Core" - the intermediate form - on haskell.org. For fusion, google for "deforestation" or "stream fusion". -- Don From bulat.ziganshin at gmail.com Sun Aug 2 04:07:52 2009 From: bulat.ziganshin at gmail.com (Bulat Ziganshin) Date: Sun Aug 2 03:49:28 2009 Subject: equivalent of EXPLAIN PLAN with GHC? In-Reply-To: References: <4c44d90b0908020012p71a6e02cqd6766c39fb207988@mail.gmail.com> Message-ID: <505030411.20090802120752@gmail.com> Hello Joshua, Sunday, August 2, 2009, 11:45:57 AM, you wrote: >> 94,604 bytes allocated in the heap > Is there any way I could find out what these 94kb of RAM were > allocated for? This seems high -- my entire program's working set > is <6kb. as Don said, compiled code works on Int# (which is the same as C int) but probably not via registers. instead, each intermediate value allocated in heap, so there are total 90 bytes per iteration = about 22 integers -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com From dons at galois.com Sun Aug 2 04:10:30 2009 From: dons at galois.com (Don Stewart) Date: Sun Aug 2 03:53:30 2009 Subject: equivalent of EXPLAIN PLAN with GHC? In-Reply-To: <505030411.20090802120752@gmail.com> References: <4c44d90b0908020012p71a6e02cqd6766c39fb207988@mail.gmail.com> <505030411.20090802120752@gmail.com> Message-ID: <20090802081030.GB9740@whirlpool.galois.com> bulat.ziganshin: > Hello Joshua, > > Sunday, August 2, 2009, 11:45:57 AM, you wrote: > > >> 94,604 bytes allocated in the heap > > > Is there any way I could find out what these 94kb of RAM were > > allocated for? This seems high -- my entire program's working set > > is <6kb. > > as Don said, compiled code works on Int# (which is the same as C int) > but probably not via registers. instead, each intermediate value > allocated in heap, so there are total 90 bytes per iteration = about > 22 integers In my example, there's no heap allocation. In the version with build/foldr though, there will be heap nodes passed to sum. The fused one results in, 32,160 bytes allocated in the heap But there's no heap allocation in the filtering code, only once we have to construct the string to print. -- Don From dons at galois.com Sun Aug 2 04:35:10 2009 From: dons at galois.com (Don Stewart) Date: Sun Aug 2 04:18:08 2009 Subject: equivalent of EXPLAIN PLAN with GHC? In-Reply-To: <20090802075013.GA9740@whirlpool.galois.com> References: <20090802075013.GA9740@whirlpool.galois.com> Message-ID: <20090802083510.GA11369@whirlpool.galois.com> dons: > Showing what transformations happened. Notably, 2 occurences of the "streamU/unstreamU" > transformation, to remove intermediate structures. > > The final code looks like: > > $s$wfold :: Int# -> Int# > $s$wfold = > \ (sc_s19l :: Int#) -> > case modInt# (-9223372036854775807) 3 of wild21_a14L { > __DEFAULT -> > case modInt# (-9223372036854775807) 5 of wild211_X159 { > __DEFAULT -> $wfold sc_s19l (-9223372036854775806); > 0 -> > $wfold > (+# sc_s19l (-9223372036854775807)) (-9223372036854775806) > }; > 0 -> > $wfold > (+# sc_s19l (-9223372036854775807)) (-9223372036854775806) > } > $wfold :: Int# -> Int# -> Int# Oh, this is also a good illustration of why you should use `rem` instead of `mod` , if you're not expecting negative numbers: The core with filterU (\x -> x `rem` 3 == 0 || x `rem` 5 == 0) avoids all the checks for -9223372036854775806. $s$wfold :: Int# -> Int# $s$wfold = \ (sc_s18m :: Int#) -> $wfold sc_s18m (-9223372036854775806) $wfold :: Int# -> Int# -> Int# $wfold = \ (ww_s17v :: Int#) (ww1_s17z :: Int#) -> case ># ww1_s17z 999 of wild_a15d { False -> case ww1_s17z of wild2_a14K { __DEFAULT -> case remInt# wild2_a14K 3 of wild1_B1 { __DEFAULT -> case remInt# wild2_a14K 5 of wild21_Xp { __DEFAULT -> $wfold ww_s17v (+# wild2_a14K 1); 0 -> $wfold (+# ww_s17v wild2_a14K) (+# wild2_a14K 1) }; 0 -> $wfold (+# ww_s17v wild2_a14K) (+# wild2_a14K 1) }; (-9223372036854775808) -> $s$wfold ww_s17v }; True -> ww_s17v } Yes, that's better code. But remember, Prelude Test.QuickCheck> quickCheck (\x -> x `mod` 3 == x `rem` 3) Falsifiable, after 2 tests: -1 -- Don From mechvel at botik.ru Sun Aug 2 05:24:01 2009 From: mechvel at botik.ru (Serge D. Mechveliani) Date: Sun Aug 2 05:11:21 2009 Subject: Re to `optimization for list' Message-ID: <20090802092401.GA7771@scico.botik.ru> Joshua Haberman writes on 2 Aug 2009 > Hello, I'm quite new to Haskell, but experienced in other languages (C, > Python, Ruby, SQL, etc). I am interested in Haskell because I've heard > that the language is capable of lots of optimizations based on laziness, > and I want to learn more about that. > > I dug in with Project Euler problem #1, and wrote: > > main = > print (show (sum [x | x <- [3..999], x `mod` 3 == 0 || x `mod` 5 == 0])) > > So far so good, but I want to have some way of observing what > optimizations GHC has performed. Most notably, in this example I want > to know if the list was ever actually constructed in memory. The "sum" > function only needs the elements one at a time, in order, so if Haskell > and GHC are everything I've heard about them, I would fully expect the > list construction to be optimized out. :) > [..] Your example is rather complex to start with, because `sum' has rather a complex definition in the Haskell library. Consider first a more pure example: main = putStr (shows (or' ys) "\n") where n = 10^8 ys = replicate n False or' :: [Bool] -> Bool or' [] = False or' (x: xs) = x || (or' xs) (the functions `||' (disjunction) and `replicate' are of the library). Compile it with ghc --make -O (for any occasion), and then, it will be evident that it takes a constant in n space (and stack) when running (run it like this: `./T +RTS -M1m -RTS' ). I think, this is not due to optimization but due to the very definition of evaluation in Haskell -- Head Normal Form. Compute this by hand: or' (replicate 9 False) = or' (False: (replicate 8 False)) = False || (or' (replicate 8 False)) = or' (replicate 8 False) = -- by definition of || ... or' [] = False. This is just the definition of evaluation in the Haskell language. In this example, the list under or' never expands further than for 2 members. Consider a more complex example (and more close to your example): main = putStr (shows (sum2 ys) "\n") -- sum' where n = 10^7 ys = [1 .. n] sum2 :: [Int] -> Int sum2 xs = sm xs 0 where sm [] s = s sm (x: xs) s = sm xs (s+x) sum' :: [Int] -> Int sum' [] = 0 sum' (x: xs) = x + (sum' xs) (the function [n .. m] is of the library). Compile it with ghc --make -O, and then, it will be evident that sum2 takes a constant in n space (and stack) when running (run it like this: `./T +RTS -M1m -RTS' ). Let us investigate this. Furst, compute this by hand with treating (+) as not of the library: sm [1 .. 4] 0 = sm (1:[2 .. 4]) 0 = sm [2 .. 4] (1+0) = sm (2: [3 .. 4]) (1+0) = sm [3 .. 4] (2+(1+0)) = ... It must accumulate an intermediate stack of size O(n). Why ghc-6.10.4 computes it in a constant space (and stack) ? The following is my guess. And let the Haskell experts correct me, please. In the Haskell-98 library it is defined that (+) is strict on the type of Int (right?). Therefore, the compiler optimizes (1+0) by replacing it immediately with 1. Taking in account that all other parts are computed lazily, the evaluation of the compiled program is like this: sm [1 .. 4] 0 = sm (1:[2 .. 4]) 0 = sm [2 .. 4] (1+0) = sm [2 .. 4] 1 = sm (2: [3 .. 4]) 1 = sm [3 .. 4] (2+1) = sm [3 .. 4] 3 = ... So, it is the result of combining of the evaluation definition with a certain compiler's optimization related to the strictness info for Int.(+). Right? As to sum', it still takes O(n) space -- in ghc-6.10.4. I think, this is because it is difficult for a compiler to guess how strictness of Int.(+) may correlate to the definition of sum'. Probably, it needs some kind of inductive reasoning about sum' to optimize it (for example) into sum2. Right ? Regards, ----------------- Serge Mechveliani mechvel@botik.ru From bertram.felgenhauer at googlemail.com Sun Aug 2 14:32:09 2009 From: bertram.felgenhauer at googlemail.com (Bertram Felgenhauer) Date: Sun Aug 2 14:13:08 2009 Subject: Working with GHC HEAD In-Reply-To: <694519c50907261309j77162c91vd973b7ce2dc01558@mail.gmail.com> References: <694519c50907261309j77162c91vd973b7ce2dc01558@mail.gmail.com> Message-ID: <4a75dbac.1c07d00a.4c90.ffff98ed@mx.google.com> Antoine Latter wrote: > I was trying to see what GHC head was like, but I've run into a few > snags compiling packages. There's a discrepancy between ghc and ghc-pkg that causes this. See http://hackage.haskell.org/trac/ghc/ticket/3410 > My existing binary for cabal-install can install quite a few packages, > but then starts giving me strange errors eventually: I believe cabal-install goes through ghc-pkg rather than manipulating the package configuration directly. So this should work in theory. > - Does anyone have a version of 'network' which builds against GHC > head? I could bludgeon in the new GHC.IO.FD.FD type myself, but I'd > thought I'd ask around first. http://int-e.home.tlink.de/haskell/network-ghc-6.11.dpatch works for me. Bertram From joshua at reverberate.org Mon Aug 3 03:05:20 2009 From: joshua at reverberate.org (Joshua Haberman) Date: Mon Aug 3 02:46:25 2009 Subject: equivalent of EXPLAIN PLAN with GHC? References: <20090802075013.GA9740@whirlpool.galois.com> Message-ID: Don Stewart galois.com> writes: > joshua: > > Is there any kind of intermediate form that a person can examine to see > > how their code is being optimized? Anything like EXPLAIN PLAN in SQL? > > It would make it much easier to understand the kinds of optimizations > > Haskell can perform. > > > > I'm not looking so much for profiling -- obvious this program is trivial > > and takes no time. I just want to better understand what kind of > > optimizations are possible given Haskell's language model. > > > > So the optimization you're looking for here is fusion of some kind. > GHC ships with build/foldr fusion, and there are libraries for an > alternative system, stream fusion. > > GHC uses an intermediate representation called 'Core', which is a > mini-Haskell, essentially, that is optimized repeatedly via > type-preserving transformations. You can inspect this with a number of > tools, including "ghc-core" (available on Hackage). Excellent Don, thanks a lot for taking the time to spell this out for me. I don't understand it all yet, but I know what to dig into next time I want to observe the compiler's behavior. Josh From r.kelsall at millstream.com Mon Aug 3 10:54:28 2009 From: r.kelsall at millstream.com (Richard Kelsall) Date: Mon Aug 3 10:35:22 2009 Subject: Current GHC core documentation. Message-ID: <4A76FA24.9060706@millstream.com> This page http://www.haskell.org/ghc/documentation.html has a link to the September 2001 (Draft for GHC 5.02) document describing GHC Core (in what is for me user-hostile .ps.gz format.) And this page http://www.haskell.org/ghc/docs/latest/html/users_guide/ext-core.html promises an easier format PDF document, but the link is broken. I did eventually find the 1st April 2009 GHC 6.10 document here http://www.haskell.org/ghc/docs/6.10.2/html/ext-core/core.pdf and a bit on this page http://www.haskell.org/ghc/docs/latest/html/users_guide/options-debugging.html#id468571 about GHC core. I haven't read these yet, but could I ask whether they constitute the complete current documentation for GHC core? (I'm just curious to get a flavour of what core does.) Richard. From redcom at fedoms.com Mon Aug 3 15:43:23 2009 From: redcom at fedoms.com (=?iso-8859-15?Q?G=FCnther_Schmidt?=) Date: Mon Aug 3 15:24:15 2009 Subject: Compiling large source files Message-ID: Hi all, I'm having trouble compiling very large source files, the compiler eats 2GB and then dies. Is there a way around it? G?nther From thomas.dubuisson at gmail.com Mon Aug 3 16:20:01 2009 From: thomas.dubuisson at gmail.com (Thomas DuBuisson) Date: Mon Aug 3 16:00:52 2009 Subject: Compiling large source files In-Reply-To: References: Message-ID: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> Can you define "very large" and "compiler"? I know an old version of GHC (6.6?) would eat lots of memory when there were absurd numbers of let statements. Thomas 2009/8/3 G?nther Schmidt : > Hi all, > > I'm having trouble compiling very large source files, the compiler eats 2GB > and then dies. Is there a way around it? > > G?nther > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users > From redcom at fedoms.com Mon Aug 3 17:09:28 2009 From: redcom at fedoms.com (=?iso-8859-15?Q?G=FCnther_Schmidt?=) Date: Mon Aug 3 16:50:21 2009 Subject: Compiling large source files In-Reply-To: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> Message-ID: Hi Thomas, yes, a source file with a single literal list with 85k elements. G?nther Am 03.08.2009, 22:20 Uhr, schrieb Thomas DuBuisson : > Can you define "very large" and "compiler"? I know an old version of > GHC (6.6?) would eat lots of memory when there were absurd numbers of > let statements. > > Thomas > > 2009/8/3 G?nther Schmidt : >> Hi all, >> >> I'm having trouble compiling very large source files, the compiler eats >> 2GB >> and then dies. Is there a way around it? >> >> G?nther >> _______________________________________________ >> Glasgow-haskell-users mailing list >> Glasgow-haskell-users@haskell.org >> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users >> From redcom at fedoms.com Mon Aug 3 17:20:01 2009 From: redcom at fedoms.com (=?iso-8859-15?Q?G=FCnther_Schmidt?=) Date: Mon Aug 3 17:00:52 2009 Subject: Compiling large source files In-Reply-To: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> Message-ID: Hi Thomas, sry, the compiler is ghc-6.10.3 G?nther From catamorphism at gmail.com Mon Aug 3 18:00:53 2009 From: catamorphism at gmail.com (Tim Chevalier) Date: Mon Aug 3 17:41:45 2009 Subject: Current GHC core documentation. In-Reply-To: <4A76FA24.9060706@millstream.com> References: <4A76FA24.9060706@millstream.com> Message-ID: <4683d9370908031500v722b3b84y2a2ccf7fd38e8d0e@mail.gmail.com> On 8/3/09, Richard Kelsall wrote: > This page > > http://www.haskell.org/ghc/documentation.html > > has a link to the September 2001 (Draft for GHC 5.02) document > describing GHC Core (in what is for me user-hostile .ps.gz format.) > > And this page > > http://www.haskell.org/ghc/docs/latest/html/users_guide/ext-core.html > > promises an easier format PDF document, but the link is broken. > Thanks, I reopened the Trac ticket on that documentation bug: http://hackage.haskell.org/trac/ghc/ticket/3135 > > I did eventually find the 1st April 2009 GHC 6.10 document here > > http://www.haskell.org/ghc/docs/6.10.2/html/ext-core/core.pdf > > and a bit on this page > > http://www.haskell.org/ghc/docs/latest/html/users_guide/options-debugging.html#id468571 > > about GHC core. I haven't read these yet, but could I ask whether they > constitute the complete current documentation for GHC core? (I'm just > curious to get a flavour of what core does.) > The PDF file is the complete documentation for External Core, along with the typechecker and interpreter (providing a static and dynamic semantics for External Core) under utils/ext-core in the GHC source tree. The version of Core described in the user manual is a little different. If you have questions about any of the abovementioned documentation, please CC both me and this list, as I don't read the GHC lists often. Cheers, Tim -- Tim Chevalier * http://cs.pdx.edu/~tjc * Often in error, never in doubt "The price one pays for pursuing any profession or calling is an intimate knowledge of its ugly side." -- James Baldwin From marlowsd at gmail.com Tue Aug 4 04:10:24 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Tue Aug 4 03:51:16 2009 Subject: Current GHC core documentation. In-Reply-To: <4A76FA24.9060706@millstream.com> References: <4A76FA24.9060706@millstream.com> Message-ID: <4A77ECF0.5040303@gmail.com> On 03/08/2009 15:54, Richard Kelsall wrote: > This page > > http://www.haskell.org/ghc/documentation.html > > has a link to the September 2001 (Draft for GHC 5.02) document > describing GHC Core (in what is for me user-hostile .ps.gz format.) > > And this page > > http://www.haskell.org/ghc/docs/latest/html/users_guide/ext-core.html > > promises an easier format PDF document, but the link is broken. > > > I did eventually find the 1st April 2009 GHC 6.10 document here > > http://www.haskell.org/ghc/docs/6.10.2/html/ext-core/core.pdf > > and a bit on this page > > http://www.haskell.org/ghc/docs/latest/html/users_guide/options-debugging.html#id468571 > > about GHC core. I haven't read these yet, but could I ask whether they > constitute the complete current documentation for GHC core? (I'm just > curious to get a flavour of what core does.) There's also the commentary page: http://hackage.haskell.org/trac/ghc/wiki/Commentary/Compiler/CoreSynType which is supposed to be the canonical place for documentation about GHC's internal Core datatype. "External Core" is slightly different: it refers to the external representation of Core that you get from the -fext-core flag. Round-tripping via External Core is supposed to be non-lossy, though, so External Core retains everything in the original Core. Cheers, Simon From marlowsd at gmail.com Tue Aug 4 04:12:37 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Tue Aug 4 03:53:37 2009 Subject: Compiling large source files In-Reply-To: References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> Message-ID: <4A77ED75.40109@gmail.com> I suggest not using Haskell for your list. Put the data in a file and read it at runtime, or put it in a static C array and link it in. Cheers, Simon On 03/08/2009 22:09, G?nther Schmidt wrote: > Hi Thomas, > > yes, a source file with a single literal list with 85k elements. > > > G?nther > > > Am 03.08.2009, 22:20 Uhr, schrieb Thomas DuBuisson > : > >> Can you define "very large" and "compiler"? I know an old version of >> GHC (6.6?) would eat lots of memory when there were absurd numbers of >> let statements. >> >> Thomas >> >> 2009/8/3 G?nther Schmidt : >>> Hi all, >>> >>> I'm having trouble compiling very large source files, the compiler >>> eats 2GB >>> and then dies. Is there a way around it? >>> >>> G?nther >>> _______________________________________________ >>> Glasgow-haskell-users mailing list >>> Glasgow-haskell-users@haskell.org >>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users >>> > > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users From Christian.Maeder at dfki.de Tue Aug 4 04:15:15 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Tue Aug 4 03:56:10 2009 Subject: use gtar and not tar under solaris Message-ID: <4A77EE13.80607@dfki.de> Hi, I've just been informed that unpacking the binary (i386) solaris distribution using bunzip2 and tar: bunzip2 -c ghc-6.10.4-i386-unknown-solaris2.tar.bz2 | tar xvf - does not work properly! Use instead: gtar jxvf ghc-6.10.4-i386-unknown-solaris2.tar.bz2 File names longer than a hundred characters are cut off, i.e. ghc-6.10.4/libraries/dph/dph-prim-seq/dist/build/Data/Array/Parallel/Unlifted/Sequential/Flat/UArr.hi is wrongly extracted as: ghc-6.10.4/libraries/dph/dph-prim-seq/dist/build/Data/Array/Parallel/Unlifted/Sequential/Flat/UArr.h leading to an installation failure: installPackage: Error: Could not find module: Data.Array.Parallel.Unlifted.Sequential.Flat.UArr with any suffix: ["hi"] gmake[1]: *** [install.library.dph/dph-prim-seq] Error 1 Ian, could you place a note about using gtar for unpacking solaris binary-dists? Cheers Christian From mechvel at botik.ru Tue Aug 4 08:30:33 2009 From: mechvel at botik.ru (Serge D. Mechveliani) Date: Tue Aug 4 08:17:53 2009 Subject: Compiling large source files In-Reply-To: <4A77ED75.40109@gmail.com> References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> <4A77ED75.40109@gmail.com> Message-ID: <20090804123033.GA8905@scico.botik.ru> On Tue, Aug 04, 2009 at 09:12:37AM +0100, Simon Marlow wrote: > I suggest not using Haskell for your list. Put the data in a file and > read it at runtime, or put it in a static C array and link it in. > > On 03/08/2009 22:09, G?nther Schmidt wrote: > >Hi Thomas, > >yes, a source file with a single literal list with 85k elements. People, when a program only defines and returns a String constant of n literals, how much memory needs ghc-6.10.4 to compile it ? O(n), or may be O(n^2), or ... Regards, ----------------- Serge Mechveliani mechvel@botik.ru From marlowsd at gmail.com Tue Aug 4 09:22:38 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Tue Aug 4 09:03:29 2009 Subject: Compiling large source files In-Reply-To: <20090804123033.GA8905@scico.botik.ru> References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> <4A77ED75.40109@gmail.com> <20090804123033.GA8905@scico.botik.ru> Message-ID: <4A78361E.7080006@gmail.com> On 04/08/2009 13:30, Serge D. Mechveliani wrote: > On Tue, Aug 04, 2009 at 09:12:37AM +0100, Simon Marlow wrote: >> I suggest not using Haskell for your list. Put the data in a file and >> read it at runtime, or put it in a static C array and link it in. >> >> On 03/08/2009 22:09, G?nther Schmidt wrote: >>> Hi Thomas, >>> yes, a source file with a single literal list with 85k elements. > > > People, > when a program only defines and returns a String constant of n > literals, how much memory needs ghc-6.10.4 to compile it ? > O(n), or may be O(n^2), or ... Certainly not O(n^2), we would class that as a bug. I can imagine it might be worse than linear however, if not in memory at least in time. Here's some very rough reasoning: GHC has to maintain various kinds of mappings to do its work. A mapping from variables is O(logn) to look up in, and we have to do at least O(n) lookups since there are O(n) variables, so compilation will be O(nlogn). We should remember that typechecking is worst-case exponential in space and time, though that is rarely an issue in practice. String constants are a special case because they are stored internally as flat arrays of bytes, so you'll probably get closer to O(n) with a much better constant factor. Cheers, Simon From slavomir.kaslev at gmail.com Tue Aug 4 13:48:25 2009 From: slavomir.kaslev at gmail.com (Slavomir Kaslev) Date: Tue Aug 4 13:29:13 2009 Subject: Data.List permutations Message-ID: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> A friend mine, new to functional programming, was entertaining himself by writing different combinatorial algorithms in Haskell. He asked me for some help so I sent him my quick and dirty solutions for generating variations and permutations: > inter x [] = [[x]] > inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) > perm [] = [[]] > perm (x:xs) = concatMap (inter x) (perm xs) > vari 0 _ = [[]] > vari _ [] = [] > vari k (x:xs) = concatMap (inter x) (vari (k-1) xs) ++ vari k xs After that I found out that nowadays there is a permutation function in the Data.List module: > permutations :: [a] -> [[a]] > permutations xs0 = xs0 : perms xs0 [] > where > perms [] _ = [] > perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations is) > where interleave xs r = let (_,zs) = interleave' id xs r in zs > interleave' _ [] r = (ts, r) > interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) ys r > in (y:us, f (t:y:us) : zs) I was surprised to find that not only my version is much simpler from the one in Data.List but it also performs better. Here are some numbers from my rather old ghc 6.8.1 running ubuntu on my box: *Main> length $ permutations [1..10] 3628800 (10.80 secs, 2391647384 bytes) *Main> length $ perm [1..10] 3628800 (8.58 secs, 3156902672 bytes) I would like to suggest to change the current implementation in Data.List with the simpler one. Also, it would be nice to add variations and combinations in the Data.List module. Cheers. -- Slavomir Kaslev From kr.angelov at gmail.com Tue Aug 4 13:53:53 2009 From: kr.angelov at gmail.com (Krasimir Angelov) Date: Tue Aug 4 13:34:42 2009 Subject: Data.List permutations In-Reply-To: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> Message-ID: Your function is not equivalent: perm _|_ = _|_ permutations _|_ = _|_ : _|_ On 8/4/09, Slavomir Kaslev wrote: > A friend mine, new to functional programming, was entertaining himself by > writing different combinatorial algorithms in Haskell. He asked me for some > help so I sent him my quick and dirty solutions for generating variations and > permutations: > > > inter x [] = [[x]] > > inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) > > > perm [] = [[]] > > perm (x:xs) = concatMap (inter x) (perm xs) > > > vari 0 _ = [[]] > > vari _ [] = [] > > vari k (x:xs) = concatMap (inter x) (vari (k-1) xs) ++ vari k xs > > After that I found out that nowadays there is a permutation function in the > Data.List module: > > > permutations :: [a] -> [[a]] > > permutations xs0 = xs0 : perms xs0 [] > > where > > perms [] _ = [] > > perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations is) > > where interleave xs r = let (_,zs) = interleave' id xs r in zs > > interleave' _ [] r = (ts, r) > > interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) ys r > > in (y:us, f (t:y:us) : zs) > > I was surprised to find that not only my version is much simpler from the one > in Data.List but it also performs better. Here are some numbers from my rather > old ghc 6.8.1 running ubuntu on my box: > > *Main> length $ permutations [1..10] > 3628800 > (10.80 secs, 2391647384 bytes) > *Main> length $ perm [1..10] > 3628800 > (8.58 secs, 3156902672 bytes) > > I would like to suggest to change the current implementation in Data.List with > the simpler one. Also, it would be nice to add variations and combinations in > the Data.List module. > > Cheers. > > -- > Slavomir Kaslev > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users > From slavomir.kaslev at gmail.com Tue Aug 4 14:22:39 2009 From: slavomir.kaslev at gmail.com (Slavomir Kaslev) Date: Tue Aug 4 14:03:28 2009 Subject: Data.List permutations In-Reply-To: References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> Message-ID: <171dfd0a0908041122v22b50bf2k9ca30183f1af03ac@mail.gmail.com> On Tue, Aug 4, 2009 at 8:53 PM, Krasimir Angelov wrote: > Your function is not equivalent: > > perm _|_ = _|_ > > permutations _|_ = _|_ : _|_ Nice catch. One can use the same trick as in permutations: > perm2 [] = [[]] > perm2 xxs@(x:xs) = xxs : tail (concatMap (inter x) (perm2 xs)) I've just noticed that permutations and perm enumerate the permutations in different order. > On 8/4/09, Slavomir Kaslev wrote: >> A friend mine, new to functional programming, was entertaining himself by >> writing different combinatorial algorithms in Haskell. He asked me for some >> help so I sent him my quick and dirty solutions for generating variations and >> permutations: >> >> > inter x [] = [[x]] >> > inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) >> >> > perm [] = [[]] >> > perm (x:xs) = concatMap (inter x) (perm xs) >> >> > vari 0 _ = [[]] >> > vari _ [] = [] >> > vari k (x:xs) = concatMap (inter x) (vari (k-1) xs) ++ vari k xs >> >> After that I found out that nowadays there is a permutation function in the >> Data.List module: >> >> > permutations ? ? ? ? ? ?:: [a] -> [[a]] >> > permutations xs0 ? ? ? ?= ?xs0 : perms xs0 [] >> > ? where >> > ? ? perms [] ? ? _ ?= [] >> > ? ? perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations is) >> > ? ? ? where interleave ? ?xs ? ? r = let (_,zs) = interleave' id xs r in zs >> > ? ? ? ? ? ? interleave' _ [] ? ? r = (ts, r) >> > ? ? ? ? ? ? interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) ys r >> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?in ?(y:us, f (t:y:us) : zs) >> >> I was surprised to find that not only my version is much simpler from the one >> in Data.List but it also performs better. Here are some numbers from my rather >> old ghc 6.8.1 running ubuntu on my box: >> >> *Main> length $ permutations [1..10] >> 3628800 >> (10.80 secs, 2391647384 bytes) >> *Main> length $ perm [1..10] >> 3628800 >> (8.58 secs, 3156902672 bytes) >> >> I would like to suggest to change the current implementation in Data.List with >> the simpler one. Also, it would be nice to add variations and combinations in >> the Data.List module. >> >> Cheers. >> >> -- >> Slavomir Kaslev >> _______________________________________________ >> Glasgow-haskell-users mailing list >> Glasgow-haskell-users@haskell.org >> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users >> > -- Slavomir Kaslev From daniel.is.fischer at web.de Tue Aug 4 14:23:46 2009 From: daniel.is.fischer at web.de (Daniel Fischer) Date: Tue Aug 4 14:05:27 2009 Subject: Data.List permutations In-Reply-To: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> Message-ID: <200908042023.46973.daniel.is.fischer@web.de> Am Dienstag 04 August 2009 19:48:25 schrieb Slavomir Kaslev: > A friend mine, new to functional programming, was entertaining himself by > writing different combinatorial algorithms in Haskell. He asked me for some > help so I sent him my quick and dirty solutions for generating variations > and > > permutations: > > inter x [] = [[x]] > > inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) > > > > perm [] = [[]] > > perm (x:xs) = concatMap (inter x) (perm xs) > > > > vari 0 _ = [[]] > > vari _ [] = [] > > vari k (x:xs) = concatMap (inter x) (vari (k-1) xs) ++ vari k xs > > After that I found out that nowadays there is a permutation function in the > > Data.List module: > > permutations :: [a] -> [[a]] > > permutations xs0 = xs0 : perms xs0 [] > > where > > perms [] _ = [] > > perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations > > is) where interleave xs r = let (_,zs) = interleave' id xs r in zs > > interleave' _ [] r = (ts, r) > > interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) > > ys r in (y:us, f (t:y:us) : zs) > > I was surprised to find that not only my version is much simpler from the > one in Data.List but it also performs better. Here are some numbers from my > rather old ghc 6.8.1 running ubuntu on my box: > > *Main> length $ permutations [1..10] > 3628800 > (10.80 secs, 2391647384 bytes) > *Main> length $ perm [1..10] > 3628800 > (8.58 secs, 3156902672 bytes) But you compare *interpreted* code here, that's not what counts. Prelude Perms> length $ perm [1 .. 10] 3628800 (1.20 secs, 1259105892 bytes) Prelude Perms> length $ permutations [1 .. 10] 3628800 (0.56 secs, 551532668 bytes) Prelude Perms> length $ perm [1 .. 11] 39916800 (13.18 secs, 14651808004 bytes) Prelude Perms> length $ permutations [1 .. 11] 39916800 (4.30 secs, 5953485728 bytes) Apparently the library code is more amenable to the optimiser (note that the actual library is faster still: Prelude Data.List> length $ permutations [1 .. 10] 3628800 (0.49 secs, 551532812 bytes) Prelude Data.List> length $ permutations [1 .. 11] 39916800 (3.73 secs, 5953485816 bytes) I have no idea why). > > I would like to suggest to change the current implementation in Data.List > with the simpler one. Also, it would be nice to add variations and > combinations in the Data.List module. > > Cheers. From slavomir.kaslev at gmail.com Tue Aug 4 14:30:58 2009 From: slavomir.kaslev at gmail.com (Slavomir Kaslev) Date: Tue Aug 4 14:11:47 2009 Subject: Data.List permutations In-Reply-To: <200908042023.46973.daniel.is.fischer@web.de> References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> <200908042023.46973.daniel.is.fischer@web.de> Message-ID: <171dfd0a0908041130u2c082817s80c6d08e1f145c86@mail.gmail.com> On Tue, Aug 4, 2009 at 9:23 PM, Daniel Fischer wrote: > Am Dienstag 04 August 2009 19:48:25 schrieb Slavomir Kaslev: >> A friend mine, new to functional programming, was entertaining himself by >> writing different combinatorial algorithms in Haskell. He asked me for some >> help so I sent him my quick and dirty solutions for generating variations >> and >> >> permutations: >> > inter x [] = [[x]] >> > inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) >> > >> > perm [] = [[]] >> > perm (x:xs) = concatMap (inter x) (perm xs) >> > >> > vari 0 _ = [[]] >> > vari _ [] = [] >> > vari k (x:xs) = concatMap (inter x) (vari (k-1) xs) ++ vari k xs >> >> After that I found out that nowadays there is a permutation function in the >> >> Data.List module: >> > permutations ? ? ? ? ? ?:: [a] -> [[a]] >> > permutations xs0 ? ? ? ?= ?xs0 : perms xs0 [] >> > ? where >> > ? ? perms [] ? ? _ ?= [] >> > ? ? perms (t:ts) is = foldr interleave (perms ts (t:is)) (permutations >> > is) where interleave ? ?xs ? ? r = let (_,zs) = interleave' id xs r in zs >> > interleave' _ [] ? ? r = (ts, r) >> > ? ? ? ? ? ? interleave' f (y:ys) r = let (us,zs) = interleave' (f . (y:)) >> > ys r in ?(y:us, f (t:y:us) : zs) >> >> I was surprised to find that not only my version is much simpler from the >> one in Data.List but it also performs better. Here are some numbers from my >> rather old ghc 6.8.1 running ubuntu on my box: >> >> *Main> length $ permutations [1..10] >> 3628800 >> (10.80 secs, 2391647384 bytes) >> *Main> length $ perm [1..10] >> 3628800 >> (8.58 secs, 3156902672 bytes) > > But you compare *interpreted* code here, that's not what counts. > > Prelude Perms> length $ perm [1 .. 10] > 3628800 > (1.20 secs, 1259105892 bytes) > Prelude Perms> length $ permutations [1 .. 10] > 3628800 > (0.56 secs, 551532668 bytes) > Prelude Perms> length $ perm [1 .. 11] > 39916800 > (13.18 secs, 14651808004 bytes) > Prelude Perms> length $ permutations [1 .. 11] > 39916800 > (4.30 secs, 5953485728 bytes) Which version of ghc are you testing on? I guess, it's more recent than mine. > Apparently the library code is more amenable to the optimiser (note that the actual > library is faster still: > > Prelude Data.List> length $ permutations [1 .. 10] > 3628800 > (0.49 secs, 551532812 bytes) > Prelude Data.List> length $ permutations [1 .. 11] > 39916800 > (3.73 secs, 5953485816 bytes) > > I have no idea why). Probably because it's compiled (and not interpreted) in this case. -- Slavomir Kaslev From daniel.is.fischer at web.de Tue Aug 4 14:43:17 2009 From: daniel.is.fischer at web.de (Daniel Fischer) Date: Tue Aug 4 14:24:58 2009 Subject: Data.List permutations In-Reply-To: <171dfd0a0908041130u2c082817s80c6d08e1f145c86@mail.gmail.com> References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> <200908042023.46973.daniel.is.fischer@web.de> <171dfd0a0908041130u2c082817s80c6d08e1f145c86@mail.gmail.com> Message-ID: <200908042043.17433.daniel.is.fischer@web.de> Am Dienstag 04 August 2009 20:30:58 schrieb Slavomir Kaslev: > On Tue, Aug 4, 2009 at 9:23 PM, Daniel Fischer wrote: > > Which version of ghc are you testing on? I guess, it's more recent than > mine. 6.10.3. But I think if you compiled it with 6.8.*, the library code would still be faster, perhaps by a smaller margin. > > > Apparently the library code is more amenable to the optimiser (note that > > the actual library is faster still: > > > > Prelude Data.List> length $ permutations [1 .. 10] > > 3628800 > > (0.49 secs, 551532812 bytes) > > Prelude Data.List> length $ permutations [1 .. 11] > > 39916800 > > (3.73 secs, 5953485816 bytes) > > > > I have no idea why). > > Probably because it's compiled (and not interpreted) in this case. All my times were from compiled (with -O2) code. The question is, why does the same source code produce slower object code in module Perms than in Data.List? I suppose it's because Data.List was compiled with different command line options, but I've no idea which. From johan.tibell at gmail.com Tue Aug 4 16:53:56 2009 From: johan.tibell at gmail.com (Johan Tibell) Date: Tue Aug 4 16:35:05 2009 Subject: Working with GHC HEAD In-Reply-To: <4a75dbac.1c07d00a.4c90.ffff98ed@mx.google.com> References: <694519c50907261309j77162c91vd973b7ce2dc01558@mail.gmail.com> <4a75dbac.1c07d00a.4c90.ffff98ed@mx.google.com> Message-ID: <90889fe70908041353h4426ec79ta968d94d7a3ac974@mail.gmail.com> On Sun, Aug 2, 2009 at 8:32 PM, Bertram Felgenhauer < bertram.felgenhauer@googlemail.com> wrote: > Antoine Latter wrote: > > - Does anyone have a version of 'network' which builds against GHC > > head? I could bludgeon in the new GHC.IO.FD.FD type myself, but I'd > > thought I'd ask around first. > > http://int-e.home.tlink.de/haskell/network-ghc-6.11.dpatch > > works for me. > I've applied this to the network HEAD. Thanks for the patch. -- Johan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.haskell.org/pipermail/glasgow-haskell-users/attachments/20090804/17ffea59/attachment.html From malcolm.wallace at cs.york.ac.uk Tue Aug 4 16:57:46 2009 From: malcolm.wallace at cs.york.ac.uk (Malcolm Wallace) Date: Tue Aug 4 16:38:37 2009 Subject: Data.List permutations In-Reply-To: References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> Message-ID: <263F269E-0035-452E-BE5B-84783ED1CF2C@cs.york.ac.uk> > Your function is not equivalent: > > perm _|_ = _|_ > > permutations _|_ = _|_ : _|_ I have a vague memory that the library version diagonalises properly, so that if you give it a lazy infinite input, it still generates sensible output lazily. If so, this important property should be noted in the haddocks. Regards, Malcolm From gale at sefer.org Wed Aug 5 07:22:06 2009 From: gale at sefer.org (Yitzchak Gale) Date: Wed Aug 5 07:03:13 2009 Subject: Data.List permutations In-Reply-To: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> Message-ID: <2608b8a80908050422q5e269319p11c893dd92a0f800@mail.gmail.com> Hi Slavomir, Slavomir Kaslev wrote: >> inter x [] = [[x]] >> inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) > >> perm [] = [[]] >> perm (x:xs) = concatMap (inter x) (perm xs) > > I was surprised to find that not only my version is much simpler from the one > in Data.List but it also performs better. > I would like to suggest to change the current implementation in Data.List with > the simpler one. Thanks for looking into this. A lot of work went into the permutations function in Data.List, most of it by Twan van Laarhoven, including studying the order in which the permutations are returned, laziness, and extensive performance testing. The result was compared against Knuth's algorithmic work on this topic. Your algorithm is indeed similar to one that was considered during that development process. See the following thread for the details: http://www.haskell.org/pipermail/libraries/2007-December/008788.html and continued in: http://www.haskell.org/pipermail/libraries/2008-January/008859.html Things do change with time, so it's always worthwhile to revisit these things and not take them for granted. If you can improve on this work, please let us know on the libraries mailing list. > Also, it would be nice to add variations and combinations in > the Data.List module. Personally, I disagree. I think those should go in a separate combinatorics library, not Data.List. As an example, take a look at the combinat package on Hackage: http://hackage.haskell.org/package/combinat But you don't have to agree with me. If you think that they should go into Data.List, propose it using the following procedure: http://www.haskell.org/haskellwiki/Library_submissions Regards, Yitz From slavomir.kaslev at gmail.com Wed Aug 5 09:04:56 2009 From: slavomir.kaslev at gmail.com (Slavomir Kaslev) Date: Wed Aug 5 08:45:44 2009 Subject: Data.List permutations In-Reply-To: <2608b8a80908050422q5e269319p11c893dd92a0f800@mail.gmail.com> References: <171dfd0a0908041048r6a52d7f5n99dff8c66ebe6d44@mail.gmail.com> <2608b8a80908050422q5e269319p11c893dd92a0f800@mail.gmail.com> Message-ID: <171dfd0a0908050604i16608cb9w3034ed0d286c4161@mail.gmail.com> Thank you for the comprehensive reply Yitzchak. On Wed, Aug 5, 2009 at 2:22 PM, Yitzchak Gale wrote: > Hi Slavomir, > > Slavomir Kaslev wrote: >>> inter x [] = [[x]] >>> inter x yys@(y:ys) = [x:yys] ++ map (y:) (inter x ys) >> >>> perm [] = [[]] >>> perm (x:xs) = concatMap (inter x) (perm xs) >> >> I was surprised to find that not only my version is much simpler from the one >> in Data.List but it also performs better. >> I would like to suggest to change the current implementation in Data.List with >> the simpler one. > > Thanks for looking into this. > > A lot of work went into the permutations function in Data.List, > most of it by Twan van Laarhoven, including studying the order > in which the permutations are returned, laziness, and extensive > performance testing. The result was compared against Knuth's > algorithmic work on this topic. > > Your algorithm is indeed similar to one that was considered > during that development process. > > See the following thread for the details: > > http://www.haskell.org/pipermail/libraries/2007-December/008788.html > > and continued in: > > http://www.haskell.org/pipermail/libraries/2008-January/008859.html Thanks for the links. I didn't realize how much effort and considerations have gone in the development of permutations. I should have guessed better. One can never determine the effort that certain code needed to be developed from the code itself. > Things do change with time, so it's always worthwhile to > revisit these things and not take them for granted. If you > can improve on this work, please let us know on the > libraries mailing list. I should have sent the message to the libraries mailing list in the first place. But I was under the (outdated) impression that Data.List is ghc specific library. For a possible improvement, I guess I have to work harder to outdo the work that went into permutations. While this is a noble goal, it wasn't my initial intention. I was just helping a friend. Although, I may get onto it when I have enough spare time (I am preparing for my graduation exam in the moment). >> Also, it would be nice to add variations and combinations in >> the Data.List module. > > Personally, I disagree. I think those should go in a separate > combinatorics library, not Data.List. As an example, take a > look at the combinat package on Hackage: > > http://hackage.haskell.org/package/combinat You are right. There's no reason to bloat Data.List with such functions, when a different module will do. Thanks for the link. Cheers. -- Slavomir Kaslev From simonpj at microsoft.com Thu Aug 6 03:22:37 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Thu Aug 6 03:03:24 2009 Subject: Compiling large source files In-Reply-To: <20090804123033.GA8905@scico.botik.ru> References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> <4A77ED75.40109@gmail.com> <20090804123033.GA8905@scico.botik.ru> Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C357F321EAEE@EA-EXMSG-C334.europe.corp.microsoft.com> It should be pretty much linear (modulo a log(n) factor, but then log(n) is practically constant (=64 or so).). But people often report that GHC is slow (perhaps non-linearly so) when compiling vast blobs of literal data. Because there is a reasonable workaround (just parse the data), which actually makes your program more flexible (since you can change the data without recompiling), we have serially failed to look hard at this problem. I wish someone would nail it, though. Simon | -----Original Message----- | From: glasgow-haskell-users-bounces@haskell.org [mailto:glasgow-haskell- | users-bounces@haskell.org] On Behalf Of Serge D. Mechveliani | Sent: 04 August 2009 13:31 | To: Simon Marlow | Cc: glasgow-haskell-users@haskell.org | Subject: Re: Compiling large source files | | On Tue, Aug 04, 2009 at 09:12:37AM +0100, Simon Marlow wrote: | > I suggest not using Haskell for your list. Put the data in a file and | > read it at runtime, or put it in a static C array and link it in. | > | > On 03/08/2009 22:09, G?nther Schmidt wrote: | > >Hi Thomas, | > >yes, a source file with a single literal list with 85k elements. | | | People, | when a program only defines and returns a String constant of n | literals, how much memory needs ghc-6.10.4 to compile it ? | O(n), or may be O(n^2), or ... | | Regards, | | ----------------- | Serge Mechveliani | mechvel@botik.ru | | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users@haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users From jason.dusek at gmail.com Thu Aug 6 03:38:08 2009 From: jason.dusek at gmail.com (Jason Dusek) Date: Thu Aug 6 03:18:53 2009 Subject: What is the mutator? Message-ID: <42784f260908060038h53d7cc0dy9f80e43f269a2faf@mail.gmail.com> I've been reading a little about GC latency and have run across statements like this: One solution to the GC synchronisation problem would be to implement a concurrent garbage collector. Typically, however, concurrent GC adds some overhead to the mutator, since it must synchronise with the collector.some thunks are never ?black-holed?, so giving a potential performance win. Unfortunately, in the parallel setting, it substantially enlarges the time window in which two or more duplicate threads might evaluate the same think, and thus -- Comparing and Optimising Parallel Haskell Implementations for Multicore Machines What is the mutator? -- Jason Dusek From Christian.Maeder at dfki.de Thu Aug 6 04:04:19 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Thu Aug 6 03:45:05 2009 Subject: gcc version for GHC 6.10.4 under Sparc/Solaris In-Reply-To: References: Message-ID: <4A7A8E83.8070502@dfki.de> Hi Ian, could you add a note on the download page that GCC version 4.3.x is not suited for: http://www.haskell.org/ghc/dist/6.10.4/maeder/ghc-6.10.4-sparc-sun-solaris2.tar.bz2 The binary-dist was compiled using gcc-4.2.2 (but also works i.e. for gcc-3.4.4) page http://hackage.haskell.org/trac/ghc/wiki/Building/Solaris says: "GCC version 4.1.2 is recommended" and "GHC has not yet been updated to understand the assembly output of GCC version 4.3.x." Maybe a ticket should be opened for gcc-4.3.x under Sparc/Solaris Did anyway use gcc-4.3.x under x86 Solaris? (I don't think, it'll have the same problem.) Cheers Christian From duncan.coutts at worc.ox.ac.uk Thu Aug 6 07:30:51 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Thu Aug 6 07:11:21 2009 Subject: use gtar and not tar under solaris In-Reply-To: <4A77EE13.80607@dfki.de> References: <4A77EE13.80607@dfki.de> Message-ID: <1249558251.20327.9352.camel@localhost> On Tue, 2009-08-04 at 10:15 +0200, Christian Maeder wrote: > Hi, > > I've just been informed that unpacking the binary (i386) solaris > distribution using bunzip2 and tar: It may work better in future if you use a non-GNU tar to pack it up in the first place. GNU tar uses a non-standard tar format by default. Solaris tar would likely have more luck unpacking a POSIX/USTAR tar format file. It's also possible to use gnu tar to make standard tar format files, using --format ustar rather than gnu tar's default of --format gnu. Duncan (who now knows an unhealthy amount about tar file formats after writing a Haskell package to read them) From duncan.coutts at worc.ox.ac.uk Thu Aug 6 07:24:58 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Thu Aug 6 07:11:27 2009 Subject: gcc version for GHC 6.10.4 under Sparc/Solaris In-Reply-To: <4A7A8E83.8070502@dfki.de> References: <4A7A8E83.8070502@dfki.de> Message-ID: <1249557898.20327.9340.camel@localhost> On Thu, 2009-08-06 at 10:04 +0200, Christian Maeder wrote: > Hi Ian, > > could you add a note on the download page that > GCC version 4.3.x is not suited for: > > http://www.haskell.org/ghc/dist/6.10.4/maeder/ghc-6.10.4-sparc-sun-solaris2.tar.bz2 > > The binary-dist was compiled using gcc-4.2.2 (but also works i.e. for > gcc-3.4.4) I should also note that there is a GHC 6.10.4 binary for Sparc/Linux that is now included with Gentoo. It's got all features turned on except for split objects (which fails due to mixing ld -r and --relax flags). In particular it's a registerised via-C build with ghci, TH and profiling working. It's a distro package not a generic relocatable GHC binary tarball so there's no point putting it on the ghc download page, but it's there nevertheless if people want it (look for the gentoo ghc ebuild). Duncan From duncan.coutts at worc.ox.ac.uk Thu Aug 6 07:37:08 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Thu Aug 6 07:17:22 2009 Subject: use gtar and not tar under solaris In-Reply-To: <1249558251.20327.9352.camel@localhost> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> Message-ID: <1249558628.20327.9364.camel@localhost> On Thu, 2009-08-06 at 12:30 +0100, Duncan Coutts wrote: > On Tue, 2009-08-04 at 10:15 +0200, Christian Maeder wrote: > > Hi, > > > > I've just been informed that unpacking the binary (i386) solaris > > distribution using bunzip2 and tar: > > It may work better in future if you use a non-GNU tar to pack it up in > the first place. GNU tar uses a non-standard tar format by default. > Solaris tar would likely have more luck unpacking a POSIX/USTAR tar > format file. It's also possible to use gnu tar to make standard tar > format files, using --format ustar rather than gnu tar's default of > --format gnu. In fact I think I'd always advocate using the USTAR tar format over the GNU tar format when distributing software, since portability is of prime concern. This is what cabal-install does. I'd recommend ghc do it too. I also filed a ticket for darcs dist about this some time ago. Duncan From bulat.ziganshin at gmail.com Thu Aug 6 07:35:38 2009 From: bulat.ziganshin at gmail.com (Bulat Ziganshin) Date: Thu Aug 6 07:26:52 2009 Subject: What is the mutator? In-Reply-To: <42784f260908060038h53d7cc0dy9f80e43f269a2faf@mail.gmail.com> References: <42784f260908060038h53d7cc0dy9f80e43f269a2faf@mail.gmail.com> Message-ID: <147602090.20090806153538@gmail.com> Hello Jason, Thursday, August 6, 2009, 11:38:08 AM, you wrote: > One solution to the GC synchronisation problem would be to > implement a concurrent garbage collector. Typically, > however, concurrent GC adds some overhead to the mutator, > since it must synchronise with the collector.some thunks are > never ?black-holed?, so giving a potential performance win. > Unfortunately, in the parallel setting, it substantially > enlarges the time window in which two or more duplicate > threads might evaluate the same think, and thus i'm not an expert, but: lazy haskell value is some expression to comupte. when this value started to evaluate, it's replaced by "black hole" - special value. attempt to compute black-holed value (in the same thread) means that we have cyclic computation dependency - exception triggered. once value of thunk is evaluated, it's written back by code called mutator -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com From malcolm.wallace at cs.york.ac.uk Thu Aug 6 07:55:24 2009 From: malcolm.wallace at cs.york.ac.uk (Malcolm Wallace) Date: Thu Aug 6 07:36:11 2009 Subject: What is the mutator? In-Reply-To: <147602090.20090806153538@gmail.com> References: <42784f260908060038h53d7cc0dy9f80e43f269a2faf@mail.gmail.com> <147602090.20090806153538@gmail.com> Message-ID: <49716545-71B5-448A-8C68-F06A89A030D0@cs.york.ac.uk> > i'm not an expert, but: once value of thunk is evaluated, it's > written > back by code called mutator Whilst that is indeed mutation, it is not what is usually referred to as the "mutator" in the context of garbage collection. Quite simply, the "mutator" is the actual running program, as opposed to the GC, which is part of the underlying runtime system. Conceptually, the mutator and GC are the two mutually-exclusive threads of control that modify the heap. Usually one must halt while the other runs. Regards, Malcolm From Christian.Maeder at dfki.de Thu Aug 6 07:58:23 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Thu Aug 6 07:39:09 2009 Subject: use gtar and not tar under solaris In-Reply-To: <1249558251.20327.9352.camel@localhost> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> Message-ID: <4A7AC55F.6080301@dfki.de> ghc's configure script set: TAR = /opt/csw/bin/gtar ZIP = zip in mk/config.mk although I've got /usr/bin/tar, too (and earlier in my path). Cheers Christian Duncan Coutts wrote: > On Tue, 2009-08-04 at 10:15 +0200, Christian Maeder wrote: >> Hi, >> >> I've just been informed that unpacking the binary (i386) solaris >> distribution using bunzip2 and tar: > > It may work better in future if you use a non-GNU tar to pack it up in > the first place. GNU tar uses a non-standard tar format by default. > Solaris tar would likely have more luck unpacking a POSIX/USTAR tar > format file. It's also possible to use gnu tar to make standard tar > format files, using --format ustar rather than gnu tar's default of > --format gnu. > > Duncan > (who now knows an unhealthy amount about tar file formats after writing > a Haskell package to read them) > From simonpj at microsoft.com Thu Aug 6 09:34:33 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Thu Aug 6 09:15:17 2009 Subject: Compiling large source files In-Reply-To: References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C357F36E7BB5@EA-EXMSG-C334.europe.corp.microsoft.com> Gunther Could you make a Trac bug report for this, and attach your source file? It'd help if you could first check that things are still bad with GHC 6.10.4. Another useful thing would be to provide data on whether the behaviour is non-linear. Eg try with 1k, 2k, 4k, 8k, etc elements in your list and see how it behaves. Providing reproducible and well-characterised bug reports greatly increases the likelihood that we'll fix it! Thanks Simon | -----Original Message----- | From: glasgow-haskell-users-bounces@haskell.org [mailto:glasgow-haskell-users- | bounces@haskell.org] On Behalf Of G?nther Schmidt | Sent: 03 August 2009 22:09 | To: Thomas DuBuisson | Cc: glasgow-haskell-users@haskell.org | Subject: Re: Compiling large source files | | Hi Thomas, | | yes, a source file with a single literal list with 85k elements. | | | G?nther | | | Am 03.08.2009, 22:20 Uhr, schrieb Thomas DuBuisson | : | | > Can you define "very large" and "compiler"? I know an old version of | > GHC (6.6?) would eat lots of memory when there were absurd numbers of | > let statements. | > | > Thomas | > | > 2009/8/3 G?nther Schmidt : | >> Hi all, | >> | >> I'm having trouble compiling very large source files, the compiler eats | >> 2GB | >> and then dies. Is there a way around it? | >> | >> G?nther | >> _______________________________________________ | >> Glasgow-haskell-users mailing list | >> Glasgow-haskell-users@haskell.org | >> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users | >> | | | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users@haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users From simonpj at microsoft.com Thu Aug 6 12:54:48 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Thu Aug 6 12:35:33 2009 Subject: GHC 6.10.2 consuming lots of memory while compiling - help? In-Reply-To: References: Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C357F36E7CE0@EA-EXMSG-C334.europe.corp.microsoft.com> Justin Presumably DeliveryManagementQueries uses TH to generate lots of glop? *** Desugar: Result size = 616,969 That's a big program! What kind of glop is it? Maybe it's the same kind of thing as one of these? http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&type=compile-time+performance+bug&order=priority Regardless, if you can make a reproducible test case that'd help us. Probably you can do this simply by generating a file with lots of repeated glop of the same kind as your TH is spitting out. Simon | -----Original Message----- | From: glasgow-haskell-users-bounces@haskell.org [mailto:glasgow-haskell-users- | bounces@haskell.org] On Behalf Of Justin Bailey | Sent: 24 July 2009 20:03 | To: glasgow-haskell-users@haskell.org | Subject: GHC 6.10.2 consuming lots of memory while compiling - help? | | I apologize in advance for the vagueness of my report here - it's one | of those situations I'm not sure how to cut it down to size yet. | | I have a module that uses HaskellDB and Template Haskell together. The | module itself depends on 23 other modules, each of which give a type | definition for a particular database table or view. I only mention | that to emphasize that the module depends on some "big" types (HList | records w/ 20+ members) and on compile-time generated code. | | My problem is this - when GHC compile the module, it consumes 1.2 GB | of memory, takes about 10 minutes, and finally produces an object | file. The memory usage seems related to template haskell, but I'm not | positive. | | I've attached verbose output from compiling the module in question. | The command line I used was: | | ghc -v --make -c DeliveryManagementQueries.hs -XEmptyDataDecls | -XTypeSynonymInstances -XTemplateHa | skell | | Now for my question - please ignore the specifics of | haskelldb/template haskell - any suggestions for figuring out what GHC | is doing, besides tried-and-true divide and conquer? | | Justin From kili at outback.escape.de Thu Aug 6 13:02:41 2009 From: kili at outback.escape.de (Matthias Kilian) Date: Thu Aug 6 12:43:54 2009 Subject: use gtar and not tar under solaris In-Reply-To: <1249558251.20327.9352.camel@localhost> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> Message-ID: <20090806170241.GB139@petunia.outback.escape.de> On Thu, Aug 06, 2009 at 12:30:51PM +0100, Duncan Coutts wrote: > > I've just been informed that unpacking the binary (i386) solaris > > distribution using bunzip2 and tar: > > It may work better in future if you use a non-GNU tar to pack it up in > the first place. GNU tar uses a non-standard tar format by default. > Solaris tar would likely have more luck unpacking a POSIX/USTAR tar > format file. It's also possible to use gnu tar to make standard tar > format files, using --format ustar rather than gnu tar's default of > --format gnu. Is there something like pax(1) available on solaris? If so, it should be be preferred, because it's a POSIX tool, so there's some hope that it behaves the same on different systems. pax(1) should be available on all BSD systems, and to my knowledge, there's a pax package available at least for Debian Linux. Ciao, Kili From berthold at Mathematik.Uni-Marburg.de Thu Aug 6 13:37:51 2009 From: berthold at Mathematik.Uni-Marburg.de (Jost Berthold) Date: Thu Aug 6 13:18:53 2009 Subject: What is the mutator? In-Reply-To: <20090806163536.C88DE32493C@www.haskell.org> References: <20090806163536.C88DE32493C@www.haskell.org> Message-ID: <4A7B14EF.3040107@mathematik.uni-marburg.de> > Message: 1 > Date: Thu, 6 Aug 2009 00:38:08 -0700 > From: Jason Dusek > Subject: What is the mutator? > To: glasgow-haskell-users@haskell.org > Message-ID: > <42784f260908060038h53d7cc0dy9f80e43f269a2faf@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > I've been reading a little about GC latency and have run > across statements like this: > > One solution to the GC synchronisation problem would be to > implement a concurrent garbage collector. Typically, > however, concurrent GC adds some overhead to the mutator, > since it must synchronise with the collector.some thunks are > never “black-holed”, so giving a potential performance win. > Unfortunately, in the parallel setting, it substantially > enlarges the time window in which two or more duplicate > threads might evaluate the same think, and thus > > -- Comparing and Optimising Parallel Haskell Implementations > for Multicore Machines > > What is the mutator? Hi Jason, as Malcolm already said, the "mutator" in this text is the/a thread evaluating some Haskell expression. Just to add some more details to the picture... In general, a Haskell expression is a computation represented as a graph in the heap. Haskell evaluates lazily and does not have to fully evaluate every part of it for the program to finish. Unevaluated parts are "thunks". As soon as one of potentially several concurrent (mutator) threads starts to evaluate a thunk, it is replaced by a blackhole, which keeps other threads out of it until the node in the graph is evaluated (say, to a list cons (:), with probably unevaluated head and tail). Then the blackhole is updated with the new value. Other threads block on the blackhole in the meantime (so not necessarily an exception in the case of concurrent mutator threads) and are woken up by the update. The passage you quote above is about two separate aspects: 1. Garbage collection and mutator running concurrently: while they usually do, they do not _have_ to exclude each other, but not doing so means that the objects they are treating have to be locked. 2. About "Blackholing": in the sequential evaluation (where hitting a blackhole indeed means to have a loop), some better performance can be gained by not blackholing a thunk immediately, so this was done in GHC earlier. However, it increases the chance for 2 mutator threads to evaluate the same thunk (double work), and we got better performance by blackholing immediately. Cheers, Jost From Christian.Maeder at dfki.de Thu Aug 6 14:54:49 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Thu Aug 6 14:35:35 2009 Subject: use gtar and not tar under solaris In-Reply-To: <20090806170241.GB139@petunia.outback.escape.de> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> <20090806170241.GB139@petunia.outback.escape.de> Message-ID: <4A7B26F9.4020501@dfki.de> Matthias Kilian wrote: > On Thu, Aug 06, 2009 at 12:30:51PM +0100, Duncan Coutts wrote: >>> I've just been informed that unpacking the binary (i386) solaris >>> distribution using bunzip2 and tar: >> It may work better in future if you use a non-GNU tar to pack it up in >> the first place. GNU tar uses a non-standard tar format by default. >> Solaris tar would likely have more luck unpacking a POSIX/USTAR tar >> format file. It's also possible to use gnu tar to make standard tar >> format files, using --format ustar rather than gnu tar's default of >> --format gnu. > > Is there something like pax(1) available on solaris? If so, it > should be be preferred, because it's a POSIX tool, so there's some > hope that it behaves the same on different systems. Yes, pax is available under solaris. I thought GNU tar is the standard packer under unix. (The usage message of pax is less clear.) Below is a part of "gtar --help" Cheers Christian Archive format selection: -H, --format=FORMAT create archive of the given format. FORMAT is one of the following: gnu GNU tar 1.13.x format oldgnu GNU format as per tar <= 1.12 pax POSIX 1003.1-2001 (pax) format posix Same as pax ustar POSIX 1003.1-1988 (ustar) format v7 old V7 tar format From kili at outback.escape.de Thu Aug 6 16:03:26 2009 From: kili at outback.escape.de (Matthias Kilian) Date: Thu Aug 6 15:44:47 2009 Subject: use gtar and not tar under solaris In-Reply-To: <4A7B26F9.4020501@dfki.de> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> <20090806170241.GB139@petunia.outback.escape.de> <4A7B26F9.4020501@dfki.de> Message-ID: <20090806200325.GA32222@petunia.outback.escape.de> On Thu, Aug 06, 2009 at 08:54:49PM +0200, Christian Maeder wrote: > > Is there something like pax(1) available on solaris? If so, it > > should be be preferred, because it's a POSIX tool, so there's some > > hope that it behaves the same on different systems. > > Yes, pax is available under solaris. I thought GNU tar is the standard > packer under unix. Depends on what `standard' means ;-) - tar has been there forever on unices, with several slightly incompatible format extensions - GNU tar is just another implementation, typically used on Linux, and it has its own incompatible format extensions. - pax is (or should be) available everywhere, its behaviour is defined by POSIX, it should (by default) create archives readable by most tar implemenations, but almost nobody knows about it ;-) http://www.opengroup.org/onlinepubs/9699919799/utilities/pax.html I wonder wether Duncan did read and understood that bit of documentation, I didn't even read all of it ;-) > (The usage message of pax is less clear.) The manpage (and of course the POSIX definition) are hard stuff, too. However, to create an archive, you can use something like $ pax -wf foo.tar directory Ciao, Kili From jason.dusek at gmail.com Thu Aug 6 20:35:32 2009 From: jason.dusek at gmail.com (Jason Dusek) Date: Thu Aug 6 20:16:13 2009 Subject: What is the mutator? In-Reply-To: <4A7B14EF.3040107@mathematik.uni-marburg.de> References: <20090806163536.C88DE32493C@www.haskell.org> <4A7B14EF.3040107@mathematik.uni-marburg.de> Message-ID: <42784f260908061735k56fa93e9wd7ea16bf55296ca0@mail.gmail.com> 2009/08/06 Jost Berthold : > as Malcolm already said, the "mutator" in this text is the/a >?thread evaluating some Haskell expression. I want to thank everyone for taking the time to clarify that to me; I'm now much more able to follow discussions of Haskell garbage collection. > 1. Garbage collection and mutator running concurrently: while > they usually do, they do not _have_ to exclude each other, > but not doing so means that the objects they are treating > have to be locked. So this is the part that actually lead me here. Say you are implementing a network server, for example -- you don't want to have big spikes in the request latency due to GC. Not that Haskell is so much worse off relative to Java, say; Erlang is the only language I'm aware of that takes concurrent GC seriously. However, it seems that this problem is hard to solve for Haskell: Parallel GC is when the whole system stops and performs multi-threaded GC, as opposed to "concurrent GC", which is when the GC runs concurrently with the program. We think concurrent GC is unlikely to be practical in the Haskell setting, due to the extra synchronisation needed in the mutator. However, there may always be clever techniques that we haven't discovered, and synchronisation might become less expensive, so the balance may change in the future. -- Simon Marlow So I wonder, to what degree is GC latency controllable in Haskell? It seems that, pending further research, we can not hope for concurrent GC. > 2. About "Blackholing": in the sequential evaluation (where > hitting a blackhole indeed means to have a loop), some > better performance can be gained by not blackholing a thunk > immediately, so this was done in GHC earlier. However, it > increases the chance for 2 mutator threads to evaluate the > same thunk (double work), and we got better performance by > blackholing immediately. Can blackholing too early could result in non-termination ("...hitting a blackhole indeed means to have a loop")? Then it's not just a matter of performance when we do it? -- Jason Dusek From jules at jellybean.co.uk Fri Aug 7 03:51:21 2009 From: jules at jellybean.co.uk (Jules Bean) Date: Fri Aug 7 03:32:08 2009 Subject: Compiling large source files In-Reply-To: <638ABD0A29C8884A91BC5FB5C349B1C357F36E7BB5@EA-EXMSG-C334.europe.corp.microsoft.com> References: <4c44d90b0908031320v3f4ba180n2ebb3102b360d799@mail.gmail.com> <638ABD0A29C8884A91BC5FB5C349B1C357F36E7BB5@EA-EXMSG-C334.europe.corp.microsoft.com> Message-ID: <4A7BDCF9.7080800@jellybean.co.uk> Gunther should first check if he thinks this is the same issue as http://hackage.haskell.org/trac/ghc/ticket/2002 (looks similar to me) If it is, he could just add comments there. Simon Peyton-Jones wrote: > Gunther > > Could you make a Trac bug report for this, and attach your source file? > > It'd help if you could first check that things are still bad with GHC 6.10.4. > > Another useful thing would be to provide data on whether the behaviour is non-linear. Eg try with 1k, 2k, 4k, 8k, etc elements in your list and see how it behaves. > > Providing reproducible and well-characterised bug reports greatly increases the likelihood that we'll fix it! > > Thanks > > Simon > > > | -----Original Message----- > | From: glasgow-haskell-users-bounces@haskell.org [mailto:glasgow-haskell-users- > | bounces@haskell.org] On Behalf Of G?nther Schmidt > | Sent: 03 August 2009 22:09 > | To: Thomas DuBuisson > | Cc: glasgow-haskell-users@haskell.org > | Subject: Re: Compiling large source files > | > | Hi Thomas, > | > | yes, a source file with a single literal list with 85k elements. > | > | > | G?nther > | > | > | Am 03.08.2009, 22:20 Uhr, schrieb Thomas DuBuisson > | : > | > | > Can you define "very large" and "compiler"? I know an old version of > | > GHC (6.6?) would eat lots of memory when there were absurd numbers of > | > let statements. > | > > | > Thomas > | > > | > 2009/8/3 G?nther Schmidt : > | >> Hi all, > | >> > | >> I'm having trouble compiling very large source files, the compiler eats > | >> 2GB > | >> and then dies. Is there a way around it? > | >> > | >> G?nther > | >> _______________________________________________ > | >> Glasgow-haskell-users mailing list > | >> Glasgow-haskell-users@haskell.org > | >> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users > | >> > | > | > | _______________________________________________ > | Glasgow-haskell-users mailing list > | Glasgow-haskell-users@haskell.org > | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users From Christian.Maeder at dfki.de Fri Aug 7 04:55:31 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Fri Aug 7 04:36:15 2009 Subject: gcc version for GHC 6.10.4 under Sparc/Solaris In-Reply-To: <1249557898.20327.9340.camel@localhost> References: <4A7A8E83.8070502@dfki.de> <1249557898.20327.9340.camel@localhost> Message-ID: <4A7BEC03.9040902@dfki.de> Duncan Coutts wrote: > I should also note that there is a GHC 6.10.4 binary for Sparc/Linux > that is now included with Gentoo. It's got all features turned on except > for split objects (which fails due to mixing ld -r and --relax flags). > In particular it's a registerised via-C build with ghci, TH and > profiling working. Does compiling using gcc-4.3.x work if -fvia-C is added? > It's a distro package not a generic relocatable GHC binary tarball so > there's no point putting it on the ghc download page, but it's there > nevertheless if people want it (look for the gentoo ghc ebuild). I've found http://packages.gentoo.org/package/dev-lang/ghc Where are ebuilds or downloadable binaries? Cheers Christian From malcolm.wallace at cs.york.ac.uk Fri Aug 7 05:04:14 2009 From: malcolm.wallace at cs.york.ac.uk (Malcolm Wallace) Date: Fri Aug 7 04:44:55 2009 Subject: What is the mutator? In-Reply-To: <42784f260908061735k56fa93e9wd7ea16bf55296ca0@mail.gmail.com> References: <20090806163536.C88DE32493C@www.haskell.org> <4A7B14EF.3040107@mathematik.uni-marburg.de> <42784f260908061735k56fa93e9wd7ea16bf55296ca0@mail.gmail.com> Message-ID: > Say you are > implementing a network server, for example -- you don't want > to have big spikes in the request latency due to GC. > > We think > concurrent GC is unlikely to be practical in the Haskell > setting, due to the extra synchronisation needed in the > mutator. > -- Simon Marlow It is perfectly possible to do real-time concurrent garbage collection for Haskell, in an incremental fashion that guarantees a small maximum bound on each packet of GC work. The tradeoff is that the percentage of time devoted to GC in total is much greater, and the mutator must do more work to co-operate with the GC. In other words, the program runs slower. This tradeoff is the same for all real-time garbage collection schemes as far as I am aware, in any language - either you can have responsiveness, or you can have better overall application speed, but you cannot have both. > So I wonder, to what degree is GC latency controllable in > Haskell? It seems that, pending further research, we can not > hope for concurrent GC. There have been several papers on real-time GC in Haskell (including one of my own). There is no technical problem, only performance worries. This is what I think Simon means by "unlikely to be practical". Regards, Malcolm From Christian.Maeder at dfki.de Fri Aug 7 05:28:31 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Fri Aug 7 05:09:11 2009 Subject: use gtar and not tar under solaris In-Reply-To: <20090806200325.GA32222@petunia.outback.escape.de> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> <20090806170241.GB139@petunia.outback.escape.de> <4A7B26F9.4020501@dfki.de> <20090806200325.GA32222@petunia.outback.escape.de> Message-ID: <4A7BF3BF.4000205@dfki.de> Matthias Kilian wrote: > However, to create an archive, you can use something like > > $ pax -wf foo.tar directory Do you think "gtar --format=posix" would be different from pax? The only question is, if we should create archives using the ustar, posix/pax, or gnu format. ustar seems to be the least common denominator. Does ustar have any disadvantages? Cheers Christian From Christian.Maeder at dfki.de Fri Aug 7 07:14:02 2009 From: Christian.Maeder at dfki.de (Christian Maeder) Date: Fri Aug 7 06:54:45 2009 Subject: use gtar and not tar under solaris In-Reply-To: <4A7BF3BF.4000205@dfki.de> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> <20090806170241.GB139@petunia.outback.escape.de> <4A7B26F9.4020501@dfki.de> <20090806200325.GA32222@petunia.outback.escape.de> <4A7BF3BF.4000205@dfki.de> Message-ID: <4A7C0C7A.6090901@dfki.de> Christian Maeder wrote: > Matthias Kilian wrote: >> However, to create an archive, you can use something like >> >> $ pax -wf foo.tar directory > > Do you think "gtar --format=posix" would be different from pax? > > The only question is, if we should create archives using the ustar, > posix/pax, or gnu format. ustar seems to be the least common > denominator. Does ustar have any disadvantages? My plain tar command under solaris cannot handle the pax files, too. So ustar archives should be created (at least under solaris). But I don't know why the ustar format can handle long file names, whereas the gnu format creates a "@LongLink" file and pax a "PaxHeader" file (when unpacked with tar). Cheers Christian From greenspan.levi at googlemail.com Fri Aug 7 10:48:42 2009 From: greenspan.levi at googlemail.com (Levi Greenspan) Date: Fri Aug 7 10:29:22 2009 Subject: FFI: Problem with Signal Handler Interruptions Message-ID: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> Apologies for re-posting this subject here since I had sent already a message to haskell-caf? 3 days ago, but I just learned about this mailing list and it seems more appropriate to ask this question here I guess. I already got a reply from Simon Marlow but I posted some further (so far unanswered) questions. The thread starts here: http://www.haskell.org//pipermail/haskell-cafe/2009-August/064880.html. Basically my remaining questions are: 1. How can one safely perform a blocking wait on a system call via FFI when compiling with -threaded and avoid signal interruptions which cause the call to return with EINTR? 2. Could 1. be achieved by blocking SIGVTALRM in the thread doing the call? For details please cf. the original maling list thread. Many thanks, Levi From bulat.ziganshin at gmail.com Fri Aug 7 11:32:39 2009 From: bulat.ziganshin at gmail.com (Bulat Ziganshin) Date: Fri Aug 7 11:18:44 2009 Subject: FFI: Problem with Signal Handler Interruptions In-Reply-To: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> References: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> Message-ID: <1222289605.20090807193239@gmail.com> Hello Levi, Friday, August 7, 2009, 6:48:42 PM, you wrote: > 1. How can one safely perform a blocking wait on a system call via > FFI when compiling with -threaded i think you should use forkOS to create OS thread dedicated to Haskell thread > and avoid signal interruptions which cause the call to return with EINTR? don't know, but may be you can retry on this retcode or disable EINTR in one specific OS thread -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com From igloo at earth.li Fri Aug 7 13:14:54 2009 From: igloo at earth.li (Ian Lynagh) Date: Fri Aug 7 12:55:35 2009 Subject: use gtar and not tar under solaris In-Reply-To: <4A77EE13.80607@dfki.de> References: <4A77EE13.80607@dfki.de> Message-ID: <20090807171454.GA23408@matrix.chaos.earth.li> On Tue, Aug 04, 2009 at 10:15:15AM +0200, Christian Maeder wrote: > > Ian, could you place a note about using gtar for unpacking solaris > binary-dists? Done! Thanks Ian From igloo at earth.li Fri Aug 7 13:42:38 2009 From: igloo at earth.li (Ian Lynagh) Date: Fri Aug 7 13:23:17 2009 Subject: gcc version for GHC 6.10.4 under Sparc/Solaris In-Reply-To: <4A7A8E83.8070502@dfki.de> References: <4A7A8E83.8070502@dfki.de> Message-ID: <20090807174238.GA24373@matrix.chaos.earth.li> On Thu, Aug 06, 2009 at 10:04:19AM +0200, Christian Maeder wrote: > Hi Ian, > > could you add a note on the download page that > GCC version 4.3.x is not suited for: Done! Thanks Ian From jason.dusek at gmail.com Fri Aug 7 16:43:07 2009 From: jason.dusek at gmail.com (Jason Dusek) Date: Fri Aug 7 16:23:46 2009 Subject: What is the mutator? In-Reply-To: References: <20090806163536.C88DE32493C@www.haskell.org> <4A7B14EF.3040107@mathematik.uni-marburg.de> <42784f260908061735k56fa93e9wd7ea16bf55296ca0@mail.gmail.com> Message-ID: <42784f260908071343n104ced6dyb3d96426bc85193a@mail.gmail.com> 2009/08/07 Malcolm Wallace : > There have been several papers on real-time GC in Haskell > (including one of my own).?There is no technical problem, only > performance worries.?This is what I think Simon means by > "unlikely to be practical". So I guess there is no "right answer" here -- a realistic solution would offer two GC options, maybe set by an option to the RTS or on a per-thread level. I guess your work was specifically on garbage collection of functional languages in embedded systems? An incremental garbage collector for embedded real-time systems ftp://ftp.cs.york.ac.uk/pub/malcolm/rtgc.html Also there is the "Non-stop Haskell" paper. Non-stop Haskell http://research.microsoft.com/en-us/um/people/simonpj/Papers/inc-gc.htm Both of these papers are from some time ago; the most recent thing I can find is: Exploring the Barrier to Entry - Incremental Generational Garbage Collection for Haskell -- Jason Dusek From axman6 at gmail.com Fri Aug 7 23:08:55 2009 From: axman6 at gmail.com (Alex Mason) Date: Fri Aug 7 22:49:41 2009 Subject: What would be required to make a LLVM backend for GHC Message-ID: <31423307-836F-49B2-ACC0-2A03406A8FA9@gmail.com> Hi all, I've been talking to one of the LLVM developers, who's working on an operating system called AuroraUX, which, among other things, is trying to use LLVM as much as possible in the system (using clang as the default compiler, compiler-rt [libgcc replacement from the LLVM team], etc.). He would like to know exactly what would need to be implemented in LLVM to allow ghc to be ported to LLVM, and he would do his best to get these features implemented. I know the LLVM guys are quite open to suggestions, and want to get people using it as much as possible, so if we can let them know what we need, hopefully they can help make life easier for us. Thanks, Alex Mason From dons at galois.com Sat Aug 8 03:47:23 2009 From: dons at galois.com (Don Stewart) Date: Sat Aug 8 03:30:32 2009 Subject: What would be required to make a LLVM backend for GHC In-Reply-To: <31423307-836F-49B2-ACC0-2A03406A8FA9@gmail.com> References: <31423307-836F-49B2-ACC0-2A03406A8FA9@gmail.com> Message-ID: <20090808074723.GA8898@whirlpool.galois.com> axman6: > Hi all, > > I've been talking to one of the LLVM developers, who's working on an > operating system called AuroraUX, which, among other things, is trying > to use LLVM as much as possible in the system (using clang as the > default compiler, compiler-rt [libgcc replacement from the LLVM team], > etc.). > > He would like to know exactly what would need to be implemented in LLVM > to allow ghc to be ported to LLVM, and he would do his best to get these > features implemented. I know the LLVM guys are quite open to > suggestions, and want to get people using it as much as possible, so if > we can let them know what we need, hopefully they can help make life > easier for us. There's a project underway at UNSW to add an LLVM backend to GHC. You should talk to them :-) -- Don From greenspan.levi at googlemail.com Sat Aug 8 19:02:10 2009 From: greenspan.levi at googlemail.com (Levi Greenspan) Date: Sat Aug 8 18:42:46 2009 Subject: FFI: Problem with Signal Handler Interruptions In-Reply-To: <1222289605.20090807193239@gmail.com> References: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> <1222289605.20090807193239@gmail.com> Message-ID: <3e62d4360908081602q297695ebk93a79b45b2334eb8@mail.gmail.com> Hi Bulat, On Fri, Aug 7, 2009 at 5:32 PM, Bulat Ziganshin wrote: > Hello Levi, > > Friday, August 7, 2009, 6:48:42 PM, you wrote: > >> 1. ?How can one safely perform a blocking wait on a system call via >> FFI when compiling with -threaded > > i think you should use forkOS to create OS thread dedicated to Haskell > thread Yes, but that wouldn't help me with my problem of getting signal interruptions. Simon Marlow wrote in the haskell-caf? thread: "The SIGVTALRM signal is delivered to one (random) thread in the program [...]" [1] >> and avoid signal interruptions which cause the call to return with EINTR? > > don't know, but may be you can retry on this retcode or disable EINTR in one > specific OS thread Yes, currently I use throwErrnoIfMinus1Retry from Foreign.C.Error, but I am not sure if this is the recommended way. Seeing ... rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 nanosleep({3, 0}, 0xbfad005c) = ? ERESTART_RESTARTBLOCK (To be restarted) --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- sigreturn() = ? (mask now []) ... many times in the output of strace made me slightly uneasy. So right now I have two options which *seem* to work ok. One is to retry on EINTR, and the other to block SIGVTALRM as described in [2]. But am still unsure whether (i) this is correct and recommended, and (ii) there is no better option. Cheers, Levi --- [1] http://www.haskell.org//pipermail/haskell-cafe/2009-August/064971.html [2] http://www.haskell.org//pipermail/haskell-cafe/2009-August/065023.html From simonpj at microsoft.com Sun Aug 9 15:42:37 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Sun Aug 9 15:23:13 2009 Subject: [Haskell-cafe] generalize RecordPuns and RecordWildCards to work with qualified names? In-Reply-To: <5ab17e790907241648x77999baft925cac5d9af116d7@mail.gmail.com> References: <2518b95d0907171556l3745912na56481698094b2dd@mail.gmail.com> <638ABD0A29C8884A91BC5FB5C349B1C34B7FA6CABB@EA-EXMSG-C334.europe.corp.microsoft.com> <5ab17e790907241648x77999baft925cac5d9af116d7@mail.gmail.com> Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C35B30DEED90@EA-EXMSG-C334.europe.corp.microsoft.com> Oh, now I get it, thanks. This message concerns design choices for record-syntax-related GHC extensions. Lennart, pls tune in. You don?t need to have read the thread to understand this message. | I think that Even refers to an example like this: | | module A where | data A = A { a :: Int } | | The following works: | | {-# LANGUAGE NamedFieldPuns #-} | module B where | import A | f (A { a }) = a | | However, if we import "A" qualified, then punning does not seem to work: | | {-# LANGUAGE NamedFieldPuns #-} | module B where | import qualified A | f (A.A { a }) = a | | This results in: Not in scope: `a' Right. What is happening is that GHC looks up the first 'a' (the one on the LHS) and finds it not in scope. If you add -XDisambiguateRecordFields, it works fine. But admittedly, the error message is unhelpful. I could improve that. Now on to the suggested change: | {-# LANGUAGE NamedFieldPuns #-} | module B where | import qualified A | | f (A.A { A.a }) = a | | This results in: Qualified variable in pattern: A.a | | Even is suggesting that instead of reporting an error, in the second | case we could use the translation: | | f (A.A { A.a }) = a --> f (A.A { A.a = a }) | | (i.e., when punning occurs with a qualified name, use just the | unqualified part of the name in the pattern) Yes, that'd be possible. But it seems debatable -- it doesn't *look* as if the pattern (A.A { A.a }) binds 'a' -- and it seems even less desirable in record construction and update. To be concrete, would you expect these to work too? g a = A.A { A.a } --> g a = A.A { A.a = a } h x a = x { A.a } --> h x a = a { A.a = a } In these cases, I think the abbreviated code looks too confusing. With -XDisambiguateRecordFields you could say g a = A.A { a } which seems better. (But there's no help for record update, since we don?t know which data constructor is involved.) So my current conclusion is: improve the error message, perhaps suggesting the flag -XDismabiguateRecordFields, but don't add the change you suggest. Comments? Simon From lennart at augustsson.net Sun Aug 9 16:39:53 2009 From: lennart at augustsson.net (Lennart Augustsson) Date: Sun Aug 9 16:20:27 2009 Subject: [Haskell-cafe] generalize RecordPuns and RecordWildCards to work with qualified names? In-Reply-To: <638ABD0A29C8884A91BC5FB5C349B1C35B30DEED90@EA-EXMSG-C334.europe.corp.microsoft.com> References: <2518b95d0907171556l3745912na56481698094b2dd@mail.gmail.com> <638ABD0A29C8884A91BC5FB5C349B1C34B7FA6CABB@EA-EXMSG-C334.europe.corp.microsoft.com> <5ab17e790907241648x77999baft925cac5d9af116d7@mail.gmail.com> <638ABD0A29C8884A91BC5FB5C349B1C35B30DEED90@EA-EXMSG-C334.europe.corp.microsoft.com> Message-ID: At a minimum I think the error message should be better. I also think it would be natural to use the DisambiguateRecordFields for the places where RecordWildcards are used. I mean, if I change from unqualified import to a qualified one, and then change all visible names to be qualified I would expect things to still work. For RecordPuns I don't have an opinion on what to do. -- Lennart On Sun, Aug 9, 2009 at 9:42 PM, Simon Peyton-Jones wrote: > Oh, now I get it, thanks. ?This message concerns design choices for record-syntax-related GHC extensions. ?Lennart, pls tune in. ?You don?t need to have read the thread to understand this message. > > | I think that Even refers to an example like this: > | > | module A where > | ? data A = A { a :: Int } > | > | The following works: > | > | {-# LANGUAGE NamedFieldPuns #-} > | module B where > | ? import A > | ? f (A { a }) = a > | > | However, if we import "A" qualified, then punning does not seem to work: > | > | {-# LANGUAGE NamedFieldPuns #-} > | module B where > | ? import qualified A > | ? f (A.A { a }) = a > | > | This results in: Not in scope: `a' > > Right. ?What is happening is that GHC looks up the first 'a' (the one on the LHS) and finds it not in scope. ?If you add -XDisambiguateRecordFields, it works fine. ?But admittedly, the error message is unhelpful. ?I could improve that. > > Now on to the suggested change: > > | {-# LANGUAGE NamedFieldPuns #-} > | module B where > | ? import qualified A > | > | ? f (A.A { A.a }) = a > | > | This results in: Qualified variable in pattern: A.a > | > | Even is suggesting that instead of reporting an error, in the second > | case we could use the translation: > | > | ? f (A.A { A.a }) = a ? --> ? f (A.A { A.a = a }) > | > | (i.e., when punning occurs with a qualified name, use just the > | unqualified part of the name in the pattern) > > Yes, that'd be possible. ? But it seems debatable -- it doesn't *look* as if the pattern (A.A { A.a }) binds 'a' -- and it seems even less desirable in record construction and update. ?To be concrete, would you expect these to work too? > > ?g a = A.A { A.a } ? ? --> ? ?g a = A.A { A.a = a } > ?h x a = x { A.a } ? ? --> ? ?h x a = a { A.a = a } > > In these cases, I think the abbreviated code looks too confusing. > > With -XDisambiguateRecordFields you could say > > ?g a = A.A { a } > > which seems better. ?(But there's no help for record update, since we don?t know which data constructor is involved.) > > > So my current conclusion is: improve the error message, perhaps suggesting the flag -XDismabiguateRecordFields, but don't add the change you suggest. > > Comments? > > Simon > > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users > > From duncan.coutts at worc.ox.ac.uk Mon Aug 10 07:13:56 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Aug 10 06:53:58 2009 Subject: gcc version for GHC 6.10.4 under Sparc/Solaris In-Reply-To: <4A7BEC03.9040902@dfki.de> References: <4A7A8E83.8070502@dfki.de> <1249557898.20327.9340.camel@localhost> <4A7BEC03.9040902@dfki.de> Message-ID: <1249902836.20327.9425.camel@localhost> On Fri, 2009-08-07 at 10:55 +0200, Christian Maeder wrote: > Duncan Coutts wrote: > > I should also note that there is a GHC 6.10.4 binary for Sparc/Linux > > that is now included with Gentoo. It's got all features turned on except > > for split objects (which fails due to mixing ld -r and --relax flags). > > In particular it's a registerised via-C build with ghci, TH and > > profiling working. > > Does compiling using gcc-4.3.x work if -fvia-C is added? There's no Sparc NCG in the 6.10 series so it's only -fvia-C. The new Sparc NCG is in 6.12. This build used gcc-4.1.2 which is the latest stable one on Gentoo for sparc. > > It's a distro package not a generic relocatable GHC binary tarball so > > there's no point putting it on the ghc download page, but it's there > > nevertheless if people want it (look for the gentoo ghc ebuild). > > I've found > http://packages.gentoo.org/package/dev-lang/ghc > > Where are ebuilds or downloadable binaries? For non-gentoo users, the ebuilds are available from cvs: http://sources.gentoo.org/viewcvs.py/gentoo-x86/dev-lang/ghc/ eg 6.10.4: http://sources.gentoo.org/viewcvs.py/gentoo-x86/dev-lang/ghc/ghc-6.10.4.ebuild?view=markup The binaries are not set up for separate distribution but if you inspect the ebuild you find: mirror://gentoo/ghc-bin-${PV}-sparc.tbz2 which means ghc-bin-6.10.4-sparc.tbz2 in the distfiles of any gentoo mirror, eg: http://www.mirrorservice.org/sites/www.ibiblio.org/gentoo/distfiles/ghc-bin-6.10.4-sparc.tbz2 It expects to unpack to / and installs under /usr however it's quite possible to unpack to a temp dir and relocate the scripts and package.conf using a bit of sed. In fact this is what the ebuild does for the bootstrapping binary. Duncan From duncan.coutts at worc.ox.ac.uk Mon Aug 10 07:27:22 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Aug 10 07:07:21 2009 Subject: use gtar and not tar under solaris In-Reply-To: <4A7C0C7A.6090901@dfki.de> References: <4A77EE13.80607@dfki.de> <1249558251.20327.9352.camel@localhost> <20090806170241.GB139@petunia.outback.escape.de> <4A7B26F9.4020501@dfki.de> <20090806200325.GA32222@petunia.outback.escape.de> <4A7BF3BF.4000205@dfki.de> <4A7C0C7A.6090901@dfki.de> Message-ID: <1249903642.20327.9438.camel@localhost> On Fri, 2009-08-07 at 13:14 +0200, Christian Maeder wrote: > Christian Maeder wrote: > > Matthias Kilian wrote: > >> However, to create an archive, you can use something like > >> > >> $ pax -wf foo.tar directory > > > > Do you think "gtar --format=posix" would be different from pax? I would expect they are the same. The USTAR format is standardised by a POSIX standard from 1988 while the pax extensions are standardised by POSIX from 2001 I think. The pax program has an -x format flag and can use pax, ustar or cpio formats. The pax format is an extension of the ustar format. > > The only question is, if we should create archives using the ustar, > > posix/pax, or gnu format. ustar seems to be the least common > > denominator. Does ustar have any disadvantages? For source code distribution I think the ustar format is ideal. This is what cabal-install's sdist mode uses. As you say it's the lowest common denominator. The limitations of the format (file sizes, lack of extended file meta-data) are not a practical problem for source code or binaries. > My plain tar command under solaris cannot handle the pax files, too. > So ustar archives should be created (at least under solaris). That's odd since pax is supposed to be a compatible extension of ustar that just adds extra meta-data entries. Older programs should either ignore those entries or extract them as if they were ordinary files. > But I don't know why the ustar format can handle long file names, The ustar format can handle file names up to 100+155 characters (bytes) long. The reason for 100+155 is that it's not simply 255. The split into a 100 and 155 field must happen on a directory separator. > whereas the gnu format creates a "@LongLink" file and pax a "PaxHeader" > file (when unpacked with tar). Right, those "files" are the extended entries. Duncan From duncan.coutts at worc.ox.ac.uk Mon Aug 10 11:38:48 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Mon Aug 10 11:42:27 2009 Subject: Haskell Platform 2009.2.0.2 In-Reply-To: <4A719C58.2080700@dfki.de> References: <20090729174205.GB18734@whirlpool.galois.com> <4A719C58.2080700@dfki.de> Message-ID: <1249918728.20327.9518.camel@localhost> On Thu, 2009-07-30 at 15:12 +0200, Christian Maeder wrote: > Don Stewart wrote: > > Heads up lads, we're about 24 hours from Haskell Platform 2009.2.0.2 > > > > http://code.haskell.org/haskell-platform/haskell-platform.cabal > > I still see > > time ==1.1.2.4, > > although ghc-6.10.4 comes with: > > time-1.1.4 > > http://trac.haskell.org/haskell-platform/ticket/74 > > Is this on purpose (possibly installing two time versions)? Yes it is on purpose. The HP policy is to keep the same API for a whole major release series. That is, the API of packages included in each minor release of 2009.2.0.x are the same. The initial release 2009.2.0 included time-1.1.2.4 and so subsequent releases must use a compatible version. Version of time in the "extralibs" tarball in ghc-6.10.x releases: * GHC-6.10.1: time-1.1.2.4 * GHC-6.10.2: none * GHC-6.10.3: time-1.1.3 * GHC-6.10.4: time-1.1.4 So we have the current situation where the HP supplies one version and ghc's extralibs tarball includes a different version. As far as I know, the windows and osx HP installers (which include ghc) only include one version of time. For the source installer of course we cannot control other pre-existing versions. > I only know that time-1.1.4 and time-1.1.3 supply more Typeable > instances (which may break existing code) Right, that's not allowed in a HP minor release as it's an API change. Duncan From alex at alexjacobson.com Fri Aug 14 14:50:48 2009 From: alex at alexjacobson.com (S. Alexander Jacobson) Date: Fri Aug 14 14:31:07 2009 Subject: what is the path to a particular module? Message-ID: <4A85B208.9010108@alexjacobson.com> Is there a way from GHCi to discover the path to a particular module that you have imported or loaded? -Alex- From marlowsd at gmail.com Fri Aug 14 17:45:17 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Fri Aug 14 17:25:36 2009 Subject: FFI: Problem with Signal Handler Interruptions In-Reply-To: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> References: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> Message-ID: <4A85DAED.3000601@gmail.com> Levi Greenspan wrote: > Apologies for re-posting this subject here since I had sent already a > message to haskell-caf? 3 days ago, but I just learned about this > mailing list and it seems more appropriate to ask this question here I > guess. I already got a reply from Simon Marlow but I posted some > further (so far unanswered) questions. The thread starts here: > http://www.haskell.org//pipermail/haskell-cafe/2009-August/064880.html. > Basically my remaining questions are: > > 1. How can one safely perform a blocking wait on a system call via > FFI when compiling with -threaded and avoid signal interruptions which > cause the call to return with EINTR? You should check for EINTR and restart the call as necessary. This is standard best practice for POSIX programming anyway, and we do it for all the calls that can return EINTR in the IO library and RTS. > 2. Could 1. be achieved by blocking SIGVTALRM in the thread doing the call? Blocking SIGVTALRM is not a good idea: the RTS relies on the signal for various things. Cheers, Simon From marlowsd at gmail.com Fri Aug 14 17:47:10 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Fri Aug 14 17:27:25 2009 Subject: FFI: Problem with Signal Handler Interruptions In-Reply-To: <1222289605.20090807193239@gmail.com> References: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> <1222289605.20090807193239@gmail.com> Message-ID: <4A85DB5E.7060103@gmail.com> Bulat Ziganshin wrote: > Hello Levi, > > Friday, August 7, 2009, 6:48:42 PM, you wrote: > >> 1. How can one safely perform a blocking wait on a system call via >> FFI when compiling with -threaded > > i think you should use forkOS to create OS thread dedicated to Haskell > thread Why do you recommend forkOS here? forkOS is normally only necessary if you need to access thread-local state in the C call. Cheers, Simon From marlowsd at gmail.com Fri Aug 14 18:02:03 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Fri Aug 14 17:42:20 2009 Subject: What is the mutator? In-Reply-To: References: <20090806163536.C88DE32493C@www.haskell.org> <4A7B14EF.3040107@mathematik.uni-marburg.de> <42784f260908061735k56fa93e9wd7ea16bf55296ca0@mail.gmail.com> Message-ID: <4A85DEDB.2010707@gmail.com> Malcolm Wallace wrote: >> Say you are >> implementing a network server, for example -- you don't want >> to have big spikes in the request latency due to GC. >> >> We think >> concurrent GC is unlikely to be practical in the Haskell >> setting, due to the extra synchronisation needed in the >> mutator. >> -- Simon Marlow > > It is perfectly possible to do real-time concurrent garbage collection > for Haskell, in an incremental fashion that guarantees a small maximum > bound on each packet of GC work. The tradeoff is that the percentage of > time devoted to GC in total is much greater, and the mutator must do > more work to co-operate with the GC. In other words, the program runs > slower. This tradeoff is the same for all real-time garbage collection > schemes as far as I am aware, in any language - either you can have > responsiveness, or you can have better overall application speed, but > you cannot have both. > >> So I wonder, to what degree is GC latency controllable in >> Haskell? It seems that, pending further research, we can not >> hope for concurrent GC. > > There have been several papers on real-time GC in Haskell (including one > of my own). There is no technical problem, only performance worries. > This is what I think Simon means by "unlikely to be practical". You're quite right, I don't really mean "practical", more like "not cheap enough to replace the existing GC as the default". My current thoughts on reducing pause times are to adopt a region-style GC, where the heap is divided into regions and any subset of the regions can be collected in any GC cycle. This generalisation of generational GC is becoming quite popular amongst the Java folks in particular. Without going to proper incremental GC, this eliminates the need to do full-heap collections (but introduces a slight danger due to cycles between regions), and leads to shorter, or even bounded, pauses. Cheers, Simon From greenspan.levi at googlemail.com Sat Aug 15 09:44:10 2009 From: greenspan.levi at googlemail.com (Levi Greenspan) Date: Sat Aug 15 09:24:26 2009 Subject: FFI: Problem with Signal Handler Interruptions In-Reply-To: <4A85DAED.3000601@gmail.com> References: <3e62d4360908070748m6bc725b5l70c08cf8757348bd@mail.gmail.com> <4A85DAED.3000601@gmail.com> Message-ID: <3e62d4360908150644n151bfcbagd81910a6cab3719c@mail.gmail.com> On Fri, Aug 14, 2009 at 11:45 PM, Simon Marlow wrote: > You should check for EINTR and restart the call as necessary. ?This is > standard best practice for POSIX programming anyway, and we do it for all > the calls that can return EINTR in the IO library and RTS. [...] > Blocking SIGVTALRM is not a good idea: the RTS relies on the signal for > various things. Alright. I'll do it this way then. Many thanks for your advice. Cheers, Levi From phercek at gmail.com Sat Aug 15 12:16:16 2009 From: phercek at gmail.com (Peter Hercek) Date: Sat Aug 15 11:56:45 2009 Subject: what is the path to a particular module? References: <4A85B208.9010108@alexjacobson.com> Message-ID: On Fri, 14 Aug 2009 14:50:48 -0400, S. Alexander Jacobson wrote: > Is there a way from GHCi to discover the path to a particular module > that you have imported or loaded? It may not be what you wanted but in the worst case you can use ghc-pkg list to see all the package names then for each name you can do ghc-pkg describe to see there it is installed and what modules it does expose. All the information is in /package.conf which just seems to be a list of InstalledPackageInfo (Distribution.InstalledPackageInfo). So it should be possible to write a function you want easily. Surprisingly when I tried to read the list of InstalledPackageInfo from my package.conf it failed despite ghc-pkg working well. I did not investigate why. Maybe there is a better way. Peter. From thomas.dubuisson at gmail.com Sat Aug 15 13:18:33 2009 From: thomas.dubuisson at gmail.com (Thomas DuBuisson) Date: Sat Aug 15 12:58:48 2009 Subject: what is the path to a particular module? In-Reply-To: <4A85B208.9010108@alexjacobson.com> References: <4A85B208.9010108@alexjacobson.com> Message-ID: <4c44d90b0908151018y205931bfje225e9e156d0b9c0@mail.gmail.com> > Is there a way from GHCi to discover the path to a particular module that > you have imported or loaded? If you do "ghci -v4" it will print out all the exposed modules right below the package that exposes them and the directory the package is in. You could just copy/paste this text and do a simple text search in your edit or choice. I don't know how to get it to print this information when you load a module, but GHCi is easy enough to read so perhaps you could make a patch and send it in! Thomas From jsch at informatik.uni-kiel.de Tue Aug 18 06:20:02 2009 From: jsch at informatik.uni-kiel.de (Jan =?iso-8859-1?Q?Schauml=F6ffel?=) Date: Tue Aug 18 06:00:10 2009 Subject: Generating valid Haskell code using the GHC API pretty printer In-Reply-To: <638ABD0A29C8884A91BC5FB5C349B1C34B7FA6CD96@EA-EXMSG-C334.europe.corp.microsoft.com> References: <20090722132828.GA11619@chuck> <638ABD0A29C8884A91BC5FB5C349B1C34B7FA6CD96@EA-EXMSG-C334.europe.corp.microsoft.com> Message-ID: <20090818102002.GA8422@chuck> Hello, sorry for the delay in replying, I have been away from my computer during the last weeks. Simon Peyton-Jones wrote: > I've fixed GHC's pretty-printer to print do-notation using braces > and semi-colons, which is much more robust. I hope that's useful This is certainly useful, and it seems to fix the issues for me. Thank you very much! Thomas Schilling wrote: > That said, if you're trying to do source-to-source transformations > you probably want to keep the original layout as much as possible. > GHC's pretty-printer isn't designed for that. GHC's syntax tree has > very accurate source locations, so you could start from there and > build your own pretty printer. Niklas Broberg wrote: > In general, unless you actually want to use any other components of > the GHC API, e.g. evaluate your code, then I see no reason to use > the GHC API for source manipulation. haskell-src-exts simply does > that better (and definitely better than haskell-src). But I couldn't > tell if that's enough for the original poster's needs. :-) Well, the application is a debugger for Concurrent Haskell programs. It reads a program and replaces certain function calls with calls to location-aware wrappers that allow stepping the program and highlighting the original source location. The result of this transformation is then compiled and executed; it is not intended to be seen by the user (who is shown the original source). So I guess what we need is not a "pretty printer" but a "valid printer". We need accurate original source locations and we need to access the syntax tree after typechecking to be able to tell which functions to replace. The GHC API looked like the way to go and it does so even more with Simon's patch. Thanks for your input, Jan -- If you're happy and you know it, syntax error! From igloo at earth.li Tue Aug 18 10:03:43 2009 From: igloo at earth.li (Ian Lynagh) Date: Tue Aug 18 09:43:49 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 Message-ID: <20090818140343.GA2835@matrix.chaos.earth.li> Hi all, This is a summary of our plans for GHC 6.12.1. We are aiming to have the first release candidate out on the 14th September 2009. Until then, we plan to focus on the bugs in the 6.12.1 milestone, marked high priority; they are listed here: http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&priority=highest&priority=high&milestone=6.12.1&order=priority If there is a bug not in that list that is important to you, please let us know. Thanks Ian From kili at outback.escape.de Tue Aug 18 13:00:08 2009 From: kili at outback.escape.de (Matthias Kilian) Date: Tue Aug 18 12:41:09 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 In-Reply-To: <20090818140343.GA2835@matrix.chaos.earth.li> References: <20090818140343.GA2835@matrix.chaos.earth.li> Message-ID: <20090818170007.GA8764@petunia.outback.escape.de> On Tue, Aug 18, 2009 at 03:03:43PM +0100, Ian Lynagh wrote: > This is a summary of our plans for GHC 6.12.1. > > We are aiming to have the first release candidate out on the 14th > September 2009. > > Until then, we plan to focus on the bugs in the 6.12.1 milestone, marked > high priority; they are listed here: > > http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&priority=highest&priority=high&milestone=6.12.1&order=priority Would it be better if I break my stable build slave in favor of the head build slave then? IIRC, I've to tweak some environment variables to make head build again, but it would break the stable builds. Ciao, Kili -- _|_ is pronounced 'bottom', and is the greatest lower bound of a complete lattice. -- Nick Williams in comp.lang.functional From catamorphism at gmail.com Tue Aug 18 19:15:08 2009 From: catamorphism at gmail.com (Tim Chevalier) Date: Tue Aug 18 18:55:13 2009 Subject: Building just a stage 1 compiler? Message-ID: <4683d9370908181615j3c0fe2d2vea1d6f4043b53a3@mail.gmail.com> Hi, Is there a way to tell the GHC build system that I only want to build a stage 1 compiler and the libraries, not a stage 2 compiler? Executing: $ sh boot $ ./configure $ make stage=1 in my build tree still builds the stage 2 compiler as well. Thanks, Tim -- Tim Chevalier * http://cs.pdx.edu/~tjc/ * Often in error, never in doubt From phercek at gmail.com Wed Aug 19 05:15:31 2009 From: phercek at gmail.com (Peter Hercek) Date: Wed Aug 19 04:57:01 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 In-Reply-To: <20090818140343.GA2835@matrix.chaos.earth.li> References: <20090818140343.GA2835@matrix.chaos.earth.li> Message-ID: Ian Lynagh wrote: > We are aiming to have the first release candidate out on the 14th > September 2009. > > Until then, we plan to focus on the bugs in the 6.12.1 milestone, marked > high priority; they are listed here: > > http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&priority=highest&priority=high&milestone=6.12.1&order=priority > > If there is a bug not in that list that is important to you, please let > us know. Could you please merge ticket #3084: allow macros to redefine builtin GHCi commands http://hackage.haskell.org/trac/ghc/ticket/3084 It was agreed for 6.12.1 and the solution is attached. The solution did not result in any merge conflicts when I updated ghc.head a week ago. It did not cause any new errors in the validate script. I use it locally from the time the ticket was opened and I'm not aware of any problems. You might also consider ticket #3434: improve vi tags (add non-exported symbols, add tag kinds, add regex tags) http://hackage.haskell.org/trac/ghc/ticket/3434 There is a link to the solution from the ticket but this one is not approved. It does not result in any new errors in the validate script. But the solution may have impact on emacs tag generation and I tested the tags for emacs only in vim (vim recognizes them too). Thanks, Peter. From phercek at gmail.com Wed Aug 19 05:18:15 2009 From: phercek at gmail.com (Peter Hercek) Date: Wed Aug 19 05:00:08 2009 Subject: few questions about GHC RTS Message-ID: Hi, I thought I may look at the :next command for GHCi debugger again. If I do, I will not make it before 6.12.1 is out. I have few questions about RTS in relation to :next implementation. If they is easy to answer I would appreciate it, if not, don't bother since I'm not sure I'll do it ... and If I decide to do it I may just figure it out myself or fail again :) Why is stg_nofoceIO_info added as last argument to IO actions in unregistered mode? Do I still need to pass it in even when (I think) my IO action does not need it? E.g. is it required for every IO action by some stack walking code or something? Are there any Cmm functions in RTS with signature (IO()) or signature (Int->IO()) which I could check out as examples to see how they are done? ... and ideally also how they are called from the stack. Are there any special things to do when adding a new field (e.g. simulated stack size) to StgTSO? Thanks, Peter. From simonpj at microsoft.com Wed Aug 19 05:28:35 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Wed Aug 19 05:08:39 2009 Subject: Building just a stage 1 compiler? In-Reply-To: <4683d9370908181615j3c0fe2d2vea1d6f4043b53a3@mail.gmail.com> References: <4683d9370908181615j3c0fe2d2vea1d6f4043b53a3@mail.gmail.com> Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C35CE71012B4@EA-EXMSG-C334.europe.corp.microsoft.com> Yes indeed http://hackage.haskell.org/trac/ghc/wiki/Building/Using#RebuildingtheGHCbinaryaftermakingchanges | -----Original Message----- | From: glasgow-haskell-users-bounces@haskell.org [mailto:glasgow-haskell-users- | bounces@haskell.org] On Behalf Of Tim Chevalier | Sent: 19 August 2009 00:15 | To: GHC Users Mailing List | Subject: Building just a stage 1 compiler? | | Hi, | | Is there a way to tell the GHC build system that I only want to build | a stage 1 compiler and the libraries, not a stage 2 compiler? | Executing: | $ sh boot | $ ./configure | $ make stage=1 | | in my build tree still builds the stage 2 compiler as well. | | Thanks, | Tim | | -- | Tim Chevalier * http://cs.pdx.edu/~tjc/ * Often in error, never in doubt | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users@haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users From marlowsd at gmail.com Wed Aug 19 06:58:40 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Aug 19 06:38:47 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 In-Reply-To: References: <20090818140343.GA2835@matrix.chaos.earth.li> Message-ID: <4A8BDAE0.1060005@gmail.com> On 19/08/2009 10:15, Peter Hercek wrote: > Ian Lynagh wrote: >> We are aiming to have the first release candidate out on the 14th >> September 2009. >> >> Until then, we plan to focus on the bugs in the 6.12.1 milestone, marked >> high priority; they are listed here: >> >> http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&priority=highest&priority=high&milestone=6.12.1&order=priority >> >> >> If there is a bug not in that list that is important to you, please let >> us know. > > Could you please merge ticket #3084: allow macros to redefine builtin > GHCi commands > http://hackage.haskell.org/trac/ghc/ticket/3084 > > It was agreed for 6.12.1 and the solution is attached. The solution did > not result in any merge conflicts when I updated ghc.head a week ago. It > did not cause any new errors in the validate script. I use it locally > from the time the ticket was opened and I'm not aware of any problems. > > You might also consider ticket #3434: improve vi tags (add non-exported > symbols, add tag kinds, add regex tags) > http://hackage.haskell.org/trac/ghc/ticket/3434 > > There is a link to the solution from the ticket but this one is not > approved. It does not result in any new errors in the validate script. > But the solution may have impact on emacs tag generation and I tested > the tags for emacs only in vim (vim recognizes them too). Sounds ok to me. If the patch goes in, I can do a quick test of :etags under emacs. Cheers, Simon From marlowsd at gmail.com Wed Aug 19 07:06:41 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Aug 19 06:46:46 2009 Subject: Building just a stage 1 compiler? In-Reply-To: <4683d9370908181615j3c0fe2d2vea1d6f4043b53a3@mail.gmail.com> References: <4683d9370908181615j3c0fe2d2vea1d6f4043b53a3@mail.gmail.com> Message-ID: <4A8BDCC1.9070303@gmail.com> On 19/08/2009 00:15, Tim Chevalier wrote: > Hi, > > Is there a way to tell the GHC build system that I only want to build > a stage 1 compiler and the libraries, not a stage 2 compiler? > Executing: > $ sh boot > $ ./configure > $ make stage=1 > > in my build tree still builds the stage 2 compiler as well. I'm surprised that the above builds stage 2. I'd expect it to fail, because stage=1 will omit the build instructions for stage 2, and then the build system will try to build things like Haddock which depend on stage 2, at which point the build should fail. There's no way at the moment to build just stage1 and the libraries from the top-level, but that could be added. You can 'cd ghc; make stage=1' and then 'cd libraries; make'. Cheers, Simon From marlowsd at gmail.com Wed Aug 19 07:21:31 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Aug 19 07:01:36 2009 Subject: few questions about GHC RTS In-Reply-To: References: Message-ID: <4A8BE03B.4020001@gmail.com> On 19/08/2009 10:18, Peter Hercek wrote: > Hi, > > I thought I may look at the :next command for GHCi debugger again. If I > do, I will not make it before 6.12.1 is out. I have few questions about > RTS in relation to :next implementation. If they is easy to answer I > would appreciate it, if not, don't bother since I'm not sure I'll do it > ... and If I decide to do it I may just figure it out myself or fail > again :) > > Why is stg_nofoceIO_info added as last argument to IO actions in > unregistered mode? Do I still need to pass it in even when (I think) my > IO action does not need it? E.g. is it required for every IO action by > some stack walking code or something? The comment in Interpreter.c says // Note [unreg]: in unregisterised mode, the return // convention for IO is different. The // stg_noForceIO_info stack frame is necessary to // account for this difference. However, the return convention for IO is now the same under both registerised and unregisterised builds (I made the change becuase of the proliferation of obscure conditional RTS code like the above), so I'm guessing the stg_noforceIO_info hack, and the above comment, are now redundant. It needs testing though. > Are there any Cmm functions in RTS with signature (IO()) or signature > (Int->IO()) which I could check out as examples to see how they are > done? ... and ideally also how they are called from the stack. See e.g. stg_catchzh in Exceptions.cmm, which needs to call an IO action. It tail-calls it though, which is perhaps not what you're after. To invoke an IO operation in the RTS, you just need to apply it to the RealWorld token, which in practice means applying it to a void argument, which you can do by loading it into R1 and tail-calling stg_ap_v_fast. > Are there any special things to do when adding a new field (e.g. > simulated stack size) to StgTSO? Just remember to rebuild enough things, because this will change the offsets of certain TSO fields, which are baked into the generated code for certain operations. I don't think GHC will force a rebuild of the libraries in this case (the build system will recompile everything, but the recompilation checker doesn't know that these offsets have changed) so you'll need to make clean in the libraries. Oh, and if the design is at all non-obvious, then writing a wiki page and asking for review would be a good first step. Adding fields to StgTSO is a pretty big change to make, especially if there are new invariants that have to be maintained, so a good justification will be needed. Cheers, Simon From igloo at earth.li Wed Aug 19 19:59:30 2009 From: igloo at earth.li (Ian Lynagh) Date: Wed Aug 19 19:39:33 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 In-Reply-To: <20090818170007.GA8764@petunia.outback.escape.de> References: <20090818140343.GA2835@matrix.chaos.earth.li> <20090818170007.GA8764@petunia.outback.escape.de> Message-ID: <20090819235930.GA5990@matrix.chaos.earth.li> On Tue, Aug 18, 2009 at 07:00:08PM +0200, Matthias Kilian wrote: > On Tue, Aug 18, 2009 at 03:03:43PM +0100, Ian Lynagh wrote: > > This is a summary of our plans for GHC 6.12.1. > > > > We are aiming to have the first release candidate out on the 14th > > September 2009. > > > > Until then, we plan to focus on the bugs in the 6.12.1 milestone, marked > > high priority; they are listed here: > > > > http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&priority=highest&priority=high&milestone=6.12.1&order=priority > > Would it be better if I break my stable build slave in favor of the > head build slave then? Yes please; the HEAD is what we're focussing on at the moment, and the next release will be from the (not yet created) 6.12 branch. Thanks Ian From simonpj at microsoft.com Thu Aug 20 10:29:23 2009 From: simonpj at microsoft.com (Simon Peyton-Jones) Date: Thu Aug 20 10:09:24 2009 Subject: [Haskell-cafe] generalize RecordPuns and RecordWildCards to work with qualified names? In-Reply-To: <2518b95d0908121558j477f061o591a4267efc40f4@mail.gmail.com> References: <2518b95d0907171556l3745912na56481698094b2dd@mail.gmail.com> <638ABD0A29C8884A91BC5FB5C349B1C34B7FA6CABB@EA-EXMSG-C334.europe.corp.microsoft.com> <5ab17e790907241648x77999baft925cac5d9af116d7@mail.gmail.com> <638ABD0A29C8884A91BC5FB5C349B1C35B30DEED90@EA-EXMSG-C334.europe.corp.microsoft.com> <2518b95d0908121558j477f061o591a4267efc40f4@mail.gmail.com> Message-ID: <638ABD0A29C8884A91BC5FB5C349B1C35CE7101966@EA-EXMSG-C334.europe.corp.microsoft.com> Evan, Lennart Thanks for the provocation. I've committed a patch that fixes all these issues. Try now! Simon Thu Aug 20 13:34:43 BST 2009 simonpj@microsoft.com * Improvements to record puns, wildcards * Make C { A.a } work with punning, expanding to C { A.a = a } * Make it so that, with -fwarn-unused-matches, f (C {..}) = x does not complain about the bindings introduced by the "..". * Make -XRecordWildCards implies -XDisambiguateRecordFields. * Overall refactoring of RnPat, which had become very crufty. In particular, there is now a monad, CpsRn, private to RnPat, which deals with the cps-style plumbing. This is why so many lines of RnPat have changed. * Refactor the treatment of renaming of record fields into two passes - rnHsRecFields1, used both for patterns and expressions, which expands puns, wild-cards - a local renamer in RnPat for fields in patterns - a local renamer in RnExpr for fields in construction and update This make it all MUCH easier to understand * Improve documentation of record puns, wildcards, and disambiguation | -----Original Message----- | From: haskell-cafe-bounces@haskell.org [mailto:haskell-cafe-bounces@haskell.org] On | Behalf Of Evan Laforge | Sent: 12 August 2009 23:59 | To: Simon Peyton-Jones | Cc: Augustsson, Lennart; haskell; GHC users | Subject: Re: [Haskell-cafe] generalize RecordPuns and RecordWildCards to work with | qualified names? | | > | Even is suggesting that instead of reporting an error, in the second | > | case we could use the translation: | > | | > | ? f (A.A { A.a }) = a ? --> ? f (A.A { A.a = a }) | > | | > | (i.e., when punning occurs with a qualified name, use just the | > | unqualified part of the name in the pattern) | > | > Yes, that'd be possible. ? But it seems debatable -- it doesn't *look* as if the | pattern (A.A { A.a }) binds 'a' -- and it seems even less desirable in record | construction and update. ?To be concrete, would you expect these to work too? | > | > ?g a = A.A { A.a } ? ? --> ? ?g a = A.A { A.a = a } | > ?h x a = x { A.a } ? ? --> ? ?h x a = a { A.a = a } | | Oh, I didn't realize that record punning included construction as | well. Yeah, that's a little funky looking. I don't mind seeing the | binding form and I think a new reader could figure it out without too | much trouble but I would be a little confused by the construction form | and think a new reader would also be confused. | | > With -XDisambiguateRecordFields you could say | > | > ?g a = A.A { a } | > | > which seems better. ?(But there's no help for record update, since we don't know | which data constructor is involved.) | | I didn't know about DisambiguateRecordFields! Looks like that also | makes the wildcard work like I want it to. | | The ghc docs for DisambiguateRecordFields don't make this very clear | to me... it talks about disambiguating names in scope, but if I say | "R.R { a = val}" I wouldn't expect it to "disambiguate" 'a', which is | not in scope at all, to 'R.a' which looks like a completely different | name. Rereading the paragraph at 7.3.11 I'm still surprised this | works. Maybe add something like: | | ... preceeding docs ... | | This also means that if you use qualified imports you can still use | unqualified field names. E.g. in the pattern @(R.R { a = a_val })@, | @a@ will be disambiguated to @R.a@, even if @R@ is imported qualified. | | I gather we're not supposed to call them "records" anymore, they're | supposed to be something I forget now, but the rest of the ghc docs | says records, so... | | > So my current conclusion is: improve the error message, perhaps suggesting the | flag -XDismabiguateRecordFields, but don't add the change you suggest. | > | > Comments? | | Sounds good to me. I'll try adding DisambiguateRecordFields and try | out the new punning, thanks! | _______________________________________________ | Haskell-Cafe mailing list | Haskell-Cafe@haskell.org | http://www.haskell.org/mailman/listinfo/haskell-cafe From p.k.f.holzenspies at utwente.nl Thu Aug 20 17:43:40 2009 From: p.k.f.holzenspies at utwente.nl (=?ISO-8859-1?Q?=22Philip_K.F._H=F6lzenspies=22?=) Date: Thu Aug 20 17:23:45 2009 Subject: Build error in users guide in HEAD of today Message-ID: Dear GHC-ers, I'm trying to put together a virtual machine with a default development for GHC hacking. To test it, I was running "sh validate" in the build tree. There were build errors in the doc/users_guide, however. Google gave me an hpaste page with the error on it, but I couldn't find the referring message. Does this ring a bell for anyone: latex failed users_guide_tmp.tex:1380: Undefined control sequence \Documents. users_guide_tmp.tex:1380: leading text: } users_guide_tmp.tex:1380: Undefined control sequence \user. users_guide_tmp.tex:1380: leading text: } users_guide_tmp.tex:5782: Undefined control sequence \Documents. users_guide_tmp.tex:5782: leading text: ...kurl{C: \Documents~And~Settings\user\ghc} users_guide_tmp.tex:5782: Undefined control sequence \user. users_guide_tmp.tex:5782: leading text: ...kurl{C: \Documents~And~Settings\user\ghc} users_guide_tmp.tex:5782: Undefined control sequence \ghc. users_guide_tmp.tex:5782: leading text: ...kurl{C: \Documents~And~Settings\user\ghc} users_guide_tmp.tex: File ended while scanning use of \hyper@n@rmalise. users_guide_tmp.tex: Emergency stop. Error: latex compilation failed make[2]: *** [docs/users_guide/users_guide.ps] Error 1 make[1]: *** [all_docs/users_guide] Error 2 I understand the LaTeX errors, but I'm not so well versed in docbook. Should the path in the ghci.xml and packages.xml be forward slashed, instead of backslashed or does this indicate there's something wrong with my docbook/latex setup? Even when I change the backslashes to forward slashes, the last error (scanning use of \hyper@n@rmalise) remains. Any ideas? Regards, Philip From p.k.f.holzenspies at utwente.nl Fri Aug 21 06:47:37 2009 From: p.k.f.holzenspies at utwente.nl (=?ISO-8859-1?Q?=22Philip_K.F._H=F6lzenspies=22?=) Date: Fri Aug 21 06:27:39 2009 Subject: Build error in users guide in HEAD of today In-Reply-To: References: Message-ID: <733F2428-5D6D-4F73-BF81-FCC97F253974@utwente.nl> On Aug 20, 2009, at 11:43 PM, Philip K.F. H?lzenspies wrote: > latex failed > users_guide_tmp.tex:1380: Undefined control sequence \Documents. > users_guide_tmp.tex:1380: leading text: } > users_guide_tmp.tex:1380: Undefined control sequence \user. > users_guide_tmp.tex:1380: leading text: } > users_guide_tmp.tex:5782: Undefined control sequence \Documents. > users_guide_tmp.tex:5782: leading text: ...kurl{C: > \Documents~And~Settings\user\ghc} > users_guide_tmp.tex:5782: Undefined control sequence \user. > users_guide_tmp.tex:5782: leading text: ...kurl{C: > \Documents~And~Settings\user\ghc} > users_guide_tmp.tex:5782: Undefined control sequence \ghc. > users_guide_tmp.tex:5782: leading text: ...kurl{C: > \Documents~And~Settings\user\ghc} > users_guide_tmp.tex: File ended while scanning use of > \hyper@n@rmalise. > users_guide_tmp.tex: Emergency stop. > Error: latex compilation failed > make[2]: *** [docs/users_guide/users_guide.ps] Error 1 > make[1]: *** [all_docs/users_guide] Error 2 Dear GHCers, It seems the last error is also due to backslashes in windows paths in the documentation. When I replace latex with a script that interrupts for user interaction and manually change all paths in /tmp/tmpXXXXXX/ users_guide_tmp.tex to have double backslashes, it all works fine. Is there a problem with the hyperref package that fails to properly escape (i.e. insufficient verbatim) the contents of the \nolinkurl{} macro, or should docbook have escaped the backslashes? Regards, Philip From christiaan.baaij at gmail.com Fri Aug 21 09:07:06 2009 From: christiaan.baaij at gmail.com (Christiaan Baaij) Date: Fri Aug 21 08:47:06 2009 Subject: Build error in users guide in HEAD of today In-Reply-To: <733F2428-5D6D-4F73-BF81-FCC97F253974@utwente.nl> References: <733F2428-5D6D-4F73-BF81-FCC97F253974@utwente.nl> Message-ID: <2EAAA643-571E-4269-80C3-CFFE0A3F3F2D@gmail.com> Hi, I had the same problem. I'm using the dblatex (version 0.2.10) from macports (on OS X 10.5.7), which isn't the latest version of dblatex. Also I'm using the MacTex 2008 Tex distribution, but I have no idea what the particular versions of the url or hyperref package are. Anyhow, I solved this particular problem by changing the style sheets in my dblatex install. I found them under: /usr/local/share/dblatex/xsl In the 'url.xsl' stylesheet I replaced the nolinkurl template with the following: \protect\nolinkurl{ } So yeah, basically it is a mechanization of your manual solution. Maybe you should bring this up with the dblatex folks? They have this bug report on it [1], but the appearently they could not reproduce it with 0.2.11. I can't verify if it is indeed solved with 0.2.11 and onwards, and have no intention of doing so as I'm satisfied with my current solution. Hope this helps you for now. Regards, Christiaan [1] http://sourceforge.net/tracker/index.php?func=detail&aid=2028163&group_id=72607&atid=535062 From ravi at bluespec.com Fri Aug 21 16:34:24 2009 From: ravi at bluespec.com (Ravi Nanavati) Date: Fri Aug 21 16:14:20 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 In-Reply-To: <20090818140343.GA2835@matrix.chaos.earth.li> References: <20090818140343.GA2835@matrix.chaos.earth.li> Message-ID: <7b977d860908211334p4bea9d57ufb29296d6bd46b55@mail.gmail.com> Hypothetically speaking (since I haven't made the call yet), if I wanted to be an "early adopter" of 6.12.1 when it is released, would now would be the time to start grabbing HEAD and playing with it (so that I know what I'm probably getting into and can offer early feedback)? Or would it be better for me to wait for the release candidate for that (since external packages and the like might be an issue at this point)? Thanks, - Ravi On Tue, Aug 18, 2009 at 10:03 AM, Ian Lynagh wrote: > > Hi all, > > This is a summary of our plans for GHC 6.12.1. > > We are aiming to have the first release candidate out on the 14th > September 2009. > > Until then, we plan to focus on the bugs in the 6.12.1 milestone, marked > high priority; they are listed here: > > http://hackage.haskell.org/trac/ghc/query?status=new&status=assigned&status=reopened&priority=highest&priority=high&milestone=6.12.1&order=priority > > If there is a bug not in that list that is important to you, please let > us know. > > > Thanks > Ian > > _______________________________________________ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users > From jvlask at hotmail.com Sat Aug 22 01:52:37 2009 From: jvlask at hotmail.com (John Lask) Date: Sat Aug 22 01:36:32 2009 Subject: ghc 6.10.4 infix declarations and '\' bug or not ? Message-ID: in declaring fixity for an operator (\\) to get it to compile using ghc 6.10.4, I needed to use the following code infixl 9 \\\ (\\) a b = etc ... where I assume the first \ escapes the second \, using infixl 9 \\ generates a syntax error infixl 9 \\ used to compile no problems with ghc 6.8.2 what is going on here ? why do I now need to add an additional \ ?, where is this change documented ?, is this a bug? From duncan.coutts at worc.ox.ac.uk Sat Aug 22 10:34:48 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Sat Aug 22 10:50:07 2009 Subject: ghc 6.10.4 infix declarations and '\' bug or not ? In-Reply-To: References: Message-ID: <1250951688.30910.2660.camel@localhost> On Sat, 2009-08-22 at 15:52 +1000, John Lask wrote: > in declaring fixity for an operator (\\) to get it to compile using ghc > 6.10.4, I needed to use the following code > > infixl 9 \\\ > (\\) a b = etc ... > > where I assume the first \ escapes the second \, using infixl 9 \\ generates > a syntax error > > infixl 9 \\ used to compile no problems with ghc 6.8.2 > > what is going on here ? Usually this problem is related to cpp, since \\ at the end of a line has special meaning to cpp. Are you sure that you're comparing like with like when you say it worked in ghc-6.8.2? If you're using cpp for the module now and in the past you were not then that would explain it. Another trick I've seen is: infixl 9 \\ -- this comment is here to defeat evil cpp mangling Duncan From jvlask at hotmail.com Sat Aug 22 20:16:50 2009 From: jvlask at hotmail.com (John Lask) Date: Sat Aug 22 20:00:39 2009 Subject: ghc 6.10.4 infix declarations and '\' bug or not ? References: <1250951688.30910.2660.camel@localhost> Message-ID: indeed it was the cpp, and the behaviour is consistent beetween versions, I must have introduced some changes that affected the eol marker. Thanks for your tip. jvl ----- Original Message ----- From: "Duncan Coutts" To: "John Lask" Cc: Sent: Sunday, August 23, 2009 12:34 AM Subject: Re: ghc 6.10.4 infix declarations and '\' bug or not ? > On Sat, 2009-08-22 at 15:52 +1000, John Lask wrote: >> in declaring fixity for an operator (\\) to get it to compile using ghc >> 6.10.4, I needed to use the following code >> >> infixl 9 \\\ >> (\\) a b = etc ... >> >> where I assume the first \ escapes the second \, using infixl 9 \\ >> generates >> a syntax error >> >> infixl 9 \\ used to compile no problems with ghc 6.8.2 >> >> what is going on here ? > > Usually this problem is related to cpp, since \\ at the end of a line > has special meaning to cpp. > > Are you sure that you're comparing like with like when you say it worked > in ghc-6.8.2? If you're using cpp for the module now and in the past you > were not then that would explain it. > > Another trick I've seen is: > > infixl 9 \\ -- this comment is here to defeat evil cpp mangling > > Duncan > > From mauricio.antunes at gmail.com Tue Aug 25 22:20:17 2009 From: mauricio.antunes at gmail.com (=?ISO-8859-1?Q?Maur=ED=ADcio_CA?=) Date: Tue Aug 25 22:04:51 2009 Subject: GHCi can't see 'atexit', can't load module Message-ID: (I left this in cafe, but I think it is actually ghc related.) I just checked it seems not to be possible to load any module in GHCi that uses FFI to wrap the standard C function 'atexit'. When trying that, we get a message saying the symbol 'atexit' can't be found. (This is not a problem when building an executable with ghc, though.) It's easy to reproduce the problem. Just try something like this: ----- module XXX where import Foreign import Foreign.C foreign import ccall atexit :: FunPtr (IO ()) -> IO CInt ----- Load it in ghci (after ':set -XForeignFunctionInterface' to get FFI) and you will get the error. If we replace the line below for the above we don't get any error message: foreign import ccall exit :: CInt -> IO () Here, we just replaced 'atexit' by 'exit', which, according to 'nm' tool, is at the same 'libc' library that 'atexit' belongs to. Thanks, Maur?cio From mauricio.antunes at gmail.com Wed Aug 26 07:49:04 2009 From: mauricio.antunes at gmail.com (=?ISO-8859-1?Q?Maur=ED=ADcio_CA?=) Date: Wed Aug 26 07:29:25 2009 Subject: GHCi can't see 'atexit', can't load module In-Reply-To: References: Message-ID: >> I just checked it seems not to be possible to load any module in >> GHCi that uses FFI to wrap the standard C function 'atexit'. (...) > A simple workaround is to just compile the module. (...) If I build a package (using cabal etc.) and try to load such package with 'ghci -package name' the problem persists. But, of course, there are many workarounds. Since I don't see this problem with other functions, and I see no reason for this to happen, I imagine this is a small bug. If it's not, I'll just remove 'atexit' from packages I have on hackage. Thanks, Maur?cio From igloo at earth.li Wed Aug 26 10:39:03 2009 From: igloo at earth.li (Ian Lynagh) Date: Wed Aug 26 10:18:46 2009 Subject: Plans for GHC 6.12.1: release candidate 14 September 2009 In-Reply-To: <7b977d860908211334p4bea9d57ufb29296d6bd46b55@mail.gmail.com> References: <20090818140343.GA2835@matrix.chaos.earth.li> <7b977d860908211334p4bea9d57ufb29296d6bd46b55@mail.gmail.com> Message-ID: <20090826143903.GA32142@matrix.chaos.earth.li> On Fri, Aug 21, 2009 at 04:34:24PM -0400, Ravi Nanavati wrote: > Hypothetically speaking (since I haven't made the call yet), if I > wanted to be an "early adopter" of 6.12.1 when it is released, would > now would be the time to start grabbing HEAD and playing with it There are currently some known problems with "make install", so I would recommend waiting for the RC to be announced. Thanks Ian From marlowsd at gmail.com Wed Aug 26 12:15:23 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Wed Aug 26 11:55:12 2009 Subject: Libraries in the repo Message-ID: <4A955F9B.904@gmail.com> Simon and I have been chatting about how we accommodate libraries in the GHC repository. After previous discussion on this list, GHC has been gradually migrating towards having snapshots of libraries kept as tarballs in the repo (currently only "time" falls into this category), but I don't think we really evaluated the alternatives properly. Here's an attempt to do that, and to my mind the outcome is different: we really want to stick to having all libraries as separate repositories. Background: * Scope: libraries that are needed to build GHC itself (aka "boot libraries") * Boot libraries are of several kinds: - INDEPENDENT: Independently maintained (e.g. time, haskeline) - COUPLED: Tightly coupled to GHC, but used by others (base) - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) * Most boot libraries are INDEPENDENT. INDEPENDENT libraries have a master repository somewhere separate from the GHC repositories. * We need a branch of INDEPENDENT libraries, so that GHC builds don't break when the upstream package is modified. * Sometimes we want to make local modifications to INDEPENDENT libraries: - when GHC adds a new warning, we need to fix instances of the warning in the library to keep the GHC build warning-free. - to check that the changes work, before pushing upstream Choices for how we deal with libraries in the GHC repository: (+) is a pro, (-) is a con. (1) Check out the library from a separate repo, using the darcs-all script. The repo may either be a GHC-specific branch [INDEPENDENT], or the master copy of the package [SPECIFIC/COUPLED]. (+) we can treat every library this way, which gives a consistent story. Consistency is good for developers. (+) [INDEPENDENT] makes it easy to push changes upstream and sync with the upstream repo (unless upstream is using a different VCS). (-) [INDEPENDENT] we have to be careful not to let our branches get too far out of sync with upstream, and we must sync before releasing GHC. (2) Put a snapshot tarball of the library in libraries/tarballs, but allow you to checkout the darcs repo instead. (-) [SPECIFIC/COUPLED] this approach doesn't really make sense, because we expect to be modifying the library often. (-) updating the snapshot is awkward (-) workflow for making a change to the library is awkward: - checkout the darcs repo - make the change, validate it - push the change upstream (bump version?) - make a new snapshot tarball - commit the new snapshot to the GHC repo. (-) having tarballs in the repository is ugly (-) we have no revision history of the library (3) The GHC repo *itself* contains every library unpacked in the tree. You are allowed to check out the darcs repo instead. (+) atomic commits to both the library and GHC. (+) doing this consistently would allow us to remove darcs-all, giving a nice easy development workflow (-) [INDEPENDENT/COUPLED] still need a separate darcs repo. (-) [INDEPENDENT/COUPLED] pushing changes upstream is hard (-) [INDEPENDENT/COUPLED] manual syncing with upstream (-) [COUPLED] (particularly base) syncing with upstream would be painful. (3) works best for SPECIFIC libraries, whereas (1) works best for INDEPENDENT/COUPLED libraries. If we want to treat all libraries the same, then the only real option is (1). Experience with Cabal and bytestring has shown that (1) can work for INDPENDENT libraries, but only if we're careful not to get too out-of-sync (as we did with bytestring). In the case of Cabal, we never have local changes in our branch that aren't in Cabal HEAD, and that works well. Comments/thoughts? Cheers, Simon From judah.jacobson at gmail.com Wed Aug 26 14:17:16 2009 From: judah.jacobson at gmail.com (Judah Jacobson) Date: Wed Aug 26 13:57:16 2009 Subject: Libraries in the repo In-Reply-To: <4A955F9B.904@gmail.com> References: <4A955F9B.904@gmail.com> Message-ID: <6d74b0d20908261117k19557d8fjdfbb5acffba6fdea@mail.gmail.com> On Wed, Aug 26, 2009 at 9:15 AM, Simon Marlow wrote: > Simon and I have been chatting about how we accommodate libraries in the GHC > repository. ?After previous discussion on this list, GHC has been gradually > migrating towards having snapshots of libraries kept as tarballs in the repo > (currently only "time" falls into this category), but I don't think we > really evaluated the alternatives properly. ?Here's an attempt to do that, > and to my mind the outcome is different: we really want to stick to having > all libraries as separate repositories. > > Background: > ?* Scope: libraries that are needed to build GHC itself (aka "boot > ? libraries") > > ?* Boot libraries are of several kinds: > ? - INDEPENDENT: Independently maintained (e.g. time, haskeline) > ? - COUPLED: Tightly coupled to GHC, but used by others (base) > ? - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) > > Choices for how we deal with libraries in the GHC repository: (+) is a > pro, (-) is a con. > > ?(1) Check out the library from a separate repo, using the darcs-all > ? ? ?script. ?The repo may either be a GHC-specific branch > ? ? ?[INDEPENDENT], or the master copy of the package > ? ? ?[SPECIFIC/COUPLED]. > > ?(2) Put a snapshot tarball of the library in libraries/tarballs, > ? ? ?but allow you to checkout the darcs repo instead. > > ?(3) The GHC repo *itself* contains every library unpacked in the > ? ? ?tree. ?You are allowed to check out the darcs repo instead. > > > (3) works best for SPECIFIC libraries, whereas (1) works best for > INDEPENDENT/COUPLED libraries. ?If we want to treat all libraries the > same, then the only real option is (1). Agreed. Also, it seems odd to have template-haskell be built-in yet something so fundamental as base be a separate repo. > Experience with Cabal and bytestring has shown that (1) can work for > INDPENDENT libraries, but only if we're careful not to get too > out-of-sync (as we did with bytestring). ?In the case of Cabal, we never > have local changes in our branch that aren't in Cabal HEAD, and that works > well. > > Comments/thoughts? I also would rather stay with (1). Although using a DVCS allows greater freedom for developers, it also creates the need for more explicit rules of process. So I propose codifying on the wiki that for certain libraries, the local ghc repo - Never has patches which are not in the library's HEAD - Pulls patches sparingly, and usually only after a tagged release of the library. (the darcs-all script could help double-check that the former is being obeyed.) We package admins would need to agree to be responsive to patch submissions from GHC devels (or grant push access to them). Thanks for your very helpful analysis, -Judah From duncan.coutts at worc.ox.ac.uk Wed Aug 26 17:32:28 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Wed Aug 26 17:12:44 2009 Subject: Libraries in the repo In-Reply-To: <4A955F9B.904@gmail.com> References: <4A955F9B.904@gmail.com> Message-ID: <1251322348.30910.10087.camel@localhost> On Wed, 2009-08-26 at 17:15 +0100, Simon Marlow wrote: > * Sometimes we want to make local modifications to INDEPENDENT > libraries: > - when GHC adds a new warning, we need to fix instances of the > warning in the library to keep the GHC build warning-free. I have to say I think this one is rather dubious. What is wrong with just allowing warnings in these independent libs until they get fixed upstream? I know ghc's build system sets -Werror on them, but I don't see that as essential, especially for new warnings added in ghc head. > Experience with Cabal and bytestring has shown that (1) can work for > INDPENDENT libraries, but only if we're careful not to get too > out-of-sync (as we did with bytestring). In the case of Cabal, we never > have local changes in our branch that aren't in Cabal HEAD, and that > works well. It requires an attentive maintainer to notice when people forget to push upstream (as they inevitably do on occasion). If it goes unnoticed for too long then ghc ends up with a forked repo that cannot sanely be synced from the upstream repo (like bytestring). I suggest if we stick with the independent repo approach that we have some automation to check that changes are indeed getting pushed upstream. Duncan From dons at galois.com Wed Aug 26 19:55:01 2009 From: dons at galois.com (Don Stewart) Date: Wed Aug 26 19:36:50 2009 Subject: Libraries in the repo In-Reply-To: <4A955F9B.904@gmail.com> References: <4A955F9B.904@gmail.com> Message-ID: <20090826235501.GL6291@whirlpool.galois.com> marlowsd: > Simon and I have been chatting about how we accommodate libraries in the > GHC repository. After previous discussion on this list, GHC has been > gradually migrating towards having snapshots of libraries kept as > tarballs in the repo (currently only "time" falls into this category), > but I don't think we really evaluated the alternatives properly. Here's > an attempt to do that, and to my mind the outcome is different: we > really want to stick to having all libraries as separate repositories. > > Background: > * Scope: libraries that are needed to build GHC itself (aka "boot > libraries") > > * Boot libraries are of several kinds: > - INDEPENDENT: Independently maintained (e.g. time, haskeline) > - COUPLED: Tightly coupled to GHC, but used by others (base) > - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) > > * Most boot libraries are INDEPENDENT. INDEPENDENT libraries have a > master repository somewhere separate from the GHC repositories. > > * We need a branch of INDEPENDENT libraries, so that GHC builds don't > break when the upstream package is modified. > > * Sometimes we want to make local modifications to INDEPENDENT > libraries: > - when GHC adds a new warning, we need to fix instances of the > warning in the library to keep the GHC build warning-free. > - to check that the changes work, before pushing upstream > > > Choices for how we deal with libraries in the GHC repository: (+) is a > pro, (-) is a con. > > (1) Check out the library from a separate repo, using the darcs-all > script. The repo may either be a GHC-specific branch > [INDEPENDENT], or the master copy of the package > [SPECIFIC/COUPLED]. > > (+) we can treat every library this way, which gives a > consistent story. Consistency is good for developers. > (+) [INDEPENDENT] makes it easy to push changes upstream and sync > with the upstream repo (unless upstream is using a different > VCS). > > (-) [INDEPENDENT] we have to be careful not to let our branches > get too far out of sync with upstream, and we must > sync before releasing GHC. > > (2) Put a snapshot tarball of the library in libraries/tarballs, > but allow you to checkout the darcs repo instead. > > (-) [SPECIFIC/COUPLED] this approach doesn't really make sense, > because we expect to be modifying the library often. > (-) updating the snapshot is awkward > (-) workflow for making a change to the library is awkward: > - checkout the darcs repo > - make the change, validate it > - push the change upstream (bump version?) > - make a new snapshot tarball > - commit the new snapshot to the GHC repo. > (-) having tarballs in the repository is ugly > (-) we have no revision history of the library > > (3) The GHC repo *itself* contains every library unpacked in the > tree. You are allowed to check out the darcs repo instead. > > (+) atomic commits to both the library and GHC. > (+) doing this consistently would allow us to remove darcs-all, > giving a nice easy development workflow > > (-) [INDEPENDENT/COUPLED] still need a separate darcs repo. > (-) [INDEPENDENT/COUPLED] pushing changes upstream is hard > (-) [INDEPENDENT/COUPLED] manual syncing with upstream > (-) [COUPLED] (particularly base) syncing with > upstream would be painful. > > > (3) works best for SPECIFIC libraries, whereas (1) works best for > INDEPENDENT/COUPLED libraries. If we want to treat all libraries the > same, then the only real option is (1). > > Experience with Cabal and bytestring has shown that (1) can work for > INDPENDENT libraries, but only if we're careful not to get too > out-of-sync (as we did with bytestring). In the case of Cabal, we never > have local changes in our branch that aren't in Cabal HEAD, and that > works well. > > Comments/thoughts? As author of bytestring, I'd prefer it if GHC used a released version direct from Hackage. I.e. GHC could snapshot a Hackage release, and get out of the business of cloning repos. Same for other INDPENDENTs. -- Don From marlowsd at gmail.com Thu Aug 27 06:16:01 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Thu Aug 27 05:55:50 2009 Subject: Libraries in the repo In-Reply-To: <1251322348.30910.10087.camel@localhost> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> Message-ID: <4A965CE1.80700@gmail.com> On 26/08/2009 22:32, Duncan Coutts wrote: > On Wed, 2009-08-26 at 17:15 +0100, Simon Marlow wrote: > >> * Sometimes we want to make local modifications to INDEPENDENT >> libraries: >> - when GHC adds a new warning, we need to fix instances of the >> warning in the library to keep the GHC build warning-free. > > I have to say I think this one is rather dubious. What is wrong with > just allowing warnings in these independent libs until they get fixed > upstream? I know ghc's build system sets -Werror on them, but I don't > see that as essential, especially for new warnings added in ghc head. True, we don't have to do that. >> Experience with Cabal and bytestring has shown that (1) can work for >> INDPENDENT libraries, but only if we're careful not to get too >> out-of-sync (as we did with bytestring). In the case of Cabal, we never >> have local changes in our branch that aren't in Cabal HEAD, and that >> works well. > > It requires an attentive maintainer to notice when people forget to push > upstream (as they inevitably do on occasion). If it goes unnoticed for > too long then ghc ends up with a forked repo that cannot sanely be > synced from the upstream repo (like bytestring). > > I suggest if we stick with the independent repo approach that we have > some automation to check that changes are indeed getting pushed > upstream. Agreed. Can you think of an easy way to automate it? Cheers, Simon From ganesh.sittampalam at credit-suisse.com Thu Aug 27 06:18:51 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Thu Aug 27 05:58:37 2009 Subject: Libraries in the repo In-Reply-To: <4A965CE1.80700@gmail.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> Simon Marlow wrote: >> >> I suggest if we stick with the independent repo approach that we have >> some automation to check that changes are indeed getting pushed >> upstream. > > Agreed. Can you think of an easy way to automate it? How about a cronjob that runs darcs send --to= ? Ganesh =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html =============================================================================== From marlowsd at gmail.com Thu Aug 27 06:21:01 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Thu Aug 27 06:00:47 2009 Subject: Libraries in the repo In-Reply-To: <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> Message-ID: <4A965E0D.70103@gmail.com> On 27/08/2009 11:18, Sittampalam, Ganesh wrote: > Simon Marlow wrote: >>> >>> I suggest if we stick with the independent repo approach that we have >>> some automation to check that changes are indeed getting pushed >>> upstream. >> >> Agreed. Can you think of an easy way to automate it? > > How about a cronjob that runs > > darcs send --to= > > ? But the requirement we want is that patches are only pushed upstream, and never pushed to the branch first. Cheers, Simon From ganesh.sittampalam at credit-suisse.com Thu Aug 27 06:24:06 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Thu Aug 27 06:04:05 2009 Subject: Libraries in the repo In-Reply-To: <4A965E0D.70103@gmail.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> Simon Marlow wrote: > On 27/08/2009 11:18, Sittampalam, Ganesh wrote: >> Simon Marlow wrote: >>>> >>>> I suggest if we stick with the independent repo approach that we >>>> have some automation to check that changes are indeed getting >>>> pushed upstream. >>> >>> Agreed. Can you think of an easy way to automate it? >> >> How about a cronjob that runs >> >> darcs send --to= >> >> ? > > But the requirement we want is that patches are only pushed upstream, > and never pushed to the branch first. I might be getting confused about something, but I'd expect this command to send an email with any changes in the branch repo that aren't in upstream. In other words if you have some patches in the branch that aren't upstream, you'll find out and can remedy the situation. Ganesh =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html =============================================================================== From jpm at cs.uu.nl Thu Aug 27 06:25:16 2009 From: jpm at cs.uu.nl (=?ISO-8859-1?Q?Jos=E9_Pedro_Magalh=E3es?=) Date: Thu Aug 27 06:05:16 2009 Subject: Libraries in the repo In-Reply-To: <4A955F9B.904@gmail.com> References: <4A955F9B.904@gmail.com> Message-ID: <52f14b210908270325h31b53bcay7494b141d0ca94f6@mail.gmail.com> Hello, On Wed, Aug 26, 2009 at 18:15, Simon Marlow wrote: > > * Boot libraries are of several kinds: > - INDEPENDENT: Independently maintained (e.g. time, haskeline) > - COUPLED: Tightly coupled to GHC, but used by others (base) > - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) Does syb fall under INDEPENDENT or COUPLED? In any case, as the syb maintainer, I'd favor (1) too. Cheers, Pedro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.haskell.org/pipermail/glasgow-haskell-users/attachments/20090827/35ddf775/attachment-0001.html From marlowsd at gmail.com Thu Aug 27 06:26:59 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Thu Aug 27 06:06:43 2009 Subject: Libraries in the repo In-Reply-To: <20090826235501.GL6291@whirlpool.galois.com> References: <4A955F9B.904@gmail.com> <20090826235501.GL6291@whirlpool.galois.com> Message-ID: <4A965F73.9050301@gmail.com> On 27/08/2009 00:55, Don Stewart wrote: > marlowsd: >> Simon and I have been chatting about how we accommodate libraries in the >> GHC repository. After previous discussion on this list, GHC has been >> gradually migrating towards having snapshots of libraries kept as >> tarballs in the repo (currently only "time" falls into this category), >> but I don't think we really evaluated the alternatives properly. Here's >> an attempt to do that, and to my mind the outcome is different: we >> really want to stick to having all libraries as separate repositories. >> >> Background: >> * Scope: libraries that are needed to build GHC itself (aka "boot >> libraries") >> >> * Boot libraries are of several kinds: >> - INDEPENDENT: Independently maintained (e.g. time, haskeline) >> - COUPLED: Tightly coupled to GHC, but used by others (base) >> - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) >> >> * Most boot libraries are INDEPENDENT. INDEPENDENT libraries have a >> master repository somewhere separate from the GHC repositories. >> >> * We need a branch of INDEPENDENT libraries, so that GHC builds don't >> break when the upstream package is modified. >> >> * Sometimes we want to make local modifications to INDEPENDENT >> libraries: >> - when GHC adds a new warning, we need to fix instances of the >> warning in the library to keep the GHC build warning-free. >> - to check that the changes work, before pushing upstream >> >> >> Choices for how we deal with libraries in the GHC repository: (+) is a >> pro, (-) is a con. >> >> (1) Check out the library from a separate repo, using the darcs-all >> script. The repo may either be a GHC-specific branch >> [INDEPENDENT], or the master copy of the package >> [SPECIFIC/COUPLED]. >> >> (+) we can treat every library this way, which gives a >> consistent story. Consistency is good for developers. >> (+) [INDEPENDENT] makes it easy to push changes upstream and sync >> with the upstream repo (unless upstream is using a different >> VCS). >> >> (-) [INDEPENDENT] we have to be careful not to let our branches >> get too far out of sync with upstream, and we must >> sync before releasing GHC. >> >> (2) Put a snapshot tarball of the library in libraries/tarballs, >> but allow you to checkout the darcs repo instead. >> >> (-) [SPECIFIC/COUPLED] this approach doesn't really make sense, >> because we expect to be modifying the library often. >> (-) updating the snapshot is awkward >> (-) workflow for making a change to the library is awkward: >> - checkout the darcs repo >> - make the change, validate it >> - push the change upstream (bump version?) >> - make a new snapshot tarball >> - commit the new snapshot to the GHC repo. >> (-) having tarballs in the repository is ugly >> (-) we have no revision history of the library >> >> (3) The GHC repo *itself* contains every library unpacked in the >> tree. You are allowed to check out the darcs repo instead. >> >> (+) atomic commits to both the library and GHC. >> (+) doing this consistently would allow us to remove darcs-all, >> giving a nice easy development workflow >> >> (-) [INDEPENDENT/COUPLED] still need a separate darcs repo. >> (-) [INDEPENDENT/COUPLED] pushing changes upstream is hard >> (-) [INDEPENDENT/COUPLED] manual syncing with upstream >> (-) [COUPLED] (particularly base) syncing with >> upstream would be painful. >> >> >> (3) works best for SPECIFIC libraries, whereas (1) works best for >> INDEPENDENT/COUPLED libraries. If we want to treat all libraries the >> same, then the only real option is (1). >> >> Experience with Cabal and bytestring has shown that (1) can work for >> INDPENDENT libraries, but only if we're careful not to get too >> out-of-sync (as we did with bytestring). In the case of Cabal, we never >> have local changes in our branch that aren't in Cabal HEAD, and that >> works well. >> >> Comments/thoughts? > > > As author of bytestring, I'd prefer it if GHC used a released version > direct from Hackage. I.e. GHC could snapshot a Hackage release, and get > out of the business of cloning repos. Same for other INDPENDENTs. Are you saying you don't want us to have a GHC branch? Even if the branch just pulls from upstream and never has local changes? We can still use released versions only, the main point about having separate repos is that we have a consistent picture of libraries from GHC's side. For bytestring I imagine we can get away without making changes between releases, or at least ensuring our changes are sent upstream and we wait for a release before pulling. For other libraries, such as Cabal, this would be too onerous I think (Cabal is really COUPLED at the moment, much as we'd like it to be INDEPENDENT). Cheers, Simon From marlowsd at gmail.com Thu Aug 27 06:29:44 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Thu Aug 27 06:09:28 2009 Subject: Libraries in the repo In-Reply-To: <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> Message-ID: <4A966018.4020603@gmail.com> On 27/08/2009 11:24, Sittampalam, Ganesh wrote: > Simon Marlow wrote: >> On 27/08/2009 11:18, Sittampalam, Ganesh wrote: >>> Simon Marlow wrote: >>>>> >>>>> I suggest if we stick with the independent repo approach that we >>>>> have some automation to check that changes are indeed getting >>>>> pushed upstream. >>>> >>>> Agreed. Can you think of an easy way to automate it? >>> >>> How about a cronjob that runs >>> >>> darcs send --to= >>> >>> ? >> >> But the requirement we want is that patches are only pushed upstream, >> and never pushed to the branch first. > > I might be getting confused about something, but I'd expect this command > to send an email with any changes in the branch repo that aren't in > upstream. In other words if you have some patches in the branch that > aren't upstream, you'll find out and can remedy the situation. Yes, it tells you that you've screwed up, rather than telling you that you're about to screw up, which would be much more convenient. After you've screwed up it might be too late to fix it, due to conflicts with upstream. Cheers, Simon From ganesh.sittampalam at credit-suisse.com Thu Aug 27 06:37:00 2009 From: ganesh.sittampalam at credit-suisse.com (Sittampalam, Ganesh) Date: Thu Aug 27 06:16:54 2009 Subject: Libraries in the repo In-Reply-To: <4A966018.4020603@gmail.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> <4A966018.4020603@gmail.com> Message-ID: <16442B752A06A74AB4D9F9A5FF076E4B03B9F982@ELON17P32001A.csfb.cs-group.com> Simon Marlow wrote: >>>> Simon Marlow wrote: >>>>>> >>>>>> I suggest if we stick with the independent repo approach that we >>>>>> have some automation to check that changes are indeed getting >>>>>> pushed upstream. [snip unhelpful suggestion from me] > > Yes, it tells you that you've screwed up, rather than telling you > that you're about to screw up, which would be much more convenient. > After you've screwed up it might be too late to fix it, due to > conflicts with upstream. Can you arrange that the only way that patches can get into the branch is via darcs pull --intersection ? Ganesh =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html =============================================================================== From marlowsd at gmail.com Thu Aug 27 09:10:41 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Thu Aug 27 08:50:25 2009 Subject: Libraries in the repo In-Reply-To: <4A955F9B.904@gmail.com> References: <4A955F9B.904@gmail.com> Message-ID: <4A9685D1.2000008@gmail.com> Incedentally, the reason I'd like us to make a decision on this now is because I'm about to add two new boot libraries: - binary, to support a binary cache of GHC's package database (INDEPENDENT) - bin-package-db, the code to read and write the binary package database (SPECIFIC, shared by ghc and ghc-pkg). I don't much like bin-package-db being a separate package, given that it's only 100 lines or so in one module, but I don't see a good alternative. Cheers, Simon On 26/08/2009 17:15, Simon Marlow wrote: > Simon and I have been chatting about how we accommodate libraries in the > GHC repository. After previous discussion on this list, GHC has been > gradually migrating towards having snapshots of libraries kept as > tarballs in the repo (currently only "time" falls into this category), > but I don't think we really evaluated the alternatives properly. Here's > an attempt to do that, and to my mind the outcome is different: we > really want to stick to having all libraries as separate repositories. > > Background: > * Scope: libraries that are needed to build GHC itself (aka "boot > libraries") > > * Boot libraries are of several kinds: > - INDEPENDENT: Independently maintained (e.g. time, haskeline) > - COUPLED: Tightly coupled to GHC, but used by others (base) > - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) > > * Most boot libraries are INDEPENDENT. INDEPENDENT libraries have a > master repository somewhere separate from the GHC repositories. > > * We need a branch of INDEPENDENT libraries, so that GHC builds don't > break when the upstream package is modified. > > * Sometimes we want to make local modifications to INDEPENDENT > libraries: > - when GHC adds a new warning, we need to fix instances of the > warning in the library to keep the GHC build warning-free. > - to check that the changes work, before pushing upstream > > > Choices for how we deal with libraries in the GHC repository: (+) is a > pro, (-) is a con. > > (1) Check out the library from a separate repo, using the darcs-all > script. The repo may either be a GHC-specific branch > [INDEPENDENT], or the master copy of the package > [SPECIFIC/COUPLED]. > > (+) we can treat every library this way, which gives a > consistent story. Consistency is good for developers. > (+) [INDEPENDENT] makes it easy to push changes upstream and sync > with the upstream repo (unless upstream is using a different > VCS). > > (-) [INDEPENDENT] we have to be careful not to let our branches > get too far out of sync with upstream, and we must > sync before releasing GHC. > > (2) Put a snapshot tarball of the library in libraries/tarballs, > but allow you to checkout the darcs repo instead. > > (-) [SPECIFIC/COUPLED] this approach doesn't really make sense, > because we expect to be modifying the library often. > (-) updating the snapshot is awkward > (-) workflow for making a change to the library is awkward: > - checkout the darcs repo > - make the change, validate it > - push the change upstream (bump version?) > - make a new snapshot tarball > - commit the new snapshot to the GHC repo. > (-) having tarballs in the repository is ugly > (-) we have no revision history of the library > > (3) The GHC repo *itself* contains every library unpacked in the > tree. You are allowed to check out the darcs repo instead. > > (+) atomic commits to both the library and GHC. > (+) doing this consistently would allow us to remove darcs-all, > giving a nice easy development workflow > > (-) [INDEPENDENT/COUPLED] still need a separate darcs repo. > (-) [INDEPENDENT/COUPLED] pushing changes upstream is hard > (-) [INDEPENDENT/COUPLED] manual syncing with upstream > (-) [COUPLED] (particularly base) syncing with > upstream would be painful. > > > (3) works best for SPECIFIC libraries, whereas (1) works best for > INDEPENDENT/COUPLED libraries. If we want to treat all libraries the > same, then the only real option is (1). > > Experience with Cabal and bytestring has shown that (1) can work for > INDPENDENT libraries, but only if we're careful not to get too > out-of-sync (as we did with bytestring). In the case of Cabal, we never > have local changes in our branch that aren't in Cabal HEAD, and that > works well. > > Comments/thoughts? > > Cheers, > Simon From marlowsd at gmail.com Fri Aug 28 05:05:41 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Fri Aug 28 04:45:24 2009 Subject: Libraries in the repo In-Reply-To: <16442B752A06A74AB4D9F9A5FF076E4B03B9F982@ELON17P32001A.csfb.cs-group.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> <4A966018.4020603@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F982@ELON17P32001A.csfb.cs-group.com> Message-ID: <4A979DE5.9040407@gmail.com> On 27/08/2009 11:37, Sittampalam, Ganesh wrote: > Simon Marlow wrote: >>>>> Simon Marlow wrote: >>>>>>> >>>>>>> I suggest if we stick with the independent repo approach that we >>>>>>> have some automation to check that changes are indeed getting >>>>>>> pushed upstream. > [snip unhelpful suggestion from me] >> >> Yes, it tells you that you've screwed up, rather than telling you >> that you're about to screw up, which would be much more convenient. >> After you've screwed up it might be too late to fix it, due to >> conflicts with upstream. > > Can you arrange that the only way that patches can get into the branch > is via darcs pull --intersection ? That's an interesting idea, I'd forgotten about --intersection. Simon From marlowsd at gmail.com Fri Aug 28 05:06:36 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Fri Aug 28 04:46:17 2009 Subject: Libraries in the repo In-Reply-To: <52f14b210908270325h31b53bcay7494b141d0ca94f6@mail.gmail.com> References: <4A955F9B.904@gmail.com> <52f14b210908270325h31b53bcay7494b141d0ca94f6@mail.gmail.com> Message-ID: <4A979E1C.6060309@gmail.com> On 27/08/2009 11:25, Jos? Pedro Magalh?es wrote: > Hello, > > On Wed, Aug 26, 2009 at 18:15, Simon Marlow > wrote: > > > * Boot libraries are of several kinds: > - INDEPENDENT: Independently maintained (e.g. time, haskeline) > - COUPLED: Tightly coupled to GHC, but used by others (base) > - SPECIFIC: Totally specific to GHC (e.g. template-haskell, DPH) > > > Does syb fall under INDEPENDENT or COUPLED? > > In any case, as the syb maintainer, I'd favor (1) too. I'd say at this stage it's INDEPENDENT. Thanks for the feedback! Cheers, Simon From marlowsd at gmail.com Fri Aug 28 06:42:00 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Fri Aug 28 06:21:42 2009 Subject: Libraries in the repo In-Reply-To: <4A979DE5.9040407@gmail.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> <4A966018.4020603@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F982@ELON17P32001A.csfb.cs-group.com> <4A979DE5.9040407@gmail.com> Message-ID: <4A97B478.4010809@gmail.com> On 28/08/2009 10:05, Simon Marlow wrote: > On 27/08/2009 11:37, Sittampalam, Ganesh wrote: >> Simon Marlow wrote: >>>>>> Simon Marlow wrote: >>>>>>>> >>>>>>>> I suggest if we stick with the independent repo approach that we >>>>>>>> have some automation to check that changes are indeed getting >>>>>>>> pushed upstream. >> [snip unhelpful suggestion from me] >>> >>> Yes, it tells you that you've screwed up, rather than telling you >>> that you're about to screw up, which would be much more convenient. >>> After you've screwed up it might be too late to fix it, due to >>> conflicts with upstream. >> >> Can you arrange that the only way that patches can get into the branch >> is via darcs pull --intersection ? > > That's an interesting idea, I'd forgotten about --intersection. I have a script that works as a prehook (below). Unfortunately it doesn't work on darcs.haskell.org, I think because we only have darcs 1.0.9 there, and it is ignoring my prehook. Can anyone think of a good reason not to upgrade darcs to 2.3.0 on darcs.haskell.org? I can think of 3 reasons to do so: - this script, for preventing accidental divergence from upstream - faster pushes, due to transfer-mode - hide those annoying "Ignore-this: xxxxx" messages Cheers, Simon #!/bin/sh -e # checkupstream.sh # Only allow applying of patches that are also in this upstream repository: UPSTREAM=$1 # echo DARCS_PATCHES_XML = $DARCS_PATCHES_XML # Take $DARCS_PATCHES_XML and turn it into a list of patch hashes # suitable for looping over. hashes=`echo $DARCS_PATCHES_XML | sed 's||\n|g' | sed -n '/hash/p' | sed "s|^.*hash='\([^']*\)'.*$|\1|"` # echo hashes: $hashes # For each patch, try pulling the patch from the upstream repo. If # the patch is not upstream, then fail. for p in $hashes; do if darcs pull --match="hash $p" $UPSTREAM --xml --dry-run | grep "$p" >/dev/null; then echo "Patch $p is upstream; ok" else echo "Patch $p is not upstream!" exit 1 fi done exit 0 From duncan.coutts at worc.ox.ac.uk Fri Aug 28 12:24:24 2009 From: duncan.coutts at worc.ox.ac.uk (Duncan Coutts) Date: Fri Aug 28 13:22:27 2009 Subject: Libraries in the repo In-Reply-To: <4A97B478.4010809@gmail.com> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> <4A966018.4020603@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F982@ELON17P32001A.csfb.cs-group.com> <4A979DE5.9040407@gmail.com> <4A97B478.4010809@gmail.com> Message-ID: <1251476664.30910.14448.camel@localhost> On Fri, 2009-08-28 at 11:42 +0100, Simon Marlow wrote: > Can anyone think of a good reason not to upgrade darcs to 2.3.0 on > darcs.haskell.org? I can think of 3 reasons to do so: > > - this script, for preventing accidental divergence from upstream > - faster pushes, due to transfer-mode > - hide those annoying "Ignore-this: xxxxx" messages By the way, people who regularly work with the ghc repos (at least on Linux) and who are thinking of upgrading to darcs-2.3.0 should heed this advice: Use "darcs get" to get your repos again. Not remotely, just locally. This switches them from darcs1 traditional format to darcs1 hashed format. If you do this, then "darcs whatsnew" gets ~4 times quicker. If you do not do this, then "darcs whatsnew" gets ~100 times slower. All times measured on Linux, local ext3 filesystem, ghc testsuite repo. All times are the second of two runs to allow for OS caching. The results may well be quite different on a different file systems, like Windows NTFS. Perhaps someone can suggest a way of doing this using the ./darcs-all script, that would not mess up what the default push/pull address is. Of course doing a get means the copy doesn't have the changes from the working directory. As far as I know darcs currently does not provide a way to do an inplace upgrade to the faster format. I've emailed the darcs list to raise this issue, that: 1. we get no warning or advice from darcs that we should switch format 2. that there is not a really convenient way of doing the switch Duncan From marlowsd at gmail.com Sat Aug 29 07:34:06 2009 From: marlowsd at gmail.com (Simon Marlow) Date: Sat Aug 29 07:13:55 2009 Subject: Libraries in the repo In-Reply-To: <1251476664.30910.14448.camel@localhost> References: <4A955F9B.904@gmail.com> <1251322348.30910.10087.camel@localhost> <4A965CE1.80700@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F97F@ELON17P32001A.csfb.cs-group.com> <4A965E0D.70103@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F980@ELON17P32001A.csfb.cs-group.com> <4A966018.4020603@gmail.com> <16442B752A06A74AB4D9F9A5FF076E4B03B9F982@ELON17P32001A.csfb.cs-group.com> <4A979DE5.9040407@gmail.com> <4A97B478.4010809@gmail.com> <1251476664.30910.14448.camel@localhost> Message-ID: <4A99122E.2070408@gmail.com> Duncan Coutts wrote: > On Fri, 2009-08-28 at 11:42 +0100, Simon Marlow wrote: > >> Can anyone think of a good reason not to upgrade darcs to 2.3.0 on >> darcs.haskell.org? I can think of 3 reasons to do so: >> >> - this script, for preventing accidental divergence from upstream >> - faster pushes, due to transfer-mode >> - hide those annoying "Ignore-this: xxxxx" messages > > By the way, people who regularly work with the ghc repos (at least on > Linux) and who are thinking of upgrading to darcs-2.3.0 should heed this > advice: > > Use "darcs get" to get your repos again. Not remotely, just > locally. This switches them from darcs1 traditional format to > darcs1 hashed format. > > If you do this, then "darcs whatsnew" gets ~4 times quicker. > If you do not do this, then "darcs whatsnew" gets ~100 times > slower. > > All times measured on Linux, local ext3 filesystem, ghc testsuite repo. > All times are the second of two runs to allow for OS caching. The > results may well be quite different on a different file systems, like > Windows NTFS. yes - on Windows things got slower with 2.3.0, even with hashed repositories: http://bugs.darcs.net/issue1585 Another thing to watch out for is that hashed repositories will automatically cache patches and pristine files in ~/.darcs/cache by default. If you home directory is NFS-mounted, this can be a bad idea. Even if you're not using NFS, the fact that pristine files are shared between all repositories means that darcs sometimes is a lot slower than it needs to be, because the timestamps on the pristine files are out of sync with the local repository (you'll see long "Reading pristine..." messages from darcs). I raised this on the darcs-users list before the 2.3.0 release, but as far as I know it isn't planned to be fixed until 2.4. Cheers, Simon