<div dir="ltr">Ryan, could you explain what you want more precisely? Specifically what you want in terms of exposed primops using the terminology / vocabulary in <a href="http://llvm.org/docs/LangRef.html#ordering">http://llvm.org/docs/LangRef.html#ordering</a> and <a href="http://llvm.org/docs/Atomics.html">http://llvm.org/docs/Atomics.html</a> ?<div>
<br></div><div> I'll first do the work for just the LLVM backend, and I"ll likely need some active guidance / monitoring for the native codegen analogues</div><div><br></div><div style>(also asked this on ticket for documentation purposes)</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jul 20, 2013 at 2:18 AM, Ryan Newton <span dir="ltr"><<a href="mailto:rrnewton@gmail.com" target="_blank">rrnewton@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Carter,<div><br></div><div>Yes, SMP.h is where I've copy pasted the duplicate functionality from (since I can't presently rely on linking the symbols).</div>
<div><br></div><div>Your proposal for the LLVM backend sounds *<b>great</b>*. But it also is going to provide additional constraints for getting "atomic-primops" right. </div>
<div> The goal of atomic-primops is to be a stable Haskell-level interface into the relevant CAS and fetch-and-add stuff. The reason this is important is that one has to be very careful to defeat the GHC optimizer in all the relevant places and make pointer equality a reliable property. I would like to get atomic-primops to work reliably in 7.4, 7.6 [and 7.8] and have more "native" support in future GHC releases, where maybe the foreign primops would become unecessary. (They are a pain and have already exposed one blocking cabal bug, fixed in upcoming 1.17.)</div>
<div><br></div><div>A couple additional suggestions for the proposal in ticket #7883:</div><div><ul><li>we should use more unique symbols than "cas", especially for this rewriting trick. How about "ghc_cas" or something?</li>
<li>it would be great to get at least fetch-and-add in addition to CAS and barriers</li><li>if we reliably provide this set of special symbols, libraries like atomic-primops may use them in the .cmm and benefit from the CMM->LLVM substitutions</li>
<li>if we include all the primops I need in GHC proper the previous bullet will stop applying ;-)</li></ul></div><div>Cheers,</div><div> -Ryan</div><div><br></div><div>P.S. Just as a bit of motivation, here are some recent performance numbers. We often wonder about how close our "pure values in a box" approach comes to efficient lock-free structures. <span style="font-size:12.727272033691406px;font-family:arial,sans-serif">Well here are some numbers about using a proper unboxed counter in the Haskell heap, vs using an IORef Int and atomicModifyIORef': </span><span style="font-size:12.727272033691406px;font-family:arial,sans-serif">Up to 100X performance difference on some platforms for microbenchmarks that hammer a counter:</span></div>
<div><div><br> <a href="https://github.com/rrnewton/haskell-lockfree-queue/blob/fb12d1121690553e4f737af258848f279147ea24/AtomicPrimops/DEVLOG.md#20130718-timing-atomic-counter-ops" target="_blank">https://github.com/rrnewton/haskell-lockfree-queue/blob/fb12d1121690553e4f737af258848f279147ea24/AtomicPrimops/DEVLOG.md#20130718-timing-atomic-counter-ops</a><br>
</div><div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px"><div><br></div><div>And here are the performance and scaling advantages of using ChaseLev (based on atomic-primops), over a traditional pure-in-a-box structure (IORef Data.Seq). The following are timings of ChaseLev/traditional respectively on a 32 core westmere:</div>
<div><br></div><div> fib(42) 1 threads: 21s</div><div> fib(42) 2 threads: 10.1s</div><div> fib(42) 4 threads: 5.2s (100%prod)</div><div> fib(42) 8 threads: 2.7s - 3.2s (100%prod) </div><div> fib(42) 16 threads: 1.28s</div>
<div> fib(42) 24 threads: 1.85s</div><div> fib(42) 32 threads: 4.8s (high variance)</div></div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px"><br></div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<div> (hive) fib(42) 1 threads: 41.8s (95% prod)</div><div> (hive) fib(42) 2 threads: 25.2s (66% prod)</div><div> (hive) fib(42) 4 threads: 14.6s (27% prod, 135GB alloc)</div><div> (hive) fib(42) 8 threads: 17.1s (26% prod)</div>
<div> (hive) fib(42) 16 threads: 16.3s (13% prod)</div><div> (hive) fib(42) 24 threads: 21.2s (30% prod)</div><div> (hive) fib(42) 32 threads: 29.3s (33% prod)</div></div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<br></div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px">And that is WITH the inefficiency of doing a "ccall" on every single atomic operation.</div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<br></div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px">Notes on parfib performance are here:</div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px"><br></div><div style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<a href="https://github.com/rrnewton/haskell-lockfree-queue/blob/d6d3e9eda2a487a5f055b1f51423954bb6b6bdfa/ChaseLev/Test.hs#L158" target="_blank">https://github.com/rrnewton/haskell-lockfree-queue/blob/d6d3e9eda2a487a5f055b1f51423954bb6b6bdfa/ChaseLev/Test.hs#L158</a><br>
</div></div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 19, 2013 at 5:05 PM, Carter Schonwald <span dir="ltr"><<a href="mailto:carter.schonwald@gmail.com" target="_blank">carter.schonwald@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">ryan, the relevant machinery on the C side is here, see ./includes/stg/SMP.h : <a href="https://github.com/ghc/ghc/blob/7cc8a3cc5c2970009b83844ff9cc4e27913b8559/includes/stg/SMP.h" target="_blank">https://github.com/ghc/ghc/blob/7cc8a3cc5c2970009b83844ff9cc4e27913b8559/includes/stg/SMP.h</a><div>
<br></div><div>(unless i'm missing something)</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 19, 2013 at 4:53 PM, Carter Schonwald <span dir="ltr"><<a href="mailto:carter.schonwald@gmail.com" target="_blank">carter.schonwald@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ryan, <div>if you look at line 270, you'll see the CAS is a C call <a href="https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L270" target="_blank">https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L270</a> </div>
<div><br></div><div>What Simon is alluding to is some work I started (but need to finish)</div><div><a href="http://ghc.haskell.org/trac/ghc/ticket/7883" target="_blank">http://ghc.haskell.org/trac/ghc/ticket/7883</a> is the relevant ticket, and I'll need to sort out doing the same on the native code gen too<br>
</div><div><br></div><div>there ARE no write barrier primops, they're baked into the CAS machinery in ghc's rts</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Jul 19, 2013 at 1:02 PM, Ryan Newton <span dir="ltr"><<a href="mailto:rrnewton@gmail.com" target="_blank">rrnewton@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Yes, I'd absolutely rather not suffer C call overhead for these functions (or the CAS functions). But isn't that how it's done currently for the casMutVar# primop?</div>
<div><br></div>
<div><a href="https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L265" target="_blank">https://github.com/ghc/ghc/blob/95e6865ecf06b2bd80fa737e4fa4a24beaae25c5/rts/PrimOps.cmm#L265</a><br>
</div>
<div><br></div><div>To avoid the overhead, is it necessary to make each primop in-line rather than out-of-line, or just to get rid of the "ccall"?</div><div><br></div>Another reason it would be good to package these with GHC is that I'm having trouble building robust libraries of foreign primops that work under all "ways" (e.g. GHCI). For example, this bug:<div>
<br></div><div> <a href="https://github.com/rrnewton/haskell-lockfree-queue/issues/10" target="_blank">https://github.com/rrnewton/haskell-lockfree-queue/issues/10</a><br></div><div><br></div><div>If I write .cmm code that depends on RTS functionality like stg_MUT_VAR_CLEAN_info, then it seems to work fine when in compiled mode (with/without threading, profiling), but I get link errors from GHCI where these symbols aren't defined.</div>
<div><br></div><div>I've got a draft of the relevant primops here:</div><div><br></div><div><a href="https://github.com/rrnewton/haskell-lockfree-queue/blob/master/AtomicPrimops/cbits/primops.cmm" target="_blank">https://github.com/rrnewton/haskell-lockfree-queue/blob/master/AtomicPrimops/cbits/primops.cmm</a><br>
</div><div><br></div><div>Which includes:</div><div><ul><li>variants of CAS for MutableArray# and MutableByteArray#</li><li>fetch-and-add for MutableByteArray#</li></ul></div><div>
Also, there are some tweaks to support the new "ticketed" interface for safer CAS:</div><div><br></div><div> <a href="http://hackage.haskell.org/packages/archive/atomic-primops/0.3/doc/html/Data-Atomics.html#g:3" target="_blank">http://hackage.haskell.org/packages/archive/atomic-primops/0.3/doc/html/Data-Atomics.html#g:3</a><br>
</div><div><br></div><div>I started adding some of these primops to GHC proper (still as out-of-line), but not all of them. I had gone with the foreign primop route instead...</div><div><br></div><div>
<a href="https://github.com/rrnewton/ghc/commits/master" target="_blank">https://github.com/rrnewton/ghc/commits/master</a><span><font color="#888888"><br></font></span></div><span><font color="#888888"><div>
<br></div><div> -Ryan</div></font></span><div><br></div><div>P.S. Where is the write barrier primop? I don't see it listed in prelude/primops.txt...</div><div><div>
<div><br></div><div><br></div><div><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 19, 2013 at 11:41 AM, Carter Schonwald <span dir="ltr"><<a href="mailto:carter.schonwald@gmail.com" target="_blank">carter.schonwald@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I guess I should find the time to finish the CAS primop work I volunteered to do then. Ill look into in a few days. <div>
<div><span></span><br><br>On Friday, July 19, 2013, Simon Marlow wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
On 18/07/13 14:17, Ryan Newton wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The "atomic-primops" library depends on symbols such as<br>
store_load_barrier and "cas", which are defined in SMP.h. Thus the<br>
result is that if the program is linked WITHOUT "-threaded", the user<br>
gets a linker error about undefined symbols.<br>
<br>
The specific place it's used is in the 'foreign "C"' bits of this .cmm code:<br>
<br>
<a href="https://github.com/rrnewton/haskell-lockfree-queue/blob/87e63b21b2a6c375e93c30b98c28c1d04f88781c/AtomicPrimops/cbits/primops.cmm" target="_blank">https://github.com/rrnewton/<u></u>haskell-lockfree-queue/blob/<u></u>87e63b21b2a6c375e93c30b98c28c1<u></u>d04f88781c/AtomicPrimops/<u></u>cbits/primops.cmm</a><br>
<br>
I'm trying to explore hacks that will enable me to pull in those<br>
functions during compile time, without duplicating a whole bunch of code<br>
from the RTS. But it's a fragile business.<br>
<br>
It seems to me that some of these routines have general utility. In<br>
future versions of GHC, could we consider linking in those routines<br>
irrespective of "-threaded"?<br>
</blockquote>
<br>
We should make the non-THREADED versions EXTERN_INLINE too, so that there will be (empty) functions to call in rts/Inlines.c. Want to submit a patch?<br>
<br>
A better solution would be to make them into primops. You don't really want to be calling out to a C function to implement a memory barrier. We have this for write_barrier(), but none of the others so far. Of couse that's a larger change.<br>
<br>
Cheers,<br>
Simon<br>
<br>
<br>
<br></div></div>
______________________________<u></u>_________________<br>
ghc-devs mailing list<br>
<a>ghc-devs@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/ghc-devs" target="_blank">http://www.haskell.org/<u></u>mailman/listinfo/ghc-devs</a><br>
</blockquote>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>