# Automatic Differentiation

### From HaskellWiki

(short explanation of automatic differentation) |
m (fixed edit error) |
||

(8 intermediate revisions by 3 users not shown) | |||

Line 1: | Line 1: | ||

− | '''Automatic Differentiation''' roughly means that a numerical value is equipped with a derivative part, |
+ | '''Automatic Differentiation''' enables you to compute both the value of a function at a point and its derivative(s) at the same time. |

− | which is updated accordingly on every function application. |
+ | |

+ | When using '''Forward Mode''' this roughly means that a numerical value is equipped with its derivative with respect to one of your input, which is updated accordingly on every function application. |
||

Let the number <math>x_0</math> be equipped with the derivative <math>x_1</math>: <math>\langle x_0,x_1 \rangle</math>. |
Let the number <math>x_0</math> be equipped with the derivative <math>x_1</math>: <math>\langle x_0,x_1 \rangle</math>. |
||

For example the sinus is defined as: |
For example the sinus is defined as: |
||

− | * \sin\langle x_0,x_1 \rangle = \langle \sin x_0, x_1\cdot\cos x_0\rangle |
+ | * <math>\sin\langle x_0,x_1 \rangle = \langle \sin x_0, x_1\cdot\cos x_0\rangle</math> |

− | You see, that's just estimating errors as in physics. |
+ | |

− | However, it becomes more interesting for vector functions. |
+ | Replacing this single derivative with a lazy list of them can enable you to compute an entire derivative tower at the same time. |

+ | |||

+ | However, it becomes more difficult for vector functions, when computing the derivatives in reverse, when computing towers, and/or when trying to minimize the number of computations needed to compute all of the kth partial derivatives of an n-ary function. |
||

+ | |||

+ | Forward mode is suitable when you have fewer arguments than outputs, because it requires multiple applications of the function, one for each input. |
||

+ | |||

+ | Reverse mode is suitable when you have fewer results than inputs, because it requires multiple applications of the function, one for each output. |
||

Implementations: |
Implementations: |
||

− | * [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/fad fad] |
+ | * [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/ad ad] (forward, forward w/ tower, reverse and other modes) |

− | * [[Vector-space]] |
+ | * [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/fad fad] (forward mode tower) |

− | * http://comonad.com/haskell/monoids/dist/doc/html/monoids/Data-Ring-Module-AutomaticDifferentiation.html |
+ | * [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/rad rad] (reverse mode) |

+ | * [[Vector-space]] (forward mode tower) |
||

+ | * [http://comonad.com/haskell/monoids/dist/doc/html/monoids/Data-Ring-Module-AutomaticDifferentiation.html Data.Ring.Module.AutomaticDifferentiation](forward mode) |
||

== Power Series == |
== Power Series == |
||

− | You may count arithmetic with power series also as Automatic Differentiation, |
+ | If you can compute all of the derivatives of a function, you can compute Taylor series from it. |

− | since this means just working with all derivatives simultaneously. |
||

Implementation with Haskell 98 type classes: |
Implementation with Haskell 98 type classes: |
||

− | http://darcs.haskell.org/htam/src/PowerSeries/Taylor.hs |
+ | http://code.haskell.org/~thielema/htam/src/PowerSeries/Taylor.hs |

With advanced type classes in [[Numeric Prelude]]: |
With advanced type classes in [[Numeric Prelude]]: |
||

Line 27: | Line 27: | ||

* [[Functional differentiation]] |
* [[Functional differentiation]] |
||

* Chris Smith in Haskell-cafe on [http://www.haskell.org/pipermail/haskell-cafe/2007-November/035477.html Hit a wall with the type system] |
* Chris Smith in Haskell-cafe on [http://www.haskell.org/pipermail/haskell-cafe/2007-November/035477.html Hit a wall with the type system] |
||

+ | * Edward Kmett in StackOverflow on [http://stackoverflow.com/questions/2744973/is-there-any-working-implementation-of-reverse-mode-automatic-differentiation-for Is there any working implementation of reverse mode automatic differentiation for Haskell?] |
||

+ | * Edward Kmett in Comonad.Reader on [http://comonad.com/reader/2010/reverse-mode-automatic-differentiation-in-haskell/ Reverse Mode Automatic Differentiation in Haskell] |
||

+ | * Conal M. Elliott in [https://vimeo.com/album/126865/video/6622658 Beautiful Diﬀerentiation](video) from. International Conference on Functional Programming (ICFP) Edinburgh 2009. Kindly recorded and posted by Malcolm Wallace. |
||

[[Category:Mathematics]] |
[[Category:Mathematics]] |

## Latest revision as of 04:23, 7 October 2013

**Automatic Differentiation** enables you to compute both the value of a function at a point and its derivative(s) at the same time.

When using **Forward Mode** this roughly means that a numerical value is equipped with its derivative with respect to one of your input, which is updated accordingly on every function application.
Let the number *x*_{0} be equipped with the derivative *x*_{1}: .
For example the sinus is defined as:

Replacing this single derivative with a lazy list of them can enable you to compute an entire derivative tower at the same time.

However, it becomes more difficult for vector functions, when computing the derivatives in reverse, when computing towers, and/or when trying to minimize the number of computations needed to compute all of the kth partial derivatives of an n-ary function.

Forward mode is suitable when you have fewer arguments than outputs, because it requires multiple applications of the function, one for each input.

Reverse mode is suitable when you have fewer results than inputs, because it requires multiple applications of the function, one for each output.

Implementations:

- ad (forward, forward w/ tower, reverse and other modes)
- fad (forward mode tower)
- rad (reverse mode)
- Vector-space (forward mode tower)
- Data.Ring.Module.AutomaticDifferentiation(forward mode)

## [edit] 1 Power Series

If you can compute all of the derivatives of a function, you can compute Taylor series from it.

Implementation with Haskell 98 type classes: http://code.haskell.org/~thielema/htam/src/PowerSeries/Taylor.hs

With advanced type classes in Numeric Prelude: http://hackage.haskell.org/packages/archive/numeric-prelude/0.0.5/doc/html/MathObj-PowerSeries.html

## [edit] 2 See also

- Functional differentiation
- Chris Smith in Haskell-cafe on Hit a wall with the type system
- Edward Kmett in StackOverflow on Is there any working implementation of reverse mode automatic differentiation for Haskell?
- Edward Kmett in Comonad.Reader on Reverse Mode Automatic Differentiation in Haskell
- Conal M. Elliott in Beautiful Diﬀerentiation(video) from. International Conference on Functional Programming (ICFP) Edinburgh 2009. Kindly recorded and posted by Malcolm Wallace.