Automatic Differentiation

In mathematics and computer algebra, automatic differentiation (AD), sometimes alternatively called algorithmic differentiation, is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, and accurate to working precision.

Automatic differentiation is not:

  • Symbolic differentiation, or
  • Numerical differentiation (the method of finite differences).

These classical methods run into problems: symbolic differentiation works at low speed, and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation. Both classical methods have problems with calculating higher derivatives, where the complexity and errors increase. Finally, both classical methods are slow at computing the partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Automatic differentiation solves all of these problems.

Read more about Automatic DifferentiationThe Chain Rule, Forward and Reverse Accumulation, Automatic Differentiation Using Dual Numbers, Implementation, Software

Other articles related to "automatic differentiation, differentiation":

Automatic Differentiation - Software
... pyderiv GPL OO automatic differentiation and (co)variance calculation CasADi LGPL OO/SCT Python front-end to CasADi..NET Package License Approach Brief Info AutoDiff GPL OO Automatic differentiation with C# operators overloading ... FuncLib MIT OO Automatic differentiation and numerical optimization, operator overloading, unlimited order of differentiation, compilation to IL code for very fast evaluation ...

Famous quotes containing the word automatic:

    The ruin of the human heart is self-interest, which the American merchant calls self-service. We have become a self- service populace, and all our specious comforts—the automatic elevator, the escalator, the cafeteria—are depriving us of volition and moral and physical energy.
    Edward Dahlberg (1900–1977)