This document presents mathematical formulas for calculating gradients and updates in reinforcement learning. It defines a formula for calculating the gradient of a value function with respect to its parameters, a formula for calculating the gradient of a policy based on the reward and value, and a formula for calculating the gradient of a parameter vector that is a weighted combination of its pre