2 Calculus for Machine Learning

Machine learning (ML) runs on calculus. Every time a later chapter says “train by minimizing the loss” (a single number scoring how wrong a model’s predictions are), it is silently using calculus. A derivative measures how the loss changes, and the gradient points the way uphill (so a step against it goes downhill). The chain rule is what makes back-propagation (the procedure that works out how the loss changes with every adjustable number) possible, and convexity is why some losses have no wrong valley to get stuck in.

This chapter builds the calculus from zero — what a derivative is, what it means, and how to compute the handful you need — so the training story in Loss Functions & Regularizers §3 and beyond reads like plain English.

What this chapter assumes: school algebra, and — for the many-variable sections (§4 to §8) — Linear Algebra §1 to §6 (vectors, the dot product and the idea of a matrix in its §1 to §4; eigenvalues in its §5 and the quadratic form and singular values in its §6, used in this chapter’s §6 and §8). Not assumed: limits (§1 builds the idea where it is needed), summation notation (glossed in §3), and trigonometry beyond two values, \(\cos0°=1\) and \(\cos90°=0\) (used in §4). Linear Algebra and Probability can be read in either order; Calculus sections 4 to 8 lean on Linear Algebra sections 1 to 6, so read Linear Algebra before them. Start here if derivatives, the chain rule, the gradient, or “why gradient descent goes downhill” are not yet second nature.

Scope. This is an operational primer: we explain and use each tool, with the rules stated, not proved from the formal definition of a limit (the so-called “epsilon–delta” definition). That rigor belongs to a course in real analysis; here we want fluency, not foundations-of-analysis.

1. What is a derivative? (and why “derivative”)

A streaming service nudges one setting of its recommender by a tiny amount, and the rating it predicts for one user moves. Does it move a lot or hardly at all? To know, you need a measure of how steeply the output responds to a small push at the exact point where you stand.

A derivative measures how fast a function changes — the slope of its graph at a point. Imagine zooming in on a curve until it looks like a straight line; the derivative is that line’s slope (the line that just touches the curve at that point is called the tangent).

Formally it is the limit of a difference quotient — rise over run as the run shrinks to zero: \[ f'(x) \;=\; \lim_{h \to 0}\frac{f(x+h)-f(x)}{h}. \]

\(f(x+h)-f(x)\) is the rise (change in output).
\(h\) is the run (a tiny change in input).
The limit \(h\to 0\) says: take the run smaller and smaller until the ratio settles.

Why the names. Derivative is from Latin derivare, “to draw off / lead from” — the new function is derived from the old one. Differentiate comes from difference: the derivative is the limit of those differences \(f(x+h)-f(x)\). The notation \(\frac{df}{dx}\) (Leibniz) literally reads “a tiny difference in \(f\) per tiny difference in \(x\).”

You will meet three spellings of the same derivative, and they mean exactly the same thing: \[ f'(x) \;=\; \frac{df}{dx} \;=\; \frac{d}{dx}f(x), \]

read “\(f\) prime of \(x\)”, “dee-\(f\) dee-\(x\)”, and “the derivative of \(f(x)\) with respect to \(x\)”. Written in front of an expression, \(\frac{d}{dx}\) means “take the derivative of this with respect to \(x\)” — so \(\frac{d}{dx}\,x^2\) is the derivative of \(x^2\).

Worked example. Take \(f(x)=x^2\) and compute the slope at \(x=3\) from the definition: \[ \frac{(3+h)^2-3^2}{h}=\frac{9+6h+h^2-9}{h}=\frac{6h+h^2}{h}=6+h \xrightarrow[h\to0]{} \boxed{6}. \]

So \(f'(3)=6\): near \(x=3\), the output grows about 6 units per 1 unit of input. (The general rule, next section, gives \(f'(x)=2x\), and \(2\cdot 3=6\) ✓.) Watch the ratio settle: with \(h=1,\ 0.1,\ 0.01\) the quotient \(6+h\) is \(7,\ 6.1,\ 6.01\).

The derivative f'(3)=6 is the slope of the tangent line (red) to f(x)=x^2 at x=3 — rise 6 per run 1. — Figure 2.1: The derivative \(f'(3)=6\) is the slope of the tangent line (red) to \(f(x)=x^2\) at \(x=3\) — rise \(6\) per run \(1\). Zooming in, the curve and this line merge (cf. §7). The tangent is at height \(15\) when \(x=4\) while the curve is already at \(16\): that gap is why the quotient is \(7\) for a run of \(1\) and only tends to \(6\) as the run shrinks.

In one line: a derivative is the slope of the tangent, the number the ratio \(\frac{f(x+h)-f(x)}{h}\) settles to as \(h\) shrinks (for \(x^2\) at \(3\): \(7,\ 6.1,\ 6.01\to6\)).

2. The rules you actually use

Training a model means asking, for every one of its settings, how steeply its output responds to a small change in that setting. Working that out from scratch each time would be hopeless, so you want a short list of ready-made answers for the few functions that keep appearing.

The rules below are that list, so you almost never go back to the limit definition of §1 by hand. First, two functions that recur in machine learning and have clean derivatives (the table below uses the first):

Exponential: \(e\approx2.718\) is a special constant (Euler’s number) — the one base for which the curve \(e^x\) is its own slope: \(\frac{d}{dx}e^x = e^x\).
Logarithm: \(\ln x\), the natural logarithm, is the power you must raise \(e\) to in order to get \(x\) (so \(\ln e=1\) and \(\ln 1=0\)), and \(\frac{d}{dx}\ln x = \frac{1}{x}\). In ML writing, \(\log\) almost always means this same \(\ln\) — which is why losses built from \(-\log\) are so convenient to differentiate (the derivative of \(-\ln p\) is just \(-1/p\); the loss was defined in the chapter’s opening; Loss Functions & Regularizers §1 builds it).

The rules, for constants \(c\) and functions \(f,g\):

Rule	Statement	Example
Constant	\(\frac{d}{dx}c = 0\)	flat line, no slope
Power	\(\frac{d}{dx}x^n = n\,x^{n-1}\)	\(\frac{d}{dx}x^2 = 2x\)
Constant multiple	\(\frac{d}{dx}\,c\,f = c\,f'\)	\(\frac{d}{dx}5x^2 = 10x\)
Sum	\((f+g)' = f' + g'\)	\(\frac{d}{dx}(x^2+x)=2x+1\)
Product	\((fg)' = f'g + fg'\)	\(\frac{d}{dx}\,x\,e^x = e^x + x e^x\)
Chain (next section)	\(\frac{d}{dx}f(g(x)) = f'(g)\,g'\)	see §3

The power rule also holds for negative \(n\): \(x^{-1}=\frac1x\) has derivative \(-1\cdot x^{-2}\), which at \(x=2\) is \(-\tfrac14\). Check with \(h=0.001\): \((\frac{1}{2.001}-\frac12)/0.001\approx-0.2499\).

Three of the rules above (the exponential, the logarithm and the product rule) are easy to doubt, so check them at \(x=1\) with a tiny run \(h=0.001\): \((e^{1.001}-e^{1})/0.001\approx2.7196\), close to \(e^1=2.718\); \(\ln(1.001)/0.001\approx0.9995\), close to \(1/1=1\); and for the product rule, \((1.001\,e^{1.001}-1\cdot e^{1})/0.001\approx5.4406\), close to \(e^1+1\cdot e^1=2e\approx5.4366\).

A point where \(f'(x)=0\) is a critical point (also stationary point): the slope is flat — a peak, a valley, or a plateau. At a minimum the derivative is zero (for a curve that has a slope everywhere), so a loss’s lowest points are among its critical points — but a zero slope can also mark a peak, or a flat spot that is neither; §6 shows how to tell them apart. Getting to a low point is the whole game in §5.

Worked example. \(f(x)=x^2-4x\) has \(f'(x)=2x-4\), which is zero at \(x=2\). There \(f(2)=4-8=-4\), the bottom of this parabola: its neighbors \(f(1.9)=f(2.1)=-3.99\) are higher. With one unknown, solving \(f'(x)=0\) is this easy; with millions of tangled unknowns (§5) it is not.

In one line: a few rules (power, sum, product, and next section’s chain rule) turn each function into its slope, and a minimum sits where that slope is \(0\) (for \(x^2-4x\), at \(x=2\)).

3. The chain rule — the engine of back-propagation

A recommender does not compute a user’s score in one step: it combines numbers, applies a fixed curve to the result, combines again, applies the curve again. If you nudge a setting at the start, how do you find its effect on the final score, when the nudge passes through every step on the way?

Functions are often nested: \(f(g(x))\) — “apply \(g\), then \(f\).” The chain rule says the derivative of the whole is the product of the derivatives along the chain: \[ \frac{d}{dx}\,f\big(g(x)\big) \;=\; \underbrace{f'\big(g(x)\big)}_{\text{outer, at the inner value}} \times \underbrace{g'(x)}_{\text{inner}}. \]

Why “chain.” The composition \(x \to g \to f\) is a chain of steps; their rates multiply link by link, like gears. A neural network is a long chain of the same kind: each layer is one simple function — it forms weighted sums of the numbers it receives (each number times an adjustable multiplier, its weight, all added up: inputs \(2\) and \(3\) with weights \(0.5\) and \(-1\) give \(2\cdot0.5+3\cdot(-1)=-2\)), then passes each sum through an activation function (a fixed, simple bend such as “replace a negative number by \(0\)”; more below) — and the last link computes the loss, \(x \to \text{layer}_1 \to \text{layer}_2 \to \dots \to \text{loss}\). So back-propagation is the chain rule applied along it — multiply the per-layer derivatives from the loss backward to each weight, which tells how the loss changes with every weight. Master this one rule and back-propagation (backprop for short) is no longer mysterious.

Why the rates multiply. Nudge \(x\) by a small \(\Delta x\). The inner function moves by \(\Delta g\approx g'\,\Delta x\), and the outer one then moves by \(\Delta f\approx f'\,\Delta g\). So \(\frac{\Delta f}{\Delta x}=\frac{\Delta f}{\Delta g}\cdot\frac{\Delta g}{\Delta x}\): the \(\Delta g\) cancels, and what is left is the product \(f'\cdot g'\).

Worked example. \(k(x)=(3x+1)^2\) (here \(k\) is just a name for this function) is a composition \(f(g(x))\): the inner function is \(g(x)=3x+1\), and the outer one squares whatever it receives, \(f(u)=u^2\) (the letter \(u\) just names the outer function’s input), so \(f'(u)=2u\) and \(g'(x)=3\). By the chain rule,

\[ k'(x) = \underbrace{f'\big(g(x)\big)}_{2(3x+1)} \times \underbrace{g'(x)}_{3} = 6(3x+1). \]

At \(x=1\): \(k'(1)=6(3\cdot1+1)=6\cdot4=\boxed{24}\). (Check: \(k(x)=9x^2+6x+1\), so \(k'(x)=18x+6\), and \(18+6=24\) ✓ — two routes, same answer. The cancellation too: nudging \(x\) from \(1\) to \(1.001\) moves \(g\) from \(4\) to \(4.003\), so \(k\) moves by about \(2\cdot4\cdot0.003=0.024\), and \(0.024/0.001=24\).)

The chain rule on a composition x to g to f (the dot in g( . — Figure 2.2: The chain rule on a composition \(x\to g\to f\) (the dot in \(g(\cdot)\) marks the slot where the box’s input goes): each box has a *local rate* (\(g'\), then \(f'\) at the inner value), and the whole derivative is their *product*, \(f'(g(x))\cdot g'(x)\). Back-propagation is this read *right-to-left* — from the loss back to each input — along a long chain of layers.

When a value feeds several paths, the rates add. In a real network a value rarely flows down a single chain — the output of one neuron (one of the numbers a layer puts out, also called a unit) fans out to many units of the next layer. The chain rule then has a sum-over-paths form: if \(x\) reaches the output \(f\) through several intermediates \(u_1,u_2,\dots\), add the contribution of each path. In symbols — where \(\frac{\partial f}{\partial u_i}\) (a curly “partial” \(\partial\), defined properly in §4) means “how fast \(f\) changes when only \(u_i\) moves, the other \(u\)’s held fixed”, and \(\sum_i\) (a capital Greek sigma, S for “sum”) means “add up the term that follows, over every path \(i\)”: \[ \frac{df}{dx} \;=\; \sum_i \frac{\partial f}{\partial u_i}\,\frac{d u_i}{dx}. \]

Each term is one path’s chain-rule product: the rate \(\frac{du_i}{dx}\) at which \(x\) moves \(u_i\), times the rate \(\frac{\partial f}{\partial u_i}\) at which \(u_i\) moves \(f\).

Worked example (a fork). Let \(a=2\) feed two branches, \(b=a^2\) and \(c=3a\), which recombine as \(\mathcal{L}=b+c\) (the curly \(\mathcal{L}\) is this book’s letter for a loss: think of it as the loss at the end of a network). Two paths reach \(a\), so add them: \[ \frac{d\mathcal{L}}{da}=\underbrace{\frac{\partial \mathcal{L}}{\partial b}\frac{db}{da}}_{1\,\cdot\,2a} +\underbrace{\frac{\partial \mathcal{L}}{\partial c}\frac{dc}{da}}_{1\,\cdot\,3}=2a+3=\boxed{7}\quad(a=2). \]

(Check: \(\mathcal{L}=a^2+3a\), so \(\frac{d\mathcal{L}}{da}=2a+3=7\) ✓ — same answer.) This summing of gradients where paths meet — a gradient, for now, is the derivative of the loss with respect to some value inside the network, like \(\frac{d\mathcal{L}}{da}\) here (§4 gives the full many-variable definition) — is exactly what back-propagation does at every value in a network that fans out, like \(a\) here. It is also why, when layers are chained, their tables of derivatives (the Jacobians of §8) combine by multiplying the tables, row times column (§8 works one out): each entry of the product adds up the paths through the middle layer.

A value that fans out: a feeds two branches (b,c) that recombine in L. — Figure 2.3: A value that *fans out*: \(a\) feeds two branches (\(b,c\)) that recombine in \(\mathcal{L}\). Each edge carries a *local rate* (blue); a path’s effect is the *product* along it, and \(\frac{d\mathcal{L}}{da}\) is the *sum* over the two paths, \(1\!\cdot\!2a+1\!\cdot\!3\). “Gradients add where paths meet” is what turns the single chain above into a whole *network*.

The two derivatives back-propagation multiplies most

The chain rule is not abstract: at every layer, backprop multiplies in the derivative of that layer’s activation function — the fixed, simple nonlinear “bend” each layer applies to its weighted sums before passing them on (nonlinear: not a straight line; without such a bend, a stack of layers would collapse into a single straight-line function, no more expressive than one layer: two layers that only multiply, \(x\to3x\to2(3x)\), are just \(6x\)). Two common ones are worth deriving once — you meet them in the Probability and Neural-Network chapters, and this is the derivation they point back to.

Sigmoid — sigmoid means “S-shaped”, after the Greek letter sigma (\(\sigma\), the Greek S) — is \(\sigma(z)=\dfrac{1}{1+e^{-z}}\): an S-shaped curve that squashes any number \(z\) into the range between \(0\) and \(1\). At \(z=0\), \(e^{0}=1\) gives \(\sigma(0)=\tfrac12\); a large positive \(z\) makes \(e^{-z}\) tiny, so \(\sigma\) is nearly \(1\); a large negative \(z\) makes \(e^{-z}\) huge, so \(\sigma\) is nearly \(0\). To differentiate it, write it as \((1+e^{-z})^{-1}\) and apply the chain rule. Outer: the power rule of §2 holds for negative powers (there, \(x^{-1}\) has slope \(-x^{-2}\)), so with \(n=-1\) the outer derivative is \(-(1+e^{-z})^{-2}\). Inner: \(\tfrac{d}{dz}(1+e^{-z})=-e^{-z}\) — itself a small chain rule, since \(e^{-z}\) is the exponential (its own slope, §2) of \(-z\), whose slope is \(-1\), so its derivative is \(e^{-z}\cdot(-1)\). Multiply the two: \[ \sigma'(z)=-(1+e^{-z})^{-2}\cdot(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^{2}} =\underbrace{\frac{1}{1+e^{-z}}}_{\sigma}\cdot\underbrace{\frac{e^{-z}}{1+e^{-z}}}_{1-\sigma} =\boxed{\sigma(z)\big(1-\sigma(z)\big)}, \]

noting that \(1-\sigma(z)=\dfrac{e^{-z}}{1+e^{-z}}\) (subtract \(\sigma=\tfrac{1}{1+e^{-z}}\) from \(1\)) — making the factorization hand-checkable.

The result is the clean \(\sigma'=\sigma(1-\sigma)\): the derivative reuses the value you already computed. (Check at \(z=2\): \(\sigma=0.8808\), so \(\sigma'=0.8808\times0.1192=0.105\) ✓.) A few values show both shapes at once — \(\sigma\) climbs through \(0.5\) like an S, while \(\sigma'\) is a symmetric bump that peaks at \(z=0\). Five values suffice because \(\sigma(-z)=1-\sigma(z)\), so \(\sigma'\) takes the same value at \(z\) and \(-z\) (the product \(p(1-p)\) does not change when \(p\) and \(1-p\) swap), and the table below shows it falling from its peak of \(0.250\) at \(z=0\) to \(0.105\) at \(z=\pm2\) and \(0.018\) at \(z=\pm4\):

\(z\)	\(-4\)	\(-2\)	\(0\)	\(2\)	\(4\)
\(\sigma(z)\)	\(0.018\)	\(0.119\)	\(0.500\)	\(0.881\)	\(0.982\)
\(\sigma'(z)\)	\(0.018\)	\(0.105\)	\(0.250\)	\(0.105\)	\(0.018\)

ReLU — short for rectified linear unit; rectified because, like an electrical rectifier, it lets the positive half of its input through and flattens the negative half to \(0\) — is \(\text{ReLU}(z)=\max(0,z)\), “the larger of \(0\) and \(z\)”. It is two straight pieces, so its derivative is each piece’s slope: \(\text{ReLU}'(z)=1\) for \(z>0\) and \(0\) for \(z<0\). For example, \(\text{ReLU}(-2)=0\) and \(\text{ReLU}(3)=3\), and the two pieces give slope \(0\) at \(z=-2\) and \(1\) at \(z=3\).

At exactly \(z=0\) the two pieces meet at a kink, so no single slope exists: the left slope is \(0\), the right slope is \(1\).

Any value from \(0\) to \(1\) is then a valid subgradient there, a stand-in slope for a point where the true derivative is undefined (sub, “under”: a line through the kink with any such slope stays on or under the graph). Deep-learning software (the libraries that train deep networks, those with many layers) just picks \(0\).

That on/off gradient is why a unit stuck at \(z<0\) stops learning — the “dead ReLU” of the Neural-Networks chapter.

A first taste of vanishing gradients — by the chain rule, with a number. How big is \(\sigma'\)? At most how big? Since \(\sigma\) always lies between \(0\) and \(1\), write \(p=\sigma(z)\): then \(\sigma'=p(1-p)=p-p^2\), a downward parabola in \(p\) whose own derivative \(1-2p\) is zero at \(p=0.5\) (reached at \(z=0\)). So \(\sigma'\) is at most \(0.5\times(1-0.5)=0.25\) — the peak of the \(\sigma'\) row in the table above. In a stack of sigmoid layers, back-propagation multiplies in one such activation factor \(\sigma'\) per layer (the chain rule above), so the activation factors alone multiply the gradient that reaches an early layer by at most \(0.25\) raised to the depth:

\[ 0.25^2=\tfrac{1}{16}=0.0625,\quad 0.25^4=\tfrac{1}{256}\approx0.0039,\quad 0.25^5=\tfrac{1}{1024}\approx0.00098 . \]

What depth does to it. After just five sigmoid layers the activation factors have cut the gradient by at least a thousand-fold (\(0.25\) is the best case, so the real cut is larger). The weights multiply in too: in a chain with one neuron per layer and the same weight \(w\) at every layer, each layer’s full factor is \(w\,\sigma'\), so a weight bigger than \(4\) could offset a factor of \(0.25\) (\(4\times0.25=1\)). A weight of size \(2\) does not rescue such a chain: each layer’s full factor is then at most \(2\times0.25=0.5\), and \(0.5^5\approx0.031\) after five layers, which is smaller still with more layers. So the early-layer gradient vanishes toward zero as depth grows, and those layers barely learn.

This product is the whole intuition behind the vanishing-gradient problem (see also §8 for the matrix form). It is one reason, a claim from practice that this chapter does not test, why ReLU (slope \(1\) on its live half, \(z>0\), so it multiplies by \(1\) instead of at most \(0.25\)) largely replaced the sigmoid in deep hidden layers (the layers between a network’s input and its output, such as the middle layer of input, middle, output) — helped by careful weight initialization: choosing the weights’ random starting values at a scale that keeps the product of the weight factors from collapsing too (with ReLU, five layers of weight \(0.5\) pass on \(0.5^5\approx0.031\) of the gradient; weight \(1\) passes on all of it).

Figure 2.4: **Vanishing gradient: sigmoid vs. ReLU, by the chain rule.** Each sigmoid layer’s activation factor \(\sigma'\) is at most \(0.25\), so after \(k\) layers the activation factors alone have scaled the gradient by at most \(0.25^k\) (the arithmetic is in the box above); a ReLU’s live-half slope is \(1\), so its factors multiply to \(1^k=1\). The weights multiply in too; the plot isolates the activation factors. The vertical axis is logarithmic: each gridline is \(10\times\) smaller than the one above, so repeated multiplication by \(0.25\) plots as a straight line.

In one line: the chain rule multiplies the local rates along a chain and adds across paths, and back-propagation is that rule run from the loss backward (for \((3x+1)^2\) at \(x=1\): \(6\cdot4=24\); for the fork: \(2a+3=7\)).

4. Partial derivatives and the gradient

A recommender’s model has not one setting but millions, and each setting, called a weight, moves the model’s mistakes up or down in its own way. Moving the weights one at a time is hopeless; you need one answer per setting, all found together.

A loss (the single number that scores the model’s mistakes) depends on all of those weights (a model’s adjustable numbers are its parameters, and nudging them to shrink the loss is training), so it is a function of many inputs. For a function \(f(x,y,\dots)\) of several variables, a partial derivative \(\frac{\partial f}{\partial x}\) is the derivative with respect to one variable, holding the others fixed. For example, for \(f(x,y)=x^2+xy\), freezing \(y\) leaves \(x^2\) (slope \(2x\)) plus \(y\) times \(x\) (slope \(y\)), so \(\frac{\partial f}{\partial x}=2x+y\).

Why “partial.” You let only part of the input change (one knob), freezing the rest — so you see only part of the total variation. The curly \(\partial\) (“partial-dee”) distinguishes it from the ordinary \(d\) of one-variable calculus.

Stack all the partials into one vector — an ordered list of numbers, written in square brackets, as in \([4,\,1]\) — and you get the gradient, written \(\nabla f\): \[ \nabla f = \left[\frac{\partial f}{\partial x},\; \frac{\partial f}{\partial y},\; \dots\right]. \]

Why “gradient” and “\(\nabla\).” Gradient is from Latin gradus, “step / slope / grade” (as in the grade of a road). It is the multivariable slope. The symbol \(\nabla\) is called nabla (after an ancient harp of that shape) or del; \(\nabla f\) reads “del \(f\)” or “grad \(f\).”

Worked example. \(f(x,y)=x^2+xy\).

\[ \frac{\partial f}{\partial x}=2x+y \quad(\text{treat }y\text{ as constant}), \qquad \frac{\partial f}{\partial y}=x \quad(\text{treat }x\text{ as constant}). \]

At the point \((x,y)=(1,2)\): \(\;\nabla f = [\,2(1)+2,\;\;1\,] = \boxed{[4,\;1]}\).

This vector is the central object of training — nudging a model’s adjustable numbers, its parameters (in a neural network, mostly its weights), step by step until the loss is as small as possible: the gradient of the loss with respect to the parameters tells the optimizer which way to move every parameter at once (the optimizer is the procedure that updates the parameters; gradient descent, §5, is the basic one).

Picture \(f(x,y)\) as a landscape whose height above the point \((x,y)\) is \(f\). A contour (or level set) is a curve along which \(f\) keeps one constant value — like a height line on a hiking map (for \(f=x^2+y^2\), the level \(f=4\) is the circle of radius \(2\)).

The figure below shows a bowl-shaped \(f\) from above, as three of its contours.

A bowl-shaped f seen from above: each gray ring is a contour (constant f). — Figure 2.5: A bowl-shaped \(f\) seen from above: each gray ring is a *contour* (constant \(f\)). The gradient \(\nabla f\) is *perpendicular* to the contour and points *uphill* (steepest increase); a descent step follows \(-\nabla f\) toward the center (the minimum). This is the picture behind §5.

Three facts about vectors (the Linear-Algebra primer §1 and §4 tell the full story). In this chapter vectors go in square brackets and points in round ones. The length of a vector \([a,b]\) is \(\sqrt{a^2+b^2}\) (Pythagoras’ theorem), written \(\lVert[a,b]\rVert\) (so \(\lVert[3,4]\rVert=5\)). A unit vector has length \(1\) — a pure direction, such as \([0.6,0.8]\) — and wears a hat, as in \(\hat{\mathbf d}\) (a bold letter marks a vector; by ML custom some vectors stay plain — the gradient \(\nabla f\) is one — and the text says so each time a plain letter stands for a vector).

The dot product multiplies matching entries and adds them up: \([4,1]\cdot[1,0]=4\cdot1+1\cdot0=4\). It also equals (length of one) \(\times\) (length of the other) \(\times\) \(\cos(\text{the angle between them})\), so for a unit direction \(\hat{\mathbf d}\) the dot product \(\nabla f\cdot\hat{\mathbf d}\) is largest when \(\hat{\mathbf d}\) points the same way as \(\nabla f\) (\(\cos0°=1\)) and zero when the two are perpendicular (\(\cos90°=0\)).

Why is \(\nabla f\) perpendicular to the contour — and why steepest? Walk along a contour and \(f\) doesn’t change, so the rate of change in that direction is \(0\). The rate of change in any unit direction \(\hat{\mathbf d}=[d_1,d_2]\) is the directional derivative \(\nabla f\cdot\hat{\mathbf d}=\frac{\partial f}{\partial x}d_1+\frac{\partial f}{\partial y}d_2\): per unit of step, moving along \(\hat{\mathbf d}\) changes \(x\) by \(d_1\) and \(y\) by \(d_2\), and each change contributes its own partial derivative times its size. By the fact just above, this is largest when \(\hat{\mathbf d}\) aligns with \(\nabla f\) and zero when it is perpendicular. So zero change is along the contour, and fastest change is the direction perpendicular to it — exactly where \(\nabla f\) points.

Worked example — directional derivative. Reuse \(f(x,y)=x^2+xy\) at \((1,2)\) where \(\nabla f=[4,1]\) (from above).

Along \(x\): take \(\hat{\mathbf d}=[1,0]\) (the unit vector pointing right). The directional derivative is \(\nabla f\cdot\hat{\mathbf d}=[4,1]\cdot[1,0]=4\cdot1+1\cdot0=4\) — which matches \(\tfrac{\partial f}{\partial x}=4\) exactly, as it should: \(\hat{\mathbf d}=[1,0]\) is the \(x\) direction.
Along the contour: rotating a 2-D vector \([a,b]\) by \(90°\) gives \([-b,a]\) (swap the entries and negate the new first one), so \([4,1]\) turns into \([-1,4]\). Its length is \(\sqrt{(-1)^2+4^2}=\sqrt{17}\), so dividing by \(\sqrt{17}\) makes it a unit vector: \(\hat{\mathbf d}_\perp=\tfrac{1}{\sqrt{17}}[-1,4]\). Then \(\nabla f\cdot\hat{\mathbf d}_\perp=[4,1]\cdot\tfrac{1}{\sqrt{17}}[-1,4]=\tfrac{-4+4}{\sqrt{17}}=0\) (the zero does not even need the rescaling).
Check that this really is the contour direction. Test it on \(f\) itself, which is \(3\) at \((1,2)\): a tiny step \(0.001\cdot[-1,4]=[-0.001,\,0.004]\) lands at \((0.999,\,2.004)\), where \(f=2.999997\) — essentially unchanged — while a step of the same length along the gradient, \(0.001\cdot[4,1]\), lands at \((1.004,\,2.001)\), where \(f=3.017\). So \(f\) stays level along \(\hat{\mathbf d}_\perp\): it is the contour direction, and it is perpendicular to \(\nabla f\) — the rate of change along the contour is zero, exactly as the box above says.

In one line: the gradient is the list of partial derivatives and points uphill (\([4,1]\) for \(x^2+xy\) at \((1,2)\)), so its negative points downhill.

5. The gradient points uphill → gradient descent

You are on a foggy hillside and want to reach the lowest ground, but the fog hides everything except the slope under your feet. You cannot work out where the valley floor is; you can only feel which way the ground tilts, take a small step the other way, and feel again. A model faces the same problem: the loss (written \(f\) from here on, \(\mathcal L\) in §3) is the hillside, and the slope under its feet is the gradient of §4, the list of slopes, one per adjustable number. This section turns the walk into a rule.

Here is the fact that makes machine learning work:

The gradient \(\nabla f\) points in the direction of steepest increase of \(f\), and its negative \(-\nabla f\) points in the direction of steepest decrease.

So to minimize a loss, repeatedly take a small step against the gradient: \[ \theta \;\leftarrow\; \theta \;-\; \eta\,\nabla f(\theta). \]

\(\theta\) — the parameters you are tuning, the whole list of them (a vector, though written as a plain letter): the weights, and in the recommenders of this book also the embeddings (the lists of numbers learned to represent each user and item; Linear-Algebra primer §1).
\(\nabla f(\theta)\) — the gradient of the loss \(f\) at the current point (which way is uphill).
\(\eta\) (the Greek letter eta; the letter is pure convention — some books write \(\alpha\) — so don’t read meaning into it) — the learning rate (step size): how far to step.
The minus sign — go downhill, opposite the uphill gradient.
\(\leftarrow\) — “becomes”: compute the right-hand side with the current \(\theta\), then overwrite \(\theta\) with the result, and repeat. It is an update, not an equation (\(\theta=\theta-\eta\nabla f\) as an equation would force \(\eta\nabla f=0\)).

Why the names. Gradient descent — you descend the slope the gradient reveals. Learning rate — \(\eta\) controls how fast the model “learns” each step: too small and training crawls; too large and it overshoots and diverges.

Why search at all? §2 said that at a minimum the derivative is zero, so you might ask why we do not simply solve \(\nabla f(\theta)=\mathbf 0\) (the bold \(\mathbf 0\) is a vector of zeros: every partial derivative zero at once) and pick the lowest solution. With one unknown that works, as in §2’s \(x^2-4x\). A loss with millions of parameters gives millions of equations in which each one involves many of the unknowns at once, and for the networks of this book no formula that solves them is known (a formula you can write down and simply evaluate, the way \(x=-b/(2a)\) gives the bottom of a parabola \(ax^2+bx+c\)). So we search instead: start somewhere and repeatedly step downhill. (The closed-form recommenders later in the book are models that do have such a formula; they take no gradient steps.)

Worked example — minimize \(f(x)=x^2\) by hand (here \(\nabla f = f'(x)=2x\)), starting at \(x_0=3\) with \(\eta=0.1\):

step	\(x\)	\(f'(x)=2x\)	update \(x-\eta f'(x)\)	\(f(x)=x^2\)
0	\(3.000\)	\(6.00\)	\(3-0.1(6.00)=2.400\)	\(9.000\)
1	\(2.400\)	\(4.80\)	\(2.4-0.1(4.80)=1.920\)	\(5.760\)
2	\(1.920\)	\(3.84\)	\(1.92-0.1(3.84)=1.536\)	\(3.686\)
3	\(1.536\)	—	—	\(2.359\)

The value marches downhill \(9 \to 5.76 \to 3.686 \to 2.359\) (the last column; the descent figure after the next two boxes plots these same points), heading for the true minimum at \(x=0\). This is the training loop behind most learned models in the book — applied not to one number \(x\) but to a whole parameter vector \(\theta\), with the gradient from §4. (Loss Functions & Regularizers §3 re-tells this story in ML terms; the engine is exactly the table above.)

In practice — stochastic gradient descent. A real loss is an average over millions of training examples (the data points a model learns from), so computing the exact \(\nabla f\) at every step is too costly.

Stochastic gradient descent (SGD) — stochastic is from the Greek stokhastikos, “able to guess”; today it simply means random — estimates the gradient on a small random minibatch (a handful of training examples drawn at random) and steps on that. Noisier, far cheaper, and what most models in the book train with (a claim from practice, not tested here).

Example. Four examples have per-example gradients \(2,4,6,8\) at the current \(\theta=3\), so the exact gradient is their average, \(5\), and a step with \(\eta=0.1\) lands at \(3-0.5=2.5\). A minibatch of two examples, \(4\) and \(8\), estimates the gradient as \(6\) and lands at \(2.4\): close, from half the work. Another pair, \(2\) and \(4\), estimates it as \(3\) and lands at \(2.7\): the estimate is cheap, but how good it is depends on the draw.

Momentum keeps a running total of recent gradients, \(v\leftarrow0.9\,v+g\) (the factor \(0.9\) is how much of the past is kept), and steps by \(\eta v\). With \(v=10\) and a new gradient \(g=4\), \(v=13\) and the step (with \(\eta=0.1\)) is \(1.3\), not the plain \(0.4\).

Adam (short for adaptive moment estimation; Kingma and Ba, 2015) divides each parameter’s step by the recent size of its own gradient, so a steady gradient of \(100\) and a steady one of \(0.01\) both move their parameters by roughly \(\eta\) per step. The Neural-Networks chapter covers momentum and Adam.

In practice — autograd, schedules, and gradient pathologies (optional on a first reading). Practitioners rarely apply the chain rule by hand.

Automatic differentiation (autograd). PyTorch and JAX — two popular software libraries, written for the Python programming language, for building and training neural networks — record each arithmetic operation as it runs, in a computation graph (the record of which value fed which: exactly the box-and-arrow pictures of §3). One command (in PyTorch, loss.backward()) then walks that graph in reverse, applying the chain rule, and stores the gradient of the loss with respect to each parameter next to that parameter (in PyTorch, in its .grad field).

Optimizers. The optimizer (§4) reads those gradients and updates the parameters: torch.optim.SGD applies exactly the step \(\theta \leftarrow \theta - \eta\,\nabla f\) derived here, and torch.optim.Adam applies Adam’s adaptive version.

Learning-rate schedules. The learning rate \(\eta\) is rarely constant: training often raises it over a short warm-up (for example \(0.02,0.04,0.06,0.08,0.1\) over five steps) and then lowers it (decay, for example \(\times0.9\) per epoch (one pass over the training data), so \(0.1\) becomes \(0.035\) after ten epochs), so that late updates settle finely; Neural Networks & Back-propagation §6 builds the common schedules on its running neuron and draws them.

Exploding gradients. Write \(\lVert\nabla f\rVert\) for the gradient’s length, the square root of the sum of the squares of its entries (§4). If it grows without bound as it is passed back through the layers — exploding gradients — clip it: cap its length at a fixed ceiling. For the gradient \([6,8]\), of length \(10\), a ceiling of \(1\) rescales it to \([0.6,0.8]\): same direction, length \(1\).

Vanishing gradients. If the gradient shrinks toward zero (§3), layer normalization (rescaling each layer’s outputs to a standard average and spread: \([1,3]\) has average \(2\) and typical distance \(1\) from it, so it becomes \([-1,1]\); Loss Functions & Regularizers §5.2 and Neural Networks & Back-propagation) helps, and so do residual connections (shortcuts that let the signal skip over layers: the output is the input plus the layer’s change, so \(5+0.3=5.3\) still carries the input \(5\) when the change is tiny), which are built in Representation Learning & the Transformer (both are claims from practice, not tested here).

Gradient descent on f(x)=x^2 from x0=3 with eta=0.1 (the table above): each red step moves against the slope, so x slides down the bowl toward the minimum at 0. — Figure 2.6: Gradient descent on \(f(x)=x^2\) from \(x_0=3\) with \(\eta=0.1\) (the table above): each red step moves *against* the slope, so \(x\) slides down the bowl toward the minimum at \(0\). Steps shrink as the slope flattens.

The learning rate is the first knob you tune. Same loss \(f(x)=x^2\), same start \(x_0=3\) — only \(\eta\) changes:

\(\eta\)	factor \(1-2\eta\)	\(x\) across steps \(0\!\to\!3\)	behavior
\(0.1\) (small)	\(0.8\)	\(3,\ 2.4,\ 1.92,\ 1.54\)	safe, but crawls toward \(0\)
\(0.5\) (large but stable)	\(0\)	\(3,\ 0,\ 0,\ 0\)	lands on \(0\) in one step here (see caveat below)
\(0.9\) (overshoots, still converges)	\(-0.8\)	\(3,\ -2.4,\ 1.92,\ -1.54\)	flips sides every step, yet shrinks
\(1\) (the edge)	\(-1\)	\(3,\ -3,\ 3,\ -3\)	bounces forever, never settles
\(1.1\) (too large)	\(-1.2\)	\(3,\ -3.6,\ 4.32,\ -5.184\)	overshoots and diverges

Each step multiplies \(x\) by \(1-2\eta\) (the slope is \(2x\)), and that factor explains every row. It is a straight line in \(\eta\): it is \(0\) at \(\eta=0.5\) and \(-1\) at \(\eta=1\), so these five rows cover every regime. A factor of \(0.8\) shrinks \(x\) slowly: about \(36\) steps are needed to get within \(0.001\) of \(0\) (since \(3\times0.8^{36}\approx0.00097\)). The behavior flips at \(\eta=1\), where the factor passes \(-1\) and each step makes \(x\) larger in size.

Too small wastes compute; too large blows up. The one-step landing at \(\eta=0.5\) is special to this quadratic: with \(\eta=1/f''\) — where \(f''\) is the second derivative, the slope of the slope (§6); here \(f''=2\) — a single step lands exactly on the bottom of a parabola. On real losses, \(\eta\) is tuned for steady descent, not a one-shot jump. §7 shows why a small-enough step lowers \(f\).

And in two dimensions it is the same picture. For the bowl \(f(x,y)=x^2+y^2\), started at \((3,3)\) with the same \(\eta=0.1\), the gradient \(\nabla f=[2x,2y]\) sends each coordinate down the very \(3\to2.4\to1.92\) path of the table above, sliding the whole point straight to the minimum:

Gradient descent on f(x,y)=x^2+y^2 from (3,3), eta=0.1: each gray ring is a contour (constant f), every step points along -nabla f (it is -etanabla f, one tenth of it) — perpendicular to the ring, str — Figure 2.7: Gradient descent on \(f(x,y)=x^2+y^2\) from \((3,3)\), \(\eta=0.1\): each gray ring is a contour (constant \(f\)), every step points along \(-\nabla f\) (it is \(-\eta\nabla f\), one tenth of it) — perpendicular to the ring, straight at the center — and *each coordinate* follows the same \(3\to2.4\to1.92\to1.536\) as the 1-D table. Multivariable descent is just the 1-D move run on every parameter at once.

In one line: gradient descent repeats \(\theta\leftarrow\theta-\eta\nabla f\), and \(\eta\) decides whether it crawls, lands or blows up (factor \(1-2\eta\): \(0.8\), \(0\), \(-1.2\) for \(\eta=0.1,\ 0.5,\ 1.1\)).

6. Curvature: second derivatives, the Hessian, convexity

You are walking over uneven ground and it suddenly feels flat underfoot. Are you at the bottom of a valley, on a hilltop, or on a level patch? A steepness of zero cannot tell the three cases apart.

Differentiate twice and you get the second derivative \(f''(x)\) — the rate of change of the slope, i.e. curvature (how the curve bends).

\(f''>0\): slope increasing → curve bends up (a valley, or “cup”).
\(f''<0\): slope decreasing → curve bends down (a hill, or cap).

For example, \(f(x)=x^2\) has \(f'(x)=2x\) and \(f''(x)=2\): at \(x=1,\ 2,\ 3\) the slope is \(2,\ 4,\ 6\), rising as we move right, so the curve bends up.

Two words we will keep using: a local minimum is lowest only within its own neighborhood (a valley floor), while a global minimum is the lowest point anywhere. For example, with two valley floors at heights about \(-0.6\) and \(-1.4\) (the two valleys the right panel of the figure below draws, not to scale), both are local minima and only the one at \(-1.4\) is global. A bumpy (non-convex) loss can have many local minima but only the deepest is global — and plain gradient descent, which only ever steps downhill, can come to rest in whichever valley it happens to fall into.

A function is convex if it curves up everywhere (\(f''\ge 0\), for a function that has a second derivative) — a bowl shape, like \(x^2\), whose \(f''=2\ge0\). Convexity is the property optimizers dream of:

A convex function has no wrong valley — so, with a small enough step, gradient descent cannot get stuck in a bad local minimum. The reason, in one variable: \(f''\ge0\) means the slope never decreases, so once the slope reaches \(0\) it can only stay \(0\) or rise — before that point the curve was falling or level, after it level or rising, so no point is lower.

(It need not have a valley at all — \(e^x\) is convex and simply keeps descending — and a flat-bottomed convex function has a whole set of minimizers, every one of them global.) Every critical point (a place where the gradient is zero — the function momentarily stops changing: a min, a max, or a saddle, a flat point that rises one way and falls another, as below) is a global minimum — there is no local minimum to get stuck in, and no saddle.

(A neural network’s loss is in general not convex in its weights, so there this guarantee is lost; that is a standard fact, not shown here.)

Worked examples (one variable).

\(f(x)=x^2 \Rightarrow f''(x)=2 \ge 0\) everywhere → convex (a single bowl; that is why §5 converged cleanly).
\(f(x)=x^3 \Rightarrow f''(x)=6x\), which is negative for \(x<0\) and positive for \(x>0\) → not convex (it changes bending; the point \(x=0\) where \(f''\) flips sign is an inflection point).

Left: a convex function — here x^2 — curves up everywhere: a single bowl, so gradient descent (with a small-enough step) cannot get stuck and heads to its one global minimum (filled dot). — Figure 2.8: **Left:** a *convex* function — here \(x^2\) — curves up everywhere: a single bowl, so gradient descent (with a small-enough step) cannot get stuck and heads to its one global minimum (filled dot). **Right:** a *non-convex* function with two valleys of *different* depth (filled dots) split by a hump (open dot = local max). The left valley is deeper — the *global* minimum; a descent that starts on the right settles into the shallower *local* minimum and stays *stuck* — the “trap” that plain gradient descent can fall into.

Many variables: the Hessian. In many variables there is no single \(f''\) to check. A function of two variables can be differentiated twice along \(x\), twice along \(y\), or once along each, so it has four second derivatives, written with subscripts:

\(f_{xx}\) — differentiate with respect to \(x\), then with respect to \(x\) again (also written \(\frac{\partial^2 f}{\partial x^2}\)): the curvature along the \(x\) direction;
\(f_{yy}\) — the same along \(y\);
\(f_{xy}\) — differentiate with respect to \(x\), then with respect to \(y\): how the \(x\)-slope changes as you nudge \(y\) — the twist between the two directions. \(f_{yx}\) is the other order; for the smooth (kink-free, differentiable) functions used in this book the order does not matter, \(f_{yx}=f_{xy}\) (it can matter only when the second partials are not continuous).

Arranged in a matrix — a rectangular table of numbers, whose entry \((i,j)\) sits in row \(i\) and column \(j\) — they form the Hessian \(H\). For \(f(x,y)=x^2+xy\) (from §4, where \(\nabla f=[2x+y,\;x]\)), differentiate each gradient entry once more: \(f_{xx}=\tfrac{\partial}{\partial x}(2x+y)=2\), \(f_{xy}=\tfrac{\partial}{\partial y}(2x+y)=1\), \(f_{yx}=\tfrac{\partial}{\partial x}(x)=1\) and \(f_{yy}=\tfrac{\partial}{\partial y}(x)=0\), so

\[ H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{bmatrix} = \begin{bmatrix} 2 & 1 \\ 1 & 0 \end{bmatrix}. \]

Per component: entry \((i,j)\) is how the slope in direction \(i\) changes as you nudge direction \(j\) — the diagonal is pure curvature along each axis, the off-diagonals are how the axes interact. The bottom-right \(0\) here says \(f=x^2+xy\) is linear in \(y\) once \(x\) is fixed (no \(y^2\) term → no curvature that way); the two \(1\)s are the \(xy\) cross-term.

Why “convex” and “Hessian.” Convex is from Latin convexus, “vaulted, arched”: the word was first used for shapes that bulge outward, and for a function it refers to the region above its graph (the inside of the bowl, which has no dents), not to the graph’s own bulge. The multivariable second-derivative object — the matrix of all second partials (\(f_{xx},f_{xy},f_{yx},f_{yy}\) for two variables; with more variables, one for each choice of a first and a second variable to differentiate by) — is the Hessian, named after the 19th-century mathematician Otto Hesse.

Classifying a critical point. At a critical point every first partial is zero, so the surface is flat there in every direction; whether it is a bottom, a top or neither is decided by the curvature. In two variables one number, together with the sign of \(f_{xx}\), settles it (unless the number is \(0\)):

The two-variable second-derivative test. At a critical point, compute

\[ D \;=\; f_{xx}\,f_{yy}\;-\;f_{xy}^{\,2}. \]

\(D>0\) and \(f_{xx}>0\): a bowl — a minimum.

\(D>0\) and \(f_{xx}<0\): a cap — a maximum.

\(D<0\): a saddle — neither: up along some directions, down along others.

\(D=0\): the test is silent — you must look closer some other way.

Per component: \(f_{xx}f_{yy}\) is positive when the two axis curvatures bend the same way, and \(f_{xy}^2\) is the twist, squared. \(D>0\) says the axes agree and outweigh the twist, so the surface bends the same way in every direction — the sign of \(f_{xx}\) says which way. \(D<0\) says either the axes already disagree (as in \(x^2-y^2\) below) or the twist outweighs them (as in \(x^2+xy\)), so some direction bends up and another down.

Worked examples — the chapter’s three functions. First find the critical point (set every first partial to zero), then read off \(D\):

\(f(x,y)\)	critical point (\(\nabla f=\mathbf 0\))	\(f_{xx},\ f_{yy},\ f_{xy}\)	\(D=f_{xx}f_{yy}-f_{xy}^2\)	verdict
\(x^2+y^2\) (the bowl of §5)	\(2x=0,\ 2y=0\): \((0,0)\)	\(2,\ 2,\ 0\)	\(2\cdot2-0^2=4\)	bowl (minimum)
\(x^2+xy\) (§4’s example)	\(2x+y=0,\ x=0\): \((0,0)\)	\(2,\ 0,\ 1\)	\(2\cdot0-1^2=-1\)	saddle
\(x^2-y^2\) (the figure below)	\(2x=0,\ -2y=0\): \((0,0)\)	\(2,\ -2,\ 0\)	\(2\cdot(-2)-0^2=-4\)	saddle

You can confirm the middle verdict by walking two straight lines through the origin: along the \(x\)-axis (\(y=0\)), \(f=x^2\) curves up; along the line \(y=-2x\), \(f=x^2+x(-2x)=-x^2\) curves down. Up one way, down another — like a horse’s saddle — so this critical point is neither a max nor a min.

Convexity in many variables. The one-variable test “\(f''\ge0\) everywhere” becomes a test on the curvature in every direction. Walking through a point along a direction \(\mathbf v=[v_1,v_2]\), the curvature you feel — the second derivative of \(f\) along that line (by the chain rule applied twice, taken on trust here) — is

\[ f_{xx}\,v_1^2+2f_{xy}\,v_1v_2+f_{yy}\,v_2^2 . \]

This is \(\mathbf v^{\top}H\mathbf v\), the quadratic form of the Linear-Algebra primer §6, with \(H\) the matrix of second derivatives (the Hessian, defined above). It grows with the length of \(\mathbf v\): doubling \(\mathbf v\) quadruples it. To get the curvature along the unit direction (length \(1\)) of the same line, divide by \(\lVert\mathbf v\rVert^2\). Two checks:

On \(x^2+xy\) with \(\mathbf v=[1,-2]\), whose length is \(\lVert\mathbf v\rVert=\sqrt5\), so \(\lVert\mathbf v\rVert^2=5\): \(2\cdot1^2+2\cdot1\cdot1\cdot(-2)+0\cdot(-2)^2=-2\), the same \(-2\) as the second derivative of \(-x^2\) (the slice along \(y=-2x\) above). Per unit direction it is \(-2/5=-0.4\).
On the bowl \(x^2+y^2\) with \(\mathbf v=[1,1]\): \(2+0+2=4\), and \(\lVert\mathbf v\rVert^2=2\), so \(4/2=2\) per unit direction, the bowl’s curvature in every direction.

A twice-differentiable \(f\) is convex exactly when this curvature is \(\ge0\) in every direction, at every point. A Hessian with that property is called positive semidefinite (PSD) — the matrix way of saying “\(\ge0\) in every direction”. If the curvature is strictly \(>0\) in every direction (never flat), the Hessian is positive definite. For a \(2\times2\) Hessian both are checked with the numbers you already have: PSD means \(f_{xx}\ge0\), \(f_{yy}\ge0\) and \(D\ge0\); positive definite means \(f_{xx}>0\) and \(D>0\). For \(x^2+y^2\) (\(f_{xx}=2\), \(D=4\)) the Hessian is positive definite; for \(x^2\) (\(f_{xx}=2\), \(D=0\)) it is positive semidefinite only. (The Linear-Algebra primer §6 gives the general matrix definition.)

So a Hessian that is PSD everywhere gives a bowl, a trough (a bowl with a flat floor line), or a surface that slopes away without ever reaching a bottom (like \(e^x\), or a tilted plane such as \(f(x,y)=x+y\)) — but never a wrong valley: every critical point is a global minimum. Positive definite everywhere also rules out the trough: if there is a bottom at all, there is exactly one.

What separates the two is a flat direction: \(f(x,y)=x^2\) has \(f_{xx}=2\), \(f_{yy}=0\), \(f_{xy}=0\), so \(D=0\) — PSD but not positive definite — and its minimizers form the whole \(y\)-axis, a trough (exactly the case where the second-derivative test is silent). In the table, \(x^2+y^2\) has the same Hessian at every point, with \(f_{xx}=2>0\) and \(D=4>0\): positive definite everywhere, a convex bowl with one bottom. If the curvature is up in some directions and down in others (\(D<0\)), the Hessian is indefinite and the critical point is a saddle — so \(x^2+xy\) and \(x^2-y^2\) are not convex.

Optional deeper view — eigenvalues (Linear-Algebra primer §5 and §6). Over all unit directions (length \(1\), §4), the curvature is largest along one direction and smallest along the perpendicular one; those two extreme curvatures are the Hessian’s eigenvalues (German eigen, “own”: the matrix’s own characteristic numbers). For a symmetric \(2\times2\) matrix \(\begin{bmatrix}a&b\\b&d\end{bmatrix}\) — symmetric meaning it equals its mirror image across the main diagonal, as the Hessian of every smooth function here does, because \(f_{xy}=f_{yx}\) — the two eigenvalues are the two solutions \(\lambda\) of

\[ \lambda^2-(a+d)\,\lambda+(ad-b^2)=0 , \]

and their product is \(ad-b^2\), which for a Hessian is exactly \(D\).

So \(D>0\) means both extreme curvatures have the same sign, and \(D<0\) means one of each — a saddle. In this language PSD means both eigenvalues are \(\ge0\), positive definite means both are \(>0\), and indefinite means one of each sign.

Worked cases. For \(x^2+xy\) (\(a=2\), \(b=1\), \(d=0\)): \(\lambda^2-2\lambda-1=0\), so \(\lambda=1\pm\sqrt2\approx2.41\) and \(-0.41\), with product \((1+\sqrt2)(1-\sqrt2)=1-2=-1=D\) ✓.

For \(x^2+y^2\): \(\lambda^2-4\lambda+4=0\), so \(\lambda=2,\,2\) (product \(4\)); for \(x^2-y^2\): \(\lambda^2-4=0\), so \(\lambda=\pm2\) (product \(-4\)).

The figure below draws the cleanest saddle, \(f(x,y)=x^2-y^2\) (the table’s third row, \(D=-4\)) — axis-aligned, so its two slices are easy to see.

A saddle point of f(x,y)=x^2-y^2 at the origin, drawn as its two slices through that point. — Figure 2.9: **A saddle point** of \(f(x,y)=x^2-y^2\) at the origin, drawn as its two slices through that point. Walk along \(x\) (blue, \(f=x^2\)) and the surface *curves up* — the point looks like a *minimum*; walk along \(y\) (red, \(f=-y^2\)) and it *curves down* — the point looks like a *maximum*. Up one way, down another: that is the horse-saddle shape, and why a critical point with \(D=f_{xx}f_{yy}-f_{xy}^2<0\) (here \(2\cdot(-2)-0^2=-4\): an *indefinite* Hessian) is neither a max nor a min. The gradient is still \(\mathbf 0\) here, so it is a critical point that gradient descent can stall near (see below).

How long does descent stall from \((1,\,0.001)\)? Take \(f=x^2-y^2\) and \(\eta=0.1\), started just off the \(x\)-axis, near the saddle at the origin. Each step multiplies \(x\) by \(0.8\) and \(y\) by \(1-0.1\cdot(-2)=1.2\). The point hugs the \(x\)-axis: \(y\) stays below \(0.01\) for the first \(12\) steps and needs about \(38\) steps to grow a thousand-fold, from \(0.001\) to about \(1\) (\(1.2^{38}\approx1000\)), before the point slides away along \(y\). (A start nearer the \(x\)-axis stalls longer; this one is chosen to show the effect.)

In one line: curvature tells a bowl from a cap from a saddle (\(D=4,\ -1,\ -4\) for the three functions above), and a convex function has no wrong valley.

7. Linear approximation and Taylor — why a downhill step works

A recommender’s builder nudges one setting of the model by a small amount and wants to know, before retraining, whether its mistakes will go down. Retraining to find out is slow. Can the steepness at the current setting predict the result of a small step, and how small must the step be to trust that prediction?

Zoom in on any differentiable curve (one with a derivative everywhere, no kinks or jumps) and it looks straight. That straight piece is the linear (or first-order Taylor) approximation: \[ f(x+\delta) \;\approx\; f(x) + f'(x)\,\delta \qquad (\text{small } \delta). \]

\(\delta\) (the Greek letter delta) — the small step you take away from \(x\) (the job \(h\) did in §1).
\(f'(x)\,\delta\) — slope \(\times\) step: the rise the tangent line predicts.
\(\approx\) — “approximately equal”; the smaller \(\delta\), the better the match.

Why “Taylor.” The approximation carries the name of the mathematician Brook Taylor. Near a point, a smooth function is approximated by its value plus slope\(\times\)step plus (curvature\(/2)\times\)step\(^2\) and further terms that shrink faster than the step. Keep just the slope term and you have the linear approximation.

(The everyday word smooth means differentiable here. It is not the stricter Lipschitz-smooth defined below: \(x^4\) is differentiable everywhere, yet its gradient \(4x^3\) changes ever faster, so it has no single such constant.)

Worked example. \(f(x)=x^2\) near \(x=3\) (\(f'(3)=6\)), estimate \(f(3.1)\): \[ f(3.1)\approx f(3)+f'(3)(0.1)=9+6(0.1)=\mathbf{9.6}, \qquad \text{actual } 3.1^2=9.61. \]

The same tangent as §1, now zoomed in near x=3: the curve (blue) and the line (red) almost coincide — that near-coincidence is the linear (first-order Taylor) approximation. — Figure 2.10: The *same* tangent as §1, now *zoomed in* near \(x=3\): the curve (blue) and the line (red) almost coincide — that near-coincidence is the linear (first-order Taylor) approximation. At \(x=3.1\) they differ by only \(0.01\); the closer you stay, the better the line.

Off by only \(0.01\). This is the idea behind gradient descent. The multivariable first-order Taylor is \(f(\theta+\delta)\approx f(\theta)+\nabla f\cdot\delta\) — now \(\theta\) is the whole list of parameters and \(\delta\) is a step vector, one entry per parameter — and the steps below show why a downhill step works (the first four give the idea, the fifth a condition under which one fixed learning rate suffices):

\(\nabla f\cdot\delta\) is the predicted change in \(f\) for a step \(\delta\) — the dot product (§4; Linear-Algebra primer §4) of the uphill direction \(\nabla f\) with where you actually move.
Choose the step straight downhill: \(\delta=-\eta\nabla f\) (against the gradient, scaled by the learning rate \(\eta\)).
Substitute: \(\nabla f\cdot(-\eta\nabla f)=-\eta\,(\nabla f\cdot\nabla f)=-\eta\lVert\nabla f\rVert^2\) — a vector dotted with itself is the sum of its squared entries, which is its squared length (e.g. \([4,1]\cdot[4,1]=16+1=17=\lVert[4,1]\rVert^2\)). So \(f(\theta-\eta\nabla f)\approx f(\theta)-\eta\lVert\nabla f\rVert^2\). For \(f=x^2+xy\) at \((1,2)\) with \(\eta=0.1\) (\(\nabla f=[4,1]\)) the prediction is \(3-0.1\cdot17=1.3\), against the true \(f(0.6,1.9)=1.5\); the gap \(0.2\) is the curvature the straight line ignores.
Why that step lowers \(f\). A squared length can’t be negative, so the predicted change is \(\le 0\) (and \(<0\) whenever \(\nabla f\neq\mathbf 0\)): the step lowers \(f\), provided \(\eta\) is small enough for the straight-line prediction to be trusted — exactly what the learning rate controls.
Formally, if the gradient doesn’t change too fast — i.e. \(\lVert\nabla f(\theta_1)-\nabla f(\theta_2)\rVert \le K\lVert\theta_1-\theta_2\rVert\) for any two parameter settings \(\theta_1,\theta_2\) and some constant \(K\) (the function is then Lipschitz-smooth, after the 19th-century German mathematician Rudolf Lipschitz) — then one fixed learning rate works at every step: any \(\eta\) between \(0\) and \(2/K\) lowers \(f\) whenever \(\nabla f\neq\mathbf 0\) (a standard result, not proved here).

(For \(f(x)=x^2\) the gradient is \(2x\), so \(K=2\) and \(2/K=1\) — the very threshold the exact check below finds. For a twice-differentiable \(f\), \(K\) can be taken as the largest curvature, in absolute value, over all points and all unit directions (§6), so a sharply curved direction forces a small \(\eta\) — the Hessian’s payoff for training.) Many losses in this book satisfy this, at least in the region training actually visits; for kinked ones (networks built from ReLUs, §3) small steps still work well in practice (claims from practice that this chapter does not test).

Check it exactly on \(f(x)=x^2\). One step from \(x\) lands at \(x-\eta\cdot2x=(1-2\eta)x\), where \(f=(1-2\eta)^2x^2\). That is below \(x^2\) precisely when \((1-2\eta)^2<1\), i.e. when \(0<\eta<1\) — which is why \(\eta=0.1\) and \(\eta=0.5\) worked in §5 (one step multiplies \(f\) by \(0.64\) and by \(0\)) while \(\eta=1.1\) blew up (it multiplies \(f\) by \(1.44\)).

How fast does it break down? The linear approximation’s error is second order — it grows like the square of the step. For \(f(x)=x^2\) linearized at \(x_0=3\) (tangent line \(T(x)=6x-9\)) the error is exactly \[ f(x)-T(x)=x^2-(6x-9)=(x-3)^2, \] the curvature term Taylor dropped. So it is tiny nearby but blows up quadratically: at \(x=3.1\) the error is \(0.01\); at \(3.5\), \(0.25\); at \(4\), \(1.0\); at \(5\), \(4.0\). That is why a downhill step is only trustworthy when it is small — double the step and you quadruple the surprise; the learning rate \(\eta\) is what keeps the step inside the region where the line is a faithful stand-in for the curve.

The linear approximation's error grows quadratically with the step. — Figure 2.11: The linear approximation’s error grows *quadratically* with the step. For \(f(x)=x^2\) linearized at \(x_0=3\), the error is exactly \((x-3)^2\) (blue), so it is negligible nearby (\(0.01\) at \(x{=}3.1\), red dots) but explodes as you move away (\(1.0\) at \(x{=}4\), \(4.0\) at \(x{=}5\)). This second-order blow-up is why gradient descent only trusts a *small* step.

In one line: a tangent line is a trustworthy stand-in only for small steps (its error is \(0.01\) at \(3.1\) and \(1.0\) at \(4\)), which is why a small \(\eta\) makes \(-\eta\nabla f\) lower \(f\).

8. The Jacobian — vector in, vector out

One layer of a network turns a list of numbers into another list of numbers. Nudge any one number going in, and every number coming out may move. How do you keep track of all those effects at once, without losing any?

Write a layer, which takes a vector in and puts a vector out, as a function \(F:\mathbb{R}^n\to\mathbb{R}^m\) — read “\(F\) takes \(n\) numbers in and gives \(m\) numbers out” (\(\mathbb{R}\) is the set of all real numbers, and \(\mathbb{R}^n\) the set of lists of \(n\) of them); its outputs are \(F_1,\dots,F_m\) and its inputs \(x_1,\dots,x_n\). Its derivative is a matrix of all the partials — the Jacobian: \[ J_{ij} = \frac{\partial F_i}{\partial x_j}, \qquad J \in \mathbb{R}^{m\times n} \;(\text{one row per output, one column per input}), \]

where \(J\in\mathbb{R}^{m\times n}\) reads “\(J\) is a table of real numbers with \(m\) rows and \(n\) columns”.

Why “Jacobian.” Named after Carl Gustav Jacob Jacobi. It is the natural generalization of the gradient: a gradient is the Jacobian of a scalar (single-number) output — one row; stack many such rows for many outputs.

Worked example. Promote §4’s two pieces, \(x^2\) and \(xy\), into one vector output \(F(x,y)=\big[\,x^2,\; xy\,\big]\) (writing \(\partial_x\) as short for \(\frac{\partial}{\partial x}\)): \[ J=\begin{bmatrix} \partial_x(x^2) & \partial_y(x^2) \\ \partial_x(xy) & \partial_y(xy)\end{bmatrix} =\begin{bmatrix} 2x & 0 \\ y & x\end{bmatrix} \;\xrightarrow{(1,2)}\; \begin{bmatrix} 2 & 0 \\ 2 & 1\end{bmatrix}. \]

Per component: \(J_{ij}\) is how much output \(i\) moves when you nudge input \(j\), so row \(i\) is the gradient of output \(i\): row 1 is \(\nabla(x^2)=[2x,\,0]\) and row 2 is \(\nabla(xy)=[y,\,x]\); add the two rows and you get §4’s \(\nabla f=[2x+y,\,x]\) for \(f=x^2+xy\). The top-right \(0\) says output \(x^2\) ignores \(y\) — the same kind of fact as the Hessian’s \(0\) corner in §6, where the \(y\)-slope \(x\) ignores \(y\). (Indeed the Hessian is the Jacobian of the gradient: the Jacobian of \([2x+y,\,x]\) is \(\begin{bmatrix}2&1\\1&0\end{bmatrix}\).)

Why it matters: the chain rule for vectors is matrix multiplication of Jacobians.

Worked example — chaining two Jacobians. Feed \(F(x,y)=[x^2,\ xy]\) from just above into the sum \(s(u,v)=u+v\) (its inputs \(u,v\) are the two outputs of \(F\), the intermediates of §3’s sum over paths). The composite \(s(F(x,y))=x^2+xy\) is §4’s \(f\), whose gradient at \((1,2)\) we already know: \([4,\ 1]\). Now get it by the chain rule instead. \(J_s=[\,1\ \ 1\,]\) (one output, two inputs: a single row), and at \((1,2)\), \(J_F=\begin{bmatrix}2&0\\2&1\end{bmatrix}\). Matrices multiply row times column: entry \(j\) of the product \(J_sJ_F\) is the dot product (§4) of the row of \(J_s\) with column \(j\) of \(J_F\) —

\[ J_sJ_F=\big[\,1\cdot2+1\cdot2,\;\;1\cdot0+1\cdot1\,\big]=[\,4,\ 1\,]\ \checkmark \]

Each entry is §3’s sum over paths: \(x\) reaches \(s\) through \(u\) (rates \(1\cdot2\)) and through \(v\) (rates \(1\cdot2\)), and \(2+2=4\).

Back-propagation through a deep network multiplies the layers’ Jacobians together in just this way, starting from the loss and working back to every weight — the §3 chain rule, now in matrix form (the Linear-Algebra primer §3 builds the matrix–vector product from the same row-times-column rule). Multiplying many such factors is double-edged: if they are small in size the product vanishes toward \(0\) (early layers stop learning): six layers that each scale by \(0.5\) leave at most \(0.5^6\approx0.016\) of the gradient.

Here a matrix’s size is its largest stretch, the most it can lengthen a vector (its largest singular value, Linear-Algebra primer §6), not the size of its entries. For example, the \(2\times2\) matrix with every entry \(0.5\) maps \([1,1]\) to \([1,1]\), so its largest stretch is at least \(1\) (it is exactly \(1\), its largest singular value) although every entry is small. If the factors are large it can explode: six layers that each scale by \(2\) multiply the gradient by at most \(2^6=64\) (exactly \(64\) when every direction is stretched by \(2\)). This is the vanishing- and exploding-gradient problem; Representation Learning & the Transformer tackles the vanishing half for long sequences.

In one line: the Jacobian is the table of all partials of a vector function, and the chain rule for vectors multiplies Jacobians (\([1\ \ 1]\begin{bmatrix}2&0\\2&1\end{bmatrix}=[4\ \ 1]\)).

9. Integration, briefly (the bridge to probability)

A predicted rating can land anywhere in a range of values, not just on one of a few fixed ones. When every value in a range is possible (a whole stretch of numbers, not a short list), any one exact value has probability zero (a rating of exactly 3.7 almost never happens), so how do you say how likely the rating is to fall between two cut-offs?

Integration is the reverse of differentiation: where the derivative takes a function to its slope, the integral \(\int_a^b f(x)\,dx\) sums infinitely many thin slices to give the area under the curve between \(a\) and \(b\). Read it piece by piece: \(\int\) is a stretched S, for “sum”; \(f(x)\,dx\) is one thin slice — height \(f(x)\) times a tiny width \(dx\); and \(a\) and \(b\) mark where the slicing starts and stops. For example, \(\int_0^1 2x\,dx\) is the triangle under \(y=2x\) between \(0\) and \(1\), with area \(\tfrac12\cdot1\cdot2=1\). A single point has no width, so the area over it is \(0\cdot f(x)=0\), which is why, when every value in a range is possible, only a range of values, never one exact value, can have a probability above zero.

Why “the reverse of differentiation”? Sweep the area from \(0\) up to a moving right edge \(x\): under \(y=2x\) it is the triangle \(\tfrac12\cdot x\cdot2x=x^2\), and the derivative of \(x^2\) is \(2x\) — the height of the curve. In general the swept area grows at a rate equal to the curve’s height, so differentiating the area gives the curve back. Why “integral”: from Latin integer, “whole” — you integrate (make whole) the slices into one total.

The bridge to probability. This is the idea continuous probability is built on. A random variable \(X\) is a number whose value is uncertain, like tomorrow’s temperature; when it can take any value in a range, the chance that \(X\) lands between \(a\) and \(b\) is the area between \(a\) and \(b\) under its probability density function (PDF) \(p(x)\) — a curve that says how densely the probability is packed near each \(x\) (for \(p(x)=2x\) on \([0,1]\), the chance of landing in \([0,0.5]\) is the area \(\tfrac12\cdot0.5\cdot1=0.25\), shaded in the figure below). Its expectation (its long-run average) is an integral too, \(\mathbb{E}[X]=\int x\,p(x)\,dx\) taken over every possible \(x\) — the continuous twin of the weighted-average sum \(\sum x\,p(x)\) (for a fair die, \(\sum x\cdot\tfrac16=3.5\); for \(p(x)=2x\) on \([0,1]\), \(\int_0^1 x\cdot2x\,dx=\int_0^1 2x^2\,dx=\tfrac23\)). The Probability primer §1 builds all of this; here we only need to know that an integral is “area = the accumulated total.”

The integral inta^b f(x) dx is the area under the curve between a and b (shaded) — “summing infinitely many thin slices.” Here f(x)=2x on the range from 0 to 0.5: a triangle of area tfrac12 . — Figure 2.12: The integral \(\int_a^b f(x) dx\) is the *area under the curve* between \(a\) and \(b\) (shaded) — “summing infinitely many thin slices.” Here \(f(x)=2x\) on the range from \(0\) to \(0.5\): a triangle of area \(\tfrac12\cdot0.5\cdot1=0.25\); the whole range \(0\) to \(1\) has area \(1\). In the Probability primer, area under a density becomes a *probability*.

In one line: an integral is the area under a curve and the reverse of differentiation (\(\int_0^1 2x\,dx=1\), and \(x^2\) differentiates back to \(2x\)).

10. Exercises

Work these by hand — the numbers are kept tiny on purpose. Full worked solutions are in the Solutions appendix at the back of the book.

(compute) — a slope from the limit. Compute the derivative of \(f(x)=x^2\) at \(x=2\) from the limit (§1): simplify the difference quotient \(\dfrac{(2+h)^2-2^2}{h}\) to a form with no \(h\) in the denominator, then let \(h\to0\). Check your answer against the power rule \(f'(x)=2x\) (§2).
(compute) — the product rule. Use the product rule \((fg)'=f'g+fg'\) (§2) on \(f(x)=x^2(x+1)\) (take the two factors \(x^2\) and \(x+1\)). At \(x=2\), evaluate \(f'(2)\). Check by first multiplying out \(f(x)=x^3+x^2\) and differentiating — the two routes should agree.
(compute) — the chain rule. Apply the chain rule (§3) to \(k(x)=(2x+1)^3\) — a composition \(f(g(x))\) with outer \(f(u)=u^3\) and inner \(g(x)=2x+1\) — to get \(k'(x)\), then evaluate \(k'(1)\). (Mirror the chapter’s \((3x+1)^2\) worked example; you can check by expanding \(k\) and differentiating.)
(compute) — sigmoid slopes. The sigmoid is \(\sigma(z)=\dfrac{1}{1+e^{-z}}\) with derivative \(\sigma'(z)=\sigma(z)\big(1-\sigma(z)\big)\) (§3). Evaluate \(\sigma\) and \(\sigma'\) at \(z=1\) (where \(\sigma(1)\approx0.7311\)) and at \(z=3\) (where \(\sigma(3)\approx0.9526\)). At which of the two points does a sigmoid layer pass back more of the gradient?
(compute) — a gradient. For \(f(x,y)=x^2y+y\) write the two partial derivatives \(\frac{\partial f}{\partial x}\) and \(\frac{\partial f}{\partial y}\), then assemble the gradient \(\nabla f\) at the point \((x,y)=(2,3)\).
(compute) — Hessians and the second-derivative test. Form the Hessian (§6) of each of \(f(x,y)=x^2+3y^2\) and \(g(x,y)=xy\). For each, find the critical point (where every first partial is zero), compute \(D=f_{xx}f_{yy}-f_{xy}^2\) (for \(g\): \(D=g_{xx}g_{yy}-g_{xy}^2\)), and use the second-derivative test (§6) to classify the point as a bowl (a minimum), a cap (a maximum) or a saddle. State which function is not convex and why. (Optional, if you read the eigenvalue box: find each Hessian’s eigenvalues and check that their product is \(D\).)
(compute) — stacked sigmoids. §3 shows a sigmoid’s activation factor is at most \(0.25\) (\(\sigma'\le0.25\)), and back-propagation multiplies one such factor per layer. Without a calculator beyond powers of \(\tfrac14\), give the largest share of the gradient the activation factors alone can pass back to the earliest layer through 6 stacked sigmoid layers, as a fraction \(1/n\). Then let every layer also carry a weight of \(3\) (the weights multiply in, as in the §3 box): with a calculator, give the largest share the whole chain can pass back as a decimal to three places, and say whether the gradient shrinks or grows.
(extend) — two descent steps. Run two steps of gradient descent on \(f(x)=x^2\) (so \(f'(x)=2x\)) from \(x_0=2\) with the chapter’s \(\eta=0.1\), using \(x\leftarrow x-\eta f'(x)\) (§5); report \(x_1\), \(x_2\), and \(f(x_2)\). Then for the first step apply the §7 argument: compute the first-order Taylor prediction of the change, \(f'(x_0)\cdot\delta\) with \(\delta=-\eta f'(x_0)\), and compare it with the actual change.
(extend) — a Jacobian. Build the Jacobian (§8) of the vector-valued \(F(x,y)=\big[\,xy,\ x+y^2\,\big]\): write the \(2\times2\) matrix \(J_{ij}=\partial F_i/\partial x_j\) symbolically, then evaluate it at \((x,y)=(2,3)\). Compute the gradient of the first output \(xy\) at \((2,3)\) and compare it with row \(1\) of your matrix.
(apply) — one training step. A model predicts \(wx\) from an input \(x\), using a single weight \(w\). One training example — a data point whose correct answer is known — has input \(x=2\) and correct answer (the target) \(t=1\). The squared-error loss \(\mathcal{L}(w)=(wx-t)^2=(2w-1)^2\) measures how far the prediction is from the target; here \(x\) and \(t\) are fixed numbers and \(w\) is the variable. Starting from \(w_0=0\) with \(\eta=0.1\): compute the gradient \(\dfrac{d\mathcal{L}}{dw}\) at \(w_0\) (use the chain rule, §3), take one gradient-descent step \(w\leftarrow w-\eta\,\frac{d\mathcal{L}}{dw}\) (§5), and compare the loss before and after by evaluating \(\mathcal{L}\) at both. This is one literal step of the training loop behind most learned models in the book.
(compute) — how fast the linear approximation breaks down. Linearize \(f(x)=x^2\) at \(x_0=2\); the tangent line is \(T(x)=4x-4\) (§7). Compute \(T\) and the true \(f\) at \(x=2.5\) and \(x=3\), and the error \(f-T\) each time. Then expand \(f(x)-T(x)\) and write it in terms of \(x-2\).
(concept) — which losses are convex? A function with a second derivative is convex when \(f''(x)\ge0\) everywhere (§6). For each, give \(f''\) and the verdict: (a) \(f(x)=x^4\), (b) \(f(x)=x^5\), (c) \(f(x)=e^{2x}\), (d) \(f(x)=-x^2\).
(apply) — the learning rate that converges. On \(f(x)=4x^2\) (\(f'=8x\)) the gradient step is \(x\leftarrow x-\eta(8x)=(1-8\eta)x\) (§5). (a) For which \(\eta\) does \(x\) shrink toward \(0\) after every step, i.e. \(|1-8\eta|<1\)? (b) What do \(\eta=0.25\) and \(\eta=0.3\) do? (c) Compare your answer to (a) with the \(2/K\) of §7, taking \(K\) as the second derivative \(f''\).
(compute) — a slope along a chosen direction. For \(f(x,y)=x^2+xy\) at \((x,y)=(1,2)\), where \(\nabla f=[4,1]\) (§4), find the directional derivative along the unit direction \(\hat{\mathbf d}=\tfrac15[3,4]\): first check that \(\hat{\mathbf d}\) has length \(1\), then compute \(\nabla f\cdot\hat{\mathbf d}\). Is \(f\) rising or falling that way, and is it rising faster or slower than along \(x\) (where the slope is \(4\))?
(compute) — curvature along a direction. For the same \(f(x,y)=x^2+xy\) (\(f_{xx}=2\), \(f_{yy}=0\), \(f_{xy}=1\)), use §6’s formula \(f_{xx}v_1^2+2f_{xy}v_1v_2+f_{yy}v_2^2\) along the unit direction \(\mathbf v=\tfrac15[3,4]\). Is the curvature that way positive or negative? (Optional, with §6’s eigenvalue box: does it lie between the two eigenvalues \(1\pm\sqrt2\)?)
(apply) — two curvatures at once. On \(f(x,y)=x^2+3y^2\), take one gradient-descent step from \((3,1)\) with \(\eta=0.1\) (§5) and report the new point and \(f\) before and after. Which coordinate moved by a larger fraction? Then find the range of \(\eta\) for which both coordinates still shrink toward \(0\) (each coordinate is multiplied by its own factor every step, as in §5).

11. Where this fits in the book

Calculus idea (here)	Where it is used	Why it matters there
Derivative / slope (§1–2)	Loss Functions & Regularizers §3 gradient descent	the slope drives the step: a step against the derivative lowers the loss
Chain rule (§3)	neural-net back-propagation; Loss Functions & Regularizers	multiply per-layer rates → train deep models
Slopes of \(e^x\) and \(\ln x\) (§2); \(\sigma'=\sigma(1-\sigma)\) and the ReLU subgradient (§3)	the activation derivatives in Neural Networks & Back-propagation; the \(-\ln p\) slope and the ReLU subgradient in Loss Functions & Regularizers	each layer’s slope factor in back-propagation is one of these; \(\ln\)’s slope \(1/x\) is why log-losses are easy to differentiate
Partial derivatives, gradient (§4)	every gradient-based training loop in the book	one direction that updates all parameters
Gradient descent (§5)	Loss Functions & Regularizers §3; training of the graph-based recommender models later in the book (LightGCN and the methods built on it)	the optimizer behind most learned embeddings (the closed-form recommenders later in the book take no gradient steps at all)
Second derivative, convexity (§6)	Loss Functions & Regularizers §9 (the BPR, Bayesian Personalized Ranking, loss curve, drawn there as convex)	convex ⇒ no bad local minima: any minimum is global
Hessian (§6)	not used again in a later chapter; it completes the picture (bowl, cap, saddle) and pairs with eigenvalues (Linear-Algebra primer §5)	bowl vs. saddle; a more sharply curved bowl tolerates only a smaller learning rate (§7’s \(K\))
Taylor / linear approx. (§7)	the idea behind a descent step	a small step against the gradient lowers the loss
Jacobian (§8)	backprop through layers (matrix chain rule)	vector-in/vector-out derivatives = matrix products
Integral = area (§9)	Probability primer (PDF → probability, \(\mathbb{E}\))	continuous probabilities and expectations

Carry this into the rest of the book: a derivative is a slope; the gradient is the slope in many dimensions and points uphill; the chain rule multiplies slopes along a chain; training is just stepping downhill (\(\theta \leftarrow \theta-\eta\nabla f\)) on a (hopefully convex) loss.

12. Glossary

Term	Plain meaning
Loss \(\mathcal{L}\)	A single number scoring how wrong a model’s predictions are; training makes it as small as possible (the chapter opener and §4; its letter \(\mathcal{L}\), §3; written \(f\) in §5 to §7).
Weight	One adjustable multiplier in a layer’s weighted sum; the main kind of parameter in a neural network (§3).
Parameters	A model’s adjustable numbers (e.g. weights, embeddings) — in a neural network mostly its weights, in a recommender also its embeddings (§5); training nudges them, step by step, until the loss is small.
Derivative \(f'(x)\)	Instantaneous rate of change = slope of the curve at a point.
Differentiate	Compute a derivative (limit of differences).
Difference quotient	\(\frac{f(x+h)-f(x)}{h}\) — rise over run before taking \(h\to0\).
Limit	The value a quantity approaches (here as the run \(h\to0\)).
Tangent	The line that just touches a curve at a point; its slope is the derivative there (§1).
Critical / stationary point	Where \(f'(x)=0\) (many variables: \(\nabla f=\mathbf 0\)): peak, valley, plateau or saddle.
Chain rule	Derivative of \(f(g(x))\) = \(f'(g)\cdot g'\); the basis of back-propagation.
Back-propagation (backprop)	The chain rule run backward from the loss, to find how the loss changes with every weight (§3).
Activation function	The fixed, simple nonlinear bend a layer applies to its weighted sums, e.g. the sigmoid or ReLU (§3).
Sigmoid \(\sigma(z)\)	\(\frac{1}{1+e^{-z}}\): an S-shaped curve squashing any \(z\) into the range between \(0\) and \(1\); \(\sigma'=\sigma(1-\sigma)\le0.25\) (§3).
ReLU (rectified linear unit)	\(\max(0,z)\); derivative \(1\) for \(z>0\) and \(0\) for \(z<0\) (§3).
Partial derivative \(\partial f/\partial x\)	Derivative with respect to one variable, the others held fixed.
Gradient \(\nabla f\)	Vector of all partials; points in the direction of steepest increase.
Nabla / del (\(\nabla\))	The symbol for the gradient operator.
Contour (level set)	A curve along which \(f\) keeps one constant value, like a height line on a map; the gradient is perpendicular to it (§4).
Gradient descent	\(\theta\leftarrow\theta-\eta\nabla f\): step against the gradient to minimize.
Learning rate \(\eta\)	Step size in gradient descent.
Optimizer	The procedure that updates the parameters from the gradient; gradient descent is the basic one (§4, §5).
Momentum	Gradient descent that steps on a running total of recent gradients, \(v\leftarrow0.9\,v+g\) (§5).
Adam	An optimizer that divides each parameter’s step by the recent size of its own gradient (§5; Kingma and Ba, 2015).
Gradient clipping	Capping the gradient’s length at a ceiling so it cannot explode; the direction is kept (§5).
Stochastic gradient descent (SGD)	Gradient descent on a noisy gradient estimated from a random minibatch, not the full dataset.
Minibatch	A handful of training examples drawn at random; SGD estimates the gradient on it (§5).
Second derivative \(f''\)	Rate of change of the slope = curvature.
Curvature	How a curve bends: \(f''\) in one variable; in many, the second derivative along a chosen direction (§6).
Convex	Curves up everywhere: for a function with second derivatives, one variable \(f''\ge0\), many variables a Hessian that is PSD at every point. A bowl, a trough, or a slope with no bottom (like \(e^x\)) — no bad local minima; any minimum is global.
Local / global minimum	Local = lowest within a neighborhood; global = lowest anywhere.
Saddle point	Critical point that is a min in some directions, a max in others (indefinite Hessian).
Positive semidefinite (PSD)	Curvature \(\ge0\) in every direction (for a \(2\times2\) Hessian: \(f_{xx}\ge0\), \(f_{yy}\ge0\), \(D\ge0\); equivalently all eigenvalues \(\ge0\)). A function whose Hessian is PSD at every point is convex.
Positive definite	Curvature strictly \(>0\) in every direction (for a \(2\times2\) Hessian: \(f_{xx}>0\) and \(D>0\); all eigenvalues \(>0\)). Positive definite everywhere ⇒ at most one minimum (§6).
Eigenvalue	One of the two extreme curvatures of a symmetric \(2\times2\) Hessian; their product is \(D\) (§6 optional box; Linear-Algebra primer §5).
Indefinite	Curvature up in some directions and down in others (\(D<0\) for a \(2\times2\) Hessian; eigenvalues of both signs) — the Hessian at a saddle.
Subgradient	A stand-in slope at a kink where the true derivative is undefined (e.g. any value in \([0,1]\) for ReLU at \(0\)).
Inflection point	Where \(f''\) changes sign (the curve switches between bending up and down).
Hessian \(H\)	Matrix of second partial derivatives \(f_{xx},f_{xy},f_{yx},f_{yy}\) (multivariable curvature).
Second-derivative test	At a critical point of \(f(x,y)\), \(D=f_{xx}f_{yy}-f_{xy}^2\): \(D>0,\ f_{xx}>0\) bowl (min); \(D>0,\ f_{xx}<0\) cap (max); \(D<0\) saddle; \(D=0\) silent (§6).
Taylor / linear approximation	\(f(x+\delta)\approx f(x)+f'(x)\delta\); a curve looks straight up close.
Smooth / differentiable	Has a derivative everywhere — no kinks or jumps; looks straight up close.
Lipschitz-smooth	The gradient changes at most in proportion to the step, \(\lVert\nabla f(\theta_1)-\nabla f(\theta_2)\rVert\le K\lVert\theta_1-\theta_2\rVert\); then one fixed learning rate, any \(\eta\) between \(0\) and \(2/K\), lowers \(f\) at every step where the gradient is not zero (§7).
Directional derivative	Rate of change of \(f\) along a unit direction \(\hat{\mathbf d}\): \(\nabla f\cdot\hat{\mathbf d}\).
Jacobian \(J\)	Matrix of partials of a vector-valued function (rows = outputs, columns = inputs).
Integral \(\int\)	Area under a curve; the reverse of differentiation; gives probabilities/expectations.
Vanishing gradient	A chain-rule product of many small factors shrinks toward zero, so early layers stop learning (§3, §8; see also the §5 box on gradient pathologies).
Product rule	\((fg)' = f'g + fg'\) — the derivative of a product is not the product of the derivatives (§2).

13. References

Cited in the text:

Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1412.6980