Backpropagation

Backpropagation

Disegnino.

Explanation of Backpropagation with a Simple Example

Given:

• $a=x+y$
• $b=\max (y,z)$
• $f=a\cdot b$

We want to compute the gradients of $f$ with respect to $x$, $y$, and $z$ for the input values:

$$x=1,y=2,z=0$$

---

Step 1: Forward pass

Calculate intermediate values:

• $a=1+2=3$
• $b=\max (2,0)=2$
• $f=a\cdot b=3\times 2=6$

---

Step 2: Backward pass (compute gradients)

We want $\frac{\partial f}{\partial x}$, $\frac{\partial f}{\partial y}$, and $\frac{\partial f}{\partial z}$.

Using the chain rule:

$$\frac{\partial f}{\partial a}=b=2$$ $$\frac{\partial f}{\partial b}=a=3$$

Gradients of $a$:

$$\frac{\partial a}{\partial x}=1,\frac{\partial a}{\partial y}=1$$

Gradients of $b$ (since $b=\max (y,z)$):

• Here, $y=2$ is greater than $z=0$, so $b$ depends only on $y$.
• Therefore:

$$\frac{\partial b}{\partial y}=1,\frac{\partial b}{\partial z}=0$$

---

Step 3: Compute final gradients

Using the chain rule again:

$$\frac{\partial f}{\partial x}=\frac{\partial f}{\partial a}\cdot \frac{\partial a}{\partial x}=2\times 1=2$$ $$\frac{\partial f}{\partial y}=\frac{\partial f}{\partial a}\cdot \frac{\partial a}{\partial y}+\frac{\partial f}{\partial b}\cdot \frac{\partial b}{\partial y}=2\times 1+3\times 1=5$$ $$\frac{\partial f}{\partial z}=\frac{\partial f}{\partial b}\cdot \frac{\partial b}{\partial z}=3\times 0=0$$

---

Summary:

Variable	Value	Gradient of $f$
$x$	1	2
$y$	2	5
$z$	0	0

This illustrates how backpropagation applies the chain rule to compute gradients efficiently.