Backpropagation for noobs

Typical Steps for Backpropagation

Let’s outline the procedure with your example:

Given functions:

• $a=x+y$
• $b=\max (y,z)$
• $f=a\cdot b$

and inputs: $x=1$, $y=2$, $z=0$.

---

Step 1: Rewrite the function

$$f(x,y,z)=(x+y)\cdot \max (y,z)$$

---

Step 2: Draw the computation graph

x -----+
---> a = x + y -------+
/ |
y ------+ |
* ---> f = a * b
y ---------------------------+ |
| |
z ---------------------------> b = max(y, z)

---

Step 3: Calculate gradients of each local function

• $\frac{\partial a}{\partial x}=1$, $\frac{\partial a}{\partial y}=1$
• $\frac{\partial b}{\partial y}=1$ (since $y>z$), $\frac{\partial b}{\partial z}=0$
• $\frac{\partial f}{\partial a}=b$
• $\frac{\partial f}{\partial b}=a$

---

Step 4: Fill in values on the graph

• $a=3$, $b=2$
• $\frac{\partial f}{\partial a}=2$
• $\frac{\partial f}{\partial b}=3$

---

Step 5: Compute total gradients with respect to inputs

Using chain rule and summation:

$$\frac{\partial f}{\partial x}=\frac{\partial f}{\partial a}\cdot \frac{\partial a}{\partial x}=2\times 1=2$$ $$\frac{\partial f}{\partial y}=\frac{\partial f}{\partial a}\cdot \frac{\partial a}{\partial y}+\frac{\partial f}{\partial b}\cdot \frac{\partial b}{\partial y}=2\times 1+3\times 1=5$$ $$\frac{\partial f}{\partial z}=\frac{\partial f}{\partial b}\cdot \frac{\partial b}{\partial z}=3\times 0=0$$

---

This matches your observed total gradients paper:

• $\frac{\partial f}{\partial x}=2$
• $\frac{\partial f}{\partial y}=5$
• $\frac{\partial f}{\partial z}=0$