Misc Exercises on part two

Ex1

1. Rewrite the Problem

• Vocabulary: ([‘i’, ‘o’, ‘c’, ‘a’])
• Embeddings: ([-2, 5, 1, 9]) (so ‘i’→-2, ‘o’→5, ‘c’→1, ‘a’→9)
• $W_h=-5$, $W_x=1$, $W_y=-1$
• Elman RNN, activation $f(x)=x$ (just identity), no bias

Task:

• Compute forward through a simple 2-timestep RNN for input [‘o’, ‘c’] (i.e. embed ‘o’, then ‘c’).
• At each step, follow the RNN equations for hidden states.
• Compute the Loss as $\text{Loss}={\text{Loss}}_1+{\text{Loss}}_2$ (details of loss not specified, but just follow the architecture for now).

2. Step-by-Step Solution:

Step 1: Initialization

• $h_0=0$ (initial hidden state)

Step 2: First input (‘o’)

• Embed ‘o’: $x_1=5$
• Elman RNN update: $h_1=W_x\cdot x_1+W_h\cdot h_0$
• Plug values:
  $h_1=1\cdot 5+(-5)\cdot 0=5+0=5$

Step 3: Second input (‘c’)

• Embed ‘c’: $x_2=1$
• Elman RNN update: $h_2=W_x\cdot x_2+W_h\cdot h_1$
• Plug values:
  $h_2=1\cdot 1+(-5)\cdot 5=1-25=-24$

(Optional) Step 4: Compute Output and Loss

• Usually, there’s one last layer for outputs ($y_t=W_y\cdot h_t$)
• For each time step:
  • $y_1=-1\cdot 5=-5$
  • $y_2=-1\cdot (-24)=24$
• Loss would be calculated here if target values were given. For now, just sum the losses at each step as in the diagram.

3. Summary Table

Step	Input	Embedding	$h_{t-1}$	$h_t$	Output $y_t$
1	‘o’	5	0	5	$-5$
2	‘c’	1	5	$-24$	$24$

General formula:
At each $t$:

$$h_t=W_x{x}_t+W_h{h}_{t-1}$$ $$y_t=W_y{h}_t$$

That’s it!

• Go forward step-by-step with the given embeddings and weights.
• Use the RNN update each time.
• Apply the output weight.
• If you had targets (labels), you could then compute ${\text{Loss}}_1$ and ${\text{Loss}}_2$ with a loss function.