MLP layers and activation functions

Each transformer block contains two sub layers: self attention and a position wise feed forward network (FFN, also called MLP). The MLP applies the same two layer network to every token position independently: FFN(x) = σ(xW₁ + b₁)W₂ + b₂.