17.3. Position-wise Feed-Forward Networks (FFN)

Section 5.3 of the survey paper highlights several research aimed at improving FFN’s performance:

• Replacing activation function:

  • Ramachandran et al., 2018 proposed replacing ReLU with $ x \cdot \text{sigmoid}(\beta x)$.
  • GPT (Radford et al., 2018) opted for Gaussian Error Linear Unit (GELU) instead of ReLU.

• Replacing FFN:

  • Lample et al., 2019 introduced product-key memory layers as an alternative to the FFN.
  • Gshard (Lepikhin et al., 2020) explored a sparsely-gated Mixture-of-Experts MoE (Mixture-of-Experts) approach in place of the FFN.