mirror of
https://github.com/labmlai/annotated_deep_learning_paper_implementations.git
synced 2025-10-30 10:18:50 +08:00
An Attention Free Transformer
This is a PyTorch implementation of the paper An Attention Free Transformer.
This paper replaces the self-attention layer
with a new efficient operation,
that has memory complexity of O(Td), where T is the sequence length
and d is the dimensionality of embeddings.
The paper introduces AFT along with AFT-local and AFT-conv. Here we have implemented AFT-local which pays attention to closeby tokens in an autoregressive model.
