learning/annotated_deep_learning_paper_implementations
1
0
Fork 0
mirror of https://github.com/labmlai/annotated_deep_learning_paper_implementations.git synced 2025-10-30 18:27:03 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
annotated_deep_learning_pap…/labml_nn/transformers/fnet
History
Varuna Jayasiri 996b58be04 paper links
2021-08-17 14:12:33 +05:30
..
__init__.py
paper links
2021-08-17 14:12:33 +05:30
experiment.py
AFT (#54)
2021-06-02 21:36:47 +05:30
readme.md
paper links
2021-08-17 14:12:33 +05:30

readme.md

FNet: Mixing Tokens with Fourier Transforms

This is a PyTorch implementation of the paper FNet: Mixing Tokens with Fourier Transforms.

This paper replaces the self-attention layer with two Fourier transforms to mix tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for BERT on GLUE benchmark.