learning/annotated_deep_learning_paper_implementations
1
0
Fork 0
mirror of https://github.com/labmlai/annotated_deep_learning_paper_implementations.git synced 2025-11-01 03:43:09 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
annotated_deep_learning_pap…/labml_nn/transformers/fnet
History
Varuna Jayasiri 07065dea92 CFR (#60)
2021-06-21 17:04:20 +05:30
..
__init__.py
CFR (#60)
2021-06-21 17:04:20 +05:30
experiment.py
AFT (#54)
2021-06-02 21:36:47 +05:30
readme.md
FNet (#53)
2021-05-26 10:56:42 +05:30

readme.md

FNet: Mixing Tokens with Fourier Transforms

This is a PyTorch implementation of the paper FNet: Mixing Tokens with Fourier Transforms.

This paper replaces the self-attention layer with two Fourier transforms to mix tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for BERT on GLUE benchmark.