annotated_deep_learning_paper_implementations

{
 "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet: Mixing Tokens with Fourier Transforms</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2105.03824\">FNet: Mixing Tokens with Fourier Transforms</a>.</p>\n<p>This paper replaces the <a href=\"https://nn.labml.ai/transformers//mha.html\">self-attention layer</a> with two <a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">Fourier transforms</a> to <em>mix</em> tokens. This is a 7X more efficient than self-attention. The accuracy loss of using this over self-attention is about 92% for <a href=\"https://paperswithcode.com/method/bert\">BERT</a> on <a href=\"https://paperswithcode.com/dataset/glue\">GLUE benchmark</a>. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformers/fnet/index.html\">FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408</a></h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a<a href=\"https://arxiv.org/abs/2105.03824\">FNet\uff1a\u5c06\u4ee3\u5e01\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408\u300b\u7684 PyTor</a> <a href=\"https://pytorch.org\">ch</a> \u5b9e\u73b0\u3002</p>\n<p>\u672c\u6587\u7528\u4e24\u4e2a<a href=\"https://en.wikipedia.org/wiki/Discrete_Fourier_transform\">\u5085\u91cc\u53f6\u53d8</a>\u6362\u53d6\u4ee3\u4e86<a href=\"https://nn.labml.ai/transformers//mha.html\">\u81ea\u6211\u6ce8\u610f\u529b\u5c42</a>\uff0c\u4ee5<em>\u6df7\u5408</em>\u4ee4\u724c\u3002\u8fd9\u6bd4\u81ea\u6211\u6ce8\u610f\u529b\u9ad87\u500d\u3002\u5728 GLUE <a href=\"https://paperswithcode.com/dataset/glue\">\u57fa\u51c6\u6d4b\u8bd5</a>\u4e2d\uff0c<a href=\"https://paperswithcode.com/method/bert\">BERT</a> \u4f7f\u7528\u5b83\u800c\u4e0d\u662f\u81ea\u6211\u6ce8\u610f\u529b\u7684\u51c6\u786e\u6027\u635f\u5931\u7ea6\u4e3a92\uff05\u3002</p>\n",
 "FNet: Mixing Tokens with Fourier Transforms": "FNet\uff1a\u5c06\u4ee4\u724c\u4e0e\u5085\u91cc\u53f6\u53d8\u6362\u6df7\u5408"
}