learning/annotated_deep_learning_paper_implementations
1
0
Fork 0
mirror of https://github.com/labmlai/annotated_deep_learning_paper_implementations.git synced 2025-10-29 09:38:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

experiment links transformer

Browse Source
This commit is contained in:
Varuna Jayasiri
2022-06-28 19:02:20 +05:30
parent e09ee89f36
commit ee5a34aa59
9 changed files with 247 additions and 240 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
labml_nn/transformers/basic/autoregressive_experiment.ipynb
View File

@ -11,6 +11,7 @@
"source": [
"[![Github](https://img.shields.io/github/stars/labmlai/annotated_deep_learning_paper_implementations?style=social)](https://github.com/labmlai/annotated_deep_learning_paper_implementations)\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb)\n",
"[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](comet.ml/labml/transformer/ea8c108c2d94434ca3c2bc2b21015082)\n",
"\n",
"## Transformer Experiment\n",
"\n",

5
labml_nn/transformers/basic/autoregressive_experiment.py
View File

@ -7,10 +7,11 @@ summary: >
# Transformer Auto-Regression Experiment
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](comet.ml/labml/transformer/ea8c108c2d94434ca3c2bc2b21015082)
This trains a simple transformer introduced in [Attention Is All You Need](https://papers.labml.ai/paper/1706.03762)
on an NLP auto-regression task (with Tiny Shakespeare dataset).
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb)
"""
import torch

5
labml_nn/transformers/mha.py
View File

@ -8,6 +8,9 @@ summary: >
# Multi-Headed Attention (MHA)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](comet.ml/labml/transformer/ea8c108c2d94434ca3c2bc2b21015082)
This is a tutorial/implementation of multi-headed attention
from paper [Attention Is All You Need](https://papers.labml.ai/paper/1706.03762)
in [PyTorch](https://pytorch.org/).
@ -17,8 +20,6 @@ Here is the [training code](basic/autoregressive_experiment.html) that uses a ba
with MHA for NLP auto-regression.
[Here is an experiment implementation](basic/autoregressive_experiment.html) that trains a simple transformer.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb)
"""
import math

3
labml_nn/transformers/models.py
View File

@ -7,6 +7,9 @@ summary: >
---
# Transformer Encoder and Decoder Models
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/basic/autoregressive_experiment.ipynb)
[![Open In Comet](https://images.labml.ai/images/comet.svg?experiment=capsule_networks&file=model)](comet.ml/labml/transformer/ea8c108c2d94434ca3c2bc2b21015082)
"""
import math