learning/annotated_deep_learning_paper_implementations
1
0
Fork 0
mirror of https://github.com/labmlai/annotated_deep_learning_paper_implementations.git synced 2025-11-02 13:00:17 +08:00
Code Issues Packages Projects Releases Wiki Activity

🚧 layer norm

Browse Source
This commit is contained in:
Varuna Jayasiri
2021-02-02 10:18:06 +05:30
parent 1eb9fe4dd6
commit c5d47067be
2 changed files with 46 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
labml_nn/normalization/layer_norm/__init__.py
View File

@ -29,20 +29,17 @@ Layer normalization does it for each batch across all elements.
Layer normalization is generally used for NLP tasks.
Here's [the training code](mnist.html) and a notebook for training
a CNN classifier that use batch normalization for MNIST dataset.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lab-ml/nn/blob/master/labml_nn/normalization/batch_norm/mnist.ipynb)
[![View Run](https://img.shields.io/badge/labml-experiment-brightgreen)](https://web.lab-ml.com/run?uuid=011254fe647011ebbb8e0242ac1c0002)
We have used layer normalization in most of the
[transformer implementations](../../transformers/gpt/index.html).
"""
import torch
from torch import nn
class BatchNorm(nn.Module):
class LayerNorm(nn.Module):
"""
## Batch Normalization Layer
## Layer Normalization
"""
def __init__(self, channels: int, *,