learning/annotated_deep_learning_paper_implementations
1
0
Fork 0
mirror of https://github.com/labmlai/annotated_deep_learning_paper_implementations.git synced 2025-11-02 21:40:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

internal covariate shift

Browse Source
This commit is contained in:
Varuna Jayasiri
2021-02-15 21:44:58 +05:30
parent fd33edc34f
commit 2e545435e8
2 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/normalization/batch_norm/index.html
View File

@ -82,7 +82,7 @@ network parameters during training.
For example, let’s say there are two layers $l_1$ and $l_2$. For example, let’s say there are two layers $l_1$ and $l_2$.
During the beginning of the training $l_1$ outputs (inputs to $l_2$) During the beginning of the training $l_1$ outputs (inputs to $l_2$)
could be in distribution $\mathcal{N}(0.5, 1)$. could be in distribution $\mathcal{N}(0.5, 1)$.
Then, after some training steps, it could move to $\mathcal{N}(0.5, 1)$. Then, after some training steps, it could move to $\mathcal{N}(0.6, 1.5)$.
This is <em>internal covariate shift</em>.</p> This is <em>internal covariate shift</em>.</p>
<p>Internal covariate shift will adversely affect training speed because the later layers <p>Internal covariate shift will adversely affect training speed because the later layers
($l_2$ in the above example) have to adapt to this shifted distribution.</p> ($l_2$ in the above example) have to adapt to this shifted distribution.</p>

2
labml_nn/normalization/batch_norm/__init__.py
View File

@ -18,7 +18,7 @@ network parameters during training.
For example, let's say there are two layers $l_1$ and $l_2$. For example, let's say there are two layers $l_1$ and $l_2$.
During the beginning of the training $l_1$ outputs (inputs to $l_2$) During the beginning of the training $l_1$ outputs (inputs to $l_2$)
could be in distribution $\mathcal{N}(0.5, 1)$. could be in distribution $\mathcal{N}(0.5, 1)$.
Then, after some training steps, it could move to $\mathcal{N}(0.5, 1)$. Then, after some training steps, it could move to $\mathcal{N}(0.6, 1.5)$.
This is *internal covariate shift*. This is *internal covariate shift*.
Internal covariate shift will adversely affect training speed because the later layers Internal covariate shift will adversely affect training speed because the later layers