langml.activations

Activations

Module Contents

Functions

gelu(x: langml.tensor_typing.Tensors) → langml.tensor_typing.Tensors	Gaussian Error Linear Units (GELUs)
relu2(x: langml.tensor_typing.Tensors) → langml.tensor_typing.Tensors

Attributes

custom_objects

langml.activations.gelu(x: langml.tensor_typing.Tensors) → langml.tensor_typing.Tensors

Gaussian Error Linear Units (GELUs) https://arxiv.org/abs/1606.08415

$GELU(x) = 0.5x(1 + tanh[sqrt(2 / Pi) (x + 0.044715x^3)])$

langml.activations.relu2(x: langml.tensor_typing.Tensors) → langml.tensor_typing.Tensors

langml.activations.custom_objects