torchbox.nn package

Submodules

torchbox.nn.activations module

torchbox.nn.activations.crelu(x)

Computes Concatenated ReLU.

Concatenates a ReLU which selects only the positive part of the activation with a ReLU which selects only the negative part of the activation. Note that as a result this non-linearity doubles the depth of the activations. Source: Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units. W. Shang, et al.

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.elu(x)

Computes exponential linear element-wise.

\[y = \left\{ {\begin{tensor}{*{20}{c}}{x,\;\;\;\;\;\;\;\;\;x \ge 0}\\{{e^x} - 1,\;\;\;x < 0}\end{tensor}} \right.. \]

See Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.leaky_relu(x, alpha=0.2)

Compute the Leaky ReLU activation function.

\[y = \left\{ {\begin{tensor}{ccc}{x, x \ge 0}\\{\alpha x, x < 0}\end{tensor}} \right. \]

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Parameters:

x (lists or tensor) – inputs
alpha (float) – $\alpha$

Returns:

outputs

Return type:

tensor

torchbox.nn.activations.linear(x)

linear activation

y = x

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.relu(x)

Computes rectified linear

{\rm max}(x, 0)

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.relu6(x)

Computes Rectified Linear 6

{\rm min}({\rm max}(x, 0), 6)

Convolutional Deep Belief Networks on CIFAR-10. A. Krizhevsky

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.selu(x)

Computes scaled exponential linear

\[y = \lambda \left\{ {\begin{tensor}{*{20}{c}}{x, x \ge 0}\\{\alpha ({e^x} - 1), x < 0}\end{tensor}} \right. \]

where, $\alpha = 1.6732632423543772848170429916717$ , $\lambda = 1.0507009873554804934193349852946$ , See Self-Normalizing Neural Networks

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.sigmoid(x)

sigmoid function

y = \frac{e^x}{e^x + 1}

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.softplus(x)

softplus function

{\rm log}(e^x + 1)

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.softsign(x)

softsign function

\frac{x} {({\rm abs}(x) + 1)}

Parameters:: x (lists or tensor) – inputs
Returns:: outputs
Return type:: tensor

torchbox.nn.activations.swish(x, beta=1.0)

Swish function

y = x\cdot {\rm sigmoid}(\beta x) = {e^{(\beta x)} \over {e^{(\beta x)} + 1}} \cdot x

See “Searching for Activation Functions” (Ramachandran et al. 2017)