pydbm.activation package

Subpackages

• pydbm.activation.interface package
  • Submodules
  • pydbm.activation.interface.activating_function_interface module
  • Module contents
• pydbm.activation.signfunction package
  • Submodules
  • pydbm.activation.signfunction.deterministic_binary_neurons module
  • pydbm.activation.signfunction.stochastic_binary_neurons module
  • Module contents
• pydbm.activation.softmaxfunction package
  • Submodules
  • pydbm.activation.softmaxfunction.log_softmax_function module
  • Module contents

Submodules

pydbm.activation.identity_function module

class pydbm.activation.identity_function.IdentityFunction

Bases: pydbm.activation.interface.activating_function_interface.ActivatingFunctionInterface

Identity function.

activate

Activate and extract feature points in forward propagation.

Parameters:np.ndarray of observed data points. (x) – Returns:np.ndarray of the activated feature points.

backward

Back propagation but not operate the activation.

Parameters:y – np.ndarray of delta. Returns:The result.

derivative

Derivative and extract delta in back propagation.

Parameters:y – np.ndarray of delta. Returns:np.ndarray of delta.

forward

Forward propagation but not retain the activation.

Parameters:np.ndarray of observed data points. (x) – Returns:The result.

pydbm.activation.logistic_function module

class pydbm.activation.logistic_function.LogisticFunction

Bases: pydbm.activation.interface.activating_function_interface.ActivatingFunctionInterface

Logistic Function.

activate

Activate and extract feature points in forward propagation.

Parameters:np.ndarray of observed data points. (x) – Returns:np.ndarray of the activated feature points.

backward

Back propagation but not operate the activation.

Parameters:y – np.ndarray of delta. Returns:The result.

derivative

Derivative and extract delta in back propagation.

Parameters:y – np.ndarray of delta. Returns:np.ndarray of delta.

forward

Forward propagation but not retain the activation.

Parameters:np.ndarray of observed data points. (x) – Returns:The result.

pydbm.activation.relu_function module

class pydbm.activation.relu_function.ReLuFunction

Bases: pydbm.activation.interface.activating_function_interface.ActivatingFunctionInterface

ReLu Function.

activate

Activate and extract feature points in forward propagation.

Parameters:np.ndarray of observed data points. (x) – Returns:np.ndarray of the activated feature points.

backward

Back propagation but not operate the activation.

Parameters:y – np.ndarray of delta. Returns:The result.

derivative

Derivative and extract delta in back propagation.

Parameters:y – np.ndarray of delta. Returns:np.ndarray of delta.

forward

Forward propagation but not retain the activation.

Parameters:np.ndarray of observed data points. (x) – Returns:The result.

pydbm.activation.sign_function module

class pydbm.activation.sign_function.SignFunction

Bases: pydbm.activation.interface.activating_function_interface.ActivatingFunctionInterface

Sign function (or Signam function).

As the sign function is non-smooth and non-convex, its gradient is zero for all nonzero inputs, and is ill-defined at zero, which makes the standard back-propagation infeasible. This is a kind of the vanishing gradient problem(Song, J. et al., 2018).

In order to tackle this problem setting, this library entries some methods such as a Straight-Through Estimator(Bengio, Y. et al., 2013) and a Binary Neurons(Oza, M., et al., 2018).

The Straight-Through Estimator is a strategy to replace the non-differentiable functions, which are used in the forward pass, by differentiable functions in the backward pass. In the backward pass, the Straight-Through Estimator simply treats the binary neurons as identify functions and ignores their gradients. A variant of the Straight-Through Estimator is the sigmoid-adjusted Straight-Through Estimator, which multiplies the gradients in the backward pass by the derivative of the sigmoid function.

The binary neurons are neurons that output binary valued predictions as a sign-like function. This library draw a distinction between deterministic binary neurons and stochastic binary neurons. The function of deterministic binary neurons is to act like neurons with hard thresholding functions as their activation functions. The stochastic binary neurons, in contrast, binarize an input according to a probability.

This class is a abstract class in Template Method Pattern, which is also useful design method to design the approximators in this library because this design pattern makes it possible to define the skeleton of an algorithm in the approximations, deferring some steps to client subclasses such as DeterministicBinaryNeurons and StochasticBinaryNeurons.

References

• Bengio, Y., Léonard, N., & Courville, A. (2013). Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432.
• Chung, J., Ahn, S., & Bengio, Y. (2016). Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704.
• Oza, M., Vaghela, H., & Srivastava, K. (2019). Progressive Generative Adversarial Binary Networks for Music Generation. arXiv preprint arXiv:1903.04722.
• Song, J., He, T., Gao, L., Xu, X., Hanjalic, A., & Shen, H. T. (2018, April). Binary generative adversarial networks for image retrieval. In Thirty-Second AAAI Conference on Artificial Intelligence.

activity_arr_list

getter

get_activity_arr_list

getter

get_memory_len

getter

memory_len

getter

set_activity_arr_list

setter

set_memory_len

setter

pydbm.activation.softmax_function module

class pydbm.activation.softmax_function.SoftmaxFunction

Bases: pydbm.activation.interface.activating_function_interface.ActivatingFunctionInterface

Softmax function.

activate

Activate and extract feature points in forward propagation.

Parameters:np.ndarray of observed data points. (x) – Returns:np.ndarray of the activated feature points.

backward

Back propagation but not operate the activation.

Parameters:y – np.ndarray of delta. Returns:The result.

derivative

Derivative and extract delta in back propagation.

Parameters:y – np.ndarray of delta. Returns:np.ndarray of delta.

forward

Forward propagation but not retain the activation.

Parameters:np.ndarray of observed data points. (x) – Returns:The result.

pydbm.activation.tanh_function module

class pydbm.activation.tanh_function.TanhFunction

Bases: pydbm.activation.interface.activating_function_interface.ActivatingFunctionInterface

Tanh function.

activate

Activate and extract feature points in forward propagation.

Parameters:np.ndarray of observed data points. (x) – Returns:np.ndarray of the activated feature points.

backward

Back propagation but not operate the activation.

Parameters:y – np.ndarray of delta. Returns:The result.

derivative

Derivative and extract delta in back propagation.

Parameters:y – np.ndarray of delta. Returns:np.ndarray of delta.

forward

Forward propagation but not retain the activation.

Parameters:np.ndarray of observed data points. (x) – Returns:The result.

Module contents