11. In many applications the units of these networks apply a " sigmoid function " as an activation function . 12. Where is the matrix of input-to-hidden-layer weights, is some activation function , and is the matrix of hidden-to-output-layer weights. 13. This result holds for a wide range of activation functions , e . g . for the sigmoidal functions. 14. Each convolutional layer is followed by a rectified linear unit, a popular activation function for deep neural networks. 15. Backpropagation requires that the activation function used by the artificial neurons ( or " nodes " ) be differentiable. 16. A generalisation of the logistic function to multiple inputs is the softmax activation function , used in multinomial logistic regression. 17. One of the first versions of the theorem was proved by George Cybenko in 1989 for sigmoid activation functions . 18. Here, \ sigma _ 1 is an element-wise activation function such as a sigmoid function or a rectified linear unit. 19. Where \ phi ^ \ prime is the derivative of the activation function described above, which itself does not vary. 20. Neurons with this kind of activation function are also called " artificial neurons " or " linear threshold units ".
