Oscillating activation functions and other recent progress in activation functions: A survey

Abstract

Neural networks are at the heart of many autonomous intelligent systems now. Originally designed to mimic human neurons forming the brain, modern neural networks have their fundamental component in the perceptron, an analog to a single neuron in the brain. A perceptron boils down to a set of input, an equal number of weights, biases, and an activation function that maps the inputs to a bounded output. A perceptron can represent the basic logic gates, except the XOR gate, popularly referred to as the XOR problem. Recent work have proposed biologically inspired oscillatory functions that solve the XOR problem with a single perceptron. We highlight the shortcomings of existing activation functions with respect to the XOR problem and how oscillating functions can solve them in this paper.