Survey on Activation Functions: A Comparative Study between state-of-the-art Activation Functions and Oscillatory Activation Functions

Abstract

Activation functions are extremely important for constructing a neural network. They assist in determining which neurons in each layer will be triggered. The main function of the activation function is to introduce non-linearity into the network. Because the majority of real-world problems are complex and non-linear, we need activation functions for the network to solve them. As a result, selecting the most appropriate activation function based on the problem statement is crucial. This paper will go over some of the most commonly used activation functions such as ReLU, Sigmoid, Swish, Mish, and so on. It will also compare these traditional activation functions with oscillatory activation functions like GCU, SQU, DSU, etc that have been inspired by biological neurons.