JOURNAL OF THE INDIAN ACADEMY OF MATHEMATICS

(Published by Indian Academy of Mathematics since 1978)

Promoting Research in Pure and Applied Mathematics

ISSN 0970-5120


HomeArchiveContents
← Back to Volume 48, Issue 1 Contents

MATHEMATICAL PROPERTIES OF ACTIVATION FUNCTIONS IN ARTIFICIAL INTELLIGENCE DEVELOPMENTS: Analysis and Implications for Deep Neural Architectures

Massimiliano Ferrara, Celeste Ciccia
Decisions LAB, University Mediterranea of Reggio Calabria, Italy.
Volume: 48Issue: 1Pages: 1–9Year: 2026ISSN: 0970-5120
⬇ Download Full PDF

Abstract

Activation functions govern the expressive power and training dynamics of deep neural networks through their analytical properties. This paper provides a rigorous mathematical analysis of six fundamental activation functions – Linear, Sigmoid, Hyperbolic Tangent, ReLU, Parametric ReLU, and Exponential Linear Unit – examining how regularity, gradient structure, and spectral properties influence representational capacity, gradient flow stability, and convergence behavior in deep architectures. We establish formal results on the representational collapse of linear activations, derive sharp gradient
decay bounds for saturating functions, prove gradient preservation theorems for piecewiselinear activations, and characterize the convergence advantages of smooth non-saturating units. Our analysis yields a unified mathematical framework connecting activation function properties to network trainability, with direct implications for the design of deep learning architectures in sequential decision-making, continuous control, and safety-critical applications.
Keywords: Activation functions, deep neural networks, gradient flow, vanishing gradients, convergence analysis, ReLU, ELU, representational capacity
MSC 2020: 68T07, 65K10, 90C26, 41A25, 60H35.

How to Cite

Massimiliano Ferrara, Celeste Ciccia, MATHEMATICAL PROPERTIES OF ACTIVATION FUNCTIONS IN ARTIFICIAL INTELLIGENCE DEVELOPMENTS: Analysis and Implications for Deep Neural Architectures, Journal of the Indian Academy of Mathematics, 48(1) (2026), 1–9.

References

  • K. Hornik, M. Stinchcombe, H. White, Multilayer feedforward networks are universal approximators, Neural Networks 2(5) (1989) 359–366.
  • I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016.
  • Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521(7553) (2015) 436–444.
  • X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: Proc. AISTATS, JMLR W&CP 15, 2011, pp. 315–323.
  • K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, in: Proc. IEEE ICCV, 2015, pp. 1026–1034.
  • D.-A. Clevert, T. Unterthiner, S. Hochreiter, Fast and accurate deep network learning by exponential linear units (ELUs), arXiv preprint arXiv:1511.07289, 2015.
  • Y. Bengio, P. Simard, P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Trans. Neural Netw. 5(2) (1994) 157–166.
  • R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
  • X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proc. AISTATS, JMLR W&CP 9, 2010, pp. 249–256.
  • Z. Allen-Zhu, Y. Li, Z. Song, A convergence theory for deep learning via over-parameterization, in: Proc. ICML, PMLR 97, 2019, pp. 242–252.
  • P. Ramachandran, B. Zoph, Q. V. Le, Searching for activation functions, arXiv preprint arXiv:1710.05941, 2017.
  • V. Mnih et al., Human-level control through deep reinforcement learning, Nature 518(7540) (2015) 529–533.

© Journal of the Indian Academy of Mathematics