Understanding the working mechanism of neural networks
DOI:
https://doi.org/10.56294/shp2025227Keywords:
Neural Networks, Machine Learning, Backpropagation, Activation Function, Gradient DescentAbstract
Neural networks are a foundational component of artificial intelligence and machine learning. This article explores the structure and functioning of neural networks, focusing on core concepts such as weight loss functions, activation functions, and backpropagation with gradient descent. The discussion includes an overview of the complete process of how neural networks operate, with detailed mathematical formulations, equations, and diagrams to aid comprehension.
References
1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
2. LeCun, Y., Bengio, Y., & Hinton, G. (2015). "Deep learning." Nature, 521(7553), 436-444. DOI: https://doi.org/10.1038/nature14539
3. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." Nature, 323(6088), 533-536. DOI: https://doi.org/10.1038/323533a0
4. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet classification with deep convolutional neural networks." Advances in Neural Information Processing Systems, 25, 1097-1105.
5. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
6. Zheng H, Zheng Z, Hu R, Xiao B, Wu Y, Yu F, et al. Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics. Nat Commun 2024;15:277. https://doi.org/10.1038/s41467-023-44614-z. DOI: https://doi.org/10.1038/s41467-023-44614-z
7. Khemani B, Patil S, Kotecha K, Tanwar S. A review of graph neural networks: concepts, architectures, techniques, challenges, datasets, applications, and future directions. J Big Data 2024;11:18. https://doi.org/10.1186/s40537-023-00876-4. DOI: https://doi.org/10.1186/s40537-023-00876-4
8. Li X, Ma Z, Yuan Z, Mu T, Du G, Liang Y, et al. A review on convolutional neural network in rolling bearing fault diagnosis. Meas Sci Technol 2024;35:072002. https://doi.org/10.1088/1361-6501/ad356e. DOI: https://doi.org/10.1088/1361-6501/ad356e
9. Chen J, Zheng L, Hu Y, Wang W, Zhang H, Hu X. Traffic flow matrix-based graph neural network with attention mechanism for traffic flow prediction. Information Fusion 2024;104:102146. https://doi.org/10.1016/j.inffus.2023.102146. DOI: https://doi.org/10.1016/j.inffus.2023.102146
10. Marmolejo-Saucedo JA, Kose U. Numerical Grad-Cam Based Explainable Convolutional Neural Network for Brain Tumor Diagnosis. Mobile Netw Appl 2024;29:109-18. https://doi.org/10.1007/s11036-022-02021-6. DOI: https://doi.org/10.1007/s11036-022-02021-6
11. Dai E, Wang S. Towards Prototype-Based Self-Explainable Graph Neural Network. ACM Trans Knowl Discov Data 2025;19:45:1-45:20. https://doi.org/10.1145/3689647. DOI: https://doi.org/10.1145/3689647
12. Chen A, Rossi RA, Park N, Trivedi P, Wang Y, Yu T, et al. Fairness-Aware Graph Neural Networks: A Survey. ACM Trans Knowl Discov Data 2024;18:138:1-138:23. https://doi.org/10.1145/3649142. DOI: https://doi.org/10.1145/3649142
13. Sharma A, Singh S, Ratna S. Graph Neural Network Operators: a Review. Multimed Tools Appl 2024;83:23413-36. https://doi.org/10.1007/s11042-023-16440-4. DOI: https://doi.org/10.1007/s11042-023-16440-4
14. Li Y, Li J, Wang H, Liu C, Tan J. Knowledge enhanced ensemble method for remaining useful life prediction under variable working conditions. Reliability Engineering & System Safety 2024;242:109748. https://doi.org/10.1016/j.ress.2023.109748. DOI: https://doi.org/10.1016/j.ress.2023.109748
15. Yan H, Wang Z, Xu Z, Wang Z, Wu Z, Lyu R. Research on Image Super-Resolution Reconstruction Mechanism based on Convolutional Neural Network. Proceedings of the 2024 4th International Conference on Artificial Intelligence, Automation and High Performance Computing, New York, NY, USA: Association for Computing Machinery; 2024, p. 142-6. https://doi.org/10.1145/3690931.3690956. DOI: https://doi.org/10.1145/3690931.3690956
16. Lyle C, Zheng Z, Khetarpal K, Hasselt H van, Pascanu R, Martens J, et al. Disentangling the Causes of Plasticity Loss in Neural Networks 2024. https://doi.org/10.48550/arXiv.2402.18762.
17. Chai X, Li S, Liang F. A novel battery SOC estimation method based on random search optimized LSTM neural network. Energy 2024;306:132583. https://doi.org/10.1016/j.energy.2024.132583. DOI: https://doi.org/10.1016/j.energy.2024.132583
18. Gui W, Liu Y, Yu L, Qian Y, Zhang Y, Liu X, et al. Neural network-like microstructures induced by 2-nitrobenzoic acid in SBS fibers for high-sensitivity triboelectric sensors. Chemical Engineering Journal 2025;509:161013. https://doi.org/10.1016/j.cej.2025.161013. DOI: https://doi.org/10.1016/j.cej.2025.161013
19. Liu Z “Leo”. Artificial Neural Networks. En: Liu Z «Leo», editor. Artificial Intelligence for Engineers: Basics and Implementations, Cham: Springer Nature Switzerland; 2025, p. 175-90. https://doi.org/10.1007/978-3-031-75953-6_7. DOI: https://doi.org/10.1007/978-3-031-75953-6_7
20. Sathish Kumar G, Premalatha K, Uma Maheshwari G, Rajesh Kanna P, Vijaya G, Nivaashini M. Differential privacy scheme using Laplace mechanism and statistical method computation in deep neural network for privacy preservation. Engineering Applications of Artificial Intelligence 2024;128:107399. https://doi.org/10.1016/j.engappai.2023.107399. DOI: https://doi.org/10.1016/j.engappai.2023.107399
21. Predić B, Jovanovic L, Simic V, Bacanin N, Zivkovic M, Spalevic P, et al. Cloud-load forecasting via decomposition-aided attention recurrent neural network tuned by modified particle swarm optimization. Complex Intell Syst 2024;10:2249-69. https://doi.org/10.1007/s40747-023-01265-3. DOI: https://doi.org/10.1007/s40747-023-01265-3
22. Shi X, Hao Z, Yu Z. SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks, 2024, p. 5610-9. DOI: https://doi.org/10.1109/CVPR52733.2024.00536
23. Stroud JP, Duncan J, Lengyel M. The computational foundations of dynamic coding in working memory. Trends in Cognitive Sciences 2024;28:614-27. https://doi.org/10.1016/j.tics.2024.02.011. DOI: https://doi.org/10.1016/j.tics.2024.02.011
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Arif Hasanov, Vugar Abdullayev (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The article is distributed under the Creative Commons Attribution 4.0 License. Unless otherwise stated, associated published material is distributed under the same licence.
