Optimizing Convolutional Neural Networks for Image Classification
Keywords:
Convolutional Neural Networks, Deep Learning, Image Classification, OptimizationAbstract
This research explores the optimization of Convolutional Neural Networks (CNNs) for image classification through a numerical experiment. A simplified CNN architecture is trained on a small dataset comprising 100 randomly generated images with a resolution of 28×28. The model incorporates key components such as convolutional layers, batch normalization, max-pooling, and dense layers. Training involves 10 epochs using the Adam optimizer and sparse categorical cross-entropy loss. The results reveal promising training accuracy of 85%, but the validation accuracy, a crucial metric for generalization, lags at 60%. The discussion emphasizes the limitations of the small and synthetic dataset, underscoring the importance of real-world, diverse datasets for meaningful experimentation. The example serves as a foundation for understanding CNN training dynamics, with implications for refining models in more realistic image classification scenarios. The conclusion calls for future research to focus on advanced techniques, larger datasets, and comprehensive validation processes to enhance the reliability and applicability of CNN models in practical applications.
References
E. Elyan et al., “Computer vision and machine learning for medical image analysis: recent advances, challenges, and way forward.,” Artif. Intell. Surg., vol. 2, 2022.
K. Sharada, W. Alghamdi, K. Karthika, A. H. Alawadi, G. Nozima, and V. Vijayan, “Deep Learning Techniques for Image Recognition and Object Detection,” in E3S Web of Conferences, EDP Sciences, 2023, p. 4032.
L. Alzubaidi et al., “Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions,” J. big Data, vol. 8, pp. 1–74, 2021.
J. Wang et al., “Generalizing to unseen domains: A survey on domain generalization,” IEEE Trans. Knowl. Data Eng., 2022.
K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Mixstyle neural networks for domain generalization and adaptation,” Int. J. Comput. Vis., pp. 1–15, 2023.
M. M. H. Shuvo, S. K. Islam, J. Cheng, and B. I. Morshed, “Efficient acceleration of deep learning inference on resource-constrained edge devices: A review,” Proc. IEEE, 2022.
B. G. F. Meng and B. Ghena, “Research on text recognition methods based on artificial intelligence and machine learning,” Prepr. under Rev., 2023.
A. Hernández-García and P. König, “Data augmentation instead of explicit regularization,” arXiv Prepr. arXiv1806.03852, 2018.
P. Chlap, H. Min, N. Vandenberg, J. Dowling, L. Holloway, and A. Haworth, “A review of medical image data augmentation techniques for deep learning applications,” J. Med. Imaging Radiat. Oncol., vol. 65, no. 5, pp. 545–563, 2021.
M. Capra, B. Bussolino, A. Marchisio, M. Shafique, G. Masera, and M. Martina, “An updated survey of efficient hardware architectures for accelerating deep convolutional neural networks,” Futur. Internet, vol. 12, no. 7, p. 113, 2020.
Q. Xu, M. Zhang, Z. Gu, and G. Pan, “Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs,” Neurocomputing, vol. 328, pp. 69–74, 2019.
Y. Kubo and T. Trappenberg, “Mitigating overfitting using regularization to defend networks against adversarial examples,” in Advances in Artificial Intelligence: 32nd Canadian Conference on Artificial Intelligence, Canadian AI 2019, Kingston, ON, Canada, May 28–31, 2019, Proceedings 32, Springer, 2019, pp. 400–405.
D. Banner et al., “Patient and public engagement in integrated knowledge translation research: are we there yet?,” Res. Involv. Engagem., vol. 5, no. 1, pp. 1–14, 2019.
M. Won, “Representation learning for music classification and retrieval: bridging the gap between natural language and music semantics.” Universitat Pompeu Fabra, 2022.
L. Chen, S. Li, Q. Bai, J. Yang, S. Jiang, and Y. Miao, “Review of image classification algorithms based on convolutional neural networks,” Remote Sens., vol. 13, no. 22, p. 4712, 2021.
A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A survey of the recent architectures of deep convolutional neural networks,” Artif. Intell. Rev., vol. 53, pp. 5455–5516, 2020.
D. R. Sarvamangala and R. V Kulkarni, “Convolutional neural networks in medical image understanding: a survey,” Evol. Intell., vol. 15, no. 1, pp. 1–22, 2022.
M. Shafiq and Z. Gu, “Deep residual learning for image recognition: A survey,” Appl. Sci., vol. 12, no. 18, p. 8972, 2022.
K. Maharana, S. Mondal, and B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Glob. Transitions Proc., vol. 3, no. 1, pp. 91–99, 2022.
C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. big data, vol. 6, no. 1, pp. 1–48, 2019.
S. Tang, S. Yuan, and Y. Zhu, “Data preprocessing techniques in convolutional neural network based on fault diagnosis towards rotating machinery,” IEEE Access, vol. 8, pp. 149487–149496, 2020.
W. Jo, S. Kim, C. Lee, and T. Shon, “Packet preprocessing in CNN-based network intrusion detection system,” Electronics, vol. 9, no. 7, p. 1151, 2020.
Y. Wu et al., “Demystifying learning rate policies for high accuracy training of deep neural networks,” in 2019 IEEE International conference on big data (Big Data), IEEE, 2019, pp. 1971–1980.
Z. Li and S. Arora, “An exponential learning rate schedule for deep learning,” arXiv Prepr. arXiv1910.07454, 2019.
M. V Narkhede, P. P. Bartakke, and M. S. Sutaone, “A review on weight initialization strategies for neural networks,” Artif. Intell. Rev., vol. 55, no. 1, pp. 291–322, 2022.
B. Hanin and D. Rolnick, “How to start training: The effect of initialization and architecture,” Adv. Neural Inf. Process. Syst., vol. 31, 2018.
C. Zhu, R. Ni, Z. Xu, K. Kong, W. R. Huang, and T. Goldstein, “Gradinit: Learning to initialize neural networks for stable and efficient training,” Adv. Neural Inf. Process. Syst., vol. 34, pp. 16410–16422, 2021.
L. Datta, “A survey on activation functions and their relation with xavier and he normal initialization,” arXiv Prepr. arXiv2004.06632, 2020.
M. Awais, M. T. Bin Iqbal, and S.-H. Bae, “Revisiting internal covariate shift for batch normalization,” IEEE Trans. Neural Networks Learn. Syst., vol. 32, no. 11, pp. 5082–5092, 2020.
W. Jung, D. Jung, B. Kim, S. Lee, W. Rhee, and J. H. Ahn, “Restructuring batch normalization to accelerate CNN training,” Proc. Mach. Learn. Syst., vol. 1, pp. 14–26, 2019.
S. H. Khan, M. Hayat, and F. Porikli, “Regularization of deep neural networks with spectral dropout,” Neural Networks, vol. 110, pp. 82–90, 2019.
I. Salehin and D.-K. Kang, “A review on dropout regularization approaches for deep neural networks within the scholarly domain,” Electronics, vol. 12, no. 14, p. 3106, 2023.
D. Newton, F. Yousefian, and R. Pasupathy, “Stochastic gradient descent: Recent trends,” Recent Adv. Optim. Model. Contemp. Probl., pp. 193–220, 2018.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Muhammad Khoiruddin Harahap, Chlap Min Xu, Kong Huang Zhu
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
