G.I. Tedeev1, A.Yu. Salnikov2

1, 2 Russian Presidential Academy of National Economy and Public Administration (Moscow, Russia)
1 g.i.tedeev@gmail.com, 2 salnikov-ay@ranepa.ru

The article presents an overview of domestic and foreign publications devoted to the application of heuristic optimization methods in the problems of training artificial neural networks. Objective is to provide information on the experience of applying methods to facilitate the selection of the most appropriate training algorithm in various problems. The analysis of the publications under consideration showed that, despite the prevalence of the error backpropagation method, the use of various heuristic optimization methods for training neural networks is widely covered in scientific papers. In particular, these methods are used to solve applied problems. The most common methods used in the context of training neural networks are the genetic algorithm, the particle swarm method, the simulated annealing method, the ant colony method, and some gradient algorithms with heuristic elements. The use of various hybrid methods is widespread.

Publications often compare the efficiency of the heuristic methods under consideration with the error backpropagation method, and some also contain a comparative analysis of several heuristic methods. However, a full comparison of all of the listed methods was not presented in the publications. Description of the application of optimization methods will help solve the problems of training artificial neural networks.

Tedeev G.I., Salnikov A.Yu. Overview of heuristic optimization methods application in training of artificial neural networks. Dynamics of complex systems. 2025. V. 19. № 4. P. 86−102. DOI: 10.18127/j19997493-202504-09 (in Russian).

1. Krug P.G. Nejronny`e seti i nejrokomp`yutery`: Uchebnoe posobie po kursu «Mikroprocessory`». M.: Izdatel`stvo ME`I; 2002. 176 s.
2. Smirnov A.V. Perspektivy` primeneniya e`vristicheskix metodov optimizacii v proektirovanii radiotexnicheskix i telekommunikacionny`x ustrojstv i sistem. Russian Technological Journal. 2017;5(6):20–33. https://doi.org/10.32362/2500-316X-2017-5-6-20-33
3. Vasenkov D.V. Metody` obucheniya iskusstvenny`x nejronny`x setej. Komp`yuterny`e instrumenty` v obrazovanii. 2007;1:20–29.
4. Kuz`min D.M. Metody` obucheniya nejronny`x setej. E`nergiya-2021: Tezisy` dokladov Shestnadczatoj vserossijskoj nauchno-texnicheskoj konferencii studentov, aspirantov i molody`x ucheny`x. V 6-ti t. 2021;4:42.
5. Dragomirov P.D., Lar`kov M.A. Metody` obucheniya nejronny`x setej. Novy`e texnologii v uchebnom processe i proizvodstve: Materialy` XXI Mezhdunarodnoj nauchno-texnicheskoj konferencii. Ryazan`. 2023; S. 659–661.
6. Feshina E.V., Omel`chenko D.A., Gonataev R.G. Analiz sposobov obucheniya nejronny`x setej. Innovacii. Nauka. Obrazovanie. 2021;28:978–982.
7. Abragin A.V. Geneticheskij algoritm obucheniya iskusstvenny`x nejronny`x setej. Potencial sovremennoj nauki. 2015;8(16):8–11.
8. Zaginajlo M.V. Obuchenie iskusstvenny`x nejronny`x setej s pomoshh`yu matematicheskogo apparata geneticheskix algoritmov. Innovacii. Nauka. Obrazovanie. 2020;12:384–389.
9. Shumkov E.A., Chistik I.K. Ispol`zovanie geneticheskix algoritmov dlya obucheniya nejronny`x setej. Politematicheskij setevoj e`lektronny`j nauchny`j zhurnal Kubanskogo gosudarstvennogo agrarnogo universiteta. 2013;91:455–464.
10. Mishhenko V.A., Korobkin A.A. Ispol`zovanie geneticheskix algoritmov v obuchenii nejronny`x setej. Sovremenny`e problemy` nauki i obrazovaniya. 2011;6:116.
11. Bozhich V.I., Lebedev O.B., Shnicer Yu.L. Razrabotka geneticheskogo algoritma obucheniya nejronny`x setej. Izvestiya Yuzhnogo federal`nogo universiteta. Texnicheskie nauki. 2001;22(4):170–174.
12. Shmatov G.P., Fomina E.E. Nejronny`e seti i geneticheskij algoritm. Tver`: Tverskoj gosudarstvenny`j texnicheskij universitet; 2019. 200 s.
13. Panchenko T.V. Geneticheskie algoritmy`: uchebno-metodicheskoe posobie. Astraxan`: Izdatel`skij dom «Astraxanskij universitet»; 2007. 87 s.
14. Gambarova E.M. Optimizaciya struktury` nejronny`x setej s ispol`zovaniem geneticheskogo algoritma dlya raspoznavaniya ob``ektov na kosmicheskix snimkax. Informaciya i kosmos. 2009;3:67–71.
15. Sánchez D., Melin P., Castillo O., Valdez F. Modular granular neural networks optimization with Multi-Objective Hierarchical Genetic Algorithm for human recognition based on iris biometric. IEEE Congress on Evolutionary Computation, Cancun, Mexico. 2013; pp. 772–778. https://doi.org/10.1109/CEC.2013.6557646
16. Klochkova K.V., Petrovich S.V., Abramova V.V. Obuchenie nejronnoj seti s primeneniem geneticheskogo algoritma dlya prognozirovaniya svojstv chugunov s vermikulyarny`m grafitom. Grani nauki. 2013;1:104–107.
17. Zhang J., Xue Q., Chen L., Deng Q. Combat Decision-Making Modeling Method Based on Genetic Neural Network. 12th International Conference on Intelligent Computation Technology and Automation. Xiangtan, China. 2019; pp. 11–15. https://doi.org/10.1109/ ICICTA49267.2019.00010
18. Trokoz D.A. Metod parametricheskoj optimizacii dlya shirokix nejronny`x setej s ispol`zovaniem geneticheskix algoritmov. Izvestiya Samarskogo nauchnogo centra Rossijskoj akademii nauk. 2021;23(2):51–56.
19. Lipanov A.M. Primenenie geneticheskogo algoritma dlya obucheniya nejronnoj seti v zadache identifikacii STM-izobrazhenij. Polzunovskij vestnik. 2010;2:216–220.
20. Bondarenko I.B., Gatchin Yu.A., Geranichev V.N. Sintez optimal`ny`x iskusstvenny`x nejronny`x setej s pomoshh`yu modificirovannogo geneticheskogo algoritma. Nauchno-texnicheskij vestnik informacionny`x texnologij, mexaniki i optiki. 2012;2(78):51–55.
21. Ayupov I.R. Parametricheskij metod obucheniya nejronnoj seti v zadache prognozirovaniya. Upravlenie e`konomicheskimi sistemami: e`lektronny`j nauchny`j zhurnal. 2015;1(73):10.
22. Zaginajlo M.V., Fatxi V.A. Ocenka e`ffektivnosti razlichny`x metodov obucheniya iskusstvenny`x nejronny`x setej. Innovacii. Nauka. Obrazovanie. 2021;35:442–447.
23. Kajornrit J. A comparative study of optimization methods for improving artificial neural network performance. 2015 7th International Conference on Information Technology and Electrical Engineering. Chiang Mai, Thailand. 2015; pp. 35–40. https://doi.org/10.1109/ ICITEED.2015.7408908
24. Lyozin I.A., Murav`yov V.V. Sravnenie algoritmov obucheniya nejronnoj seti s binarny`mi vxodami. Izvestiya Samarskogo nauchnogo centra Rossijskoj akademii nauk. 2016;18(4-4):766–769.
25. Derisma, Silvana M., Imelda. Optimization of Neural Network with Genetic Algorithm for Breast Cancer Classification. 2018 Internatio­nal Conference on Information Technology Systems and Innovation. Bandung, Indonesia. 2018; pp. 398–403. https://doi.org/10.1109/ ICITSI.2018.8696014
26. Cherny`x V.S. Sravnenie e`ffektivnosti razlichny`x metodov obucheniya nejronny`x setej. Nauchny`j rezul`tat. Informacionny`e texnologii. 2023;8(1):83–93.
27. Larionov V.S., Maleev O.G. E`ffektivnost` rasparallelivaniya metoda na osnove roya chasticz pri optimizacii obucheniya nejronny`x setej. Sovremennaya nauka: aktual`ny`e problemy` teorii i praktiki. Ser.: Estestvenny`e i texnicheskie nauki. 2022;7:71–77.
28. Korolev S.A., Majkov D.V. Modifikaciya algoritma roya chasticz na osnove metoda analiza ierarxij. Vestnik VGU. Ser.: Sistemny`j analiz i informacionny`e texnologii. 2019;4:36–46.
29. Shajdurov R.S., Andreeva K.A. Primenenie metoda roya chasticz i geneticheskogo algoritma dlya resheniya zadachi opredeleniya ploshhadi lesnogo pozhara. Aktual`ny`e problemy` aviacii i kosmonavtiki. 2016;1(12):670–672.
30. Chen X., Hu N. Effectiveness evaluation for UAV air-to-ground attack based on PSO-BP neural network. 36th Chinese Control Confe­rence. Dalian, China. 2017; pp. 3864–3869. https://doi.org/10.23919/ChiCC.2017.8027961
31. Yin Z., Guo X., Tseng S.P., Tang L., Chen Y. Research on Prediction of Medical Training Effect Based on PSO-BP Neural Network. 10th International Conference on Orange Technology. Shanghai, China. 2022; pp. 1–4. https://doi.org/10.1109/ ICOT56925.2022.10008122
32. Shi L., Tang X., Lv J. PCA-based PSO-BP neural network optimization algorithm. The 27th Chinese Control and Decision Conference. Qingdao, China. 2015; pp. 1720–1725. https://doi.org/10.1109/CCDC.2015.7162197
33. Kazakova E.M. Obuchenie iskusstvennoj nejronnoj seti s ispol`zovaniem gibridnogo algoritma optimizacii PSOJaya. Izvestiya Kabardino-Balkarskogo nauchnogo centra RAN. 2023;6(116):95–102.
34. Pang X., Ma H., Su P., Tang G.Y. TPPMA: New Adaptive BP Neural Network Based on PSO and PCA Algorithms. IEEE 27th International Symposium on Industrial Electronics. Cairns, QLD, Australia. 2018; pp. 637–642. https://doi.org/10.1109/ISIE.2018.8433786
35. Kolosov A.A. Mnogopotochnoe obuchenie nejronny`x setej s pomoshh`yu metoda roya chasticz. Novy`e matematicheskie metody` i komp`yuterny`e texnologii v proektirovanii, proizvodstve i nauchny`x issledovaniyax: Materialy` XXIII Respublikanskoj nauchnoj konferencii studentov i aspirantov. Gomel`: Gomel`skij gosudarstvenny`j universitet im. Franciska Skoriny`. 2020; S. 245–246.
36. Larionov V.S., Safiullina L.X. E`ffektivnost` primeneniya metodov na osnove roya chasticz pri optimizacii ICLPSO obucheniya nejronny`x setej. Sovremennaya nauka: aktual`ny`e problemy` teorii i praktiki. Ser.: Estestvenny`e i texnicheskie nauki. 2022;4(2):81–87.
37. Chastikova V.A., Vlasov K.A., Kartamy`shev D.A. Obnaruzhenie DDoS-atak na osnove nejronny`x setej s primeneniem metoda roya chasticz v kachestve algoritma obucheniya. Fundamental`ny`e issledovaniya. 2014;8(4):829–832.
38. Kotel`nikova A.Yu., Vanin A.S. Metody` obucheniya nejronny`x setej dlya kratkosrochnogo prognozirovaniya nagruzki v intellektual`ny`x e`lektricheskix setyax. E`lektroe`nergetika glazami molodezhi 2016: Materialy` VII Mezhdunarodnoj molodyozhnoj nauchno-texnicheskoj konferencii. V 3 t. T. 2. Kazan`: Kazanskij gosudarstvenny`j e`nergeticheskij universitet. 2016; S. 264–267.
39. Gupta S., Singhal. Phishing URL detection by using artificial neural network with PSO. 2nd International Conference on Telecommunication and Networks. Noida, India. 2017; pp. 1–6. https://doi.org/10.1109/TEL-NET.2017.8343553
40. Tityunnikov A.V., Besschetnov A.V. Primenenie metoda roya chasticz v kachestve obucheniya nejronny`x setej. Problemy` nauki. 2019;5(41):50–52.
41. Ermakov B.S. Optimizaciya roem chasticz v obuchenii iskusstvenny`x nejronny`x setej. Sistemny`j analiz i logistika. 2017;1:3–9.
42. Rakityanskaya A.S., E`ngel`brext A.P. Obuchenie iskusstvenny`x nejrosetej s pomoshh`yu dinamicheskogo algoritma roya chasticz. Matematicheskoe modelirovanie. 2012;24(12):107–112.
43. Basalin P.D., Belokry`lov P.Yu., Zgurskij D.S. Sintez sxem proizvol`noj kombinacionnoj logiki v nejrosetevom bazise s primeneniem metoda imitacii otzhiga. Vestnik Nizhegorodskogo universiteta im. N.I. Lobachevskogo. 2008;5:126–130.
44. Gavrilov A.V., Kangler V.M. Ispol`zovanie iskusstvenny`x nejronny`x setej dlya analiza danny`x. Sbornik nauchny`x trudov NGTU. 1999;3:16.
45. Lyozin I.A., Kir`yakov S.A. Ispol`zovanie algoritma imitacii otzhiga dlya obucheniya seti TSK. Izvestiya Samarskogo nauchnogo centra Rossijskoj akademii nauk. 2015;17(2-5):1041–1043.
46. Kozy`rev A.S., Legotin D.L. Ispol`zovanie geneticheskix algoritmov obucheniya nejronny`x setej dlya resheniya zadach prognozirovaniya. Aktual`ny`e problemy` prepodavaniya informacionny`x i estestvenno-nauchny`x disciplin: Materialy` XV Vserossijskoj nauchno-metodicheskoj konferencii. Kostroma: Kostromskoj gosudarstvenny`j universitet. 2022; S. 125–129.
47. Turovskij Ya.A., Adamenko A.A. Sravnitel`ny`j analiz e`volyucionnogo metoda s ispol`zovaniem «izolyatov» i metoda imitacii otzhiga pri obuchenii iskusstvenny`x nejronny`x setej. Programmnaya inzheneriya. 2018;9(4):185–190.
48. Nikolaeva N.A., Lezina I.V. Ispol`zovanie algoritma imitacii otzhiga dlya obucheniya mnogoslojnogo perseptrona pri reshenii zadachi identifikacii zakonov raspredeleniya. Prikladnaya matematika i informatika: sovremenny`e issledovaniya v oblasti estestvenny`x i texnicheskix nauk: Materialy` III nauchno-prakticheskoj vserossijskoj konferencii (shkoly`-seminara) molody`x ucheny`x. Tol`yatti: Izdatel` Kachalin Aleksandr Vasil`evich. 2017; S. 437–439.
49. Maklachkova V.V. Sravnitel`ny`j analiz algoritmov obratnogo rasprostraneniya oshibki i imitacii otzhiga. REDS: Telekommunikacionny`e ustrojstva i sistemy`. 2023;13(1):26–32.
50. Ni W., Li R. Research on Recognition Technology of Fruit based on Simulated Annealing Algorithm and Neural Network. IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers. Dalian, China. 2021; pp. 877 880. https://doi.org/10.1109/ IPEC51340.2021.9421261
51. Zhang H., Wang Y., Deng C. Application of gesture recognition based on simulated annealing BP neural network. Proceedings of 2011 International Conference on Electronic & Mechanical Engineering and Information Technology. Harbin, China. 2011; pp. 178–181. https://doi.org/10.1109/EMEIT.2011.6022891
52. Kurejchik V.M., Mileshko L.P., Spiridonov O.B., Shtuchny`j A.M. Problema poiska perspektivny`x algoritmov obucheniya nejronny`x setej. Innovacii v informacionny`x texnologiyax, mashinostroenii i avtotransporte: Sbornik materialov V Mezhdunarodnoj nauchno-prakticheskoj konferencii. Kemerovo: Kuzbasskij gosudarstvenny`j texnicheskij universitet imeni T.F. Gorbacheva. 2021; S. 100–102.
53. Zhang C.T., Zhao A.X. Using adaptive ant colony algorithm optimized BP neural network to identify the DGA fault. IEEE International Conference of IEEE Region 10. Xian, China. 2013; pp. 1–4. https://doi.org/10.1109/TENCON.2013.6719070
54. Kotlyarov E.V. Obuchenie nejronnoj seti na osnove algoritma murav`inoj kolonii dlya zadachi klassifikacii. E`lektrotexnicheskie i komp`yuterny`e sistemy`. 2012;8(84):122–129.
55. Valdez F., Castillo O., Melin P. Ant colony optimization for the design of Modular Neural Networks in pattern recognition. International Joint Conference on Neural Networks. Vancouver, BC, Canada. 2016; pp. 163–168. https://doi.org/10.1109/ IJCNN.2016.7727194
56. Liu L. Research on Optimization Design of RBF Neural Network Based on Ant Colony Algorithm. 2nd International Conference on 3D Immersion, Interaction and Multi-sensory Experiences. Madrid, Spain. 2023; pp. 11–15. https://doi.org/10.1109/ICDIIME59043. 2023.00008
57. Belyavskij G.I., Puchkov E.V., Lila V.B. Algoritm i programmnaya realizaciya gibridnogo metoda obucheniya iskusstvenny`x nejronny`x setej. Programmny`e produkty` i sistemy`. 2012;4:96–100.
58. Protalinskij O.M., Shherbatov I.A., Belyaev I.O. Gibridny`j metod obucheniya nejronny`x setej dlya klassifikacii kataliticheskoj stadii processa Klausa. Vestnik Saratovskogo gosudarstvennogo texnicheskogo universiteta. 2010;4(2):38–43.
59. Yang R., Hu X., He L. Prediction of Shanghai air quality index based on BP neural network optimized by genetic algorithm. 13th International Symposium on Computational Intelligence and Design. Hangzhou, China. 2020; pp. 205–208. https://doi.org/10.1109/ ISCID51228.2020.00052
60. Ry`bak L.A., Mamaev Yu.A., Virabyan L.G. Sintez algoritma korrekcii traektorii dvizheniya vy`xodnogo zvena roboto-geksapoda na osnove teorii iskusstvenny`x nejronny`x setej. Vestnik Belgorodskogo gosudarstvennogo texnologicheskogo universiteta im. V.G. Shuxova. 2016;12:142–151.
61. Zhang Y., Niu B., Zhuang X., Liao H. Water content ratio measurement with neural network based on simulated annealing. Seventh International Conference on Natural Computation. Shanghai, China. 2011; pp. 878–881. https://doi.org/10.1109/ ICNC.2011.6022215
62. Kotlyarov E.V., Petrushina T.I. Gibridny`j metod obucheniya iskusstvennoj nejronnoj seti na osnove modificirovannogo algoritma murav`ya. Vostochno-Evropejskij zhurnal peredovy`x texnologij. 2012;5(4-59):16–21.
63. Lyozin I.A., Kanabeev S.P. Optimizaciya obucheniya nejronny`x setej metodom kombinacii algoritmov obucheniya. Perspektivny`e informacionny`e texnologii: Mezhdunar. nauch.-texn. konf. 2017; S. 14–16.
64. Utkarsh A., Kantha A.S., Praveen J., Kumar J.R. Hybrid GA-PSO trained functional link artificial neural network based channel equalizer. 2nd International Conference on Signal Processing and Integrated Networks. Noida, India. 2015; pp. 285–290. https://doi.org/10.1109/SPIN.2015.7095331
65. Yulkova V.M., Shilovskij G.V. Iskusstvenny`e nejronny`e seti. Glubokoe obuchenie. Monitoring. Nauka i texnologii. 2019;4(42):68–72.
66. Ye X., Yang K. Optimizing neural networks for public opinion trends prediction. 11th International Conference on Natural Computation. Zhangjiajie, China. 2015; pp. 31–36. https://doi.org/10.1109/ICNC.2015.7377961
67. Chiba Z., Abghour N., Moussaid K., Omri A.E., Rida M. A Hybrid Optimization Framework Based on Genetic Algorithm and Simulated Annealing Algorithm to Enhance Performance of Anomaly Network Intrusion Detection System Based on BP Neural Network. International Symposium on Advanced Electrical and Communication Technologies. Rabat, Morocco. 2018; pp. 1–6. https://doi.org/10.1109/ ISAECT.2018.8618804
68. Buly`ga F.S. E`vristicheskie algoritmy` obucheniya svertochny`x nejronny`x setej v ramkax texnologii raspoznavaniya licz. StudNet. 2021;4(5).
69. Dopira R.V. Metod identifikacii texnicheskogo sostoyaniya radiotexnicheskix sredstv s primeneniem texnologij iskusstvenny`x nejronny`x setej. Programmny`e produkty` i sistemy`. 2019;32(4):628–638.
70. Maxotilo K.V., Voronenko D.I. Modifikaciya algoritma Levenberga–Markvardta dlya povy`sheniya tochnosti prognosticheskix modelej svyaznogo potrebleniya e`nergoresursov v by`tu. Vestnik Nacional`nogo texnicheskogo universiteta Xar`kovskij politexnicheskij institut. Ser.: Informatika i modelirovanie. 2005;56:83–90.
71. Parxomenko S.S., Ledenyova T.M. Obuchenie nejronny`x setej metodom Levenberga–Markvardta v usloviyax bol`shogo kolichestva danny`x. Vestnik Voronezhskogo gosudarstvennogo universiteta. Ser.: Sistemny`j analiz i informacionny`e texnologii. 2014;2:98–106.
72. Kryuchin O.V., Arzamascev A.A. Parallel`ny`e algoritmy` obucheniya iskusstvennoj nejronnoj seti QuickProp i RPROP. Vestnik rossijskix universitetov. Matematika. 2012;17(1):175–178.
73. Zapryagaev S.A., Karpushin A.A. Vy`chislenie i obuchenie iskusstvenny`x nejronny`x setej pryamogo rasprostraneniya na graficheskom processore. Vestnik Voronezhskogo gosudarstvennogo universiteta. Ser.: Sistemny`j analiz i informacionny`e texnologii. 2011;1:157–164.
74. Osovskij S. Nejronny`e seti dlya obrabotki informacii. M.: Finansy` i statistika; 2002. 344 s.