A.I. Vlasov1, A.N. Orekhov2, L.V. Tetik3

1 Bauman Moscow State Technical University (Moscow, Russia)
2 Foundation for Assistance to the Creation and Implementation of Computer Psyche (Moscow, Russia)
3 Moscow State University M.V. Lomonosov (Moscow, Russia)
 

Problem setting. The work considers the methods and results of probabilistic and semantic tests of the AlNikOr computer system, the world's first computer personality that solves non-standard problems. The problem of this kind of testing has two aspects. Firstly, since the algorithms of AlNikOr implement nomothetic models of the human psyche, and nomothetic models of the psyche contain in a reduced form mathematical maps not only of cognitive processes, but also of personal characteristics, are probabilistic in nature and become dependent on the semantic characteristics of words describing the problem. Therefore, any external interference, in particular changing the patterns of words or their semantic characteristics, can make the behavior of the system inadequate. Secondly, methods used to test the characteristics of expert systems, large linguistic models, deep learning algorithms, etc. for systems based on nomothetic models of the psyche do not give reliably interpreted results. The improvement of large linguistic models, deep learning algorithms, especially the widespread use of positive and negative reinforcement in them makes the problem of interpreting results more and more significant.

Target. Development and testing of methods for checking probabilistic and semantic characteristics of computer systems based on nomothetic models of the psyche.

Results. The proposed methods, important fragments of which are partial blocking of semantic memory, as well as replacement of words with their semantic equivalents, made it possible to obtain adequate and interpreted results in probabilistic and semantic tests. Probabilistic tests confirmed that the error rate is evenly distributed across the tests and the most likely sources of error are identified. Semantic trials were conducted by replacing words with their semantic equivalents at different levels of cognitive complexity. It has been shown that at a low level of cognitive complexity, replacing even one word with a sense equivalent leads to a sharp increase in the error rate, while at an average level of cognitive complexity, replacing even several words with sense equivalents does not affect the error rate.

Practical significance. The developed methods make it possible to determine significant probabilistic and semantic characteristics of computer systems based on nomothetic models of the psyche. The use of the developed methods makes it possible to more reasonably interpret the results of the work of advanced artificial intelligence systems using negative and positive reinforcement. The characteristics of the AlNikOr system, confirmed during probabilistic and semantic tests, make it possible to create practically useful systems on its basis in different fields. The characteristics of algorithms confirmed in this work based on nomothetic models make it possible to create on their basis more advanced algorithms of systems based on rules, large linguistic models, deep learning, in particular using positive and negative reinforcement, and so on.

Vlasov A.I., Orekhov A.N., Tetik L.V. Probabilistic and semantic tests of a computer system solving non-standard tasks. Neuro­computers: development, application. 2023. T. 25. № 1. С.31–45. DOI: https://doi.org/10.18127/j19998554-202301-01

1. Orekhov A.N. Solving non-standard problems with a computer system. Neurocomputers: development, application. 2021. V. 23 № 3.
   P. 43–62. DOI 10.18127/j19998554-202103-05 (in Russian).
2. Tom B. Brown et al. Language Models are Few-Shot Learners – Johns Hopkins University, Open AI. 2020. 75 p.
3. Bazhin V.A. Fine-tuning BERT and GPT-3 to solve the problem of generating Russian-language news. Current scientific research in the modern world. 2021. № 5-2 (73). P. 43–58 (in Russian).
4. Dulnev V.V. Analysis of the use of neural networks to generate program code using the example of GPT-3. Innovations. Science. Education. 2021. № 35. P. 408–411 (in Russian).
5. Dotsenko S.A. Reflection of the problems of inequality and communication of modern society in generative language neural networks using the example of GPT-3. Philosophy and culture of the information society. Abstracts of the reports of the Ninth International Scientific and Practical Conference. St. Petersburg, 2021. P. 43–45 (in Russian).
6. Schrittwieser J., Antonoglou I., Hubert T. et al. Mastering Atari, Go, chess and shogi by planing with a learned model. Nature. 2020. 588. P. 604–609. https://doi.org/10.1038/s41586-020-03051-4.
7. Kozharinov A.S., Kirichenko Yu.A., Afanasyev I.V., Vlasov A.I., Labuz N.P. Methods Analysis of cognitive searches and concepts of automation of intelligent systems of their detective analysis. Neurocomputers: development, application. 2022. V. 24. №4. P. 38–74. DOI 10.18127/j19998554-202204-04 (in Russian).
8. Peregudov F.I., Tarasenko F.P. Introduction to systematic analysis: ucheb. a manual of universities. M.: Vyysh. School, 1989 (in Russian).
9. Glushko A.A., Busov V.D., Peredelerin K.D. Methods of algorithmic design of artificial intelligence. Engineering and information system technologies. 2019. № 2. P. 72–88 (in Russian).
10. Peshkov D.V., Minitaeva A.M. Neural network for determining the precaution of recording. Engineering and information system technologies. 2019. № 4. P. 10–19 (in Russian).
11. Vlasov A.I., Konkova A.F. Medical and diagnostic expert systems for assessing the adequacy of the body's adaptive response to the effects of extreme factors. Conversion. 1995. № 9-10. P. 18–21 (in Russian).
12. Vasilova E.V., Vlasov A.I., Evdokimov G.M. Non-verbal communications of the animal world: systematic analysis of sign languages. International Research Journal. 2017. № 5-3(59). P. 14–23 (in Russian).
13. Vasilova E.V., Vlasov A.I., Evdokimov G.M. Non-verbal communications of the animal world: mapping elements of sign languages. International Research Journal. 2017. № 6-3(60). P. 102–110 (in Russian).
14. Vlasov A.I., Larionov I.T., Orekhov A.N., Tetik L.V. System of automatic analysis of methods for recognizing the emotional state of a person. Neurocomputers: development, application. 2021. V. 23. № 5. P. 33–50. DOI 10.18127/j19998554-202105-03 (in Russian).
15. Alfimtsev A.N., Sakulin S.A., Bolshakov V.E., Bykov N.V., Tovarnov M.S., Vlasova N.S. Method of solving social dilemmas based on reputation and deep multi-agent training with reinforcement. Neurocomputers: development, application. 2022. V. 24. № 1. P. 5–15. DOI 10.18127/j19998554-202201-01 (in Russian).
16. Bolshakov V.E. Overview of internal motivation methods for machine learning with reinforcement. Fundamental and applied research of young scientists. Collection of materials of the V International Scientific and Practical Conference of students, graduate students and young scientists. Omsk, 2021. P. 305–311 (in Russian).
17. Orekhov A.N. Modeling of mental and socio-psychological processes: nomothetic approach: Autoref. diss. ... Doc. psychol. sciences. M.: 2006 (in Russian).
18. Orekhov A.N. Computer personality: advantages and disadvantages. Tr. XV All-Russian. scientific. conf. «Neurocomputers and their applications». 2017. P. 125–126 (in Russian).
19. Kosheleva L.Yu. Nomothetic and idiographic methods in the science of criminal law. Education. Science. Scientific personnel. 2011. № 2.
    P. 51–56 (in Russian).
20. Kalashnikov S.G. Nomothetic types of explanation in historical science. In the collection: Discourse. philosophical reflections. Collection of scientific articles by teachers of the Department of Philosophy, History, Political Science and Law. Kolomna, 2006. P. 8–25 (in Russian).
21. Chuchupal V.V. Mathematical methods and models in social sciences. Actual directions of scientific research of the XXI century: theory and practice. 2015. V. 3. № 5-2 (16-2). P. 343–346 (in Russian).
22. Momjyan K.H. Nomothetic cognition in the social and human sciences. Epistemology and philosophy of science. 2015. V. 45. № 3.
    P. 16–22 (in Russian).
23. Budnik P.V., Shegelman I.R. On the issue of classification of heuristic methods for finding new solutions in engineering activities. Engineering Bulletin of the Don. 2017. № 4 (47). P. 61 (in Russian).
24. Noss I.N. Methods of interpretation of empirical data in psychological research, refraction of interpretation and their control. Human capital. 2014. № 7 (67). P. 16–22 (in Russian).
25. Burkov V.N., Novikov D.A. Theory of active systems: state and prospects. M.: Sinteg, 1999. – 128 p. (in Russian).
26. Burkov V.N., Korgin N.A., Novikov D.A. Introduction to the theory of management of organizational systems: Textbook. M.: Librocom, 2009 (in Russian).
27. Burkov V.N., Kondratyev V.V. Mechanisms of functioning of organizational systems. M.: Science, 1981.
28. Lebedev S.A. Scientific knowledge: the demarcation problem. European Journal of Philosophical Research. 2016. № 1 (5). P. 27–34.
29. Shakhnov V.A., Kurnosenko A.E., Demin A.A., Vlasov A.I. Industry 4.0 visual tools for digital twin system design. Software Engineering Perspectives in Intelligent Systems. Proceedings of 4th Computational Methods in Systems and Software. 2020. V.2. P. 864-875.
30. Bortakur T, Gogoi B. Research of some tests for homogeneity and their effectiveness. OSR Journal of Mathematics. 2017. V. 13,
    Is. 1 Ver. III. P. 6-19.
31. Cook S. Complexity of theorem proof procedures. Proceedings of the Third Annual ACM Symposium on Computational Theory. 1971.
    P. 151–158.