A.I. Vlasov1, A.N. Orekhov2, L.V. Tetik3

1 Bauman Moscow State Technical University (Moscow, Russia)

2 Foundation for Assistance to the Creation and Implementation of Computer Psyche (Moscow, Russia)

3 Moscow State University M.V. Lomonosov (Moscow, Russia)

1 vlasovai@bmstu.ru, 2 alexandernorekhov@yandex.ru, 3 liliatetik@rambler.ru

Problem setting. The work considers the methods and results of information tests of the AlNikOr computer system, the world's first computer personality that solves non-standard problems. The ability of the system to solve problems in the context of the receipt of redundant information, as well as interfering information, was checked. It is known that messages containing interfering information create conditions for the appearance of errors, and in particular cognitive distortion. Such errors prevent the solution of the problem, or significantly, often exponentially, increase the time of the solution. In artificial intelligence systems based on large language models such as GPT-4 ("multimodal large language model"), interfering information leads to errors when searching for an answer in standard tasks. Effective and universal methods for assessing the behavior of a computer system in the event of excessive or interfering information are received have not been developed.

Target. Development of a method for information testing of a computer personality, assessment, with its use, of the computer system's ability to recover from excessive or interfering information and testing the hypothesis of a polynomial increase in the time of solving the problem depending on the number of meaningful words, as well as the meanings generated by the system.

Results. The developed method of information tests of a computer personality makes it possible to determine its characteristics in conditions of external, input to the system, and internal, generated by the system itself, destructive with respect to the process of solving a non-standard problem. The developed technique allows you to interpret the signs of gross errors (hallucinations) when learning, and using large language models. This will speed up and reduce the cost of developing computer systems based on large language models, as well as hybrid artificial intelligence systems. Demonstrating the capabilities of computer personality algorithms to effectively recover from redundant and interfering information will enable developers of hybrid systems to use these algorithms as components of the upper level of behavior regulation in solving problems.

Practical significance. The developed method of information testing of a computer personality makes it possible to determine its characteristics in conditions of external, input to the system, and internal, generated by the system itself, destructive regarding the process of solving a non-standard problem. The use of the developed methodology allows us to more reasonably interpret the signs of possible gross errors (hallucinations) both in the learning process and in the process of using large language models. This will speed up and reduce the cost of developing computer systems based on large language models, as well as hybrid artificial intelligence systems. Demonstrating the capabilities of computer personality algorithms to effectively recover from redundant and interfering information will enable developers of hybrid systems to use these algorithms as necessary components of the upper level of behavior regulation in solving problems.

Vlasov A.I., Orekhov A.N., Tetik L.V. Information tests of a computer personality solving a non-standard physical problem. Neurocomputers. 2024. V. 26. № 1. Р. 32-44. DOI: https://doi.org/10.18127/j19998554-202401-04 (In Russian)

1. Orekhov A.N. Solving non-standard problems by a computer system. Neurocomputers: development, application. 2021. V. 23. № 3. Р. 44−63. DOI 10.18127/ j19998554-202103-05. (in Russian)
2. Kozharinov A.S., Kirichenko Yu.A., Afanasyev I.V., Labuz N.P., Vlasov A.I. Methods of analysis of cognitive distortions and the concept of an automated intelligent system for their detection. Neurocomputers: development, application. 2022. V. 24. № 4. Р. 38–74. DOI 10.18127/j19998554-202204-04. (in Russian)
3. Abdulaev B.K., Vlasov A.I., Fatkhutdinov T.M. Automated neural network system for analyzing the emotional state of a per-son-operator. Neurocomputers: development, application. 2023. V. 25. № 2. Р. 41–57. DOI 10.18127/j19998554-202302-04. (In Russian)
4. Vlasov A.I., Larionov I.T., Orekhov A.N., Tetik L.V. System of automatic analysis of methods for recognizing the emotional state of a person. Neurocomputers: development, application. 2021. V. 23. № 5. P. 33–50. DOI 10.18127/j19998554-202105-03. (In Russian)
5. Peregudov F.I., Tarasenko F.P. Introduction to system analysis: a textbook for universities. M.: Higher School. 1989. (in Russian)
6. Schrittwieser J., Antonoglou I., Hubert T., Simonyan K., Sifre L., Schmitt S., Guez A., Lockhart E., Hassabis D., Graepel T., Lillicrap T., Silver D. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature. 2020. V. 588. № 7839. P. 604–609. DOI 10.1038/s41586-020-03051-4.
7. Glushko A.A., Busov V.D., Perederin K.D. Methods of algorithmic design of artificial intelligence. Technologies of engineering and information systems. 2019. № 2. P. 72–88. (in Russian)
8. Vlasov A.I., Konkova A.F. Medical diagnostic expert systems for assessing the adequacy of the adaptive response of the body to the effects of extreme factors. Conversion. 1995. № 9-10. P. 18–21. (in Russian)
9. Vlasov A.I., Orekhov A.N., Tetik L.V. Probabilistic and semantic tests of a computer system solving non-standard tasks. Neurocomputers: development, application. 2023. V. 25. № 1. P. 31–45. DOI 10.18127/j19998554-202301-01. (in Russian)
10. Demin A.A., Karpunin A.A., Ganeev Yu.M. Methods of verification and validation of complex software systems. Software products and systems. 2014. № 4. P. 229–233. (in Russian)
11. MI 2891-2004 GSI. General requirements for the software of measuring instruments. (in Russian)
12. Sozinov A.A. The effect of interference and the reorganization of memory in learning: dis. ... cand. psychology. sciences. M.: IP RAS. 2008. 191 p. (in Russian)
13. Tomlinson T.D., Huber D.E., Rieth C.A., Davelaar E.J. An interference account of cue-independent forgetting in the no-think paradigm. Proceedings of the National Academy of Sciences. 2009. V. 106. № 37. P. 15588–15593. DOI 10.1073/pnas.0813370106.
14. Averyanikhin A.E., Vlasov A.I., Malevany A.Yu. The concept of test environment management in conditions of synchronous design technologies. Proceedings of the international symposium "Reliability and Quality". 2019. V. 2. P. 135–142. (in Russian)
15. Kolesnikov D.V., Petrov A.Yu., Khramov V.Yu. Methodology for assessing the security of special software during testing of automated systems. Bulletin of the Voronezh State University. Series: System analysis and Information Technology. 2010. № 1. P. 74–79. (in Russian)
16. Fedukhin A.V., Mukha A.A., Cespedes Garcia N.V. Proof of computer system security. Mathematical machines and systems. 2016. № 3. P. 93–101. (in Russian)
17. Cognitive psychology: history and modernity. A textbook. Edited by M. Falikman and V. Spiridonov. M.: Lomonosov. 2011. 384 p. ISBN 978-5-91678-008-6. (in Russian)
18. Benson B. Cognitive bias cheet sheet. [Electronic resource] – Access mode: https://cs.brown.edu/courses/cs180/sour-ces/2016_01_06_Medium_CognitiveBiasCheatSheet_BetterHumans.pdf, date of reference 05.12.2021.
19. Ilyin E.P. Psychology of creativity, creativity, giftedness. St. Petersburg: Peter. 2009. 444 p. ISBN 978-5-49807-239-5. (in Russian)
20. Zefirov T.L., Ziyatdinova N.I., Kuptsova A.M. Physiological foundations of memory. Memory development in children and adolescents. Kazan: KFU. 2015. 40 p. (in Russian)
21. ISO 14598-1-6: 1998-2000. Evaluation of the software product. Part 1. Overview. Part 2. Planning and management. Part 3. Processes for developers. Part 4. Processes for buyers. Part 5. Processes for appraisers. Part 6. Documentation and evaluation of modules.
22. Leffingwell D., Widrig D. Principles of working with software requirements. Translated from English by N.A. Orekhova. M.: Williams. 2002. 445 p. (in Russian)
23. Hutcheson L. Software Testing Fundamentals: Methods and Metrics by Marnie. John Wiley & Sons. 2003. 408 p.
24. Kan S.H. Metrics and Models in Software Quality Engineering. 2nd Ed. Addison Wesley Professional. 2002. 560 p.
25. Crispin L., Gregory J. Flexible testing: a practical guide for software testers and flexible teams. M.: Williams. 2010. 464 p. (in Russian)
26. Lipaev V.V. To the development of models of a dynamic external environment for testing complex software products. Program engineering. 2017. V. 8. № 2. P. 51–57. (in Russian)