A.N. Timofeev1, S.S. Mikhaylova2

1 East Siberian State University of Technology and Management (Ulan-Ude, Russia)

2 Financial University under the Government of the Russian Federation (Moscow, Russia)

1 89021632777@mail.ru, 2 ssmihajlova@fa.ru

Problem. The problems that arise when using large language models in code generation problems and existing methods of solving them are investigated.

Target. Improve the quality of program code generation

Results. An approach to quality improvement based on verification of generation results using a model that includes ontologies and knowledge bases is proposed.

Practical significance. Among the possible approaches to the application of ontologies and knowledge bases in code generation tasks, the following can be distinguished: checking the code for possible errors, generating explanations, generating hints, preparing tasks, evaluating results. The proposed approach is aimed at verifying and enriching the semantics of intermediate or final results of the large language model (LLM), as well as to improve the quality of manually written code.

Timofeev A.N., Mikhaylova S.S. Approach to improving the quality of program code generation by large language models. Neurocomputers. 2024. V. 26. № 4. Р. 18-26. DOI: https://doi.org/10.18127/j19998554-202404-02 (In Russian)

1. Goldman Sachs allowed the full automation of 300 million jobs thanks to ChatGPT. [Electronic resource] – Access mode: https://www.forbes.ru/tekhnologii/486786-goldman-sachs-dopustil-polnuu-avtomatizaciu-300-mln-rabocih-mest-blagodara-chatgpt, date of reference 20.01.2024. (In Russian)
2. Mashkoor A., Menzies T., Egyed A., Ramler R. Artificial Intelligence and Software Engineering: Are We Ready? Computer. 2022. V. 55. № 3. P. 24–28. DOI 10.1109/MC.2022.3144805.
3. Bull C., Kharrufa A. Generative Artificial Intelligence Assistants in Software Development Education: A Vision for Integrating Generative Artificial Intelligence Into Educational Practice, Not Instinctively Defending Against It. IEEE Software. 2024. V. 41. № 2. P. 52–59. DOI 10.1109/MS.2023.3300574.
4. Dunder N., Lundborg S., Wong J., Viberg O. Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence. [Electronic resource] – Access mode: https://arxiv.org/pdf/2312.01109, date of reference 20.01.2024.
5. Ivanova G.S., Martynyuk P.A. Analysis of neural network language models for solving problems of text data processing. Neurocomputers. 2023. V. 25. № 2. Р. 5–20. DOI 10.18127/j19998554-202302-01 (In Russian)
6. Oh S., Lee K., Park S., Kim D., Kim H. Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models. [Electronic resource] – Access mode: https://arxiv.org/pdf/2312.06227, date of reference 20.01.2024.
7. Generative AI for Coding. [Electronic resource] – Access mode: https://toloka.ai/blog/generative-ai-for-coding/, date of reference 22.01.2024.
8. Cámara J., Troya J., Burgueño L., Vallecillo A. On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML. Software and Systems Modeling. 2023. V. 22. № 3. P. 781–793. DOI 10.1007/s10270-023-01105-5.
9. AI code-generation software: What it is and how it works. [Electronic resource] – Access mode: https://www.ibm.com/blog/ai-code-generation/, date of reference 22.01.2024.
10. How I use Giga Chat for Frontend development. [Electronic resource] – Access mode: https://habr.com/ru/arti-cles/787866 /, date of reference 22.01.2024. (In Russian)
11. Romera-Paredes B., Barekatain M., Novikov A., Balog M., Kumar M.P., Dupont E., Ruiz F.J.R., Ellenberg J.S., Wang P., Fawzi O., Kohli P., Fawzi A. Mathematical discoveries from program search with large language models. Nature. 2024. V. 625. № 7995. P. 468–475. DOI 10.1038/s41586-023-06924-6.
12. Finnie-Ansley J., Denny P., Becker B. A., Luxton-Reilly A., Prather J. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. Australasian Computing Education Conference. 2022. P. 10–19. DOI 10.1145/3511861.3511863.
13. Denny P., Kumar V., Giacaman N. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. [Electronic resource] – Access mode: https://arxiv.org/pdf/2210.15157, date of reference 22.01.2024.
14. Denny P., Leinonen J., Prather J., Luxton-Reilly A., Amarouche T., Becker B.A., Reeves B.N. Prompt Problems: A New Programming Exercise for the Generative AI Era. Proceedings of the 55th ACM Technical Symposium on Computer Science Education. 2023. V. 1. P. 296–302. DOI 10.1145/3626252.3630909.
15. Kiesler N., Schiffner D. Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments. [Electronic resource] – Access mode: https://arxiv.org/pdf/2308.08572, date of reference 22.01.2024.
16. Savelka J., Agarwal A., Bogart C., Song Y., Sakr M. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses? [Electronic resource] – Access mode: https://arxiv.org/pdf/2303.09325, date of reference 22.01.2024.
17. Hassany M. Ke J., Brusilovsky P., Lekshmi Narayanan A.B., Akhuseyinoglu K. Authoring Worked Examples for Java Programming with Human-AI Collaboration. Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing. 2024. P. 101–103. DOI 10.1145/3605098.3636160.
18. Unleashing developer productivity with generative AI. [Electronic resource] – Access mode: https://www.mckinsey.com/capa-biliies/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai, date of reference 25.01.2024.
19. Heyman G., Huysegems R., Justen P., Van Cutsem T. Natural language-guided programming. [Electronic resource] – Access mode: https://arxiv.org/pdf/2108.05198, date of reference 25.01.2024.
20. Jiang E., Toh E., Molina A., Olson K., Kayacik C., Donsbach A., Cai C.J., Terry M. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. Proceedings of the CHI Conference on Human Factors in Computing Systems. 2022. № 386. P. 1–19. DOI 10.1145/3491102.3501870.
21. Vaithilingam P., Zhang T., Glassman E.L. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 2022. № 332. P. 1–7. DOI 10.1145/3491101.3519665.
22. Cipriano B.P., Alves P. GPT-3 vs Object Oriented Programming Assignments: An Experience Report. Proceedings of the Conference on Innovation and Technology in Computer Science Education. 2023. V. 1. P. 61–67. DOI 10.1145/3587102.3588814.
23. Kazemitabaar M., Chow J., Ma C.K.T., Ericson B.J., Weintrop D., Grossman T. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. Proceedings of the CHI Conference on Human Factors in Computing Systems. 2023. № 455. P. 1–23. DOI 10.1145/3544548.3580919.
24. Timofeev A.N., Evdokimova I.S., Khaptakhaeva N.B., Cenotrusova A.A. Semantic modeling of the software development domain: tools, methods, knowledge (Part 1). Modern science: actual problems of theory and practice. Series: Natural and Technical Sciences. 2022. № 12. P. 85–91. DOI 10.37882/2223-2966.2022.12.31. (In Russian)
25. Timofeev A.N., Evdokimova I.S., Khaptakhaeva N.B. Semantic modeling of the software development domain: tools, methods, knowledge (Part 2). Modern science: actual problems of theory and practice. Series: Natural and Technical Sciences. 2023. № 4-2. P. 122–129. DOI 10.37882/2223-2966.2023.04-2.30. (In Russian)
26. Zhuge M., Liu H., Faccio F., Ashley D.R., Csordás R., Gopalakrishnan A., Hamdi A., Al Kader Hammoud H.A., Herrmann V., Irie K., Kirsch L., Li B., Li G., Liu S., Mai J., Piękos P., Ramesh A., Schlag I.l, Shi W., Stanić A., Wang W., Wang Y., Xu M., Fan D.-P., Ghanem B., Schmidhuber J. Mindstorms in Natural Language-Based Societies of Mind. [Electronic resource] – Access mode: https://arxiv.org/pdf/2305.17066, date of reference 25.01.2024.
27. Qian C., Dang Y., Li J., Liu W., Xie Z., Wang Y., Chen W., Yang C., Cong X., Che X., Liu Z., Sun M. Experiential Co-Learning of Software-Developing Agents. [Electronic resource] – Access mode: https://arxiv.org/pdf/2312.17025, date of reference 25.01.2024.
28. Newell A. The Knowledge Level. Artificial Intelligence. 1982. V. 18. № 1. P. 87–127. DOI 10.1016/0004-3702(82)90012-1.
29. Timofeev A. Development of a Competence Model in the Field of Information Technology Using Ontologies and Knowledge Bases. 16th International Conference Management of large-scale system development (MLSD). 2023. P. 1–4. DOI 10.1109/MLSD58227. 2023.10303922.
30. Timofeev A.N. Development of a competence model in the field of information technology based on the integration of ontologies and knowledge bases. Modern science: actual problems of theory and practice. Series: Natural and Technical Sciences. 2023. № 6-2. P. 153–159. DOI 10.37882/2223-2982.2023.6-2.37. (In Russian)