Metrological foundation of a cascade quality assessment system for LLM-generated text

D. Teterevenkov

doi:10.18127/j19997493-202605-10

Metrological foundation of a cascade quality assessment system for LLM-generated text

research article UDC 004.9 Pages: 71-76 doi.org/10.18127/j19997493-202605-10

¹ Financial University under the Government of the Russian Federation (Moscow, Russia)

Keywords: Large language models metrological complex cascade assessment errors of the first and second kind validity Bayes' theorem mathematical modeling

Abstract

The rapid development of large language models (LLM) has raised the problem of quantitative evaluation of generated text quality. Existing metrics (BLEU, ROUGE, BERTScore) assess particular aspects of quality but do not provide a holistic metrologically grounded characterization of the evaluation system itself. The question of the error of the evaluation instrument remains open.

Goal – to develop a mathematical model of a cascade quality assessment system for LLM-generated text based on the metrological approach, formally characterizing the errors of evaluation and establishing a link between artificial-intelligence metrics and classical metrological characteristics.

An approach is proposed that treats the evaluation system as a measuring instrument with a metrological complex (errors of the first and second kind, accuracy, reliability, validity, amount of information). Formulas for the transformation of errors through a cascade of modules are derived by analogy with a chain of measuring transducers. A formal correspondence between AI metrics and metrological characteristics is established. A numerical example of a four-module cascade is given.

The proposed apparatus enables the use of automatic LLM quality assessment systems within mission-critical information-analytical systems with formally verifiable accuracy and validity characteristics.

Received: 07.09.2026
Approved after review: 15.09.2026
Accepted: 01.10.2026

How to cite

Тетеревенков Д. Метрологическое обоснование каскадной системы оценки качества генерации текста большими языковыми моделями // Динамика сложных систем - XXI век. 2026. Т. 20. № 5. С. 71−76. DOI: https://doi.org/10.18127/j19997493-202605-10

References

Papineni K., Roukos S., Ward T., Zhu W.-J. BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). Philadelphia. 2002. P. 311–318.
Lin C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Barcelona. 2004. P. 74–81.
Zhang T., Kishore V., Wu F., Weinberger K.Q., Artzi Y. BERTScore: Evaluating Text Generation with BERT. Proceedings of the International Conference on Learning Representations (ICLR). 2020.
Novickij P.V., Zograf I.A. Otsenka pogreshnostey rezul'tatov izmereniy. 2nd ed., rev. and enl. L.: Energoatomizdat. 1991. 304 p. (in Russian).
Prokopchina S.V. Bajesovskie intellektual'nye tehnologii v zadachah modelirovaniya zakona raspredeleniya v usloviyah neopredelennosti: monografiya. M.: Izdatel'skij dom «NAUCHNAYA BIBLIOTEKA». 2020. 292 p. (in Russian).
Bernardo J.M., Smith A.F.M. Bayesian Theory. Chichester: Wiley. 2000. 586 p.
Zheng L., Chiang W.-L., Sheng Y. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS). 2023.
Hanu L. Unitary team. Detoxify. 2020. URL: https://github.com/unitaryai/detoxify (data obrashcheniya: 15.01.2026).
Hoeffding W. Probability Inequalities for Sums of Bounded Random Variables. Journal of the American Statistical Association. 1963. V. 58. № 301. P. 13–30.
Shannon C.E. A Mathematical Theory of Communication. The Bell System Technical Journal. 1948. V. 27. № 3. P. 379–423.
Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I. Language Models are Unsupervised Multitask Learners. OpenAI Blog. 2019.
Reimers N., Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2019. P. 3982–3992.
He P., Liu X., Gao J., Chen W. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. Proceedings of the International Conference on Learning Representations (ICLR). 2021.
Liu Y., Ott M., Goyal N. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. 2019.
Powers D.M.W. Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies. 2011. V. 2. № 1. P. 37–63.

[1] Papineni K., Roukos S., Ward T., Zhu W.-J. BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). Philadelphia. 2002. P. 311–318.

[2] Lin C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Barcelona. 2004. P. 74–81.

[3] Zhang T., Kishore V., Wu F., Weinberger K.Q., Artzi Y. BERTScore: Evaluating Text Generation with BERT. Proceedings of the International Conference on Learning Representations (ICLR). 2020.

[4] Novickij P.V., Zograf I.A. Otsenka pogreshnostey rezul'tatov izmereniy. 2nd ed., rev. and enl. L.: Energoatomizdat. 1991. 304 p. (in Russian).

[5] Prokopchina S.V. Bajesovskie intellektual'nye tehnologii v zadachah modelirovaniya zakona raspredeleniya v usloviyah neopredelennosti: monografiya. M.: Izdatel'skij dom «NAUCHNAYA BIBLIOTEKA». 2020. 292 p. (in Russian).

[6] Bernardo J.M., Smith A.F.M. Bayesian Theory. Chichester: Wiley. 2000. 586 p.

[7] Zheng L., Chiang W.-L., Sheng Y. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS). 2023.

[8] Hanu L. Unitary team. Detoxify. 2020. URL: https://github.com/unitaryai/detoxify (data obrashcheniya: 15.01.2026).

[9] Hoeffding W. Probability Inequalities for Sums of Bounded Random Variables. Journal of the American Statistical Association. 1963. V. 58. № 301. P. 13–30.

[10] Shannon C.E. A Mathematical Theory of Communication. The Bell System Technical Journal. 1948. V. 27. № 3. P. 379–423.

[11] Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I. Language Models are Unsupervised Multitask Learners. OpenAI Blog. 2019.

[12] Reimers N., Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2019. P. 3982–3992.

[13] He P., Liu X., Gao J., Chen W. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. Proceedings of the International Conference on Learning Representations (ICLR). 2021.

[14] Liu Y., Ott M., Goyal N. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. 2019.

[15] Powers D.M.W. Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies. 2011. V. 2. № 1. P. 37–63.

Abstract

How to cite

Related articles