Stochastic model of a closed metrological chain for quality control of large language model generation

Problem statement. In a closed metrological chain, a text generated by a large language model (LLM) is evaluated by a cascade measuring instrument with certified probabilities of type I and type II errors, after which the text is published, regenerated or replaced by a fallback text. The number of model calls, the share of fallbacks and, above all, the risk of publishing a defective text have so far been described only qualitatively, and the regeneration limit has been set empirically.

Goal. To build a stochastic model of the "generation – cascade evaluation – regeneration" cycle, derive its characteristics in closed form and formulate a stopping rule consistent with the metrological passport of the instrument.

Results. The cycle is formalized as an absorbing Markov chain; the coverage of requests, the expected number of model calls, the distribution of the number of attempts and the residual risk are obtained in closed form. For independent attempts the residual risk equals the complement of the certified validity and does not depend on the number of regenerations, so the problem of the attempt limit degenerates. For dependent attempts (a mixture of "easy" and "hard" requests) the validity of a text accepted at the j-th attempt decreases monotonically in j and the tail of the number of attempts becomes heavier, whereas feedback through the diagnostic package raises the validity of late attempts. A stopping rule combining an economic criterion with a validity norm is formulated; in the monotone case a threshold policy with a finite number of attempts is optimal. A numerical example based on the certified characteristics of the Metrolog AI 1.00 complex is confirmed by Monte Carlo simulation.

Practical significance. The results extend the certification procedure to closed chains: the passport is supplemented with the acceptance probability, the residual risk and a justified regeneration limit, and the validity of each published text is recomputed taking the attempt number into account.