A.V. Sayapin1

1 Federal State Budgetary Educational Institution of Higher Education “Ogarev Mordovian State University” (Saransk, Russian)
1 mrzxfy@gmail.com

This study aims to evaluate the applicability of specialized Large Language Models (LLMs) within the financial domain and analyze their effectiveness in predicting sentiment based on stock price changes following news publication.

Historical data for Russian stocks was collected via the Moscow Exchange API covering the period from 2011 to 2024, alongside news data from Lenta.ru. A comparative US dataset was sourced from Reddit WorldNews. Sentiment labeling was algorithmically derived from stock price dynamics rather than subjective expert opinion. For Russian assets, a threshold-based approach was utilized where sentiment was determined if price changes exceeded 2%, while US assets were labeled based on closing price changes. To accommodate English-centric LLMs, Russian news texts were automatically translated using the OPUS-MT model.

Several architectures were compared, including FinBERT, FinBERT-tone, Llama-Open-Finance-8B, and fine-tuned variations, alongside a stacked ensemble incorporating Logistic Regression, Random Forest, and LightGBM. Model performance was assessed using Accuracy, Precision, Recall, F1-score, and ROC-AUC, with SHAP values employed for interpretability. Results indicate that models trained on manual expert labeling do not predict price dynamics with high accuracy. On the Russian dataset, original FinBERT outperformed fine-tuned versions, suggesting overfitting risks with limited data. US dataset yielded higher accuracy (up to 69.45% for the ensemble), likely due to larger data volume and the use of headlines reducing noise. SHAP analysis confirmed that models identify semantically significant words but fail to capture market expectations fully.

The analysis confirms that textual information alone is insufficient for high-precision prediction of stock price dynamics, as market reactions depend on expectations beyond textual tone. Consequently, future improvements require integrating quantitative features such as financial indicators, macroeconomic context, and trading volumes into predictive architectures. The development of such systems offers significant practical value, potentially providing alternative signals for stop-loss and take-profit orders. While current limitations exist regarding text-only inputs, further research into integrating textual and quantitative data remains a promising direction for enhancing predictive analytics in the securities market.

Sayapin A.V. Assessing large language models for financial news sentiment analysis // Nonlinear World. 2026. V. 24. № 3.
P. 14–20. DOI: https:// doi.org/10.18127/ j20700970-202603-02

1. Vaswani A. et al. Attention is all you need // Advances in neural information processing systems. 2017. Т. 30.
2. Araci D. Finbert: Financial sentiment analysis with pre-trained language models // arXiv preprint arXiv:1908.10063. 2019.
3. Malo P. et al. Good debt or bad debt: Detecting semantic orientations in economic texts // Journal of the Association for Information Science and Technology. 2014. V. 65. № 4. P. 782–796.
4. Kang J. W., Choi S. Y. Comparative investigation of GPT and FinBERT’s sentiment analysis performance in news across different sectors // Electronics. 2025. V. 14. № 6. P. 1090.
5. Muhammad I., Rospocher M. On assessing the performance of Llms for target-level sentiment analysis in financial news headlines // Algorithms. 2025. V. 18. № 1. P. 46.
6. Wu S. et al. Bloomberggpt: A large language model for finance // arXiv preprint arXiv:2303.17564. 2023.
7. URL: https://huggingface.co/DragonLLM/Llama-Open-Finance-8B (data obrashchenija: 16.02.2026)
8. Correia F., Madureira A. M., Bernardino J. Deep Neural Networks applied to stock market sentiment analysis // Sensors. 2022. V. 22. № 12. P. 4409.
9. Teplova T.V. i dr. Sentiment chastnyh investorov v ob"yasnenii razlichij v birzhevyh harakteristikah akcij rossijskogo rynka // Zhurnal Novoj ekonomicheskoj associacii. 2022. T. 1. № 53. S. 53–84 (In Russian).
10. Hvatkov V.I. Sovremennye tekhnologii analiza tonal'nosti teksta kak instrumentarij povedencheskih finansov // Rossijskij ekonomicheskij internet-zhurnal. 2024. № 2. S. 1 (In Russian).
11. Gavrilov V.S. Razrabotka programmnogo kompleksa i modeli mashinnogo obucheniya dlya avtomaticheskogo analiza novostnyh potokov v finansovoj otrasli // Nelinejnyj mir. 2025. T. 23. № 3. S. 6–14. DOI: https://doi.org/10.18127/ j20700970-202503-02 (In Russian).
12. Tiedemann J., Thottingal S. OPUS-MT–Building open translation services for the World // Annual Conference of the European Association for Machine Translation. European Association for Machine Translation. 2020. P. 479–480.