D.V. Isaev1, V.G. Feklin2, A.A. Kochkarov3

1-3 Financial University under the Government of the Russian Federation (Moscow, Russia)

The article is devoted to the application of neural network architecture to time series. The main object of study was credit default data, characterized by accompanying transactional data.

Neural networks used for sequential data are taken as basic models for comparative analysis. Customer transactions preceding the onset of a credit default were used as a time series.

As a result of the study of the applicability of transformers for credit scoring, the following was revealed:

the application of the time structure in the problem of credit scoring has been little studied, so this direction is new in this area;

the transformer is applicable to a sequence of data, however, it is sensitive to the volume of data, since its complex structure makes it prone to retraining;

the use of a transformer should not be used as the first approach to model training.

Isaev D.V., Feklin V.G., Kochkarov A.A. Transformers and their application in credit scoring. Neurocomputers. 2022. V. 24. № 3. Р. 29-36. DOI: https://doi.org/10.18127/j19998554-202203-03 (in Russian)

1. Sutskever I., Vinyals O., Quoc V.L. Sequence to Sequence Learning with Neural Networks. arXiv. 2014.
2. Filonov P., Lavrentyev A., Vorontsov A. Multivariate Industrial Time Series with Cyber-Attack Simulation: Fault Detection Using an LSTM-based Predictive Data Model. arXiv. 2016.
3. Ashish V., Noam S., Niki P., Jakob U., Llion J., Aidan N. G., Lukasz K. Attention Is All You Need. arXiv. 2017.
4. Neo W., Bradley G., Xue B., Xue B. Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case. arXiv. 2020.
5. Transformers for Time-series Forecasting. Medium URL: https://medium.com/mlearning-ai/transformer-implementation-for-time-series-forecasting-a9db2db5c820 (data obrashhenija 15.02.2022).
6. Timeseries classification with a Transformer model. Keras URL: https://keras.io/examples/timeseries/timeseries_transformer_classification/ (data obrashhenija 24.02.2022).
7. Tianyang L., Yuxin W., Xiangyang L., Xipeng Q. A Survey of Transformers. arXiv. 2021.
8. Neural Machine Translation: Inner Workings, Seq2Seq, and Transformers. towards data science URL: https://towards-datascience.com/neural-machine-translation-inner-workings-seq2seq-and-transformers-229faff5895b (data obrashhenija 27.03.2022).
9. Yi T., Mostafa D., Dara B., Donald M. Efficient Transformers: A Survey. arXiv. 2022.