D. I. Ozornin - Post-graduate Student, Yandex LLC, Moscow. E-mail: denioz.25@gmail.com

Statistical language models based on recurrent neural networks are one of the most popular methods of language representation nowadays. However, training the model is a very time-consuming process, in spite of all of the advances in computational power. One of the bottlenecks is the size of the output layer which is as large as the vocabulary. In this paper we are suggesting a new technique based on Huffman codes to decrease training time. It does so by training smartly chosen part of the weights between hidden and output layers so that overall quality will not suffer much. To ensure that quality stays on the same level we suggest using a momentum method for optimizing recurrent neural network. It allows us to tune the model more precisely and use less number of epochs to achieve convergence, which also speeds up the time used for training. Momentum method is a powerful optimization method of stochastic gradient decent, which is based on accumulating the error gradient from one training example to another. Then this accumulated gradient is used for the weight tuning during iterations with the current error gradient. This allows the model to converge more quickly and go through less epochs of training. In conclusion we analyse the experiments with this two methods on two corpuses of different sizes: English and Russian. We are able to show that technique for speed increase combined with momentum method can produce an insignificant quality decrease with a good increase in speed of training. There are a lot of other means and methods to optimize this types of models, which will be investigated in future papers.