A.V. Solopiy1, R.K. Klassen2

1,2 Kazan National Research Technical University n. a. A. N. Tupolev (KAI) (Kazan, Russia)
1 entityfx@yandex.ru, 2 klassen.rk@gmail.com

This article describes load testing common IoT message brokers under various configurations. The list of message brokers is represented by Mosquitto, EMQX, Active MQ, and Aedes. The NBomber library was chosen as the main tool for load testing. Based on this library, scenarios for stress testing brokers were developed. Each scenario was applied to the corresponding number of connections, message size, and QoS level. A custom client (called "message listener") was developed to handle errors. The collected data included the number of requests processed per second, the latency of sending messages, the number of errors, and the CPU utilization. Based on the obtained data, a normalized broker rating was proposed and applied. The best message broker can be selected by calculating the normalized broker score for each load scenario and merging all the calculated results into a pivot table. The best message broker is selected from the result table.

Solopiy A.V., Klassen R.K. Choosing a message broker for Internet of things based on load testing. Dynamics of complex systems. 2025. V. 19. № 3. P. 44−51. DOI: 10.18127/j19997493-202503-04 (in Russian).

1. Ashton K. et al. That ‘internet of things’ thing. RFID journal. 2009. V. 22. № 7. P. 97–114.
2. Rose K., Eldridge S., Chapin L. The internet of things: An overview. The internet society (ISOC). 2015. V. 80. № 15. P. 1–53.
3. Kaur J., Kaur K. Internet of things: A review on technologies, architecture, challenges, applications, future trends. International Journal of Computer Network and Information Security. 2017. V. 9. № 4. P. 57.
4. Mansour M. et al. Internet of things: a comprehensive overview on protocols, architectures, technologies, simulation tools, and future directions. Energies. 2023. V. 16. № 8. P. 3465.
5. Bender M., Kirdan E., Pahl M.-O., Carle G. Open-Source MQTT Evaluation. 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). Las Vegas. NV. USA. 2021. P. 1–4. DOI: 10.1109/CCNC49032.2021.9369499.
6. Tatarnikova T.M., Palkin I.I. Primenenie paradigmy interneta veshchej v koncepcii «umnogo goroda» // Uchenye zapiski Rossijskogo gosudarstvennogo gidrometeorologicheskogo universiteta. 2019. № 57. S. 187–197. DOI: 10.33933/2074-2762-2019-57-187-197 (in Russian).
7. Fan Wang. Comparison of Open Source MQTT Brokers. 2023. URL: https://www.emqx.com/en/blog/a-comprehensive-comparison-of-open-source-mqtt-brokers-in-2023 (дата обращения: 30.04.2024).
8. Antonielli F. Development and comparison of MQTT distributed algorithms for HiveMQ. 2021.
9. Naik N. Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP. 2017 IEEE international systems engineering symposium (ISSE). IEEE. 2017. P. 1–7.
10. Krylovskij A. MQTT benchmarking tool. URL: https://github.com/krylovsk/mqtt-benchmark (data obrashcheniya: 30.04.2024) (in Russian).
11. Lightweight MQTT Benchmark Tool written in Erlang. URL: https://github.com/emqx/emqtt-bench (дата обращения: 30.04.2024).
12. XMeter. URL: https://www.emqx.com/en/products/xmeter (дата обращения: 30.04.2024).
13. Halili E.H. Apache JMeter. 2008.
14. NBomber overview. URL: https://nbomber.com/docs/getting-started/overview/ (дата обращения: 30.04.2024).
15. Gheorghe-Pop I.D. et al. A performance benchmarking methodology for MQTT broker implementations. 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE. 2020. P. 506–513.
16. Getting Started with MQTT Performance Testing: A Primer on Scenarios and Metrics // emqx.com URL: https://www.emqx.com/en/blog/getting-started-with-mqtt-performance-testing-a-primer-on-scenarios-and-metrics (дата обращения: 11.08.2024).
17. Mishra B., Mishra B., Kertesz A. Stress-testing MQTT brokers: A comparative analysis of performance measurements. Energies. 2021. V. 14. № 18. P. 5817.
18. Standard O. MQTT version 3.1.1. URL: http://docs.oasis-open.org/mqtt/mqtt/v3. 2014. V. 1. P. 29.
19. Lee S. et al. Correlation analysis of MQTT loss and delay according to QoS level. The International Conference on Information Networking 2013 (ICOIN). IEEE. 2013. P. 714–717.
20. Quincozes S., Emilio T., Kazienko J. MQTT protocol: fundamentals, tools and future directions. IEEE Latin America Transactions. 2019. V. 17. № 09. P. 1439–1448.
21. Solopij A.V. EntityFX Mqtt Benchmark, Slushatel' topikov. URL: https://github.com/EntityFX/MqttBenchmark/tree/main/src/EntityFX. MqttBenchmark.Counters (data obrashcheniya: 30.04.2024) (in Russian).
22. Turnbull J. Monitoring with Prometheus. Turnbull Press, 2018.
23. Van Rossum G. et al. Python Programming Language. USENIX annual technical conference. 2007. V. 41. № 1. P. 1–36.
24. Solopij A.V. EntityFX Mqtt Benchmark, Rezul'taty izmereniya proizvoditel'nosti brokerov MQTT. URL: https://github.com/EntityFX/MqttBenchmark/ tree/main/results (data obrashcheniya: 30.04.2024) (in Russian).
25. Solopij A.V. EntityFX Mqtt Benchmark. URL: https://github.com/EntityFX/MqttBenchmark (data obrashcheniya: 30.04.2024).
26. Solopij A.V. EntityFX Mqtt Stand. URL: https://github.com/EntityFX/mqs-stand (data obrashcheniya: 30.04.2024).