Bang Khuc1, A.L. Gelgor2

1, 2 Peter the Great St. Petersburg Polytechnic University (St. Petersburg, Russia) 1 huk.tb@edu.spbstu.ru; 2 agelgor@spbstu.ru

Formulation of the problem. For next-generation mobile communication systems, the application of FBMC/OQAM with OTFS pre-processing (FBMC/OQAM-OTFS) has been considered [1–7]. This waveform combination offers several key advantages by combining OTFS and FBMC/OQAM modulations, including: (i) a relatively low level of out-of-band emissions (OOBE), (ii) improved spectral efficiency due to the absence of a cyclic prefix, and (iii) enhanced robustness against channel impairments in highly dynamic channels. Despite these benefits, a major challenge in the practical deployment of OTFS, as well as FBMC/OQAM-OTFS, lies in channel estimation in the delay–Doppler (DD) domain [1–4]. This difficulty primarily arises from the limited resolution of the DD domain and the presence of inter-path interference (IPI). A variety of DD domain channel estimation algorithms have been proposed in the literature, including threshold-based algorithms [8–12] and two-stage estimation algorithms [13–15]. For threshold-based algorithm, the presence of IPI may lead to inaccurate estimation of path parameters, which in turn leads to degraded BER performance. In two-stage channel estimation algorithms, the first stage is performing preliminary detection of propagation paths using a threshold-based algorithm as in [11, 12], while the second stage aims at refining the path parameters obtained in the initial estimation. At this stage, the channel estimation problem is often formulated as a global optimization problem, which is solved using numerical optimization techniques such as the Quasi-Newton method [13] or the limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithm [14, 15]. However, the channel estimation methods presented in [13, 14] rely on optimization techniques that are inherently designed for local rather than global optimization. As a result, in practical scenarios the optimization process may converge to local extrema, leading to reduced accuracy in channel parameter estimation. In contrast, a genetic algorithm (GA) constitutes an effective class of global optimization methods. Therefore, a two-stage channel estimation algorithm incorporating a GA is expected to enable more effective suppression of IPI, which is a key factor limiting channel estimation accuracy.

Objective. To enhance the robustness of FBMC/OQAM-OTFS signal reception in scenarios with high-dynamic channels by applying a genetic algorithm to refine the parameters of the propagation paths.

Results. A two-stage channel estimation algorithm in the DD domain based on a genetic algorithm is developed for the FBMC/OQAM-OTFS technology. Using simulation modeling in the MATLAB environment with the proposed system model, the bit error rate (BER) performance of FBMC/OQAM-OTFS signals employing the proposed algorithm is investigated under channel conditions ranging from moderately to highly dynamic. In addition, the BER performance is compared for different GA parameter settings. The simulation results demonstrate that, with appropriate parameter selection, the proposed channel estimation algorithm achieves superior BER performance compared to existing algorithms. FBMC/OQAM-OTFS signals employing the proposed channel estimation algorithm exhibit energy losses not exceeding 0.5 dB relative to the perfect channel estimation case for standard multipath channel profiles EPA, EVA, and ETU, as well as under both moderate and high mobility conditions. By contrast, the other benchmark algorithms show significantly higher energy losses in these scenarios.

Practical significance. The obtained results indicate that FBMC/OQAM-OTFS offers advantages over CP-OFDM for 5G NR and beyond mobile communication systems when operating under the specific conditions of highly time-varying channels.

Bang Khuc, Gelgor A.L. A two-stage genetic algorithm-based channel estimation algorit hm for FBMC/OQAM with OTFS preprocessing. Radiotekhnika. 2026. V. 90. № 3. P. 41−52. DOI: https://doi.org/10.18127/j00338486-202603-04 (In Russian)

1. Hammoodi L. Audah. Green coexistence for 5G waveform candidates: A review. IEEE Access. 2019. vol. 7. P. 10103−10126.
2. Hadani R. et al. Orthog onal time frequency space modulation. IE EE Wireless Communications and Networking Conference (WCNC). 2017.
3. Xiao L. et al. An overview of OTFS for internet of things: conc epts, benefits, and challenges. IEEE Internet of Things Journal. 2022. V. 9. № 10. P. 7596−7618.
4. Zhiqiang Wei, et al. Orthogonal time-frequency space modulation: a promising next-generation waveform. IEEE Wireless Communica tions. 2021. V. 28. № 4. P. 136−144.
5. Junior R.P. et al. A two-dimensional FFT precoded filter bank s cheme. IEEE Transactions on Wire less Communications. 2023. V. 2 2. № 11. P. 8366−8377.
6. Khuc et al. Channel estimation and equalization for OTFS system s over high-mobility channels. 2023 International Conference on Electrical Engineering and Photonics (EExPolytech). St. Petersburg. Russian Federation. 2023. P. 205−208.
7. Bhat V.S. et al. Input-output relation and performance of RIS-Aided OTFS with fractional delay−Doppler. IEEE Communications Letters. 2023. V. 27. № 1. P. 337−341.
8. Raviteja P. et al. Embedded pilot-aided channel estimation for OTFS in delay–Doppler channels. IEEE Transactions on Vehicular Technology. 2019. V. 68. № 5. P. 4906−4917.
9. Zhou M. et al. Iterative channel estimation for OTFS systems ba sed on low-PAPR hybrid superimposed pilots. IEEE Communications Letters. 2024. V. 28. № 8. P. 1939−1943. 1
10. Lee H.-G. et al. Single-tone-based channel estimation method fo r OTFS systems. IEEE Transactions on Communications. 2025. V. 7 3. № 3. P. 1638−1651. 1
11. Hashimoto N. et al. Channel estimation and equalization for CP- OFDM-based OTFS in fractional Doppler channels. 2021 IEEE Inter national Conference on Communications Workshops (ICC Workshops). 2021. 1
12. Khan A., Mohammed S.K. A low-complexity OTFS channel estimation m e t h o d f o r f r a c t i o n a l d e l a y - D o p p l e r s c e n a r i o s. IEEEWireless Communications Letters. Sept. 2023. V. 12. № 9. P. 1484−1488. 1
13. He X. et al. A Two-stage channel estimation algorithm for OTFS in fractional Doppler channels. IEEE Communications Letters. 20 23. V. 27. № 7. P. 1839−1843. 1
14. Jia et al. Low complexity doubly fractional OTFS channel estima tion based on L-BFGS method. Pro c. IEEE Wireless Commun. Netw. Conf. (WCNC). Glasgow. U.K. 2023. P. 1–6. 1
15. Yang Y. et al. An OTFS channel estimation scheme based on itera tive path peak search and inter-path interference mitigation. I EEE Wireless Communications Letters. July 2024. V. 13. № 7. P. 1828−1832. 1
16. Seyed Mahmoud Pishvaei et al. Design and performance evaluation of FBMC-based Orthogonal Time–Frequency Space (OTFS) system. Physical Communication. 2022. V. 53. Р. 101723. 1
17. Man K.F. et al. Genetic algorithms: concepts and applications [in engineering design]. IEEE Transactions on Industrial Electro nics. 1996. V. 43. № 5. P. 519−534. 1
18. Hong Y. et al. Delay-Doppler communications: principles and applications. Elsevier. 2022. 1
19. Gel'gor A.L., Gorlov A.I., Nguen V.F. Povyshenie spektral'noj i energeticheskoj effektivnosti signalov SEFDM putem ispol'zovan iya optimal'nyh impul'sov v kachestve formy spektrov podnesushchih. Radiotekhnika. 2018. T. 82. № 1. S. 49−56 (in Russian). 2
20. Gel'gor A.L., Tan N.F.H. Sintez spektral'no-effektivnyh signalo v pri nalichii ogranicheniya v vide spektral'noj maski. Radiote khnika. 2018. T. 82. № 12. S. 49−57. DOI: 10.18127/j00338486-201812-06 (in Russian). 2
21. Makarov S.B., Zav'yalov S.V. Optimizaciya spektral'no-effektivn yh mnogochastotnyh neortogonal'nyh signalov. Radiotekhnika. 201 6. T. 80. № 12. S. 121−133 (in Russian). 2
22. Van N.F., Gel'gor A.L., Tan N.F.H. Preodolenie spektral'noj eff ektivnosti signalov OFDM putem ispol'zovaniya optimal'nyh impul 'sov i primeneniya algoritma demodulyacii M-BCJR. Radiotekhnika. 2018. T. 82. № 12. S. 95−102. DOI: 10.18127/j00338486-201812-12. 2
23. Rashich A.V., Gorbunov S.V., Urvancev A.S. Pomekhoustojchivost' priema SEFDM-signalov v kanale s chastotno-selektiv-nymi relee vskimi zamiraniyami. Radiotekhnika. 2018. T. 82. № 1. S. 57−62 (in Russian).