A.L. Gorbunov – Ph.D.(Eng.), Professor, Adviser of Rector Office, 

Moscow State Technical University of Civil Aviation

E-mail: a.gorbunov@mstuca.aero

E.E. Nechaev – Dr.Sc.(Eng.), Professor, Head of Department of Air Traffic Control,  Moscow State Technical University of Civil Aviation

E-mail: e.nechaev@mstuca.aero

The combination of the following tendencies in today’s air traffic control (ATC): a) a growth of flight intensity in airline hubs, b) the development of area navigation methods, c) the implementation of continuous descent/climb trajectories and d) an increase of the computational capabilities of control systems, will inevitably lead to automatic ATC in the near future: the sequences of take-offs and landings, arbitrarily laid continuous descent/climb trajectories; changes of sequence and flight paths will be set without human participation. However, there is no doubt that a human controller will retain the function of visual monitoring for air traffic in order to control the actions of the automatic control system and pilots. In any method of flight control, a human pilot will retain a similar function of visually monitoring the flight path.

The currently used display devices of the ATC systems, providing 2D or 3D image on mono flat screens, are totally inadequate to the task of monitoring arbitrarily laid continuously variable trajectories that can dynamically change. The traditional 2D route plan on the flat screen does not allow one to control the continuous change of altitude and the height intersection of flight trajectories; a 3D-image on a flat screen can mislead the observer about the geometry of the image elements and has a limited field of view, which complicates orientation while observing the dynamically changing routing. The same considerations apply to air surveillance devices for pilots.

The solution is the observation of air traffic in virtual and augmented reality (VR/AR) by means of autonomous wearable devices. Augmented reality technology is a derivative form of virtual reality. AR retains all of the possibilities of VR, but beyond this, it has considerable advantages that stem from the parallel presence of virtual and real objects in one scene. The VR/AR use in ATC domain embraces applications for the controllers working at control towers in airports (linking of real aircraft and their flight parameters for a situation awareness improvement, x-ray vision in low visibility conditions); remote towers systems (video panorama solutions and 3D modelling); training systems for air traffic controllers (safe modelling of dangerous situations in the real environment). The works published last 15 years are presented in the review.

1. Azuma R., Daily M. Advanced human-computer interfaces for air traffic management and simulation. Proceedings of the 1996 AIAA Flight Simulation Technologies Conference. San Diego, CA (USA). 29−31 July 1996.
2. Bagassi S., De Crescenzio F., Lucchi F., Masotti N. Augmented and Virtual Reality in the Airport Control Tower. 30th Congress of the International Council of the Aeronautical Sciences. 25−30 September 2016. Daejeon (South Korea). http://www.icas.org/ICAS_ARCHIVE/ICAS2016/data/papers/2016_0719_paper.pdf. Poslednee poseshchenie 16.12.19.
3. Bagassi S., De Crescenzio F., Piastra S. Augmented reality technology selection based on integrated QFD-AHP model. International Journal on Interactive Design and Manufacturing (IJIDeM). 2019. https://doi.org/10.1007/s12008-019-00583-6.
4. Brown M., Van Benthem K., Howell J., Poisson J., Arburthnot S., Herdman C. Virtual Reality and 2D Interfaces: A Comparison of Visual Search Task Performance. 19th International Symposium on Aviation Psychology. 2017. P. 71−76. https://corescholar.libraries.wright.edu/isap_2017/87.
5. De Piano R., Ferreira A., Terenzi V., Pozzi S., Aricò P., Borghini G., Di Flumeri G., et al. Virtual Reality platform to assess air traffic controllers’ performance in Control Tower Operations. 2017. http://www.staff.science.uu.nl/~telea001/uploads/PAPERS/VSMM17/paper.pdf. Poslednee poseshchenie 16.12.19.
6. Ellejmi M., Bagassi S., Piastra S., Persiani A. Evaluation of Augmented Reality Tools for the provision of Tower Air Traffic Control using An Ecological Interface Design. Modeling and Simulation Technologies Conference. AIAA AVIATION Forum. (AIAA 2018-2939). 2018. https://doi.org/10.2514/6.2018-2939. Poslednee poseshchenie 16.12.19.
7. Ellis S. Towards determination of visual requirements for augmented reality displays and virtual environments for the airport tower. Virtual Media for Military Applications RTO-MP-HFM-136. P. 31-1−31-10. RTO, Neuilly-sur-Seine (2006).
8. Gorbunov A., Kaurov F., Eliseev B., Gorbunova A. Augmented reality for driving simulators. Proceedings of IEEE SAI Computing Conference. 13−15 July 2016. Londo (UK). P. 238−242. ISBN 978-1-4673-8460-5 (Xplore). IEEE Catalog Number CFP16SAA.
9. Gorbunov A. Stereoscopic Augmented Reality in Visual Interface for Flight Control. Aerospace Science and Technology (Elsevier). October 2014. V. 38. P. 116−123.
10. Gurluk H., Gluchshenko O., et al. Assessment of Risks and Benefits of Context-Adaptive Augmented Reality for Aerodrome Control Towers. IEEE/AIAA 37th Digital Avionics Systems Conference (DASC). 2018. DOI: 10.1109/DASC.2018.8569859.
11. Hagl M., Friedrich M., Papenfuss A., Scherer-Negenborn N., Jakobi J., Rambau T., Schmidt M. Augmented Reality in a Remote Tower Environment Based on VS/IR Fusion and Optical Tracking. Engineering Psychology and Cognitive Ergonomics (EPCE).  Harris D. (eds). 2018. Lecture Notes in Computer Science. V. 10906. Springer. Cham.
12. Han K., Shah S., Lee J. Holographic Mixed Reality System for Air Traffic Control and Management. Appl. Sci. 2019, 9(16), 3370; https://doi.org/10.3390/app9163370.
13. Handbook of Human Factors in Air Transportation Systems. Steven James Landry (eds). CRC Press. Boca Raton (US). 2018.
14. Haring K., Finomore V. Analysis of Using Virtual Reality (VR) for Command and Control Applications of Multi-Robot Systems. https://www.researchgate.net/profile/Kerstin_Haring/publication/323642340_Analysis_of_Using_Virtual_Reality_VR_for_Command_an d_Control_Applications_of_Multi-Robot_Systems/links/5aa16df3aca272d448b3704e/Analysis-of-Using-Virtual-Reality-VR-for-Commandand-Control-Applications-of-Multi-Robot-Systems.pdf. Poslednee poseshchenie 16.12.19.
15. Hofmann T., König C., Röbig A. Steps towards an Augmented Reality workspace for Air Traffic Controllers. Proceedings of 19th Triennial Congress of the IEA. 9−14 August 2015. Melbourne. https://www.researchgate.net/profile/Thomas_Hofmann5/ publication/281619869_Steps_towards_an_Augmented_Reality_workspace_for_Air_Traffic_Controllers/links/5675c2a908ae502c99ce0a 05/Steps-towards-an-Augmented-Reality-workspace-for-Air-Traffic-Controllers.pdf. Poslednee poseshchenie 16.12.19.
16. Jessica A. Updegrove and Shafagh Jafer. Optimization of Air Traffic Control Training at the Federal Aviation Administration Academy. Aerospace. 2017. 4 (4). 50.
17. Lopez-Araquistain J., Campana I. Experimental prototype for remote tower systems design. Integrated Communications, Navigation and Surveillance Conference (ICNS). 2017. DOI: 10.1109/ICNSURV.2017.8011891.
18. Masotti N., De Crescenzio F., Bagassi S. Augmented Reality in the Control Tower: A Rendering Pipeline for Multiple Head-Tracked Head-up Displays. Augmented Reality, Virtual Reality, and Computer Graphics (AVR). De Paolis L., Mongelli A. (eds). 2016. Lecture Notes in Computer Science. V. 9768. Springer. Cham. DOI: https://doi.org/10.1007/978-3-319-40621-3_23.
19. Masotti N., Persiani F. On the history and prospects of three-dimensional human–computer interfaces for the provision of air traffic control services. CEAS Aeronautical Journal. June 2016. V. 7. № 2. P. 149−166.
20. Maxime Cordeil, Tim Dwyer, Christophe Hurter. Immersive solutions for future Air Traffic Control and Management. ACM Companion on Interactive Surfaces and Spaces (ISS Companion). November 2016. Niagara Falls (Canada). P. 25−31. ISBN: 978-1-4503-4530-9. DOI: 10.1145/3009939.3009944.
21. Next Generation Air Transportation System (FAA NextGen). Joint Implementation Planyu 2017. https://www.faa.gov/nextgen/media/NGPriorities-2017.pdf. Poslednee poseshchenie 16.12.19.
22. Pat. US 7129887. Mitchell S. Augmented reality traffic control center. 2007.
23. Reisman R., Brown D. Design of Augmented Reality Tools for Air Traffic Control Towers. Proceedings of 6th AIAA Aviation Technology, Integration and Operation Conference. 2006.
24. Robinson T. Wearable cockpits – the ultimate human-machine interface - https://www.aerosociety.com/news/wearable-cockpits-theultimate-human-machine-interface/. Poslednee poseshchenie 16.12.19.
25. Rohacs J., Rohacs D., Jankovics I. Conceptual development of an advanced air traffic controller workstation based on objective workload monitoring and augmented reality. Proceedings of the Institution of Mechanical Engineers. Part G: Journal of Aerospace Engineering. 2016. V. 230. № 9. DOI: 10.1177/0954410016636154.
26. Rottermanner G., Wagner M. et al. Requirements Analysis & Concepts for Future European Air Traffic Control Systems. Publication forthcoming in Proc. Workshop Vis in Practice-Visualization Solutions in the Wild (VIP 2017). IEEE 2017. The authors' accepted version (postprint). http://mc.fhstp.ac.at/sites/default/files/publications/Rottermanner_2017_Requirements.pdf. Poslednee poseshchenie 16.12.19.
27. SESAR. Master Plan Level 3 2016 Implementation Plan (ESSIP Plan – Edition 2016. https://www.atmmasterplan.eu/news/. Poslednee poseshchenie 16.12.19.
28. Shah S., Han K., Lee J. Interaction Paradigms for Air Traffic Control and Management in Mixed Reality. Virtual, Augmented and Mixed Reality. Applications and Case Studies. Chen J., Fragomeni G. (eds). HCII 2019. Lecture Notes in Computer Science. V. 11575. Springer. Cham.
29. Sun L., Marchese V. et al. Exploring the Best Practices of Virtual Training Delivery for the Prospective Air Traffic Controllers. National Training Aircraft Symposium (NTAS). 2018. https://commons.erau.edu/ntas/2018/presentations/2/. Poslednee poseshchenie 16.12.19.
30. Thales Group. Technology in the skies of tomorrow. https://www.thalesgroup.com/en/singapore/magazine/technology-skies-tomorrow. Poslednee poseshchenie 16.12.19.
31. The RETINA project. http://www.retina-atm.eu/index.html. Poslednee poseshchenie 16.12.19.
32. Truschzinskia M. et al. Emotional and cognitive influences in air traffic controller tasks: An investigation using a virtual environment?. Applied Ergonomics. May 2018. V. 69. P. 1−9. https://doi.org/10.1016/j.apergo.2017.12.019. Poslednee poseshchenie 16.12.19.
33. Pat. US9728006. 2017. Computer-aided system for 360°heads up display of safety/mission critical data. Varga K.
34. Virtual and Remote Control Tower: Research, Design, Development and Validation. N. Fürstenau (eds). Springer. 2016. 336 p.
35. Wickens C., Dempsey G., Pringle A., Kazansky L., Hutka S. The Joint Tactical Air Controller: cognitive modeling and augmented reality HMD design. 20th International Symposium on Aviation Psychology. 2019 P. 163−168. https://corescholar.libraries.wright.edu/isap_2019/28. Poslednee poseshchenie 16.12.19.
36. Zorzal E., Fernandes A., Castro B. Using augmented Reality to overlapping information in live airport cameras. 19th Symposium on Virtual and Augmented Reality (SVR). 2017. DOI: 10.1109/SVR.2017.53.
37. Gorbunov A.L. Treningovaya sistema dopolnennoi realnosti dlya aviadispetcherov. Prikladnaya informatika (Moskva: NOU «Sinergiya»). 2014. 5(53). S. 80−87. (in Russian)
38. Pat. na izobretenie RU 2493606. 2013. Sposob obucheniya aviadispetcherov dispetcherskikh punktov ruleniya, starta i posadki na realnom letnom pole. Gorbunov A.L., Eliseev B.P., Nechaev E.E. (in Russian)
39. Gorbunov A.L., Nechaev E.E. Udalennyi monitoring aviatrafika. Radiotekhnika. 2018. № 11. S. 162−166. DOI: 10.18127/j00338486201811-28. (in Russian)
40. Gorbunov A.L., Nechaev E.E., Surint P.S. Udalennaya aviadispetcherskaya vyshka v virtualnoi realnosti. Radiotekhnika. 2017. № 12. S. 85−92. (in Russian)
41. Strategiya razvitiya Aeronavigatsionnoi sistemy Rossiiskoi Federatsii do 2030 g. http://gkovd.ru/upload/strategiya-razvitiyaans/Presentation_strategy_development_ANS.pdf. Poslednee poseshchenie 16.12.19. (in Russian)
42. Tekhnicheskie kharakteristiki Microsoft HoloLens 2. https://www.microsoft.com/ru-ru/hololens/hardware. Poslednee poseshchenie 16.12.19. (in Russian)