INESC TEC leads European project that combines radio communications with computer vision towards 6G

02nd March 2023

It is a new area of research that aims to generate knowledge from merging wireless communications, computer vision, sensing and machine learning. Concerning the CONVERGE project, Luís Pessoa, researcher at INESC TEC, guest professor at the Faculty of Engineering of the University of Porto (FEUP) and project coordinator, stated that "the goal is to develop a set of tools for research infrastructures that promote the creation of a new research area aligned with the see-to-communicate and communicate-to-see paradigm".

Through the combination of data generated by radiofrequency (RF) communications systems and video cameras, it will be possible "to obtain new datasets that will be made available openly to the scientific community, promoting the creation of new knowledge and new discoveries".

"Communications systems that operate in higher radio frequency bands will depend much more on the line-of-sight to operate. These systems will benefit from computer vision provided by video cameras to better predict the dynamics of the communications channel, like anticipating line-of-sight loss due to an obstacle, or helping build three-dimensional maps to improve mobile terminal position estimation". Moreover, the researcher mentioned that "computer vision applications may become more robust against challenges like occlusions or low light when supported by RF data".

This set of tools is unprecedented in the world; on the one hand, it will provide the scientific community with a series of exclusive and open data, and, on the other hand, it will improve the competitiveness of research infrastructures and the companies involved.

"The community, featuring thousands of users, will benefit from the results of CONVERGE over the next 10 years", said Manuel Ricardo, full professor at the Faculty of Engineering of the University of Porto (FEUP), Associate Director at INESC TEC and scientific coordinator of the project. Among the various expected results are the emergence of new areas of research, the development of new industrial applications capable of combining video information, sensing, radio and data traffic, and the improvement of the skills of employees, students and industrial users of research infrastructures, who will be trained to use the new set of tools.

Areas like healthcare, industry, the automotive sector, telecommunications and audio-visual content distribution services will be some of the main beneficiaries of the research supported by these tools, namely through the automatic assessment of patient posture and prosthesis alignment in physical rehabilitation (in the case of healthcare); better understanding of the factory floor (industry); improvement in the perception of the surrounding environment of the vehicle, as well as external conditions that may affect the quality of autonomous driving (concerning the automotive sector); the possibility of controlling the electromagnetic response of the environment, as well as high-precision location and sensing services of high-resolution 3D environment (telecommunications); and use of computer vision to help decrease computational complexity in the radio frequency modelling of objects and people (multimedia).

"Currently, only the COSMOS infrastructure (United States of America), a partner of the CONVERGE consortium, provides access to cameras as a way to enhance research that exploits the information captured by these equipment to better understand the surrounding environment, thus developing new paradigms and communications solutions", explained Luís Pessoa - emphasising that "these cameras do not have the same line of sight as the base stations, thus being unable to provide the complete repeatability and reproducibility that is sought in CONVERGE".

The CONVERGE project is coordinated by INESC TEC and features 15 more partners from six countries – Portugal, Finland, Spain, France, the United Kingdom, and the United States of America. In addition to INESC TEC, ALLBESMART and Adapttech are the other Portuguese institutions that are part of the consortium. The European Commission allocated €9M to fund the project.