People, bicycles, cars or road, sky, grass: Which pixels of an image belong to people or objects in the foreground of the environment of a self-propelled car, and which pixels represent the urban scenery? This task (panoptical segmentation) is a fundamental problem in many fields such as self-propelled cars, robotics, augmented reality and even in biomedical image analysis.
This type of task is often solved using DL methods (deep learning), and how good the algorithms are can be measured with benchmarks. "For many years, research teams from companies like Google and Uber have been competing for the top spot in these benchmarks," says Rohit Mohan from Valada's team. However, the new method developed by the Freiburg computer scientists has now reached first place in Cityscapes, a benchmark for methods for understanding scenes in autonomous driving. EfficientPS is also listed in other benchmark data sets such as KITTI, Mapillary Vistas and IDD.
On the project's website, Valada shows examples of how the team has trained different AI models on different data sets. The results are superimposed on the respective image captured with the camera, with the colours showing which object class the model assigns the respective pixel to. For example, cars are marked blue, people red, trees green and buildings grey. In addition, the AI model also draws a frame around each object, which it views as a separate entity. The Freiburg researchers have succeeded in training the model to transfer the learned information of urban scenes from Stuttgart to New York City. Although the AI model did not know what a city in the USA might look like, it was able to accurately recognize scenes from New York City.
Most of the previous methods that address this problem require large amounts of data and are too computationally intensive for use in real-world applications such as robotics, which are highly resource-constrained, Valada explains: "Our EfficientPS not only achieves high output quality, it is also the most computationally efficient and fastest method. This significantly extends the areas of application in which EfficientPS can be used".