3D Simulation in Autonomous Driving

The vehicles of tomorrow that are to achieve the goal of CO2-neutral mobility will be autonomous, electric, connected - and software-defined. In the Elektronik-automotive interview, Gaurav Tomar from MathWorks talked about the importance of 3D simulations for autonomous driving.

The software defined, autonomous, connected, and electric vehicle of tomorrow promises to help realize the shared carbon neutral mobility. Organizations are investing in digital capabilities to make the shift from designing best piece of machinery to providing state-of-the-art end-to-end mobility & digital services.

The vision is a user experience, which will largely be defined through software. To enable this vision, the industry is reshaping itself around software as a future value driver. This software is going to drive our car while also booking us a table at our favorite restaurant, as we finish our work calls, all in the car. An important component of this software is the autonomous driving software (ADS) which should work flawlessly so that the rider (sans Driver) has time for other activities.

Elektronik automotive: What are the biggest challenges AV teams face today?

Gaurav Tomar: The autonomous vehicle area has a number of challenges and obstacles which need to be maneuvered around before Level 4 (L4) or Level 5 (L5) autonomous driving is possible. As the recent IAA has shown us, some companies are closer to solving this challenge than others and soon will be starting with trials in Europe as well, and not just in USA. I believe the following challenges are facing the engineers in this space:

• Proving that the designed ADS is safe enough, secure and reliable. For the acceptance of autonomous vehicles in the public it is paramount that AVs on the road do not cause accidents. For this to become a reality the ADS should work reliably and safely in its operational design domains (ODDs). How to make the ADS safe, secure and reliable is the most important challenge being faced by the AV teams today.
• Testing vehicles virtually to cover large sets of scenarios for diverse ODDs. It is well accepted that validating the performance of the ADS in the real world alone is not realistic. So, the engineers must turn towards testing the ADS virtually, where they scale the tests to thousands of simultaneous simulations and to gain evidence that the ADS makes the right decisions in all possible and probably scenarios for that particular ODD.

Elektronik automotive: What does 3D Driving Simulation solve?

Gaurav Tomar: Our observations – compiled while working with all major automotive organizations around the globe – show that automotive organizations have established capabilities and maturity to address they key challenges. These challenges however have been exacerbated in the recent times by technology advances which have resulted into new architectures, and new functionality that often relies on combinations of other functions. Organizations are pivoting to large scale virtual testing of the embedded software. The 3D Driving Simulation software helps design scenes and scenarios to in which the ADAS can be tested. This primarily helps with

• Accelerating development cycles by enabling closed loop testing of the ADS early, including the Perception, planning and control software.
• Reducing development costs by testing virtually and providing a safe way to test all possible edge and dangerous scenarios.
• Validating the ADS virtually using scenes that correlate with the real world to fulfill requirements of ISO 26262 and SOTIF.

Elektronik automotive: Can you walk us through the steps needed to set-up such a driving simulation?

Gaurav Tomar: In broad strokes the following tasks need to be taken care of:

• Prepare a virtual model of the vehicle. Often tools like Simulink are used for this.
• Select and place the right sensor models on this virtual vehicle model. It is important that the sensor models chosen have appropriate fidelity levels, so that they perceive the 3D world like how the physical prototype would perceive the real world.
• Build realistic scenes using 3D scene design tools like RoadRunner and ensure that these scenes correspond to the target ODD of the autonomous vehicle. One might need access to HD Maps and other road & environment artifacts.
• Design driving scenarios based on the test strategy. To ensure a quality ADS, the simulated scenarios should highly correlate with real world situations. For this, some companies are already implementing restimulation frameworks, where virtual scenarios are built using data from real test drives and then the ADS and virtual vehicle is tested for all possible variations of that scenario.
• Import the scenes, scenarios and the virtual vehicle model into a driving simulation software and simulate the ADS.

It is important to know that creating scenes, selecting appropriate sensor models and designing scenarios which cover all possible and probable situations, is a time intensive exercise often running into weeks if not months. Many existing solutions fall short of providing efficient scene building capabilities, while virtual validation continues to demand high scene and artifact quality.

Elektronik automotive: So, it seems building 3D scenes correctly is critical for the testing of the ADS. How are these 3D scenes built today?

Gaurav Tomar: Yes, that is correct. Building accurate 3D scenes is one critical step in the 3D Driving Simulation. Point cloud scan of the real world, Ortho imagery and other Vector data help as a reference while designing the 3D Scene. A simple process for creating such scenes using a tool like RoadRunner could look as follows:

Matchmaker+ Anbieter zum Thema

zu Matchmaker+

• Import scene reference from either of the following sources: OpenDRIVE road network, or HD Maps, or use GIS assets to create real world from scratch, or existing virtual scene.
• Edit road networks and lanes in this import for the area of interest, which often could run in 10s or 100s of kilometers.
• Enrich the scene with pre-built 3D assets like buildings, vegetation, road artifacts etc.
• Ensure correctness of the scene by using visual reference like Satellite imagery.
• And finally export scenes created in RoadRunner to industry standard formats, including OpenDRIVE, FBX, glTF, OpenFlight, OpenSceneGraph, OBJ, and USD.

Elektronik automotive: What are the major advantages using a tool like RoadRunner to design scenes?

Gaurav Tomar: There are quite a few advantages of using a dedicated to tool for road network design and scene building. The top 3 would be:

• The scenes created in RoadRunner can be easily exported into the common formats which can then be ingested by most driving simulators out there. This makes the teams highly flexible. Organizations which design scenes as a business can target multiple customers from a single tool. RoadRunner Scene Builder helps automatically generate 3D road models from HD maps. RoadRunner scenes exports to a variety of simulators and gaming engines, including CARLA, Vires VTD, NVIDIA DRIVE Sim, Cognata, Metamoto LGSVL, Baidu Apollo, Unity, and Unreal Engine.
• RoadRunner Asset Library lets you quickly populate your 3D scenes with a large set of realistic and visually consistent 3D models.

Elektronik automotive: Which type of information is typically needed by driving simulators?

Gaurav Tomar: Typically, a scene design tool will export two broad categories of information. First is geometric mesh, for example roads, curbs, terrain, and objects in the scene. This information is used by sensor models like cameras, lidar, and radar. Second is semantics information which typically is exported as OpenDRIVE format. Semantic information is commonly used to design or validate planning algorithms.
In my experience, oftentimes different groups within a single organization are using different driving simulators. In such cases it helps to have way to create scene once and use this as single source of truth for all simulators.

Elektronik automotive: How important are the highly detailed maps in this area?
Gaurav Tomar: As car makes traverse the ladder of SAE autonomous driving levels, majority of the AV Teams today consider HD Maps, an integral part of the autonomous driving infrastructure. HD maps (or High-Definition Maps) provide highly accurate road and lane map including high environment fidelity like accurate position of pedestrian crossings, traffic lights, curvature of the road, turn radius, barriers etc. These maps are often accurate to a centimeter or less. Autonomous vehicles need HD maps to deal with inconsistencies in perception algorithms just like human do when while following a GPS.

Broadly HD maps help in the following ways:

1. Localization: As a reference to the environment model created by perception and sensor fusion algorithm.
2. Planning: The detailed information in HD maps helps driving strategy functions to plan route based on current scenario.
3. Software Testing: While developing Software, HD Maps help in creating virtual scenes which represent the real world and are used to test the ADS.

In past decade, multiple new organizations have sprung up with new business models for HD Maps for both engineering teams and for fleet owners (the OEMs). But HD maps are still not ubiquitous due to costs involved in creating and maintaining them, because HD Maps unlike the normal GPS maps in our vehicles, would need revisions every few weeks rather than months.

Elektronik automotive: Are there any customer use cases (references) you can share?

Gaurav Tomar: In this years’ MathWorks Automotive Conference, Porsche Engineering Services talked about how the engineers at the Porsche Engineering ADAS Testing Centre (PEVATeC) bring together the best tools, data sources, and experience in automotive and ADAS to create and provide a flexible modular simulation platform that supports end-to-end ADAS/AD development.

Elektronik automotive: How exactly can 3D simulations be used to train algorithms for autonomous driving?

Gaurav Tomar: The limitation of recorded data is that it can only be used for the scenario which was actually recorded. Algorithm developers need synthetic data in addition to their recorded data to test cases that are difficult, dangerous or seldom encountered. With appropriate models of Camera, Lidar, Radar and other sensors like IMU and GPS, engineers can capture synthetic data (from the 3D Simulations) by integrating 3D scene with their development environment, e.g., with MATLAB & Simulink. Developers would also like to recreate recorded data to enable testing of variations of the recorded case.

Elektronik automotive: If you were to look around the corner, what is coming next for MathWorks in this area?

Gaurav Tomar: There are several trends that are driving MathWorks’ investments. The first is the need for virtual worlds. The industry continues to do more with simulation in order to reduce vehicle testing as well as explore scenarios that may be difficult or unsafe to reproduce in the real world. MathWorks provides tools that allow you to design and simulate scenes, scenarios, sensors, and dynamics.

The second trend driving MathWorks investments is the need for multidisciplinary skills. Developing an automated driving application requires a variety of skills from planning and controls to perception disciplines, like detection, localization, tracking, and fusion. MathWorks provides dedicated toolboxes to help you learn and apply these disciplines. These toolboxes metrics and analysis capabilities that you can use throughout your development process.

The third trend driving investments is the need for software. Our investments here are focused on strengthening our platform tools to enable you to design, deploy, and test embedded software.

ADAS/AD Virtual Platform for End-to-End Software Development and Testing Video - MATLAB & Simulink (mathworks.com)
 

The interview partner

Gaurav Tomar
is Automotive Industry Marketing Manager for EMEA at MathWorks, focusing on the German market. With many years of experience in the automotive industry, he helps decision makers maximize engineering efficiency in areas such as ADAS/AD, electrification, and vehicle software. He outlines how MathWorks technology can be applied to automotive processes and methodologies today, and identifies future growth areas for the company. He also has hands-on experience developing perception, sensor fusion, and localization algorithms at BMW, Porsche, and Honda. He has Master degree in Electrical Engineering from TH Rosenheim and a Bachelor degree in Electrical Engineering from India.