Elektroniknet Logo

Interoperability in embedded vision

Khronos and EMVA cooperate

Khronos ist eine US-Industriekonsortium . Die EMVA ist das europäische Standardisierungsgremium für industrielle Bildverarbeitung
Khronos is a US industry consortium with around 150 member companies and develops open standards for 3D accelerators (e.g. OpenGL, SYCL) and machine learning. EMVA is the European standardization body for machine vision and coordinates, among others, the development of GenICam and the worldwide G3 standards, which also include GigE Vision, USB3 Vision and CameraLink.
© Khronos | EMVA

Details about the cooperation are one of the topics that Khronos President Neil Trevett talks about in his keynote at the embedded world Conference. In this interview Trevett gives a preview of his Keynote, the need for open standards and where the Khronos standards are moving to.

?   Mr. Trevett, in your Keynote you will present several different open standards for the deployment of embedded vision. Why did you choose  this topic?

!   Neil Trevett: The speed with which parallel processing and machine learning are are developing is creating new opportunities for deploying devices and applications that leverage compute-intensive techniques, such as neural network inferencing, for never before seen levels of vision-based functionality and accuracy. However, the industry’s rapid evolution has created a confusing landscape of processors, accelerators, compilers, APIs and libraries. Open standards have an important role to play in helping hardware and software vendors successfully navigate this complex technology environment. The Keynote explains the need for open interoperability standards and their role in reducing costs, time to market, and barriers to using inferencing and vision acceleration in compelling real-world products.

Neil Trevett, Vice President of Developer Ecosystems at NVIDIA
Neil Trevettis Vice President of Developer Ecosystems at NVIDIA and the elected President of the Khronos Group, where he initiated the OpenGL ES standard now used by billions of people worldwide every day.
© NVIDIA

Every industry can benefit from open standards that provide increased interoperability between the elements of its own ecosystem. Open standards and proprietary technologies have complex and interdependent relationships. Proprietary APIs and interfaces are often the Darwinian testing ground and can remain dominant in the hands of an early smart market leader and that is as it should be. Strong open standards result from a wider need for a proven technology  and can provide healthy, motivating competition. In the long view, an open standard that is not controlled by, or dependent on, any single company can often be the thread of continuity for industry forward progress as technologies, platforms and market positions swirl and evolve.

We are presenting the subject of Khronos standards at embedded world because we believe  these standards can help grow market opportunities in some of the most dynamic segments of the embedded industry.

?   You focus on the standards developed in the Khronos Group. Do they cover all relevant areas for the embedded developer?

!   Trevett: There are many standards organizations in the industry each with its own area of expertise. Khronos’ specific focus is the creation of open, royalty-free API standards that enable software applications libraries and engines to harness the power of silicon acceleration for demanding use cases. These use cases include; 3D graphics, parallel computation, vision processing and inferencing. Khronos has productive liaison relationships with many other standards organizations to avoid duplication of effort and to ensure that standards from different organizations work well together.

In fast evolving areas, such as embedded vision, it is important that standards organizations pay close attention to evolving industry requirements and »hot spot« areas of industry friction where new interoperability standards may be able to create and accelerate market opportunities. Khronos has a proven exploratory group process that enables Khronos members, and the wider industry, engage in constructive dialogue to assess the need for new interoperability standards, best practices, and guidelines.

It is important that standards organizations pay close attention to evolving industry requirements.
Neil Trevett, Khronos Group
© NVIDIA

In fact, we are currently assessing industry interest in new interoperability standards for embedded vision! Camera sensors are increasingly being closely integrated with image, vision and inferencing accelerators in self-contained systems. Thus the increasingly urgent industry question is, whether innovation and efficiency in the embedded vision market is being constrained by the lack of open cross-vendor camera control API standards to reduce development and integration costs. Such API standards could potentially help control of a wide range of camera sensors, depth sensors, camera arrays and ISP hardware to generate sophisticated image streams for downstream local processing by diverse accelerators.

We are happy to announce here at embedded world that Khronos and the European Machine Vision Association (EMVA) are cooperating to create the new Embedded Camera Exploratory Group to gather perspectives from the processor, sensor, machine vision and open-source communities. The group's goal will be to discuss use cases and requirements  as well as evaluating the need for embedded camera/sensor interoperability APIs to reduce application complexity, minimize development and integration costs, and accelerate time to market. This new Exploratory Group will be open to any interested company at no cost.

?   Where are the main differences between the  various Khronos standards?

!   Trevett: Broadly, the Khronos open standards that I discuss in my keynote can be divided into two groups:

  1. High-level programming APIs such as SYCL and OpenVX, that focus on ease of programming with effective performance portability across multiple hardware architectures.
  2. By contrast, low-level APIs such as OpenCL and Vulkan SC provide direct, explicit access to hardware resources for maximum flexibility and control.
“We are currently assessing industry interest in new interoperability standards for embedded vision.
Neil Trevett, Khronos Group
© NVIDIA

It is important that developers understand which level of API will best suit their development needs. The high-level standards include SYCL, (pronounced »sickle«) which enables code for heterogeneous processors to be written in a »single-source« file using standard C++ for portable acceleration on a wide range of hardware. SYCL will typically compile the part of the C++ application to be offloaded onto an acceleration processor into a lower-level API such as OpenCL. OpenVX streamlines the development of vision and inferencing software by providing a graph-level abstraction that allows a programmer to construct his/her required functionality by connecting a set of functions or »Nodes«. This high-level of abstraction enables silicon vendors to optimize their OpenVX drivers for efficient execution on almost any processor architecture.

Khronos offers two low-level APIs for compute acceleration in the embedded market. Vulkan SC is the safety critical variant of the new generation Vulkan 3D API that can accelerate compute operations on any compatible GPU using compute shaders alongside graphics rendering. OpenCL is a low-level standard for cross-platform, parallel programming of diverse heterogeneous processors found in PC, servers, mobile devices and embedded devices, including GPUs, DSPs, and FPGAs. OpenCL provides C and C++-based languages for the construction of  kernel programs that can be compiled and executed in parallel across any processors in a system with an OpenCL compiler, giving the programmer explicit control over which kernels are executed on which processors.

?   Your Keynote will also include estimations about  which direction each standard will move in – can you give us a short preview?

!   Trevett: Yes, all Khronos standards are being very actively developed to meet evolving industry needs.

Earlier this month SYCL released the SYCL 2020 Final Specification, which incorporates feedback collected following the provisional specification, released in June 2020, and adds significantly more features and fixes for easier integration with C++17. These updates make the coding language more compact to enable smaller code size and simplified common patterns. SYCL 2020 simplifies interoperability and makes it easier for developers to port standard C++ applications to SYCL, greatly improving programmability.

These updates make the coding language more compact and  enable smaller code size and simplified common patterns.
Neil Trevett, Khronos Group
© NVIDIA

The number of OpenVX nodes has been growing steadily over time and the recent release of OpenVX 1.3 is partitioned into feature sets for vertical market segments, such as inferencing and has a reduced area profile for safety critical systems. Other new functionality includes sophisticated graph control, such as conditional execution, that selectively executes portions of a graph as well as increasing use of interop with OpenCL that enables fully accelerated custom Nodes within an OpenVX graph.

OpenCL 3.0, released in September 2020, enables hardware vendors to focus their resources on driver functionality that their customers need by making all functionality beyond OpenCL 1.2 optional through API queries and language macros. Several new extensions were released with OpenCL 3.0, including Asynchronous DMA extensions.

These extensions enable DSP-like embedded processors to transfer data between global and local memories via DMA transactions. This complements  previous one-dimensional operations by supporting complex memory transfers and use of fences for synchronization of multiple asynchronous transactions running in parallel. There is significant interest in continuing to extend OpenCL functionality for DSPs and other embedded processors.

With Vulkan SC, Khronos is defining a safety-critical subset of the new-generation Vulkan GPU API. Vulkan SC is in development with an expected release during 2021. Khronos’ experience in defining safety-critical graphics APIs, started with OpenGL SC 1.0 that took a minimal subset of OpenGL ES 1.0 in 2005 for easier certification to the DO-178B standard for avionics displays. When OpenGL ES 2.0 introduced programmable shaders, OpenGL SC 2.0 integrated that functionality in 2016 for ISO 26262 certification for automotive displays. And now as the need for safety-certified embedded GPU graphics and compute expands into many new market areas, Vulkan SC will reduce driver size and increase determinism and error handling, together with MISRA C compliance.

?   Who should attend to your talk?

!   Trevett: This talk will be valuable to anyone looking for a state-of-the-art overview  regarding open standard APIs for embedded vision and inferencing. System integrators, imaging and vision engineers, program managers, application developers, researchers and educators will gain insights as to whether open standards can add value to their projects and which APIs may best suit their needs.

?   Being a Keynote-Speaker is also a chance to direct the attention of the community. What is the most important thing the embedded vision community should focus on in 2021?

Khronos and the European Machine Vision Association (EMVA) are cooperating to create the  new Embedded Camera Exploratory Group.
Neil Trevett, Khronos Group
© NVIDIA

!   Trevett: I think a underlying trend, that will continue to accelerate in 2021, is convergence. For example, the previously separate markets for machine vision and embedded processors are converging to create new tightly integrated embedded vision products and opportunities. The fields of classical image and vision processing are converging with the fast-moving field of machine learning, transforming the domain of embedded vision with stunning new capabilities. Conventional hardware-centric API designs are now converging with sophisticated compiler technology, opening-up significant new levels of programmable flexibility and portability for any developer tapping into silicon acceleration. Everyone working in the embedded industry should be alert to new areas of convergence, and the opportunities they bring!


  1. Khronos and EMVA cooperate
  2. Session Keynote: Embedded Vision

Das könnte Sie auch interessieren

Verwandte Artikel

NVIDIA Corporate, EMVA European Machine Vision Association

Anbieterkompass