The company claims these are the world's first image sensors with AI processors on board, although numerous startups have been prototyping event-based sensors that minimize data transfer and are well suited to machine vision.
The ability to put AI processing on the image sensor enables high-speed edge processing and provides the opportunity to develop smart cameras and industrial and commercial IoT applications, Sony said.
The sensors are made with a two-die, stacked configuration with pixel chip and logic chip. The signal acquired by the pixel sensor is analysed by the AI processor eliminating the need for high-performance processors or external memory and minimizing data transfer both within the system and also up to the cloud.
The sensor component output can then be metadata belonging to the image. Different AI models and machine learning can be used to customize the image sensor to different applications, lighting conditions and locations.
The iMX500 has 12.3 megapixel back-side illuminated image sensor and a Sony-developed logic chip including memory and image signal processor and DSP for AI signal processing. The ISP is used to select region-of-interest and to format images (YUV/RGB).
These latest sensors can perform AI processing in 3.1 milliseconds (MobileNet v1) on the logic chip. As this is within a single frame period it allows real-time tracking of objects while recording video at 30fps.
Users can write and rewrite their own AI models to the embedded memory allowing the sensor to be used for such applications as people-counting, stock monitoring, heat mapping of people in a store
The iMX500 is the bare die version of the product. The iMX501 is housed in a ceramic package 12.5mm by 15.0mm. The sample price of the bare die version is 10,000 yen (about US$94) and of the packaged version is 20,000 yen (about US$187).
Next: Key specificatons of iMX500.