A Camera with Brain - Embedding Machine Learning in 3D Sensors
Burhan Ahmad Mudassara, Priyabrata Sahab, Yun Longc, Muhammad Faisal Amird, Evan Gebhardte, Taesik Naf, Jong Hwan Kog, Marilyn Wolfh and Saibal Mukhopadhyayi
School of ECE, Georgia Institute of Technology, Atlanta, Georgia, USA
aburhan.mudassar@gatech.edu
bpriyabratasaha@gatech.edu
cyunlong@gatech.edu
dmfamir@gatech.edu
eegebhardt6@gatech.edu
ftaesik.na@gatech.edu
gjonghwan.ko@gatech.edu
hwolf@ece.gatech.edu
isaibal@ece.gatech.edu
ABSTRACT
The cameras today are designed to capture signals with highest possible accuracy to most faithfully represent what it sees. However, many mission-critical autonomous applications ranging from traffic monitoring to disaster recovery to defense requires quality of information, where useful information depends on the tasks and is defined using complex features, rather than only changes in captured signal. Such applications require cameras that capture useful information from a scene with highest quality while meeting system constraints such as power, performance, and bandwidth. This paper will discuss the feasibility of a camera that learns how to capture taskdependent information with highest quality, paving the pathway to design a camera with brain. 3D integration of digital pixel sensors with massively parallel computing platform for machine learning creates a hardware architecture for such a camera. The paper will discuss embedded machine learning algorithms that can run on such platform to enhance quality of useful information by real-time control of the sensor parameters. We conclude by identifying critical challenges as well as opportunities for hardware and algorithmic innovations to enable machine learning in the feedback loop of a 3D image sensor based camera.
Keywords: Smart cameras, Embedded vision, Machine learning based Feedback, 3D-stacked sensor, Digital Pixel Sensor
