doi: 10.7873/DATE.2015.0369

SAPPHIRE: An Always-on Context-Aware Computer Vision System for Portable Devices

Swagath Venkataramani1, Victor Bahl2,a, Xian-Sheng Hua2,b, Jie Liu2,c, Jin Li2,d, Matthai Phillipose2,e, Bodhi Priyantha2,f and Mohammed Shoaib2,g

1School of Electrical and Computer Engineering Purdue University, West Lafayette, IN 47907, USA .

2Microsoft Research, One Microsoft Way, Redmond WA 98052, USA.


Being aware of objects in the ambient provides a new dimension of context awareness. Towards this goal, we present a system that exploits powerful computer vision algorithms in the cloud by collecting data through always-on cameras on portable devices. To reduce comunication-energy costs, our system allows client devices to continually analyze streams of video and distill out frames that contain objects of interest. Through a dedicated image-classification engine SAPPHIRE, we show that if an object is found in 5% of all frames, we end up selecting 30% of them to be able to detect the object 90% of the time: 70% data reduction on the client device at a cost of ≤ 60mW of power (45nm ASIC). By doing so, we demonstrate system-level energy reductions of ≥ 2×. Thanks to multiple levels of pipelining and parallel vector-reduction stages, SAPPHIRE consumes only 3.0 mJ/frame and 38 pJ/OP – estimated to be lower by 11.4× than a 45 nm GPU – and a slightly higher level of peak performance (29 vs. 20 GFLOPS). Further, compared to a parallelized sofware implementation on a mobile CPU, it provides a processing speed up of up to 235× (1.81 s vs. 7.7 ms/frame), which is necessary to meet the real-time processing needs of an always-on context-aware system.

Keywords: Always on, Portable devices, Computer vision, Object recognition, Hardware acceleration, Energy efficiency

Full Text (PDF)