ActiveSight: Enabling Continuous Wearable Perception through Algorithm-Hardware Co-Designed Multimodal Semantic Gating

Abstract

Continuous wearable AI perception must capture transient, meaningful events throughout the day, yet always-on visual processing rapidly exhausts batteries of compact wearables. We present ActiveSight, an algorithm-hardware co-designed multimodal gating system that exploits the complementarity between audio and vision: audio is energy-efficient but perceptually limited, while vision is accurate but too power-hungry to run continuously. ActiveSight uses a low-power audio front-end as an always-on trigger and activates a unified CLIP-based detector only when visual semantic filtering is needed. The detector performs keyword-guided semantic filtering with reconfigurable text anchors, transmitting only relevant frames without on-device retraining. To enable on-wearable execution, we design a custom FPGA accelerator featuring the PatchDW dataflow, which mitigates the memory bottleneck of depthwise convolutions and achieves 44.3 GOPS/W in PL-side active power. Evaluation on Ego4D shows that ActiveSight achieves 87.7% event hit rate at 453 mW average power, reduces average power by 3× compared with continuous visual streaming, and reaches 88.5% hit rate in live smart-glasses deployment.