This article covers new MIT research that brings non-line-of-sight imaging capabilities to affordable consumer-grade lidar systems. Video from MIT can be seen below.
Lidar technology is rapidly becoming ubiquitous, powering everything from our smartphones and Augmented Reality (AR) headsets to warehouse robotics and autonomous vehicles. Despite their impressive capabilities, even the most sophisticated lidar systems are still limited by the basic physical requirement of an unobstructed line of sight. This constraint means they can typically only map what their emitted light can directly reach and reflect from, making the visualization of objects around corners or over horizons difficult to impossible. However, these limitations are currently undergoing a significant shift.
MIT Pushes Consumer LiDAR Beyond Line-of-Sight Limitations
New research from MIT, published in Nature by Somasundaram et al., is shattering this boundary and fundamentally changing what’s possible with the affordable “Time-of-Flight” (ToF) depth sensors already widely in use. This breakthrough moves advanced Non-Line-of-Sight (NLOS) imaging from bulky, expensive lab environments to widely available, sub-$100 hardware with plug-and-play usability.
The challenge of achieving NLOS on consumer-grade sensors is not insignificant. Unlike research-grade systems, mobile lidars operate under strict constraints like eye-safe, low laser power and short exposure times, leading to a low Signal-to-Noise Ratio (SNR). Constraints on size and power result in low spatial resolution and concurrent object and camera motion can cause debilitating motion blur. These constraints conspire to produce measurements insufficient for traditional NLOS reconstruction.
The Motion-Induced Aperture Sampling (MAS) Model
The core innovation is a multi-frame fusion strategy, which the team calls the “Motion-Induced Aperture Sampling (MAS)” model. This approach takes inspiration from two powerful imaging techniques: burst photography, which improves SNR through redundancy and frame stacking, and synthetic aperture radar, which uses diverse viewpoints created by motion to dramatically improve resolution.
The MAS model serves as a unified measurement framework that achieves a crucial mathematical feat: it decouples the complex, time-varying effects of object shape, object motion, and camera motion. By performing this separation, the system can efficiently process numerous low-quality individual frames, stitching them into a collective, high-utility estimate of the hidden scene. Critically, the inference process is designed to be computationally tractable—recovering just one unknown variable (shape, position, or camera position) at a time—making it suitable for real-time operation on mobile platforms.
Reconstructing Hidden Objects with Smartphone-Grade LiDAR
Leveraging the MAS model, the researchers demonstrated three key NLOS capabilities directly on smartphone-grade lidar, providing immediate and valuable solutions for lidar-based applications.
First, static objects hidden around a corner can be reconstructed in three dimensions. This is achieved by exploiting the natural, unstructured motion of a handheld camera, effectively using movement to generate a larger synthetic aperture and increase sampling of the “virtual mirror,” or light field reflected off a relay surface.
The system can also perform real-time, 3D object tracking of single and multiple moving objects of known shape. This relies on a particle filtering framework that significantly improves robustness against noise by incorporating motion priors from past frames, allowing for confident tracking even when individual frames are ambiguous.
Real-Time Tracking and Camera Localization
Finally, camera localization using hidden objects is a major advancement for the fields of robotics and AR. Hidden objects can be used as reliable visual cues to localize the camera, proving invaluable in feature-poor environments where conventional visual odometry and computer vision techniques like structure from motion or iterative closest point fail.
This breakthrough dramatically lowers the barrier to entry for non-line-of-sight imaging. By validating these techniques on commercial sensors like the STMicroelectronics VL53L8CX with minimal calibration, the potential for integrating advanced NLOS features into everyday robotics, navigation, and computer vision systems is no longer a distant possibility, but a rapidly approaching reality.
Why This Breakthrough Matters for Robotics, AR, and Navigation
The implications of this research extend far beyond academic experimentation. Affordable consumer lidar capable of non-line-of-sight imaging could dramatically improve robotic navigation in cluttered environments, enhance AR systems with hidden-scene awareness, and provide safer perception systems for autonomous platforms operating in dynamic spaces.
As smartphone-grade depth sensors continue to improve, techniques like Motion-Induced Aperture Sampling may become foundational components in next-generation computer vision pipelines. What once required expensive laboratory hardware may soon become standard functionality in everyday devices.
More Recent News: Webinar: LGSx to Deliverable – Leica + Mach9
