Light field detection in the age of artificial intelligence

Abstract

Light carries information across intensity, phase, spectrum, polarization, space and time, yet a conventional photodetector typically produces only a scalar photocurrent. Computational light field detection addresses this mismatch by encoding multidimensional optical information into compact measurements that can be reconstructed algorithmically. Artificial intelligence is playing a central role in reshaping both sides of this process. For the optical front end, learned surrogate and generative models not only reduce the computational burden of nanophotonic design but also open much larger design spaces, including complex, non-intuitive structures. Reconstruction presents a different challenge. No single model class is optimal across all sensing regimes, but machine learning offers flexible routes to recover multidimensional information from sparse or compressed measurements. Despite these advances, hardware and algorithms are still largely optimized separately, limiting system-level performance. This limitation motivates a shift towards co-design. Differentiable digital twins provide a framework for this integration through co-optimization of encoder parameters and reconstruction models. Physics-based formulations can go further, incorporating experimental error sources and estimating reconstruction uncertainty. The prospect is compelling, indicating a new generation of compact, adaptable detectors for multidimensional light sensing. Their real-world impact will ultimately depend on generalization, physical consistency, interpretability and robustness to discrepancies between digital models and physical hardware. Artificial intelligence (AI) is reshaping computational light field detection by improving optical encoding and algorithmic reconstruction. This Review examines AI-enabled encoder design, multidimensional reconstruction and differentiable digital twins for joint hardware–software optimization, alongside the challenges that must be overcome for reliable deployment. Computational light field detection recovers multidimensional optical information from low-dimensional detector outputs by combining information-rich encoding with reconstruction. Its effectiveness depends not only on the number of measurements but also on the distinction in captured information. No single machine learning architecture is optimal. The choice of model depends on training data, forward model availability, latency and tolerance for reconstruction errors. Artificial intelligence (AI)-based surrogate models make designing high-dimensional photonic encoders more practical by replacing repeated electromagnetic simulations with learned predictions. Combined with generative and differentiable optimization, they enable exploration of non-intuitive, fabrication-constrained designs. AI-based decoders can recover spectral, phase, polarization and temporal information from sparse or compressed measurements. Progress in spectral and temporal reconstruction is more mature than phase and polarization retrieval. A new opportunity is the differentiable digital twin for light field detection, which enables joint optimization of the physical encoder and the computational decoder. When grounded in physics, the framework could additionally provide uncertainty estimates. Reliable detection does not solely depend on accurate reconstruction. Trustworthy deployment demands physics-based models and hardware-in-the-loop optimization that generalize beyond training data, remain physically consistent and quantify uncertainty. Computational light field detection recovers multidimensional optical information from low-dimensional detector outputs by combining information-rich encoding with reconstruction. Its effectiveness depends not only on the number of measurements but also on the distinction in captured information. No single machine learning architecture is optimal. The choice of model depends on training data, forward model availability, latency and tolerance for reconstruction errors. Artificial intelligence (AI)-based surrogate models make designing high-dimensional photonic encoders more practical by replacing repeated electromagnetic simulations with learned predictions. Combined with generative and differentiable optimization, they enable exploration of non-intuitive, fabrication-constrained designs. AI-based decoders can recover spectral, phase, polarization and temporal information from sparse or compressed measurements. Progress in spectral and temporal reconstruction is more mature than phase and polarization retrieval. A new opportunity is the differentiable digital twin for light field detection, which enables joint optimization of the physical encoder and the computational decoder. When grounded in physics, the framework could additionally provide uncertainty estimates. Reliable detection does not solely depend on accurate reconstruction. Trustworthy deployment demands physics-based models and hardware-in-the-loop optimization that generalize beyond training data, remain physically consistent and quantify uncertainty.