\(\begin{align} \newcommand{transp}{^\intercal} \newcommand{F}{\mathcal{F}} \newcommand{Fi}{\mathcal{F}^{-1}} \newcommand{inv}{^{-1}} \newcommand{stochvec}[1]{\mathbf{\tilde{#1}}} \newcommand{argmax}[1]{\underset{#1}{\mathrm{arg\, max}}\,} \newcommand{argmin}[1]{\underset{#1}{\mathrm{arg\, min}}\,} \end{align}\)

Computational Imaging


Coded Aperture Spectral Snapshot Imaging#

Content#

  • Introduction

  • Conventional RGB-camera

  • Popular CASSI architectures

Introduction#

Most camera systems capable of capturing color information filter incident light by means of three spectral filters with peaks in the red, green and blue regions of the spectrum. This design has been chosen to mimic the human visual system with the three different cone cells responsible for human perception of color.

However, for many fields of applications, such as remote sensing, materials science, biophotonics, environmental monitoring, smart agriculture or food, a finer sampling of the spectral information can be of utter importance.

Simultaneously capturing high-resolution spatial and spectral information within the classical Nyquist-Shannon sampling regime is currently only possible by performing time-multiplexed measurements.

Typical classical sensor concepts for capturing spectral information are

  • point spectrometers,

  • hyperspectral line scan cameras and

  • multispectral color filter cameras.

Point spectrometers and hyperspectral line scan cameras use prisms or diffraction gratings to map the spectral dimension of the incident light onto a spatial dimension by exploiting the fact that prisms and gratings transmit light with different angles depending on its wavelength.

The resulting spectrally decomposed light is then captured by a sensor oriented in such a way so that one of its spatial coordinates encodes the wavelength of the incident light.

Point spectrometers capture the spectrum of a single measurement point whereas hyperspectral line scan cameras retain one dimension of the sensor for spatial information so that the spectrum of a line in the measurement field can be acquired.

Multispectral cameras sacrifice spatial resolution to gain some spectral resolution by employing larger filter arrays, e.g., \(5 \times 5\), similar to the red, green and blue filters of common color cameras.

So-called coded aperture spectral snapshot imaging (CASSI) cameras use computational imaging methods to undersample the high-dimensional spatio-spectral data and then computationally reconstruct the sought data by incorporating suitable prior information (i.e., regularizers).

They are called snapshot imaging systems, as they only require to capture a single measurement (i.e., a single image).

Typical CASSI setups modulate the image formation process in two ways: they spatially modulate the amplitude via binary coded aperture masks and they shear the image with respect to the present wavelengths via prisms or diffraction gratings.

Conventional RGB-camera#

The following figure shows the sampling scheme of a conventional RGB-camera for a single spatial coordinate \(x\) and the spectral coordinate \(\lambda\):

The modulation is performed by the RGB Bayer filter pattern on the sensor and the projection along the spectral dimension is performed by the sensor elements which integrate over the modulated spectra.

Let \(\mathbf{X}\in\mathbb{R}^{S\times N}\) contain the spectra with length \(S\) as columns for each of the \(N\) spatial positions and let \(\mathbf{x}\in \mathbb{R} ^{S\cdot N}\) denote its vectorized form.

The modulation can be modeled via a diagonal matrix \(\mathrm{diag}(\mathbf{p})\in \mathbb{R} ^{S\cdot N \times S\cdot N}\) with the modulation pattern \(\mathbf{p}\in \mathbb{R}^{S\cdot N} \) containing the spectral filter coefficients for every spatial and spectral sampling point.

The modulated spectra are then given by \(\mathrm{diag}(\mathbf{p})\mathbf{x}\).

The projection along the spectral dimension can be modeled via a matrix \(\boldsymbol{\Sigma}\in \mathbb{R} ^{N\times S\cdot N}\) containing blocks of ones of length \(S\) in every row that are shifted by \(S\) from row to row.

Hence, the overall forward model is given by:

\(\begin{align} \mathbf{y} &= \underbrace{\boldsymbol{\Sigma} \mathrm{diag}(\mathbf{p})}_{=:\mathbf{A}} \mathbf{x} \\ &= \mathbf{Ax} \,. \end{align}\)