Oversampled binary image sensor: Difference between revisions

Revision as of 06:11, 11 December 2020

Latest revision as of 14:42, 8 June 2025

(13 intermediate revisions by 13 users not shown)

Line 1:

{{short description|none}}

{{short description|Image sensor with non-linear response capabilities reminiscent of traditional photographic film}}

An '''oversampled binary image sensor''' is an [[image sensor]] with non-linear response capabilities reminiscent of traditional [[photographic film]].<ref name="Gigavision">[httphttps://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=4959778&contentType=Conference+Publications L. Sbaiz, F. Yang, E. Charbon, S. Süsstrunk and M. Vetterli, The Gigavision Camera, ''Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)'', pp. 1093 - 1096, 2009.]</ref><ref name="bitsfromphotons">[httphttps://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6104150 F. Yang, Y.M. Lu, L. Saibz and M. Vetterli, Bits from Photons: Oversampled Image Acquisition Using Binary Poisson Statistics, ''IEEE Transactions on Image Processing'', vol. 21, issue 4, pp.1421-1436, 2012.]</ref> Each pixel in the sensor has a binary response, giving only a one-bit quantized measurement of the local light intensity. The response function of the image sensor is non-linear and similar to a logarithmic function, which makes the sensor suitable for [[high dynamic range imaging]].<ref name="Gigavision" />

==Working principle==

Before the advent of digital image sensors, photography, for the most part of its history, used film to record light information. At the heart of every photographic film are a large number of light-sensitive grains of [[silver-halide]] crystals.<ref name="filmphotography">T. H. James, The Theory of The Photographic Process, 4th ed., New York: Macmillan Publishing Co., Inc., 1977.</ref> During exposure, each micron-sized grain has a binary fate: Either it is struck by some incident photons and becomes "exposed", or it is missed by the photon bombardment and remains "unexposed". In the subsequent film development process, exposed grains, due to their altered chemical properties, are converted to silver metal, contributing to opaque spots on the film; unexposed grains are washed away in a chemical bath, leaving behind the transparent regions on the film. Thus, in essence, photographic film is a binary imaging medium, using local densities of opaque silver grains to encode the original light intensity information. Thanks to the small size and large number of these grains, one hardly notices this quantized nature of film when viewing it at a distance, observing only a continuous gray tone.

The oversampled binary image sensor is reminiscent of photographic film. Each pixel in the sensor has a binary response, giving only a one-bit quantized measurement of the local light intensity. At the start of the exposure period, all pixels are set to 0. A pixel is then set to 1 if the number of photons reaching it during the exposure is at least equal to a given threshold ''q''. One way to build such binary sensors is to modify standard memory chip technology, where each memory bit cell is designed to be sensitive to visible light.<ref name="RAMsensor">S. A. Ciarcia, A 64K-bit dynamic RAM chip is the visual sensor in this digital image camera, ''Byte Magazine'', pp.21-31, Sep. 1983.</ref> With current CMOS technology, the level of integration of such systems can exceed 10⁹~10¹⁰ (i.e., 1 giga to 10 giga) pixels per chip. In this case, the corresponding pixel sizes (around 50~nm <ref name="DRAM">Y. K. Park, S. H. Lee, J. W. Lee et al., Fully integrated 56nm DRAM technology for 1Gb DRAM, in ''IEEE Symposium on VLSI Technology'', Kyoto, Japan, Jun. 2007.</ref>) are far below the diffraction limit of light, and thus the image sensor is ''[[oversampling]]'' the optical resolution of the light field. Intuitively, one can exploit this spatial redundancy to compensate for the information loss due to one-bit quantizations, as is classic in oversampling [[delta-sigma converter]] conversionss.<ref name="ODSC">J. C. Candy and G. C. Temes, Oversamling Delta-Sigma Data Converters-Theory, Design and Simulation. New York, NY: IEEE Press, 1992.</ref>

Building a binary sensor that emulates the photographic film process was first envisioned by [[Eric Fossum|Fossum]],<ref name="DFS">[[Eric Fossum|E. R. Fossum]], What to do with sub-diffraction-limit (SDL) pixels? - A proposal for a gigapixel digital film sensor (DFS), in ''IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors'', Nagano, Japan, Jun. 2005, pp.214-217.</ref> who coined the name ''digital film sensor'' (now referred to as a ''quanta image sensor''<ref>E.R. Fossum, J. Ma, S. Masoodian, L. Anzagira, and R. Zizza, ''The quanta image sensor: every photon counts'', MDPI Sensors, vol. 16, no. 8, 1260; August 2016. {{doi:|10.3390/s16081260|doi-access=free}} (Special Issue on Photon-Counting Image Sensors)</ref>). The original motivation was mainly out of technical necessity. The [[miniaturization]] of camera systems calls for the continuous shrinking of pixel sizes. At a certain point, however, the limited [[full-well capacity]] (i.e., the maximum photon-electrons a pixel can hold) of small pixels becomes a bottleneck, yielding very low [[signal-to-noise ratio]]s (SNRs) and poor [[dynamic range]]s. In contrast, a binary sensor whose pixels need to detect only a few photon-electrons around a small threshold ''q'' has much less requirement for full-well capacities, allowing pixel sizes to shrink further.

==Imaging model==

===The lensLens===

[[File:Oversampled binary sensor imaging model.jpg|thumb|right|250px|Fig.1 The imaging model. The simplified architecture of a diffraction-limited imaging system. Incident light field <math>\lambda_0(x)</math> passes through an optical lens, which acts like a linear system with a diffraction-limited point spread function (PSF). The result is a smoothed light field <math>\lambda(x)</math>, which is subsequently captured by the image sensor.]]

Consider a simplified camera model shown in Fig.1. The <math>\lambda_0(x)</math> is the incoming light intensity field. By assuming that light intensities remain constant within a short exposure period, the field can be modeled as only a function of the spatial variable <math>x</math>. After passing through the optical system, the original light field <math>\lambda_0(x)</math> gets filtered by the lens, which acts like a linear system with a given [[impulse response]]. Due to imperfections (e.g., aberrations) in the lens, the impulse response, a.k.a. the [[point spread function]] (PSF) of the optical system, cannot be a Dirac delta, thus, imposing a limit on the resolution of the observable light field. However, a more fundamental physical limit is due to light [[diffraction]].<ref name="Optics">M. Born and E. Wolf, ''[[Principles of Optics]]'', 7th ed. Cambridge: Cambridge University Press, 1999</ref> As a result, even if the lens is ideal, the PSF is still unavoidably a small blurry spot. In optics, such diffraction-limited spot is often called the [[Airy disk]],<ref name="Optics"/> whose radius <math>R_a</math> can be computed as

:<math>R_a = 1.22 \, w f,</math>

Line 19:

where <math>w</math> is the [[wavelength]] of the light and <math>f</math> is the [[F-number]] of the optical system. Due to the [[lowpass]] (smoothing) nature of the PSF, the resulting <math>\lambda(x)</math> has a finite spatial-resolution, i.e., it has a finite number of [[Degrees of freedom (physics and chemistry)|degrees of freedom]] per unit space.

===The sensorSensor===

[[File:binary sensor model.svg|thumb|right|480px|Fig.2 The model of the binary image sensor. The pixels (shown as "buckets") collect photons, the numbers of which are compared against a quantization threshold ''q''. In the figure, we illustrate the case when ''q'' = 2. The pixel outputs are binary: <math>b_m = 1</math> (i.e., white pixels) if there are at least two photons received by the pixel; otherwise, <math>b_m = 0</math> (i.e., gray pixels).]]

Fig.2 illustrates the binary sensor model. The <math>s_m</math> denote the exposure values accumulated by the sensor pixels. Depending on the local values of <math>s_m</math>, each pixel (depicted as "buckets" in the figure) collects a different number of photons hitting on its surface. <math>y_m</math> is the number of photons impinging on the surface of the <math>m</math>th pixel during an [[exposure (photography)|exposure]] period. The relation between <math>s_m</math> and the photon count <math>y_m</math> is stochastic. More specifically, <math>y_m</math> can be modeled as realizations of a Poisson [[random variable]], whose intensity parameter is equal to <math>s_m</math>,

As a [[photosensitive]] device, each pixel in the image sensor converts photons to electrical signals, whose amplitude is proportional to the number of photons impinging on that pixel. In a conventional sensor design, the analog electrical signals are then quantized by an [[Analog-to-digital converter|A/D converter]] into 8 to 14 bits (usually the more bits the better). But in the binary sensor, the quantizer is 1 bit. In Fig.2, <math>b_m</math> is the quantized output of the <math>m</math>th pixel. Since the photon counts <math>y_m</math> are drawn from random variables, so are the binary sensor output <math>b_m</math>.

===Spatial and temporal oversampling===

If it is allowed to have temporal oversampling, i.e., taking multiple consecutive and independent frames without changing the total exposure time <math>\tau</math>, the performance of the binary sensor is equivalent to the sensor with same number of spatial oversampling under certain condition.<ref name="bitsfromphotons" /> It means that people can make trade off between spatial oversampling and temporal oversampling. This is quite important, since technology usually gives limitation on the size of the pixels and the exposure time.

==Advantages over traditional sensors==

Line 33:

==Reconstruction==

[[File:SPAD EPFL BINARY IMAGES.png|thumb|right|268px|Fig.4 Reconstructing an image from the binary measurements taken by a SPAD<ref name="SPADS">L. Carrara, C. Niclass, N. Scheidegger, H. Shea, and E. Charbon, A gamma, X-ray and high energy proton radiation-tolerant CMOS image sensor for space applications, in ''IEEE International Solid-State Circuits Conference'', Feb. 2009, pp.40-41.</ref> sensor, with a spatial resolution of 32×32 pixels. The final image (lower-right corner) is obtained by incorporating 4096 consecutive frames, 11 of which are shown in the figure.]]

One of the most important challenges with the use of an oversampled binary image sensor is the reconstruction of the light intensity <math>\lambda(x)</math> from the binary measurement <math>b_m</math>. [[Maximum likelihood|Maximum likelihood estimation]] can be used for solving this problem.<ref name="bitsfromphotons" /> Fig. 4 shows the results of reconstructing the light intensity from 4096 binary images taken by [[single photon avalanche diode]]s (SPADs) camera.<ref name="SPADS" /> A better reconstruction quality with fewer temporal measurements and faster, hardware friendly implementation, can be achieved by more sophisticated algorithms.<ref>{{Cite journal|title = Image reconstruction from dense binary pixels|arxiv = 1512.01774|journal = Signal Processing with Adaptive Sparse Structured Representations (SPARS 2015)|date = 2015-12-06|first = Or|last = Litany|first2 = Tal|last2 = Remez|first3 = Alex|last3 = Bronstein|bibcode = 2015arXiv151201774L}}</ref>