Digital Photo Processes

-- Introduction -- Lens & image -- A/D conversion -- Lumix & Leica --  Sharpness -- CCD noise1 -- CCD noise2 -- 5 years later -- 

Analog to digital conversion

The CCD or CMOS sensor in a digital camera converts light photons into electrons. Each of the millions of sensor pixels accumulates an electric charge that is proportional to the number of photons which it collects. Each pixel has a red, green or blue filter in front of it. The filters are arranged in a Bayer mosaic pattern consisting of two green, one red and one blue filter and a micro lens for each. After a picture has been taken and the sensor has been exposed to light, the charge from each pixel is read out and converted to a voltage. The pixel charge is reset to zero, i.e. to black. The conversion process adds electrical circuit noise to the light voltage. Further noise is added when the voltage is amplified in order to work with the camera in low light at increased ISO. The voltage which represents the luminance of each pixel is then converted into a binary number by a 12-bit analog-to-digital converter. Information about a pixel's x and y location is added, as are metadata about the sensor's color filters and chosen camera white balance setting. This becomes the RAW data file that is stored in the camera memory on a removable SD card, for example.

The 12-bit pixel luminance data D1 from the ADC are converted and compressed inside the camera into a JPG file, D2, with 8-bit Red, Green and Blue luminance information for each pixel. This creates a much smaller data file than the corresponding RAW file, D3. It saves memory space, and more importantly allows a much faster rate of taking pictures. The JPG compression is optimized by the user chosen setting for its Quality, Contrast, Sharpness, Saturation, Noise reduction and White balance. Information is always lost in the process. Especially White balance, Noise reduction and Sharpening cannot be undone in post-processing of the JPG file, though they can be changed..

The light which is reflected from a scene can span a wide range of intensities, like 10000:1 between highlights and shadows. A large CCD sensor may be able to capture that range without saturating or clipping the highlights and losing shadow detail in its noise. The output voltages from the various pixels would simultaneously vary linearly over that same 10000:1 range. The human eye and brain system does not work the same way and compresses that wide scene range into a much smaller perceptual range depending upon where the eye is focused at the moment..

The CCD transfer function relates photon input to electron output. It is shown with log-log scales to cover the wide ranges.  For example, for an input light range 1 from dark to light, D-L,  the output voltage changes from D1 to L1. Normal exposure for an average of 18% gray scene reflectivity is set to fall slightly above the middle of the D1-L1 range.  At low light levels electronic circuit noise X-Y is dominant. At high light levels the photon noise Y-Z, which is proportional to the square root of the number of collected photons, takes over. Thus the signal-to-noise ratio is governed by electronic noise at low illumination and by photon noise at high illumination in this example.  Larger sensors with larger pixel areas can collect more photons before a pixel saturates. They produce a larger signal output voltage, while their electronic noise is not that different from a smaller sensor. Thus photon noise determines the ultimate S/N ratio that can be realized in larger sensor cameras.

If the same scene D-L as 1 above is photographed with a higher ISO setting, then the number of photons impinging upon the sensor is reduced. To obtain the same 18% gray output voltage level the gain after the CCD charge conversion is increased. This also increases the noise and degrades S/N.  At ISO 100 both circuit, X-Y, and photon noise, Y-Z, determine S/N depending upon the relative number of photons collected. At ISO 400 the S/N ratio is solely determined by circuit noise as the lowest line shows. Doubling ISO degrades S/N by a factor of two in the circuit noise range, but only by a factor of square root of two in the photon noise dominated range.

The large range of output voltages V1 that is generated by the photon conversion process is, for example, digitized by a 12-bit linear ADC for storage as a RAW file. The JPG file, though, has only 8-bit luminance information, corresponding to a light range from 0 to 255. It is derived from the 0 to 4095 range of the 12-bit ADC output D1. In a direct 12 bit to 8 bit conversion the first 2048 levels would be represented by the first 95 levels of the 0 to 255 range. The next 1024 levels would be represented by 60 levels and so on as in the table below. This type of conversion would assign too many levels to the highlights of a picture and not enough to the dark regions. As a trade-off the conversion is done with a slope of 1/2.2 = 0.45 which is called gamma = 2.2. The last column in the table shows the corresponding distribution and the graph shows the 12 to 8 conversion on log-log scale. 

An 8-bit JPG file is also used to generate the luminance histogram on the camera's LCD and therefore under-represent what is captured in the RAW file in the critical highlight area to the right edge of the histogram.

Input level log2	12-bit level diff. 12/12 slope	8-bit level diff. 8/12 slope	8-bit level diff. 1/2.2 slope	(Source data: pixel-stats1.xls)
12	2048	95	69
11	1024	60	50
10	512	38	37
9	256	24	27
8	128	15	20
7	64	9	14
6	32	6	11
5	16	4	8
4	8	2	6
3	4	1	4
2	2	1	3

Data file post-processing and rendering

The RAW or JPG picture files from the camera are further processed in a computer for eventual presentation on a CRT or LCD display, for a paper print, or for the Internet. Each has different requirements. The LCD can render a wider range of luminance than the reflected light from a print and the gamma compression is undone. Their color spaces are different. The Internet file has to be small for speedy transmission, low storage requirements and to fit the allocated size in the display. 

The RAW file with its 12-bit luminance and Metadata information must first be converted into an 8-bit RGB and Luminance format which can then be rendered in standard fashion for the different final output displays.  The in-camera white balance setting had no effect upon the pixel luminance data and is only carried over as information that can be used in rendering the RAW file. Color and lighting parameters can be freely adjusted without destroying the original data. Only noise reduction and sharpening correlate adjacent pixels in a non-reversible way. Raw converters differ in their capabilities and performance for rendering the original data file.  The JPG file has been rendered in-camera. It can still be changed in post-processing, but picture pixel information has already been combined or lost and cannot be retrieved.

Measured Light-to-Data transfer functions

Using a constant intensity light source, and then taking pictures of it at constant aperture and stepped shutter times, gives a series of photographs which can then be evaluated numerically in post-processing. Over 40 RAW files were collected for a Lumix FZ50 camera. They were converted in ACR and then an area of 350 x 350 pixels was analyzed for the mean value of the 8-bit RGB data and the standard deviation of their luminance.

f:3.2 and 1/4 s, 1/6 s, 1/15 s, 1/25 s at 0 relative exposure  Panasonic Lumix FZ50

The luminance output from ACR has an 8-bit range, 0 to 255. The exposure scale is in 1-stop steps relative to normal exposure, e.g. f:3.2 and 1/4 s for ISO 100. The transfer function has a leaning S-shape as it bends into CCD saturation and clipping at high exposure and into darkness and electronic noise at low exposure. The curves for different ISO values look similar in this log-lin presentation.  The 8-bit data are already compressed approximately like to the gamma = 2.2 curve which was discussed above and is superimposed here. The desired shape of the S-curve is implicitly specified by the camera manufacturer in the RAW file Metadata and imbedded in the camera's JPG output file.

The four S-curves above are the result of one way of plotting the measured data points. When plotted on log-log scales and against the actual exposure times, then their shape looks different, see below, and reveals low luminance level information more clearly. In particular, the signal-to-noise ratio at different ISO values can be seen. S/N is the ratio of the luminance mean value to its standard deviation. Even with the measurement uncertainties involved, it can be seen that S/N has a slope of 1. This indicates that the noise is additive as in the circuit noise region discussed above. The small sensor in the FZ50 is far from being photon noise limited. 

f:3.2

f:3.2

Plotting the same measured data again on log-log scales, but against relative exposure, highlights changes in the shape of the S-curve and the degradation of S/N with increasing ISO. A relative exposure of 0 below corresponds to 1/4 s for ISO 100, 1/6 s for ISO 200, 1/15 s for ISO 400 and 1/25 s for ISO 800 in the two graphs above. This data presentation describes the change in luminance and S/N when a picture is taken of a scene for which the light ranges from -6 to +2 stops and the average corresponds to 18% gray at 0.   

f:3.2, 1/4 s	f:3.2, 1/15 s
f:3.2, 1/6 s	f:3.2, 1/25 s

It would take more experience than I have to relate the measured curves and their shape to details of the final image quality. Note, though, the change in the slope of the luminance curves below -3 stops. It becomes shallower as ISO increases. Contrast and blackness are lost as noise takes over. The effect will be different when viewed on an LCD with high contrast ratio or when a color print is made on a specific art paper. The curves illustrate the well-known degradation that can be expected from increases in ISO. 
(Source data: S-N-tests1.xls)

The analog to digital conversion process description above should have made it clear that RAW capture is essential in order to minimze data manipulation in a 3 x 8-bit RGB preprocessed data space. The highest image quality is obtained if the ACR converter is used to optimize all the various image parameters before generating 8-bit RGB data for further processing in PS Elements 6. No such conversion takes place in Adobe Photoshop Lightroom and the RAW data are processed with full dynamic range. Unfortunately I could not find access to the mean and standard deviation values in 12-bit space to see if the 8-bit graphs above misrepresent the the inherent capability of the camera. I doubt it though. 

References:

1 - Making the Transition from Film to Digital, AdobeTechnical Paper
2 - Understanding Digital Raw Capture, Adobe Whitepaper
3 - Raw Capture, Linear Gamma, and Exposure, Adobe Whitepaper
4 - Rendering the Print: the Art of Photography, Adobe Technical Paper
5 - CCD Image Sensor Noise Sources, Kodak Application Note
6 - Introduction to CMOS Image Sensors, Optical Microscopy Primer
7 - CCD Noise Sources and Signal-to-Noise Ratio, Hamamatsu Review Articles
8 - CCD Signal-to-Noise Ratio, Nikon MicroscopyU

-- Introduction -- Lens & image -- A/D conversion -- Lumix & Leica --  Sharpness -- CCD noise1 -- CCD noise2 -- 5 years later --