Re-inventing the eye is the area where
we've had the most success
in the field of computer vision.
Over the past few decades,
we have created sensors and image processors that
match and in some ways exceed
the capabilities of the human eyes,
with larger more optically perfect lenses
and semiconductor pixels fabricated at
manometer scales the precision and
sensitivity of modern cameras
is nothing short of incredible.
After starting from one or more light sources,
reflecting of one or more surfaces
in the world and passing through
the camera's optics light
finally reaches the imaging sensor.
How are the photos from
the light sources arriving in the sensor converted
into the digital RGB values
that we observe in the digital image?
We need to develop a simple model that accounts for
the most important effects such as exposure,
nonlinear mappings, sampling and
aliasing and noise associated with the cameras.
Once the light from a scene reaches the camera
it must still pass through
the lens before reaching the sensor.
For many applications, it suffices to treat the lens as
an ideal pinhole that simply
projects all rays through a common center of projection.
However, if we want to deal with the issues such as
focus exposure, vignetting or aberration.
We must develop a more sophisticated model
which is where the study of optics comes in.
Light falling on an imaging sensor
is usually picked up by
an active sensing area
integrated for the duration of the exposure.
This is usually expressed as the shutter speed.
The two main kinds of sensors that are
used in still and video cameras today are charged
coupled devices or CCD and
complementary metal oxide on silicon or CMOS.
In CCD photons are
accumulated in each active well during the exposure time.
The main factors affecting the performance of
digital image sensors are shutter speed,
sampling pitch, chip size, analog gain,
sensor noise, and the quality
of analog-to-digital converter.
Let us look at these factors in more detail.
Here is a digital imaging pipeline
where the object is illuminated and is
captured using a digital camera
and then there is some kind of post-processing,
but our main focus will be on the digital camera.
The key components of a digital camera are
optics and the imaging sensor
as well as the image processors.
Moving away from the simple pinhole camera model,
digital cameras are equipped
with sophisticated lens optics.
Hence, it is important for us to
take them into account in understanding digital images.
Here is a visual that shows chromatic aberration
caused by a lens which needs to be accounted for.
Vignetting is another effect caused by
the lens optics which needs to be compensated for.
Light after passing through
the optics hits the imaging sensor.
Photoelectric effect is
the principle behind digital imaging.
Intensity at an image pixel is dependent on
several factors one of
the main factors being exposure time or shutter speed,
it is also dependent on
the quantum efficiency of the photo-sensor.
As you can see smaller sensor pixels
saturate faster than the larger sensor pixels.
Hence, we are going to list the pixel size chip size as
well as the sampling pitch as
the factors that affect the intensity and density pixel.
Finally, noise as the integral part
of any imaging process.
Shot noise and dark noise are
few key factors that affect
the intensity at an image pixel.
Let us look at the digital camera pipeline.
We have discussed optics,
so light travels through
the optics and it is controlled by the aperture,
the size of the aperture decides
how much light enters
and the shutter decides the exposure time,
the time for which
the image sensor is exposed to the light.
Finally, light reaches the sensor.
This is just the tip of
the iceberg in the digital camera pipeline.
There is a huge lineup of
processes that have to take place before you get to save.
The key processes we are going to explore are gain,
demosaicing, denoising, white balance,
gamma correction, and compression.
All these processes are going to be
covered in much more detail later on.
But just think about the progress that
has been made in digital imaging.
Just with a click of a button,
all these processes happen and finally,
you see the image saved on your camera.