Image processor: Difference between revisions

Revision as of 21:17, 31 July 2024

Latest revision as of 20:11, 23 May 2025

(12 intermediate revisions by 7 users not shown)

Line 2:

[[File:Nikon D90 - board 0 - Nikon Expeed EI-149-1769.jpg|thumb|[[EXPEED|Nikon EXPEED]], a [[system on a chip]] including an [[Color image pipeline|image processor]], [[Video_scaler#Video_processor|video processor]], [[digital signal processor]] (DSP) and a [[32-bit]] [[microcontroller]] controlling the chip]]

An '''image processor''', also known as an '''image processing engine''', '''image processing unit''' ('''IPU'''), or '''image signal processor''' ('''ISP'''), is a type of [[media processor]] or specialized [[digital signal processor]] (DSP) used for [[image processing]], in [[digital camera]]s or other devices.<ref>[http://www.robots.ox.ac.uk/~sjrob/Teaching/B4_SP/b4_sp.pdf DIGITAL SIGNAL & IMAGE PROCESSING]</ref><ref>[http://homepages.inf.ed.ac.uk/rbf/BOOKS/VERNON/Chap004.pdf Fundamentals of digital image processing]</ref>

Image processors often employ [[parallel computing]] even with [[Single instruction, multiple data|SIMD]] or [[Multiple instruction, multiple data|MIMD]] technologies to increase speed and efficiency.<ref>{{Cite journal |last1=Merigot |first1=Alain |last2=Petrosino |first2=Alfredo |date=2008-12-01 |title=Parallel processing for image and video processing: Issues and challenges |url=https://www.sciencedirect.com/science/article/abs/pii/S0167819108000999 |journal=Parallel Computing |volume=34 |issue=12 |pages=694–699 |doi=10.1016/j.parco.2008.09.009 |issn=0167-8191|url-access=subscription }}</ref> The [[digital image]] processing engine can perform a range of tasks.

To increase the system integration on [[Embedded system|embedded devices]], often it is a [[system on a chip]] with [[multi-core processor]] architecture.

Line 8:

{{Unreferenced section|date=October 2017}}

=== Bayer transformation ===

The [[photodiode]]s employed in an [[image sensor]] are color-blind by nature: they can only record [[Shades of gray|shades of grey]]. To get [[color]] into the picture, they are covered with different color filters: [[red]], [[green]] and [[blue]] ([[RGB]]) according to the pattern designated by the [[Bayer filter]] - named after its inventor. As each photodiode records the color information for exactly one [[pixel]] of the image, without an image processor there would be a green pixel next to each red and blue pixel. (Actually, with most [[Sensor|sensors]] there are two green for each blue and red diode.)

This process, however, is quite complex, and involves a number of different operations. Its quality depends largely on the effectiveness of the [[Algorithm|algorithms]] applied to the raw data coming from the sensor. The mathematically manipulated data becomes the recorded photo file recorded.

=== Demosaicing ===

As stated above, the image processor evaluates the color and [[brightness]] data of a given pixel, compares them with the data from neighboring pixels, and then uses a [[demosaicing]] algorithm to produce an appropriate colourcolor and brightness value for the pixel.<ref>{{Cite journal |last=Chang |first=Lanlan |date=Mar 2006 |title=Hybrid color filter array demosaicking for effective artifact suppression |url=http://www3.ntu.edu.sg/home5/CHAN0069/JEI013003.pdf |journal=Journal of Electronic Imaging |volume=15 |issue=1 |pages=013003-1 - 013003-17|doi=10.1117/1.2183325 |bibcode=2006JEI....15a3003C |archive-url=https://web.archive.org/web/20091229063407/http://www3.ntu.edu.sg/home5/CHAN0069/JEI013003.pdf |archive-date=29 December 2009 }}</ref> The image processor also assesses the whole picture to guess at the correct distribution of [[Contrast (vision)|contrast]]. By adjusting the [[Gamma correction|gamma]] value (heightening or lowering the contrast range of an image's mid-tones), subtle tonal gradations, such as in [[human skin]] or the blue of the [[sky]], become much more realistic.

=== Noise reduction ===

[[Image noise|Noise]] is a phenomenon found in any [[Electronic circuit|electronic circuitry]]. In [[digital photography]] its effect is often visible as random spots of obviously wrong colourcolor in an otherwise smoothly-colouredcolored area. Noise increases with temperature and [[Exposure (photography)|exposure]] times. When higher [[ISO 5800:1987#Digital camera ISO speed and exposure index|ISO]] settings are chosen the electronic signal in the image sensor is amplified, which at the same time increases the noise level, leading to a lower [[signal-to-noise ratio]]. The image processor attempts to separate the noise from the image information and to remove it. This can be quite a challenge, as the image may contain areas with fine textures which, if treated as noise, may lose some of their definition.

=== Image sharpening ===

As the color and brightness values for each pixel are [[interpolated]] some [[image sharpening]] is applied to even out any fuzziness that has occurred. To preserve the impression of [[Depth perception|depth]], clarity and fine details, the image processor must sharpen edges and contours. It therefore must [[Edge detection|detect edges]] correctly and reproduce them smoothly and without over-sharpening.

==Models==

Line 34:

*Google - [[Pixel Visual Core]]<ref>{{cite news|last1=Amadeo|first1=Ron|title=Surprise! The Pixel 2 is hiding a custom Google SoC for image processing|url=https://arstechnica.com/gadgets/2017/10/the-pixel-2-contains-a-custom-google-soc-the-pixel-visual-core/|access-date=19 October 2017|publisher=Ars Technica|date=17 October 2017}}</ref>

*HTC - ImageSense

*Intel - IPU<ref>{{Cite web |title=7.8. Intel Image Processing Unit 6 (IPU6) Input System driver — The Linux Kernel documentation |url=https://docs.kernel.org/admin-guide/media/ipu6-isys.html |access-date=2024-08-30 |website=docs.kernel.org}}</ref>

*MediaTek - Imagiq

*Minolta / Konica Minolta - [[SUPHEED]] with CxProcess

Line 42 ⟶ 43:

*Panasonic - [[Venus Engine]] (based on Panasonic [[MN103]]/MN103S)

*Pentax - PRIME (Pentax Real IMage Engine) (newer variants based on Fujitsu [[Milbeaut]])

*Qualcomm - [[Qualcomm Spectra]] (based on [[Qualcomm Snapdragon]])

*Ricoh - GR engine (GR digital), Smooth Imaging Engine

*Samsung - DRIMe (based on [[Samsung Electronics|Samsung]] [[Exynos]])

Line 54 ⟶ 55:

=== Speed ===

With the ever-higher pixel count in image sensors, the image processor's speed becomes more critical: [[Photographer|photographers]] don't want to wait for the camera's image processor to complete its job before they can carry on shooting - they don't even want to notice some processing is going on inside the camera. Therefore, image processors must be optimised to cope with more data in the same or even a shorter period of time.

== Software ==