Image processor: Difference between revisions

Revision as of 09:23, 16 August 2022

Latest revision as of 20:11, 23 May 2025

(22 intermediate revisions by 12 users not shown)

Line 2:

[[File:Nikon D90 - board 0 - Nikon Expeed EI-149-1769.jpg|thumb|[[EXPEED|Nikon EXPEED]], a [[system on a chip]] including an [[Color image pipeline|image processor]], [[Video_scaler#Video_processor|video processor]], [[digital signal processor]] (DSP) and a [[32-bit]] [[microcontroller]] controlling the chip]]

An '''image processor''', also known as an '''image processing engine''', '''image processing unit''' ('''IPU'''), or '''image signal processor''' ('''ISP'''), is a type of [[media processor]] or specialized [[digital signal processor]] (DSP) used for [[image processing]], in [[digital camera]]s or other devices.<ref>[http://www.robots.ox.ac.uk/~sjrob/Teaching/B4_SP/b4_sp.pdf DIGITAL SIGNAL & IMAGE PROCESSING]</ref><ref>[http://homepages.inf.ed.ac.uk/rbf/BOOKS/VERNON/Chap004.pdf Fundamentals of digital image processing]</ref>

Image processors often employ [[parallel computing]] even with [[Single instruction, multiple data|SIMD]] or [[Multiple instruction, multiple data|MIMD]] technologies to increase speed and efficiency.<ref>{{Cite journal |last1=Merigot |first1=Alain |last2=Petrosino |first2=Alfredo |date=2008-12-01 |title=Parallel processing for image and video processing: Issues and challenges |url=https://www.sciencedirect.com/science/article/abs/pii/S0167819108000999 |journal=Parallel Computing |volume=34 |issue=12 |pages=694–699 |doi=10.1016/j.parco.2008.09.009 |issn=0167-8191|url-access=subscription }}</ref> The [[digital image]] processing engine can perform a range of tasks.

To increase the system integration on [[Embedded system|embedded devices]], often it is a [[system on a chip]] with [[multi-core processor]] architecture.

==Function==

{{Unreferenced section|date=October 2017}}

=== Bayer transformation ===

The [[photodiode]]s employed in an [[image sensor]] are color-blind by nature: they can only record [[Shades of gray|shades of grey]]. To get [[color]] into the picture, they are covered with different color filters: [[red]], [[green]] and [[blue]] ([[RGB]]) according to the pattern designated by the [[Bayer filter]] - named after its inventor. As each photodiode records the color information for exactly one [[pixel]] of the image, without an image processor there would be a green pixel next to each red and blue pixel. (Actually, with most sensors there are two green for each blue and red diodes.)

This process, however, is quite complex, and involves a number of different operations. Its quality depends largely on the effectiveness of the [[Algorithm|algorithms]] applied to the raw data coming from the sensor. The mathematically manipulated data becomes the recorded photo file recorded.

=== Demosaicing ===

As stated above, the image processor evaluates the color and [[brightness]] data of a given pixel, compares them with the data from neighboring pixels, and then uses a [[demosaicing]] algorithm to produce an appropriate colourcolor and brightness value for the pixel.<ref>{{Cite journal |last=Chang |first=Lanlan |date=Mar 2006 |title=Hybrid color filter array demosaicking for effective artifact suppression |url=http://www3.ntu.edu.sg/home5/CHAN0069/JEI013003.pdf |journal=Journal of Electronic Imaging |volume=15 |issue=1 |pages=013003-1 - 013003-17|doi=10.1117/1.2183325 |bibcode=2006JEI....15a3003C |archive-url=https://web.archive.org/web/20091229063407/http://www3.ntu.edu.sg/home5/CHAN0069/JEI013003.pdf |archive-date=29 December 2009 }}</ref> The image processor also assesses the whole picture to guess at the correct distribution of [[Contrast (vision)|contrast]]. By adjusting the [[Gamma correction|gamma]] value (heightening or lowering the contrast range of an image's mid-tones), subtle tonal gradations, such as in [[human skin]] or the blue of the [[sky]], become much more realistic.

=== Noise reduction ===

[[Image noise|Noise]] is a phenomenon found in any [[Electronic circuit|electronic circuitry]]. In [[digital photography]] its effect is often visible as random spots of obviously wrong colourcolor in an otherwise smoothly-colouredcolored area. Noise increases with temperature and [[Exposure (photography)|exposure]] times. When higher [[ISO 5800:1987#Digital camera ISO speed and exposure index|ISO]] settings are chosen the electronic signal in the image sensor is amplified, which at the same time increases the noise level, leading to a lower [[signal-to-noise ratio]]. The image processor attempts to separate the noise from the image information and to remove it. This can be quite a challenge, as the image may contain areas with fine textures which, if treated as noise, may lose some of their definition.

=== Image sharpening ===

As the color and brightness values for each pixel are [[interpolated]] some [[image softeningsharpening]] is applied to even out any fuzziness that has occurred. To preserve the impression of [[Depth perception|depth]], clarity and fine details, the image processor must sharpen edges and contours. It therefore must [[Edge detection|detect edges]] correctly and reproduce them smoothly and without over-sharpening.

==Models==

Line 34:

*Google - [[Pixel Visual Core]]<ref>{{cite news|last1=Amadeo|first1=Ron|title=Surprise! The Pixel 2 is hiding a custom Google SoC for image processing|url=https://arstechnica.com/gadgets/2017/10/the-pixel-2-contains-a-custom-google-soc-the-pixel-visual-core/|access-date=19 October 2017|publisher=Ars Technica|date=17 October 2017}}</ref>

*HTC - ImageSense

*Intel - IPU<ref>{{Cite web |title=7.8. Intel Image Processing Unit 6 (IPU6) Input System driver — The Linux Kernel documentation |url=https://docs.kernel.org/admin-guide/media/ipu6-isys.html |access-date=2024-08-30 |website=docs.kernel.org}}</ref>

*MediaTek - [https://www.mediatek.com/features/imagiq-camera Imagiq]

*MediaTek - Imagiq

*Minolta / Konica Minolta - [[SUPHEED]] with CxProcess

*Leica - MAESTRO (based on Fujitsu [[Milbeaut]])<ref>[http://www.fujitsu.com/hk/news/pr/20080926.html Fujitsu Microelectronics-Leica's Image Processing System Solution For High-End DSLR] {{Webarchive|url=https://web.archive.org/web/20081007190028/http://www.fujitsu.com/hk/news/pr/20080926.html |date=208-10-04 }}</ref>

*Nikon - [[Expeed]] (based on Fujitsu [[Milbeaut]])<ref>[http://www.dslrbodies.com/newsviews/milbeaut-and-expeed.html Milbeaut and EXPEED] {{Webarchive|url=https://web.archive.org/web/20160521154952/http://www.dslrbodies.com/newsviews/milbeaut-and-expeed.html |date=2016-05-21 }} byThom</ref>

*Olympus - TruePic (based on Panasonic [[MN103]]/MN103S)

*OPPO - [https://www.oppo.com/en/newsroom/press/oppo-imaging-npu-marisilicon-x/ MariSilicon] X

*Panasonic - [[Venus Engine]] (based on Panasonic [[MN103]]/MN103S)

*Pentax - PRIME (Pentax Real IMage Engine) (newer variants based on Fujitsu [[Milbeaut]])

*Qualcomm - [[Qualcomm Spectra]] (based on [[Qualcomm Snapdragon]])

*Ricoh - GR engine (GR digital), Smooth Imaging Engine

*Samsung - DRIMe (based on [[Samsung Electronics|Samsung]] [[Exynos]])

Line 51 ⟶ 52:

*Sony - [[Bionz]]

*THine - THP series [https://www.thine.co.jp/en/products/camera_processor/Image-Signal-Processor/] with compatible SDK Kit for developing firmware [https://www.thinesolutions.com/camera-development-kit-cdk]

*UNISOC - [https://www.youtube.com/watch?v=VREjRuI73jU Vivimagic]

*Vivo - [https://www.gizmochina.com/2021/09/06/vivo-v1-isp-chip-launched/ Vivo V1]

=== Speed ===

With the ever-higher pixel count in image sensors, the image processor's speed becomes more critical: [[Photographer|photographers]] don't want to wait for the camera's image processor to complete its job before they can carry on shooting - they don't even want to notice some processing is going on inside the camera. Therefore, image processors must be optimised to cope with more data in the same or even a shorter period of time.

== Software ==

[[libcamera]] is a software library that supports using image signal processors for the capture of pictures.

== See also ==