Neural processing unit: Difference between revisions

Revision as of 15:23, 6 May 2025

Latest revision as of 20:06, 8 August 2025

(24 intermediate revisions by 14 users not shown)

Line 3:

{{Use mdy dates|date=October 2021}}

A '''neural processing unit''' ('''NPU'''), also known as '''AI accelerator''' or '''deep learning processor,''', is a class of specialized [[hardware acceleration|hardware accelerator]]<ref>{{cite web |url=https://www.v3.co.uk/v3-uk/news/3014293/intel-unveils-movidius-compute-stick-usb-ai-accelerator |title=Intel unveils Movidius Compute Stick USB AI Accelerator |date=July 21, 2017 |access-date=August 11, 2017 |archive-url=https://web.archive.org/web/20170811193632/https://www.v3.co.uk/v3-uk/news/3014293/intel-unveils-movidius-compute-stick-usb-ai-accelerator |archive-date=August 11, 2017 }}</ref> or computer system<ref>{{cite web |url=https://insidehpc.com/2017/06/inspurs-unveils-gx4-ai-accelerator/ |title=Inspurs unveils GX4 AI Accelerator |date=June 21, 2017}}</ref><ref>{{citation |title=Neural Magic raises $15 million to boost AI inferencing speed on off-the-shelf processors |last=Wiggers |first=Kyle |date=November 6, 2019 |url=https://venturebeat.com/2019/11/06/neural-magic-raises-15-million-to-boost-ai-training-speed-on-off-the-shelf-processors/ |publication-date=November 6, 2019 |orig-date=2019 |archive-url=https://web.archive.org/web/20200306120524/https://venturebeat.com/2019/11/06/neural-magic-raises-15-million-to-boost-ai-training-speed-on-off-the-shelf-processors/ |archive-date=March 6, 2020 |access-date=March 14, 2020}}</ref> designed to accelerate [[artificial intelligence]] (AI) and [[machine learning]] applications, including [[artificial neural network]]s and [[computer vision]].

==Use==

They can be used either to efficiently execute already trained AI models (inference) or for training AI models. Typical applications include [[algorithm]]s for [[robotics]], [[Internet of Things]], and other [[data (computing)|data]]-intensive or sensor-driven tasks.<ref>{{cite web |url=https://www.eetimes.com/google-designing-ai-processors/ |title=Google Designing AI Processors|date=May 18, 2016 }} Google using its own AI accelerators.</ref> They are often [[Manycore processor|manycore]] designs and generally focus on [[precision (computer science)|low-precision]] arithmetic, novel [[dataflow architecture]]s or [[in-memory computing]] capability. {{As of|2024}}, a typical AI [[integrated circuit]] chip [[transistor count|contains tens of billions]] of [[MOSFET]]s.<ref>{{cite web|url=https://www.datacenterdynamics.com/en/news/nvidia-reveals-new-hopper-h100-gpu-with-80-billion-transistors/|title=Nvidia reveals new Hopper H100 GPU, with 80 billion transistors|last=Moss|first=Sebastian|date=2022-03-23|website=Data Center Dynamics|access-date=2024-01-30}}</ref>

Their purpose is either to efficiently execute already trained AI models (inference) or to train AI models. Their applications include [[algorithm]]s for [[robotics]], [[Internet of things]], and [[data (computing)|data]]-intensive or sensor-driven tasks.<ref>{{cite web |url=https://www.eetimes.com/google-designing-ai-processors/ |title=Google Designing AI Processors|date=May 18, 2016 }} Google using its own AI accelerators.</ref> They are often [[Manycore processor|manycore]] or [[Spatial architecture|spatial]] designs and focus on [[precision (computer science)|low-precision]] arithmetic, novel [[dataflow architecture]]s, or [[in-memory computing]] capability. {{As of|2024}}, a typical datacenter-grade AI [[integrated circuit]] chip, the H100 GPU, [[transistor count|contains tens of billions]] of [[MOSFET]]s.<ref>{{cite web|url=https://www.datacenterdynamics.com/en/news/nvidia-reveals-new-hopper-h100-gpu-with-80-billion-transistors/|title=Nvidia reveals new Hopper H100 GPU, with 80 billion transistors|last=Moss|first=Sebastian|date=2022-03-23|website=Data Center Dynamics|access-date=2024-01-30}}</ref>

=== Consumer devices ===

AI accelerators are used in mobile devices such as Apple [[iPhone]]s and [[Huawei]] cellphones,<ref>{{Cite web|url=https://consumer.huawei.com/en/press/news/2017/ifa2017-kirin970|title=HUAWEI Reveals the Future of Mobile AI at IFA}}</ref> and personal computers such as [[Intel]] laptops,<ref>{{Cite web|url=https://www.intel.com/content/www/us/en/newsroom/news/intels-lunar-lake-processors-arriving-q3-2024.html|title=Intel's Lunar Lake Processors Arriving Q3 2024|website=Intel|date=May 20, 2024 }}</ref> [[AMD]] laptops<ref>{{cite web|title=AMD XDNA Architecture|url=https://www.amd.com/en/technologies/xdna.html}}</ref> and [[Apple silicon]] [[Mac (computer)|Macs]].<ref>{{Cite web |title=Deploying Transformers on the Apple Neural Engine |url=https://machinelearning.apple.com/research/neural-engine-transformers |access-date=2023-08-24 |website=Apple Machine Learning Research |language=en-US}}</ref> Accelerators are used in [[cloud computing]] servers, including [[tensor processing unit]]s (TPU) in [[Google Cloud Platform]]<ref>{{Cite journal|date=2017-06-24|title=In-Datacenter Performance Analysis of a Tensor Processing Unit|journal=ACM SIGARCH Computer Architecture News|volume=45|issue=2|pages=1–12|language=EN|doi=10.1145/3140659.3080246|doi-access=free |last1=Jouppi |first1=Norman P. |last2=Young |first2=Cliff |last3=Patil |first3=Nishant |last4=Patterson |first4=David |last5=Agrawal |first5=Gaurav |last6=Bajwa |first6=Raminder |last7=Bates |first7=Sarah |last8=Bhatia |first8=Suresh |last9=Boden |first9=Nan |last10=Borchers |first10=Al |last11=Boyle |first11=Rick |last12=Cantin |first12=Pierre-luc |last13=Chao |first13=Clifford |last14=Clark |first14=Chris |last15=Coriell |first15=Jeremy |last16=Daley |first16=Mike |last17=Dau |first17=Matt |last18=Dean |first18=Jeffrey |last19=Gelb |first19=Ben |last20=Ghaemmaghami |first20=Tara Vazir |last21=Gottipati |first21=Rajendra |last22=Gulland |first22=William |last23=Hagmann |first23=Robert |last24=Ho |first24=C. Richard |last25=Hogberg |first25=Doug |last26=Hu |first26=John |last27=Hundt |first27=Robert |last28=Hurt |first28=Dan |last29=Ibarz |first29=Julian |last30=Jaffey |first30=Aaron |display-authors=1 |arxiv=1704.04760 }}</ref> and [[Trainium]] and [[Inferentia]] chips in [[Amazon Web Services]].<ref>{{cite web | title = How silicon innovation became the 'secret sauce' behind AWS's success| website = Amazon Science| date = July 27, 2022| url = https://www.amazon.science/how-silicon-innovation-became-the-secret-sauce-behind-awss-success| access-date = July 19, 2024}}</ref> Many vendor-specific terms exist for devices in this category, and it is an [[emerging technologies|emerging technology]] without a [[dominant design]].

AI accelerators are used in mobile devices such as Apple [[iPhone]]s, AMD [[AI engine|AI engines]]<ref>{{Cite book |last=Brown |first=Nick |chapter=Exploring the Versal AI Engines for Accelerating Stencil-based Atmospheric Advection Simulation |date=2023-02-12 |title=Proceedings of the 2023 ACM/SIGDA International Symposium on Field Programmable Gate Arrays |chapter-url=https://dl.acm.org/doi/10.1145/3543622.3573047 |series=FPGA '23 |___location=New York, NY, USA |publisher=Association for Computing Machinery |pages=91–97 |doi=10.1145/3543622.3573047 |isbn=978-1-4503-9417-8|arxiv=2301.13016 }}</ref> in Versal and NPUs, [[Huawei]], and [[Google Pixel]] smartphones,<ref>{{Cite web|url=https://consumer.huawei.com/en/press/news/2017/ifa2017-kirin970|title=HUAWEI Reveals the Future of Mobile AI at IFA}}</ref> and seen in many [[Apple silicon]], [[Qualcomm]], [[Samsung]], and [[Google Tensor]] smartphone processors.<ref>{{Cite web| title=Snapdragon 8 Gen 3 mobile platform | url=https://docs.qualcomm.com/bundle/publicresource/87-71408-1_REV_B_Snapdragon_8_gen_3_Mobile_Platform_Product_Brief.pdf | archive-url=https://web.archive.org/web/20231025162610/https://docs.qualcomm.com/bundle/publicresource/87-71408-1_REV_B_Snapdragon_8_gen_3_Mobile_Platform_Product_Brief.pdf | archive-date=2023-10-25}}</ref>

It is more recently (circa 2022) added to computer processors from [[Intel]],<ref>{{Cite web|url=https://www.intel.com/content/www/us/en/newsroom/news/intels-lunar-lake-processors-arriving-q3-2024.html|title=Intel's Lunar Lake Processors Arriving Q3 2024|website=Intel|date=May 20, 2024 }}</ref> [[AMD]],<ref>{{cite web|title=AMD XDNA Architecture|url=https://www.amd.com/en/technologies/xdna.html}}</ref> and Apple silicon.<ref>{{Cite web |title=Deploying Transformers on the Apple Neural Engine |url=https://machinelearning.apple.com/research/neural-engine-transformers |access-date=2023-08-24 |website=Apple Machine Learning Research |language=en-US}}</ref> All models of Intel [[Meteor Lake]] processors have a built-in ''versatile processor unit'' (''VPU'') for accelerating [[statistical inference|inference]] for computer vision and deep learning.<ref>{{Cite web|url=https://www.pcmag.com/news/intel-to-bring-a-vpu-processor-unit-to-14th-gen-meteor-lake-chips|title=Intel to Bring a 'VPU' Processor Unit to 14th Gen Meteor Lake Chips|website=PCMAG|date=August 2022 }}</ref>

[[Graphics processing units]] designed by companies such as [[Nvidia]] and [[AMD]] often include AI-specific hardware, and are commonly used as AI accelerators, both for [[Machine learning|training]] and [[Inference engine|inference]].<ref>{{cite web| last1 = Patel| first1 = Dylan| last2 = Nishball| first2 = Daniel| last3 = Xie| first3 = Myron| title = Nvidia's New China AI Chips Circumvent US Restrictions| url=https://www.semianalysis.com/p/nvidias-new-china-ai-chips-circumvent| website = SemiAnalysis| date=2023-11-09| access-date=2024-02-07}}</ref>

On consumer devices, the NPU is intended to be small, power-efficient, but reasonably fast when used to run small models. To do this they are designed to support low-bitwidth operations using data types such as INT4, INT8, FP8, and FP16. A common metric is trillions of operations per second (TOPS), though this metric alone does not quantify which kind of operations are being done.<ref>{{cite web |title=A guide to AI TOPS and NPU performance metrics |url=https://www.qualcomm.com/news/onq/2024/04/a-guide-to-ai-tops-and-npu-performance-metrics}}</ref>

== History ==

Computer systems have frequently complemented the [[central processing unit|CPU]] with special-purpose accelerators for specialized tasks, known as [[coprocessor]]s. Notable [[application-specific integrated circuit|application-specific]] [[expansion card|hardware units]] include [[video card]]s for [[computer graphics|graphic]]s, [[sound card]]s, [[graphics processing unit]]s and [[digital signal processor]]s. As [[deep learning]] and [[artificial intelligence]] workloads rose in prominence in the 2010s, specialized hardware units were developed or adapted from existing products to [[hardware acceleration|accelerate]] these tasks.

=== Early attemptsDatacenters ===

Accelerators are used in [[cloud computing]] servers, including [[tensor processing unit]]s (TPU) in [[Google Cloud Platform]]<ref>{{Cite journal|date=2017-06-24|title=In-Datacenter Performance Analysis of a Tensor Processing Unit|journal=ACM SIGARCH Computer Architecture News|volume=45|issue=2|pages=1–12|language=EN|doi=10.1145/3140659.3080246|doi-access=free |last1=Jouppi |first1=Norman P. |last2=Young |first2=Cliff |last3=Patil |first3=Nishant |last4=Patterson |first4=David |last5=Agrawal |first5=Gaurav |last6=Bajwa |first6=Raminder |last7=Bates |first7=Sarah |last8=Bhatia |first8=Suresh |last9=Boden |first9=Nan |last10=Borchers |first10=Al |last11=Boyle |first11=Rick |last12=Cantin |first12=Pierre-luc |last13=Chao |first13=Clifford |last14=Clark |first14=Chris |last15=Coriell |first15=Jeremy |last16=Daley |first16=Mike |last17=Dau |first17=Matt |last18=Dean |first18=Jeffrey |last19=Gelb |first19=Ben |last20=Ghaemmaghami |first20=Tara Vazir |last21=Gottipati |first21=Rajendra |last22=Gulland |first22=William |last23=Hagmann |first23=Robert |last24=Ho |first24=C. Richard |last25=Hogberg |first25=Doug |last26=Hu |first26=John |last27=Hundt |first27=Robert |last28=Hurt |first28=Dan |last29=Ibarz |first29=Julian |last30=Jaffey |first30=Aaron |display-authors=1 |arxiv=1704.04760 }}</ref> and [[Trainium]] and [[Inferentia]] chips in [[Amazon Web Services]].<ref>{{cite web | title = How silicon innovation became the 'secret sauce' behind AWS's success| website = Amazon Science| date = July 27, 2022| url = https://www.amazon.science/how-silicon-innovation-became-the-secret-sauce-behind-awss-success| access-date = July 19, 2024}}</ref> Many vendor-specific terms exist for devices in this category, and it is an [[emerging technologies|emerging technology]] without a [[dominant design]].

First attempts like [[Intel]]'s ETANN 80170NX incorporated analog circuits to compute neural functions.<ref name=ICH_1>{{cite web| title=Inside Track| author=Dvorak, J.C.| url=https://archive.org/details/PC_Magazine_1990_05_29_v9n10/page/n83/mode/2up| publisher=PC Magazine| volume=9| issue=10| date=29 May 1990| access-date=26 December 2023}}</ref>

[[Graphics processing units]] designed by companies such as [[Nvidia]] and [[AMD]] often include AI-specific hardware, and are commonly used as AI accelerators, both for [[Machine learning|training]] and [[Inference engine|inference]].<ref>{{cite web| last1 = Patel| first1 = Dylan| last2 = Nishball| first2 = Daniel| last3 = Xie| first3 = Myron| title = Nvidia's New China AI Chips Circumvent US Restrictions| url=https://www.semianalysis.com/p/nvidias-new-china-ai-chips-circumvent| website = SemiAnalysis| date=2023-11-09| access-date=2024-02-07}}</ref>

Later all-digital chips like the Nestor/Intel [[Ni1000]] followed. As early as 1993, [[digital signal processor]]s were used as neural network accelerators to accelerate [[optical character recognition]] software.<ref>{{cite web |url=https://www.youtube.com/watch?v=FwFduRA_L6Q |title=convolutional neural network demo from 1993 featuring DSP32 accelerator|website=[[YouTube]] |date=June 2, 2014 }}</ref>

== Programming ==

By 1988, Wei Zhang et al. had discussed fast optical implementations of convolutional neural networks for alphabet recognition.<ref name="wz1988">{{cite journal |last=Zhang |first=Wei |date=1988 |title=Shift-invariant pattern recognition neural network and its optical architecture |journal=Proceedings of Annual Conference of the Japan Society of Applied Physics}}</ref><ref name="wz1990">{{cite journal |last=Zhang |first=Wei |date=1990 |title=Parallel distributed processing model with local space-invariant interconnections and its optical architecture |journal=Applied Optics |volume=29 |issue=32 |pages=4790–7 |doi=10.1364/AO.29.004790 |pmid=20577468 |bibcode=1990ApOpt..29.4790Z}}</ref>

Mobile NPU vendors typically provide their own [[application programming interface]] such as the Snapdragon Neural Processing Engine. An operating system or a higher-level library may provide a more generic interface such as TensorFlow Lite with LiteRT Next (Android) or CoreML (iOS, macOS).

Consumer CPU-integrated NPUs are accessible through vendor-specific APIs. AMD (Ryzen AI), Intel (OpenVINO), Apple silicon (CoreML){{efn|MLX builds atop the CPU and GPU parts, not the Apple Neural Engine (ANE) part of Apple Silicon chips. The relatively good performance is due to the use of a large, fast [[unified memory]] design.}} each have their own APIs, which can be built upon by a higher-level library.

In the 1990s, there were also attempts to create parallel high-throughput systems for workstations aimed at various applications, including neural network simulations.<ref name=DCS_1>{{cite journal| title=Designing a connectionist network supercomputer| author1=Asanović, K.| author2=Beck, J.| author3=Feldman, J.| author4=Morgan, N.| author5=Wawrzynek, J.| url=https://www.researchgate.net/publication/15149042| journal=[[International Journal of Neural Systems]]| publisher=ResearchGate| volume=4| issue=4| pages=317–26| date=January 1994| access-date=26 December 2023| doi=10.1142/S0129065793000250| pmid=8049794}}</ref><ref name="krste general purpose">{{cite web |title=The end of general purpose computers (not) | website=[[YouTube]] | date=April 17, 2015 |url=https://www.youtube.com/watch?v=VtJthbiiTBQ}}</ref>

GPUs generally use existing [[GPGPU]] pipelines such as CUDA and OpenCL adapted for lower precisions. Custom-built systems such as the Google TPU use private interfaces.

[[field-programmable gate array|FPGA]]-based accelerators were also first explored in the 1990s for both inference and training.<ref name="fpga-inference">{{cite web| title=Space Efficient Neural Net Implementation| author1=Gschwind, M.| author2=Salapura, V.| author3=Maischberger, O.| url=https://www.researchgate.net/publication/2318589| date=February 1995| access-date=26 December 2023}}</ref><ref name="fpga-training">{{cite book |chapter=A Generic Building Block for Hopfield Neural Networks with On-Chip Learning |year=1996 |doi=10.1109/ISCAS.1996.598474 |s2cid=17630664 |title=1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96 |last1=Gschwind |first1=M. |last2=Salapura |first2=V. |last3=Maischberger |first3=O. |pages=49–52 |isbn=0-7803-3073-0}}</ref>

==Notes==

In 2014, Chen et al. proposed DianNao (Chinese for "electric brain"),<ref name=":1">{{Cite journal|last1=Chen|first1=Tianshi|last2=Du|first2=Zidong|last3=Sun|first3=Ninghui|last4=Wang|first4=Jia|last5=Wu|first5=Chengyong|last6=Chen|first6=Yunji|last7=Temam|first7=Olivier|date=2014-04-05|title=DianNao|journal=ACM SIGARCH Computer Architecture News|volume=42|issue=1|pages=269–284|doi=10.1145/2654822.2541967|issn=0163-5964|doi-access=free}}</ref> to accelerate deep neural networks especially. DianNao provides 452 Gop/s peak performance (of key operations in deep neural networks) in a footprint of 3.02&nbsp;mm² and 485&nbsp;mW. Later, the successors (DaDianNao,<ref name=":2">{{Cite book|last1=Chen|first1=Yunji|last2=Luo|first2=Tao|last3=Liu|first3=Shaoli|last4=Zhang|first4=Shijin|last5=He|first5=Liqiang|last6=Wang|first6=Jia|last7=Li|first7=Ling|last8=Chen|first8=Tianshi|last9=Xu|first9=Zhiwei|last10=Sun|first10=Ninghui|last11=Temam|first11=Olivier|title=2014 47th Annual IEEE/ACM International Symposium on Microarchitecture |chapter=DaDianNao: A Machine-Learning Supercomputer |date=December 2014|pages=609–622|publisher=IEEE|doi=10.1109/micro.2014.58|isbn=978-1-4799-6998-2|s2cid=6838992}}</ref> ShiDianNao,<ref name=":3">{{Cite journal|last1=Du|first1=Zidong|last2=Fasthuber|first2=Robert|last3=Chen|first3=Tianshi|last4=Ienne|first4=Paolo|last5=Li|first5=Ling|last6=Luo|first6=Tao|last7=Feng|first7=Xiaobing|last8=Chen|first8=Yunji|last9=Temam|first9=Olivier|date=2016-01-04|title=ShiDianNao|journal=ACM SIGARCH Computer Architecture News|volume=43|issue=3S|pages=92–104|doi=10.1145/2872887.2750389|issn=0163-5964}}</ref> PuDianNao<ref name=":4">{{Cite journal|last1=Liu|first1=Daofu|last2=Chen|first2=Tianshi|last3=Liu|first3=Shaoli|last4=Zhou|first4=Jinhong|last5=Zhou|first5=Shengyuan|last6=Teman|first6=Olivier|last7=Feng|first7=Xiaobing|last8=Zhou|first8=Xuehai|last9=Chen|first9=Yunji|date=2015-05-29|title=PuDianNao|journal=ACM SIGARCH Computer Architecture News|volume=43|issue=1|pages=369–381|doi=10.1145/2786763.2694358|issn=0163-5964}}</ref>) were proposed by the same group, forming the DianNao Family<ref>{{Cite journal|last1=Chen|first1=Yunji|last2=Chen|first2=Tianshi|last3=Xu|first3=Zhiwei|last4=Sun|first4=Ninghui|last5=Temam|first5=Olivier|date=2016-10-28|title=DianNao family|journal=Communications of the ACM|volume=59|issue=11|pages=105–112|doi=10.1145/2996864|s2cid=207243998|issn=0001-0782}}</ref>

{{notelist}}

[[Smartphone]]s began incorporating AI accelerators starting with the [[Qualcomm Snapdragon 820]] in 2015.<ref>{{Cite web|title=Qualcomm Helps Make Your Mobile Devices Smarter With New Snapdragon Machine Learning Software Development Kit|url=https://www.qualcomm.com/news/releases/2016/05/02/qualcomm-helps-make-your-mobile-devices-smarter-new-snapdragon-machine |website=Qualcomm}}</ref><ref>{{Cite web|last=Rubin|first=Ben Fox|title=Qualcomm's Zeroth platform could make your smartphone much smarter|url=https://www.cnet.com/tech/mobile/qualcomms-zeroth-platform-could-make-your-smartphone-much-smarter/|access-date=September 28, 2021|website=CNET|language=en}}</ref>

=== Heterogeneous computing ===

{{Main|Heterogeneous computing}}

Heterogeneous computing incorporates many specialized processors in a single system, or a single chip, each optimized for a specific type of task. Architectures such as the [[Cell (microprocessor)|Cell microprocessor]]<ref name="cell">{{cite journal |title=Synergistic Processing in Cell's Multicore Architecture |year=2006 |doi=10.1109/MM.2006.41 |s2cid=17834015 |last1=Gschwind |first1=Michael |last2=Hofstee |first2=H. Peter |last3=Flachs |first3=Brian |last4=Hopkins |first4=Martin |last5=Watanabe |first5=Yukio |last6=Yamazaki |first6=Takeshi |journal=IEEE Micro |volume=26 |issue=2 |pages=10–24}}</ref> have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic, [[dataflow architecture]], and prioritizing throughput over latency. The Cell microprocessor has been applied to a number of tasks<ref>{{cite journal |title=Performance of Cell processor for biomolecular simulations |journal=Computer Physics Communications |volume=176 |issue=11–12 |pages=660–664 |arxiv=physics/0611201 |doi=10.1016/j.cpc.2007.02.107 |year=2007 |last1=De Fabritiis |first1=G. |bibcode=2007CoPhC.176..660D |s2cid=13871063}}</ref><ref>{{cite book |title=Video Processing and Retrieval on Cell architecture |citeseerx=10.1.1.138.5133}}</ref><ref>{{cite book |doi=10.1109/RT.2006.280210 |title=2006 IEEE Symposium on Interactive Ray Tracing |pages=15–23 |year=2006 |last1=Benthin |first1=Carsten |last2=Wald |first2=Ingo |last3=Scherbaum |first3=Michael |last4=Friedrich |first4=Heiko |isbn=978-1-4244-0693-7 |citeseerx=10.1.1.67.8982 |s2cid=1198101}}</ref> including AI.<ref>{{Cite web|url=https://www.teco.edu/~scholz/papers/ScholzDiploma.pdf|title=Development of an artificial neural network on a heterogeneous multicore architecture to predict a successful weight loss in obese individuals|access-date=November 14, 2017|archive-date=August 30, 2017|archive-url=https://web.archive.org/web/20170830041003/http://www.teco.edu/~scholz/papers/ScholzDiploma.pdf|url-status=dead}}</ref><ref>{{cite book |doi=10.1109/ccnc08.2007.235 |title=2008 5th IEEE Consumer Communications and Networking Conference |pages=1030–1034 |year=2008 |last1=Kwon |first1=Bomjun |last2=Choi |first2=Taiho |last3=Chung |first3=Heejin |last4=Kim |first4=Geonho |isbn=978-1-4244-1457-4 |s2cid=14429828}}</ref><ref>{{cite book |doi=10.1007/978-3-540-85451-7_71 |title=Euro-Par 2008 – Parallel Processing |volume=5168 |pages=665–675 |series=Lecture Notes in Computer Science |year=2008 |last1=Duan |first1=Rubing |last2=Strey |first2=Alfred |isbn=978-3-540-85450-0}}</ref>

In the 2000s, [[central processing unit|CPU]]s also gained increasingly wide [[SIMD]] units, driven by video and gaming workloads; as well as support for packed low-precision [[data type]]s.<ref>{{cite web |title=Improving the performance of video with AVX |url=https://software.intel.com/content/www/us/en/develop/articles/improving-the-compute-performance-of-video-processing-software-using-avx-advanced-vector-extensions-instructions.html |date=February 8, 2012}}</ref> Due to the increasing performance of CPUs, they are also used for running AI workloads. CPUs are superior for [[deep neural network|DNNs]] with small or medium-scale parallelism, for sparse DNNs and in low-batch-size scenarios.

=== Use of GPUs ===

[[Graphics processing unit]]s or GPUs are specialized hardware for the manipulation of images and calculation of local image properties. The mathematical basis of neural networks and [[graphics pipeline|image manipulation]] are similar, [[embarrassingly parallel]] tasks involving matrices, leading GPUs to become increasingly used for machine learning tasks.<ref name=HPC>{{cite web| title=High Performance Convolutional Neural Networks for Document Processing| author1=Chellapilla, K.| author2=Sidd Puri| author3=Simard, P.| url=https://inria.hal.science/inria-00112631/document| publisher=10th International Workshop on Frontiers in Handwriting Recognition| date=23 October 2006| access-date=23 December 2023}}</ref><ref name=INC_1>{{cite journal| title=ImageNet Classification with Deep Convolutional Neural Networks| author1=Krizhevsky, A.| author2=Sutskever, I.| author3=Hinton, G.E.| journal=Communications of the ACM| volume=60| issue=6| pages=84–90| date=24 May 2017| doi=10.1145/3065386| doi-access=free}}</ref>

In 2012, Alex Krizhevsky adopted two GPUs to train a deep learning network, i.e., AlexNet,<ref>{{Cite journal|last1=Krizhevsky|first1=Alex|last2=Sutskever|first2=Ilya|last3=Hinton|first3=Geoffrey E|date=2017-05-24|title=ImageNet classification with deep convolutional neural networks|journal=Communications of the ACM|language=EN|volume=60|issue=6|pages=84–90|doi=10.1145/3065386|doi-access=free}}</ref> which won the champion of the ISLVRC-2012 competition. During the 2010s, GPU manufacturers such as [[Nvidia]] added deep learning related features in both hardware (e.g., INT8 operators) and software (e.g., cuDNN Library).

Over the 2010s GPUs continued to evolve in a direction to facilitate deep learning, both for training and inference in devices such as [[self-driving car]]s.<ref name=ND_1>{{cite web| title=Nvidia in the Driver's Seat for Deep Learning| author=Roe, R.| url=https://insidehpc.com/2016/05/nvidia-driving-the-development-of-deep-learning| publisher=insideHPC| date=17 May 2023| access-date=23 December 2023}}</ref><ref name=NIS_1>{{cite web| title=Nvidia announces 'supercomputer' for self-driving cars at CES 2016| author=Bohn, D.| url=https://www.theverge.com/2016/1/4/10712634/nvidia-drive-px2-self-driving-car-supercomputer-announces-ces-2016| publisher=Vox Media| date=5 January 2016| access-date=23 December 2023}}</ref> GPU developers such as Nvidia [[NVLink]] are developing additional connective capability for the kind of dataflow workloads AI benefits from. As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated [[neural network]]-[[application-specific integrated circuit|specific]] hardware to further accelerate these tasks.<ref>"[https://www.researchgate.net/publication/329802520_A_Survey_on_Optimized_Implementation_of_Deep_Learning_Models_on_the_NVIDIA_Jetson_Platform A Survey on Optimized Implementation of Deep Learning Models on the NVIDIA Jetson Platform]", 2019</ref><ref name="CUDA9">{{cite web |first=Mark |last=Harris |url=https://developer.nvidia.com/blog/cuda-9-features-revealed/ |title=CUDA 9 Features Revealed: Volta, Cooperative Groups and More |date=May 11, 2017 |access-date=August 12, 2017}}</ref> [[Tensor core]]s are intended to speed up the training of neural networks.<ref name="CUDA9"/>

GPUs continue to be used in large-scale AI applications. For example, [[Summit (supercomputer)|Summit]], a supercomputer from IBM for [[Oak Ridge National Laboratory]],<ref name=SOR_1>{{cite web| title=Summit: Oak Ridge National Laboratory's 200 petaflop supercomputer| url= https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit| publisher=[[United States Department of Energy]]| date=2024| access-date=8 January 2024}}</ref> contains 27,648 [[Nvidia Tesla]] V100 cards, which can be used to accelerate deep learning algorithms.

=== Use of FPGAs ===

Deep learning frameworks are still evolving, making it hard to design custom hardware. [[Reconfigurable computing|Reconfigurable]] devices such as [[field-programmable gate array]]s (FPGA) make it easier to evolve hardware, frameworks, and software [[integrated design|alongside each other]].<ref>{{cite book |last1=Sefat |first1=Md Syadus |last2=Aslan |first2=Semih |last3=Kellington |first3=Jeffrey W |last4=Qasem |first4=Apan |title=2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS) |chapter=Accelerating HotSpots in Deep Neural Networks on a CAPI-Based FPGA |date=August 2019 |chapter-url=https://ieeexplore.ieee.org/document/8855410 |pages=248–256 |doi=10.1109/HPCC/SmartCity/DSS.2019.00048 |isbn=978-1-7281-2058-4 |s2cid=203656070}}</ref><ref name="fpga-inference" /><ref name="fpga-training" /><ref>{{cite web |url=http://www.nextplatform.com/2016/08/23/fpga-based-deep-learning-accelerators-take-asics/ |title=FPGA Based Deep Learning Accelerators Take on ASICs |date=August 23, 2016 |website=The Next Platform |access-date=September 7, 2016}}</ref>

Microsoft has used FPGA chips to accelerate inference for real-time deep learning services.<ref>{{Cite web|title=Microsoft unveils Project Brainwave for real-time AI|website=[[Microsoft]] |date=22 August 2017 |url=https://www.microsoft.com/en-us/research/blog/microsoft-unveils-project-brainwave/}}</ref>

=== Use of NPUs ===

Neural Processing Units (NPU) are another more native approach. Since 2017, several CPUs and SoCs have on-die NPUs: for example, [[Meteor Lake (microarchitecture)|Intel Meteor Lake]], [[Lunar Lake]], and [[Apple A11]].

=== Emergence of dedicated AI accelerator ASICs ===

While GPUs and FPGAs perform far better than CPUs for AI-related tasks, a factor of up to 10 in efficiency<ref>{{cite web |url=https://techreport.com/news/30155/google-boosts-machine-learning-with-its-tensor-processing-unit/ |title=Google boosts machine learning with its Tensor Processing Unit |date=May 19, 2016 |access-date=September 13, 2016}}</ref><ref>{{cite web |url=https://www.sciencedaily.com/releases/2016/02/160203134840.htm |title=Chip could bring deep learning to mobile devices |date=February 3, 2016 |website=www.sciencedaily.com |access-date=September 13, 2016}}</ref> may be gained with a more specific design, via an [[application-specific integrated circuit]] (ASIC).<ref>{{cite web | url=https://techcrunch.com/2023/08/29/google-cloud-announces-the-5th-generation-of-its-custom-tpus/ | title=Google Cloud announces the 5th generation of its custom TPUs | date=August 29, 2023 }}</ref> These accelerators employ strategies such as optimized [[cache-aware model|memory use]]{{citation needed |date=November 2017}} and the use of [[minifloat|lower precision arithmetic]] to accelerate calculation and increase [[throughput]] of computation.<ref name="lowprecision">{{Cite web|url=http://proceedings.mlr.press/v37/gupta15.pdf|title=Deep Learning with Limited Numerical Precision}}</ref><ref>{{cite arXiv |title=XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks |eprint=1603.05279 |last1=Rastegari |first1=Mohammad |last2=Ordonez |first2=Vicente |last3=Redmon |first3=Joseph |last4=Farhadi |first4=Ali |class=cs.CV |year=2016}}</ref> Some low-precision [[floating-point format]]s used for AI acceleration are [[half-precision floating-point format|half-precision]] and the [[bfloat16 floating-point format]].<ref name="toms_Inte">{{cite web |title=Intel To Launch Spring Crest, Its First Neural Network Processor, In 2019 |author=Lucian Armasu |work=Tom's Hardware |date=May 23, 2018 |access-date=May 23, 2018 |url=https://www.tomshardware.com/news/intel-neural-network-processor-lake-crest,37105.html |quote=Intel said that the NNP-L1000 would also support bfloat16, a numerical format that's being adopted by all the ML industry players for neural networks. The company will also support bfloat16 in its FPGAs, Xeons, and other ML products. The Nervana NNP-L1000 is scheduled for release in 2019.}}</ref><ref name="arxiv_1711.10604">{{cite report |title=TensorFlow Distributions |author=Joshua V. Dillon |author2=Ian Langmore |author3=Dustin Tran |author4=Eugene Brevdo |author5=Srinivas Vasudevan |author6=Dave Moore |author7=Brian Patton |author8=Alex Alemi |author9=Matt Hoffman |author10=Rif A. Saurous |date=November 28, 2017 |id=Accessed May 23, 2018 |arxiv=1711.10604 |quote=All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts |bibcode=2017arXiv171110604D}}</ref> [[Cerebras|Cerebras Systems]] has built a dedicated AI accelerator based on the largest processor in the industry, the second-generation Wafer Scale Engine (WSE-2), to support deep learning workloads.<ref>{{Cite web |last=Woodie |first=Alex |date=2021-11-01 |title=Cerebras Hits the Accelerator for Deep Learning Workloads |url=https://www.datanami.com/2021/11/01/cerebras-hits-the-accelerator-for-deep-learning-workloads/ |access-date=2022-08-03 |website=Datanami}}</ref><ref>{{Cite web |date=2021-04-20 |title=Cerebras launches new AI supercomputing processor with 2.6 trillion transistors |url=https://venturebeat.com/2021/04/20/cerebras-systems-launches-new-ai-supercomputing-processor-with-2-6-trillion-transistors/ |access-date=2022-08-03 |website=VentureBeat |language=en-US}}</ref>[[Amazon Web Services]] NeuronCores are heterogenous compute-units that power Tranium, Tranium2, Inferentia, and Inferentia2 chips consisting of 4 main engines: Tensor, Vector, Scalar, and GPSIMD, with on-chip software-managed SRAM memory to manage data locality and data prefetch.<ref>{{Cite web |date=2024-12-27 |title=AWS NeuronCore Architecture |url=https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-hardware/neuroncores-arch.html |access-date=2024-12-27 |website=readthedocs-hosted|language=en-US}}</ref>

== Ongoing research ==

=== In-memory computing architectures ===

{{Expand section|date=October 2018}}

In June 2017, [[IBM]] researchers announced an architecture in contrast to the [[Von Neumann architecture]] based on [[in-memory processing|in-memory computing]] and [[phase-change memory]] arrays applied to temporal [[Correlation (statistics)|correlation]] detection, intending to generalize the approach to [[heterogeneous computing]] and [[massively parallel]] systems.<ref>{{cite journal |arxiv=1706.00511 |author=Abu Sebastian |author2=Tomas Tuma |author3=Nikolaos Papandreou |author4=Manuel Le Gallo |author5=Lukas Kull |author6=Thomas Parnell |author7=Evangelos Eleftheriou |title=Temporal correlation detection using computational phase-change memory |journal=Nature Communications |volume=8 |doi=10.1038/s41467-017-01481-9 |year=2017 |issue=1 |page=1115 |pmid=29062022 |pmc=5653661|bibcode=2017NatCo...8.1115S }}</ref> In October 2018, IBM researchers announced an architecture based on in-memory processing and [[neuromorphic engineering|modeled on the human brain's synaptic network]] to accelerate [[deep neural network]]s.<ref>{{cite news |url=https://phys.org/news/2018-10-brain-inspired-architecture-advance-ai.html |title=A new brain-inspired architecture could improve how computers handle data and advance AI |date=October 3, 2018 |work=American Institute of Physics |access-date=October 5, 2018}}</ref> The system is based on phase-change memory arrays.<ref>{{cite journal |arxiv=1801.06228 |author=Carlos Ríos |author2=Nathan Youngblood |author3=Zengguang Cheng |author4=Manuel Le Gallo |author5=Wolfram H.P. Pernice |author6=C. David Wright |author7=Abu Sebastian |author8=Harish Bhaskaran |title=In-memory computing on a photonic platform |journal=Science Advances |year=2018|volume=5 |issue=2 |pages=eaau5759 |doi=10.1126/sciadv.aau5759 |pmid=30793028 |pmc=6377270 |bibcode=2019SciA....5.5759R |s2cid=7637801 }}</ref>

=== In-memory computing with analog resistive memories ===

In 2019, researchers from Politecnico di Milano found a way to solve systems of linear equations in a few tens of nanoseconds via a single operation. Their algorithm is based on [[in-memory computing]] with analog resistive memories which performs with high efficiencies of time and energy, via conducting [[matrix–vector multiplication]] in one step using Ohm's law and Kirchhoff's law. The researchers showed that a feedback circuit with cross-point resistive memories can solve algebraic problems such as systems of linear equations, matrix eigenvectors, and differential equations in just one step. Such an approach improves computational times drastically in comparison with digital algorithms.<ref>{{cite journal |title=Solving matrix equations in one step with cross-point resistive arrays |year=2019 |author=Zhong Sun |author2=Giacomo Pedretti |author3=Elia Ambrosi |author4=Alessandro Bricalli |author5=Wei Wang |author6=Daniele Ielmini |journal=Proceedings of the National Academy of Sciences |volume=116 |issue=10 |pages=4123–4128 |doi=10.1073/pnas.1815682116 |pmid=30782810 |pmc=6410822|bibcode=2019PNAS..116.4123S |doi-access=free }}</ref>

=== Atomically thin semiconductors ===

In 2020, Marega et al. published experiments with a large-area active channel material for developing logic-in-memory devices and circuits based on [[floating-gate]] [[field-effect transistor]]s (FGFETs).<ref name="atomthin">{{cite journal |title=Logic-in-memory based on an atomically thin semiconductor |year=2020 |doi=10.1038/s41586-020-2861-0 |last1=Marega |first1=Guilherme Migliato |last2=Zhao |first2=Yanfei |last3=Avsar |first3=Ahmet |last4=Wang |first4=Zhenyu |last5=Tripati |first5=Mukesh |last6=Radenovic |first6=Aleksandra |last7=Kis |first7=Anras |journal=Nature |volume=587 |issue=2 |pages=72–77 |pmid=33149289 |pmc=7116757|bibcode=2020Natur.587...72M }}</ref> Such atomically thin [[semiconductor]]s are considered promising for energy-efficient [[machine learning]] applications, where the same basic device structure is used for both logic operations and data storage. The authors used two-dimensional materials such as semiconducting [[molybdenum disulphide]] to precisely tune FGFETs as building blocks in which logic operations can be performed with the memory elements.<ref name="atomthin"/>

=== Integrated photonic tensor core ===

In 1988, Wei Zhang et al. discussed fast optical implementations of [[convolutional neural networks]] for alphabet recognition.<ref name="wz1988"/><ref name="wz1990"/>

In 2021, J. Feldmann et al. proposed an integrated [[photonic]] [[hardware accelerator]] for parallel convolutional processing.<ref name="photonic">{{cite journal |title=Parallel convolutional processing using an integrated photonic tensor |year=2021 |doi=10.1038/s41586-020-03070-1 |last1=Feldmann |first1=J. |last2=Youngblood|first2=N. |last3=Karpov |first3=M. | last4=Gehring |first4=H. | display-authors=3 | journal=Nature |volume=589 |issue=2 |pages=52–58|pmid=33408373 |arxiv=2002.00281 |s2cid=211010976 }}</ref> The authors identify two key advantages of integrated photonics over its electronic counterparts: (1) massively parallel data transfer through [[wavelength]] division [[multiplexing]] in conjunction with [[frequency comb]]s, and (2) extremely high data modulation speeds.<ref name="photonic"/> Their system can execute trillions of multiply-accumulate operations per second, indicating the potential of [[Photonic integrated circuit|integrated]] [[photonics]] in data-heavy AI applications.<ref name="photonic"/> Optical processors that can also perform backpropagation for artificial neural networks have been experimentally developed.<ref>{{cite web | url=https://spectrum.ieee.org/backpropagation-optical-ai | title=Photonic Chips Curb AI Training's Energy Appetite - IEEE Spectrum }}</ref>

== Nomenclature ==

As of 2016, the field is still in flux and vendors are pushing their own marketing term for what amounts to an "AI accelerator", in the hope that their designs and [[application programming interface|APIs]] will become the [[dominant design]]. There is no consensus on the boundary between these devices, nor the exact form they will take; however several examples clearly aim to fill this new space, with a fair amount of overlap in capabilities.

In the past when consumer [[graphics accelerator]]s emerged, the industry eventually adopted [[Nvidia]]'s self-assigned term, "the GPU",<ref>{{cite web |url=http://www.nvidia.com/object/IO_20020111_5424.html |title=NVIDIA launches the World's First Graphics Processing Unit, the GeForce 256|archive-url=https://web.archive.org/web/20160227145622/http://www.nvidia.com/object/IO_20020111_5424.html |archive-date=February 27, 2016 }}</ref>

as the collective noun for "graphics accelerators", which had taken many forms before {{Clarify|text=settling on an overall [[graphics pipeline|pipeline]] implementing a model presented by [[Direct3D]]|date=July 2024}}.

All models of Intel [[Meteor Lake]] processors have a ''Versatile Processor Unit'' (''VPU'') built-in for accelerating [[statistical inference|inference]] for computer vision and deep learning.<ref>{{Cite web|url=https://www.pcmag.com/news/intel-to-bring-a-vpu-processor-unit-to-14th-gen-meteor-lake-chips|title=Intel to Bring a 'VPU' Processor Unit to 14th Gen Meteor Lake Chips|website=PCMAG|date=August 2022 }}</ref>

== Deep learning processors (DLPs) ==

Inspired from the pioneer work of DianNao Family, many DLPs are proposed in both academia and industry with design optimized to leverage the features of deep neural networks for high efficiency. At ISCA 2016, three sessions (15%) of the accepted papers, focused on architecture designs about deep learning. Such efforts include Eyeriss (MIT),<ref name=":5">{{Cite journal|last1=Chen|first1=Yu-Hsin|last2=Emer|first2=Joel|last3=Sze|first3=Vivienne|author3-link=Vivienne Sze|date=2017|title=Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks|journal=IEEE Micro|pages=1|doi=10.1109/mm.2017.265085944|issn=0272-1732|hdl=1721.1/102369|hdl-access=free}}</ref> EIE (Stanford),<ref name=":6">{{Cite book|last1=Han|first1=Song|title=EIE: Efficient Inference Engine on Compressed Deep Neural Network| last2=Liu|first2=Xingyu|last3=Mao|first3=Huizi|last4=Pu|first4=Jing|last5=Pedram|first5=Ardavan|last6=Horowitz|first6=Mark A.|last7=Dally|first7=William J.|date=2016-02-03|oclc=1106232247}}</ref> Minerva (Harvard),<ref>{{Cite book|last1=Reagen|first1=Brandon|last2=Whatmough|first2=Paul|last3=Adolf|first3=Robert|last4=Rama|first4=Saketh|last5=Lee|first5=Hyunkwang|last6=Lee|first6=Sae Kyu|last7=Hernandez-Lobato|first7=Jose Miguel|last8=Wei|first8=Gu-Yeon|last9=Brooks|first9=David|title=2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) |chapter=Minerva: Enabling Low-Power, Highly-Accurate Deep Neural Network Accelerators |date=June 2016|___location=Seoul|publisher=IEEE|pages=267–278|doi=10.1109/ISCA.2016.32|isbn=978-1-4673-8947-1}}</ref> Stripes (University of Toronto) in academia,<ref>{{Cite journal|last1=Judd|first1=Patrick|last2=Albericio|first2=Jorge|last3=Moshovos|first3=Andreas|date=2017-01-01|title=Stripes: Bit-Serial Deep Neural Network Computing|journal=IEEE Computer Architecture Letters|volume=16|issue=1|pages=80–83|doi=10.1109/lca.2016.2597140|s2cid=3784424|issn=1556-6056}}</ref> TPU (Google),<ref name=":0">{{cite book| title=In-Datacenter Performance Analysis of a Tensor Processing Unit| author1=Jouppi, N.| author2=Young, C.| author3=Patil, N.| author4=Patterson, D.| publisher=[[Association for Computing Machinery]]| pages=1–12| date=24 June 2017| doi=10.1145/3079856.3080246| s2cid=4202768| doi-access=free| isbn=9781450348928}}</ref> and MLU ([[Cambricon]]) in industry.<ref>{{Cite web|title=MLU 100 intelligence accelerator card|url=https://www.cambricon.com/index.php?m=content&c=index&a=lists&catid=21| publisher=Cambricon| language=Japanese| date=2024| access-date=8 January 2024}}</ref> We listed several representative works in Table 1.

{| class="wikitable"

! colspan="8" |Table 1. Typical DLPs

|-

!Year

!DLPs

!Institution

!Type

!Computation

!Memory Hierarchy

!Control

!Peak Performance

|-

| rowspan="2" |2014

|DianNao<ref name=":1" />

|ICT, CAS

|digital

|vector [[Multiply–accumulate operation|MACs]]

|scratchpad

|[[very long instruction word|VLIW]]

|452 Gops (16-bit)

|-

|DaDianNao<ref name=":2" />

|ICT, CAS

|digital

|vector MACs

|scratchpad

|VLIW

|5.58 Tops (16-bit)

|-

| rowspan="2" |2015

|ShiDianNao<ref name=":3" />

|ICT, CAS

|digital

|scalar MACs

|scratchpad

|VLIW

|194 Gops (16-bit)

|-

|PuDianNao<ref name=":4" />

|ICT, CAS

|digital

|vector MACs

|scratchpad

|VLIW

|1,056 Gops (16-bit)

|-

| rowspan="5" |2016

|DnnWeaver

|Georgia Tech

|digital

|Vector MACs

|scratchpad

| -

| -

|-

|EIE<ref name=":6" />

|Stanford

|digital

|scalar MACs

|scratchpad

| -

|102 Gops (16-bit)

|-

|Eyeriss<ref name=":5" />

|MIT

|digital

|scalar MACs

|scratchpad

| -

|67.2 Gops (16-bit)

|-

|Prime<ref name=":7">{{Cite book|last1=Chi|first1=Ping|last2=Li|first2=Shuangchen|last3=Xu|first3=Cong|last4=Zhang|first4=Tao|last5=Zhao|first5=Jishen|last6=Liu|first6=Yongpan|last7=Wang|first7=Yu|last8=Xie|first8=Yuan|title=2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) |chapter=PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory |date=June 2016|pages=27–39|publisher=IEEE|doi=10.1109/isca.2016.13|isbn=978-1-4673-8947-1}}</ref>

|UCSB

|hybrid

|[[In-memory processing|Process-in-Memory]]

|ReRAM

| -

| -

|-

|Orlando<ref>{{Cite book |last1=Desoli |first1=Giuseppe |last2=Chawla |first2=Nitin |last3=Boesch |first3=Thomas |last4=Singh |first4=Surinder-pal |last5=Guidetti |first5=Elio |last6=De Ambroggi |first6=Fabio |last7=Majo |first7=Tommaso |last8=Zambotti |first8=Paolo |last9=Ayodhyawasi |first9=Manuj |last10=Singh |first10=Harvinder |last11=Aggarwal |first11=Nalin |chapter=14.1 a 2.9TOPS/W deep convolutional neural network SoC in FD-SOI 28nm for intelligent embedded systems |date=2017-02-05 |title=2017 IEEE International Solid-State Circuits Conference (ISSCC) |chapter-url=https://ieeexplore.ieee.org/document/7870349 |publisher=IEEE |pages=238–239 |doi=10.1109/ISSCC.2017.7870349 |isbn=978-1-5090-3758-2 |via=IEEEXplore}}</ref>

|STMicroelectronics

|digital

|Convolution accelerator + DSP

|scratchpad

|RISC

| 676 Gops (16 bits)

|-

| rowspan="4" |2017

|TPU<ref name=":0" />

|Google

|digital

|scalar MACs

|scratchpad

|[[complex instruction set computer|CISC]]

|92 Tops (8-bit)

|-

|PipeLayer<ref name=":8" />

|U of Pittsburgh

|hybrid

|Process-in-Memory

|ReRAM

| -

|

|-

|FlexFlow

|ICT, CAS

|digital

|scalar MACs

|scratchpad

| -

|420 Gops ()

|-

|DNPU<ref>{{Cite book |chapter-url=https://ieeexplore.ieee.org/document/7870350 |access-date=2023-08-24 |doi=10.1109/ISSCC.2017.7870350 |s2cid=206998709 |chapter=14.2 DNPU: An 8.1TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks |title=2017 IEEE International Solid-State Circuits Conference (ISSCC) |date=2017 |last1=Shin |first1=Dongjoo |last2=Lee |first2=Jinmook |last3=Lee |first3=Jinsu |last4=Yoo |first4=Hoi-Jun |pages=240–241 |isbn=978-1-5090-3758-2 }}</ref>

|KAIST

|digital

|scalar MACS

|scratchpad

| -

|300 Gops(16bit)

1200 Gops(4bit)

|-

| rowspan="3" |2018

|MAERI

|Georgia Tech

|digital

|scalar MACs

|scratchpad

| -

|

|-

|PermDNN

|City University of New York

|digital

|vector MACs

|scratchpad

| -

|614.4 Gops (16-bit)

|-

|UNPU<ref>{{Cite book |chapter-url=https://ieeexplore.ieee.org/document/8310262 |access-date=2023-11-30 |doi=10.1109/ISSCC.2018.8310262 |s2cid=3861747 |chapter=UNPU: A 50.6TOPS/W unified deep neural network accelerator with 1b-to-16b fully-variable weight bit-precision |title=2018 IEEE International Solid - State Circuits Conference - (ISSCC) |date=2018 |last1=Lee |first1=Jinmook |last2=Kim |first2=Changhyeon |last3=Kang |first3=Sanghoon |last4=Shin |first4=Dongjoo |last5=Kim |first5=Sangyeob |last6=Yoo |first6=Hoi-Jun |pages=218–220 |isbn=978-1-5090-4940-0 }}</ref>

|KAIST

|digital

|scalar MACs

|scratchpad

| -

|345.6 Gops(16bit)

691.2 Gops(8b)

1382 Gops(4bit)

7372 Gops(1bit)

|-

| rowspan="2" |2019

|FPSA

|Tsinghua

|hybrid

|Process-in-Memory

|ReRAM

| -

|

|-

|Cambricon-F

|ICT, CAS

|digital

|vector MACs

|scratchpad

|FISA

|14.9 Tops (F1, 16-bit)

956 Tops (F100, 16-bit)

|}

=== Digital DLPs ===

The major components of DLPs architecture usually include a computation component, the on-chip memory hierarchy, and the control logic that manages the data communication and computing flows.

Regarding the computation component, as most operations in deep learning can be aggregated into vector operations, the most common ways for building computation components in digital DLPs are the [[Multiply–accumulate operation|MAC]]-based (multiplier-accumulation) organization, either with vector MACs<ref name=":1" /><ref name=":2" /><ref name=":4" /> or scalar MACs.<ref name=":0" /><ref name=":3" /><ref name=":5" /> Rather than [[Single instruction, multiple data|SIMD]] or [[Single instruction, multiple threads|SIMT]] in general processing devices, deep learning ___domain-specific parallelism is better explored on these MAC-based organizations. Regarding the memory hierarchy, as deep learning algorithms require high bandwidth to provide the computation component with sufficient data, DLPs usually employ a relatively larger size (tens of kilobytes or several megabytes) on-chip buffer but with dedicated on-chip data reuse strategy and data exchange strategy to alleviate the burden for memory bandwidth. For example, DianNao, 16 16-in vector MAC, requires 16 × 16 × 2 = 512 16-bit data, i.e., almost 1024&nbsp;GB/s bandwidth requirements between computation components and buffers. With on-chip reuse, such bandwidth requirements are reduced drastically.<ref name=":1" /> Instead of the widely used cache in general processing devices, DLPs always use scratchpad memory as it could provide higher data reuse opportunities by leveraging the relatively regular data access pattern in deep learning algorithms. Regarding the control logic, as the deep learning algorithms keep evolving at a dramatic speed, DLPs start to leverage dedicated ISA (instruction set architecture) to support the deep learning ___domain flexibly. At first, DianNao used a VLIW-style instruction set where each instruction could finish a layer in a DNN. Cambricon<ref>{{Cite book|last1=Liu|first1=Shaoli|last2=Du|first2=Zidong|last3=Tao|first3=Jinhua|last4=Han|first4=Dong|last5=Luo|first5=Tao|last6=Xie|first6=Yuan|last7=Chen|first7=Yunji|last8=Chen|first8=Tianshi|title=2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA) |chapter=Cambricon: An Instruction Set Architecture for Neural Networks |date=June 2016|pages=393–405|publisher=IEEE|doi=10.1109/isca.2016.42|isbn=978-1-4673-8947-1}}</ref> introduces the first deep learning ___domain-specific ISA, which could support more than ten different deep learning algorithms. TPU also reveals five key instructions from the CISC-style ISA.

=== Hybrid DLPs ===

Hybrid DLPs emerge for DNN inference and training acceleration because of their high efficiency. Processing-in-memory (PIM) architectures are one most important type of hybrid DLP. The key design concept of PIM is to bridge the gap between computing and memory, with the following manners: 1) Moving computation components into memory cells, controllers, or memory chips to alleviate the memory wall issue.<ref name=":8">{{Cite book|last1=Song|first1=Linghao|last2=Qian|first2=Xuehai|last3=Li|first3=Hai|author3-link=Hai Li|last4=Chen|first4=Yiran|title=2017 IEEE International Symposium on High Performance Computer Architecture (HPCA) |chapter=PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning |date=February 2017|pages=541–552|publisher=IEEE|doi=10.1109/hpca.2017.55|isbn=978-1-5090-4985-1|s2cid=15281419}}</ref><ref name=":9">{{Cite journal|last1=Ambrogio|first1=Stefano|last2=Narayanan|first2=Pritish|last3=Tsai|first3=Hsinyu|last4=Shelby|first4=Robert M.|last5=Boybat|first5=Irem|last6=di Nolfo|first6=Carmelo|last7=Sidler|first7=Severin|last8=Giordano|first8=Massimo|last9=Bodini|first9=Martina|last10=Farinha|first10=Nathan C. P.|last11=Killeen|first11=Benjamin|date=June 2018|title=Equivalent-accuracy accelerated neural-network training using analogue memory|journal=Nature|volume=558|issue=7708|pages=60–67|doi=10.1038/s41586-018-0180-5|pmid=29875487|bibcode=2018Natur.558...60A |s2cid=46956938|issn=0028-0836}}</ref><ref>{{Cite book|last1=Chen|first1=Wei-Hao|last2=Lin|first2=Wen-Jang|last3=Lai|first3=Li-Ya|last4=Li|first4=Shuangchen|last5=Hsu|first5=Chien-Hua|last6=Lin|first6=Huan-Ting|last7=Lee|first7=Heng-Yuan|last8=Su|first8=Jian-Wei|last9=Xie|first9=Yuan|last10=Sheu|first10=Shyh-Shyuan|last11=Chang|first11=Meng-Fan|title=2017 IEEE International Electron Devices Meeting (IEDM) |chapter=A 16Mb dual-mode ReRAM macro with sub-14ns computing-in-memory and memory functions enabled by self-write termination scheme |date=December 2017|pages=28.2.1–28.2.4|publisher=IEEE|doi=10.1109/iedm.2017.8268468|isbn=978-1-5386-3559-9|s2cid=19556846}}</ref> Such architectures significantly shorten data paths and leverage much higher internal bandwidth, hence resulting in attractive performance improvement. 2) Build high efficient DNN engines by adopting computational devices. In 2013, HP Lab demonstrated the astonishing capability of adopting ReRAM crossbar structure for computing.<ref>{{Cite journal|last1=Yang|first1=J. Joshua|last2=Strukov|first2=Dmitri B.|last3=Stewart|first3=Duncan R.|date=January 2013|title=Memristive devices for computing|url=https://www.nature.com/articles/nnano.2012.240|journal=Nature Nanotechnology|language=en|volume=8|issue=1|pages=13–24|doi=10.1038/nnano.2012.240|pmid=23269430|bibcode=2013NatNa...8...13Y |issn=1748-3395}}</ref> Inspiring by this work, tremendous work are proposed to explore the new architecture and system design based on ReRAM,<ref name=":7" /><ref>{{Cite journal|last1=Shafiee|first1=Ali|last2=Nag|first2=Anirban|last3=Muralimanohar|first3=Naveen|last4=Balasubramonian|first4=Rajeev|last5=Strachan|first5=John Paul|last6=Hu|first6=Miao|last7=Williams|first7=R. Stanley|last8=Srikumar|first8=Vivek|date=2016-10-12|title=ISAAC|journal=ACM SIGARCH Computer Architecture News|volume=44|issue=3|pages=14–26|doi=10.1145/3007787.3001139|s2cid=6329628|issn=0163-5964}}</ref><ref>{{Cite book|last=Ji, Yu Zhang, Youyang Xie, Xinfeng Li, Shuangchen Wang, Peiqi Hu, Xing Zhang, Youhui Xie, Yuan|title=FPSA: A Full System Stack Solution for Reconfigurable ReRAM-based NN Accelerator Architecture|date=2019-01-27|oclc=1106329050}}</ref><ref name=":8" /> phase change memory,<ref name=":9" /><ref>{{Cite book|last1=Nandakumar|first1=S. R.|last2=Boybat|first2=Irem|last3=Joshi|first3=Vinay|last4=Piveteau|first4=Christophe|last5=Le Gallo|first5=Manuel|last6=Rajendran|first6=Bipin|last7=Sebastian|first7=Abu|last8=Eleftheriou|first8=Evangelos|title=2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS) |chapter=Phase-Change Memory Models for Deep Learning Training and Inference |date=November 2019|pages=727–730|publisher=IEEE|doi=10.1109/icecs46596.2019.8964852|isbn=978-1-7281-0996-1|s2cid=210930121}}</ref><ref>{{Cite journal|last1=Joshi|first1=Vinay|last2=Le Gallo|first2=Manuel|last3=Haefeli|first3=Simon|last4=Boybat|first4=Irem|last5=Nandakumar|first5=S. R.|last6=Piveteau|first6=Christophe|last7=Dazzi|first7=Martino|last8=Rajendran|first8=Bipin|last9=Sebastian|first9=Abu|last10=Eleftheriou|first10=Evangelos|date=2020-05-18|title=Accurate deep neural network inference using computational phase-change memory|journal=Nature Communications|volume=11|issue=1|page=2473|doi=10.1038/s41467-020-16108-9|arxiv=1906.03138|pmid=32424184|pmc=7235046|bibcode=2020NatCo..11.2473J |issn=2041-1723|doi-access=free}}</ref> etc.

== Benchmarks ==

Benchmarks such as MLPerf and others may be used to evaluate the performance of AI accelerators.<ref>{{cite web | url=https://www.theregister.com/2022/09/09/nvidia_hopper_mlperf/ | title=Nvidia claims 'record performance' for Hopper MLPerf debut }}</ref> Table 2 lists several typical benchmarks for AI accelerators.

{| class="wikitable"

|+Table 2. Benchmarks.

!Year

!NN Benchmark

!Affiliations

!# of microbenchmarks

!# of component benchmarks

!# of application benchmarks

|-

|2012

|BenchNN

|ICT, CAS

|N/A

|12

|N/A

|-

|2016

|Fathom

|Harvard

|N/A

|8

|N/A

|-

|2017

|BenchIP

|ICT, CAS

|12

|11

|N/A

|-

|2017

|DAWNBench

|Stanford

|8

|N/A

|N/A

|-

|2017

|DeepBench

|Baidu

|4

|N/A

|N/A

|-

|2018

|AI Benchmark

|ETH Zurich

|N/A

|26

|N/A

|-

|2018

|MLPerf

|Harvard, Intel, and Google, etc.

|N/A

|7

|N/A

|-

|2019

|AIBench

|ICT, CAS and Alibaba, etc.

|12

|16

|2

|-

|2019

|NNBench-X

|UCSB

|N/A

|10

|N/A

|}

== Potential applications ==

*[[Agricultural robot]]s, for example, herbicide-free weed control.<ref>{{cite web |title=Development of a machine vision system for weed control using precision chemical application |website=University of Florida |citeseerx = 10.1.1.7.342 |url=http://www.abe.ufl.edu/wlee/Publications/ICAME96.pdf |archive-url=https://web.archive.org/web/20100623062608/http://www.abe.ufl.edu/wlee/Publications/ICAME96.pdf|archive-date=June 23, 2010}}</ref>

*[[Vehicular automation|Autonomous vehicles]]: Nvidia has targeted their [[Drive PX-series]] boards at this application.<ref>{{cite web |url=https://www.nvidia.com/en-us/self-driving-cars/ |title=Self-Driving Cars Technology & Solutions from NVIDIA Automotive |website=NVIDIA}}</ref>

*[[Computer-aided diagnosis]]

*[[Industrial robot]]s, increasing the range of tasks that can be automated, by adding adaptability to variable situations.

*[[Machine translation]]

*[[Military robot]]s

*[[Natural language processing]]

*[[Search engine]]s, increasing the [[energy efficiency in computing|energy efficiency]] of [[data center]]s and the ability to use increasingly advanced [[information retrieval|queries]].

*[[Unmanned aerial vehicle]]s, e.g. navigation systems, e.g. the [[Movidius Myriad 2]] has been demonstrated successfully guiding autonomous drones.<ref>{{cite web |title=movidius powers worlds most intelligent drone |url=https://www.siliconrepublic.com/machines/movidius-dji-drone |date=March 16, 2016}}</ref>

*[[Voice user interface]], e.g. in mobile phones, a target for Qualcomm [[Zeroth (software)|Zeroth]].<ref>{{cite web |title=Qualcomm Research brings server class machine learning to everyday devices–making them smarter [VIDEO] |url=https://www.qualcomm.com/news/onq/2015/10/01/qualcomm-research-brings-server-class-machine-learning-everyday-devices-making |date=October 2015}}</ref>

== See also ==

*[[Cognitive computer]]

*[[Neuromorphic engineering]]

*[[Optical neural network]]

*[[Physical neural network]]

*[[UALink]]

== References ==

{{Reflist|32em}}

== External links ==

*[https://www.nextplatform.com/2016/04/05/nvidia-puts-accelerator-metal-pascal/ Nvidia Puts The Accelerator To The Metal With Pascal.htm], The Next Platform

*[http://eyeriss.mit.edu/ Eyeriss Project], MIT

*https://alphaics.ai/

{{Hardware acceleration}}

[[Category:Application-specific integrated circuits]]

[[Category:Neural processing units| ]]