Apache Pinot: Difference between revisions

Revision as of 07:23, 30 June 2021

Latest revision as of 01:11, 15 August 2025

(42 intermediate revisions by 17 users not shown)

Line 1:

{{Short description|Open-source distributed data store}}

{{AfC submission|||ts=20210630054902|u=Forlornacorn|ns=118}}

{{Infobox software

| name = Apache Pinot

| logo = [[File:Pinot LogoApache_Pinot_Crimson_logotype,_2023.svgpng|frameless|alt=Pinot Logo]]

| screenshot =

| caption =

| author = {{ubl|Kishore Gopalakrishna|Xiang Fu}}

| developer = Apache Pinot

| latest release version = 01.72.10

| latest release date = {{Start date and age|df=yes|20212024|0308|1821}}

| repo = [https://gitboxgithub.com/apache.org/repos/asf?p=incubator-pinot.git Pinot repository]

| programming language = [[Java (programming language)|Java]]

| operating system = [[Cross-platform]]

Line 21:

}}

'''Apache Pinot''' is a [[Column-oriented DBMS|column-oriented]], [[open-source software|open-source]], [[Distributed database|distributed]] [[data store]] written in [[Java (programming language)|Java]]. Pinot is designed to execute [[Online analytical processing|OLAP]] queries with low latency.<ref>{{cite journalbook |last1=Cui |first1=Tingting |last2=Peng |first2=Lijun |last3=Pardoe |first3=David |last4=Liu |first4=Kun |last5=Agarwal |first5=Deepak |last6=Kumar |first6=Deepak |title=Proceedings of the ADKDD'17 |chapter=Data-Driven Reserve Prices for Social Advertising Auctions at LinkedIn |date=14 August 2017 |pages=1–7 |doi=10.1145/3124749.3124759 |chapter-url=https://dl.acm.org/doi/abs/10.1145/3124749.3124759 |publisher=Association for Computing Machinery|isbn=9781450351942 |s2cid=12327343 }}</ref><ref>{{cite book |last1=Rosa |first1=Marcello La |title=ADVANCED INFORMATION SYSTEMS ENGINEERING: 33rd International Conference |date=2021 |publisher=Springer Nature |isbn=978-3-030-79382-1 |url=https://books.google.com/books?id=Q7k0EAAAQBAJ&dq=Apache+Pinot+-wikipedia&pg=PA384 |language=en}}</ref><ref>{{cite book |last1=Chin |first1=Francis Y. L. |last2=Chen |first2=C. L. Philip |last3=Khan |first3=Latifur |last4=Lee |first4=Kisung |last5=Zhang |first5=Liang-Jie |title=Big Data – BigData 2018: 7th International Congress, Held as Part of the Services Conference Federation, SCF 2018, Seattle, WA, USA, June 25–30, 2018, Proceedings |date=20 June 2018 |publisher=Springer |isbn=978-3-319-94301-5 |page=153 |url=https://books.google.com/books?id=eSVhDwAAQBAJ |language=en}}</ref><ref>{{cite book |last1=Im |first1=Jean-François |last2=Gopalakrishna |first2=Kishore |last3=Subramaniam |first3=Subbu |last4=Shrivastava |first4=Mayank |last5=Tumbde |first5=Adwait |last6=Jiang |first6=Xiaotian |last7=Dai |first7=Jennifer |last8=Lee |first8=Seunghyun |last9=Pawar |first9=Neha |last10=Li |first10=Jialiang |last11=Aringunram |first11=Ravi |title=Pinot: Realtime OLAP for 530 Million Users |journal=Proceedings of the 2018 International Conference on Management of Data |chapter=Pinot: Realtime OLAP for 530 Million Users |series=Sigmod '18 |date=2018-05-27 |pages=583–594 |doi=10.1145/3183713.3190661 |url=https://dl.acm.org/doi/10.1145/3183713.3190661#d13801648e1 |publisher=Association for Computing Machinery|isbn=9781450347037 |s2cid=44083085 }}</ref><ref>{{cite web |title=The Apache Software Foundation Announces Apache® Pinot™ as a Top-Level Project |url=https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces76 |website=blogs.apache.org|date=2 August 2021 }}</ref> It is suited in contexts where fast analytics, such as aggregations, are needed on immutable data, possibly, with real-time data ingestion.<ref>{{cite arXiv |last1=Rogers |first1=Ryan |last2=Subramaniam |first2=Subbu |last3=Peng |first3=Sean |last4=Durfee |first4=David |last5=Lee |first5=Seunghyun |last6=Kancha |first6=Santosh Kumar |last7=Sahay |first7=Shraddha |last8=Ahammad |first8=Parvez |title=LinkedIn's Audience Engagements API: A Privacy Preserving Data Analytics System at Scale |date=16 November 2020 |class=cs.CR |eprint=2002.05839}}</ref><ref>{{cite book |last1=Javadi |first1=Seyyed Ahmad |last2=Gupta |first2=Harsh |last3=Manhas |first3=Robin |last4=Sahu |first4=Shweta |last5=Gandhi |first5=Anshul |title=2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) |chapter=EASY: Efficient Segment Assignment Strategy for Reducing Tail Latencies in Pinot |date=July 2018 |pages=1432–1437 |doi=10.1109/ICDCS.2018.00144 |isbn=978-1-5386-6871-9 |s2cid=21659844 }}</ref><ref name="pinot-joins-apache-foundation">Pawar, Neha. [https://engineering.linkedin.com/blog/2019/03/pinot-joins-apache-incubator "Pinot Joins Apache Incubator"] {{Webarchive|url=https://web.archive.org/web/20190402090136/https://engineering.linkedin.com/blog/2019/03/pinot-joins-apache-incubator |date=2019-04-02 }}, ''LinkedIn Engineering'', 01 April 2019</ref> The name Pinot comes from the [[Pinot grape]] vines that are pressed into liquid that is used to produce a variety of different wines. The founders of the database chose the name as a metaphor for analyzing vast quantities of data from a variety of different file formats or streaming data sources.<ref name="open-sourcing-pinot">{{cite web |last1=Gopalakrishna |first1=Kishore |title=Open Sourcing Pinot: Scaling the Wall of Real-Time Analytics |url=https://engineering.linkedin.com/pinot/open-sourcing-pinot-scaling-wall-real-time-analytics |website=engineering.linkedin.com |publisher=LinkedIn |accessdate=3 September 2020 |archiveurl=https://web.archive.org/web/20150910081445/http://engineering.linkedin.com/pinot/open-sourcing-pinot-scaling-wall-real-time-analytics |archivedate=10 September 2015 |language=en}}</ref>

Pinot was first created at [[LinkedIn]] after the engineering staff determined that there were no off the shelf solutions that met the social networking site's requirements like predictable low latency, data freshness in seconds, fault tolerance and scalability.<ref name="open-sourcing-pinot" /> Pinot is used in production by technology companies such as [[Uber]],<ref name="pinot-at-uber">{{cite webnews |last1=WangYegulalp |first1=HaiboSerdar |title=EngineeringLinkedIn SQLfills Supportanother SQL-on-Hadoop Apache Pinot at Uberniche |url=https://engwww.uberinfoworld.com/engineeringarticle/2934506/linkedins-sqlpinot-supportfills-another-sql-on-apachehadoop-pinot/niche.html |websitework=Uber Engineering BlogInfoWorld |publisherdate=Uber2015-06-11 |accessdatelanguage=3 September 2020 |date=15 January 2020en}}</ref> [[Microsoft]],<refPinot name="pinot-joins-apache-foundation"is />used in andproduction by technology companies such as [[Factual|FactualUber]],<ref name="pinot-at-factual">{{cite webbook |last1=MelzFu |first1=EricYupeng |last2=Soman |first2=Chinmay |title=PinotProceedings @of Factualthe 2021 International Conference on Management of Data |urlchapter=https://medium.com/@ericmelz/pinot-factualReal-6118ebaf64b3time Data Infrastructure at Uber |websiteseries=MediumSigmod/Pods '21 |accessdatedate=39 SeptemberJune 20202021 |archiveurlpages=2503–2516 |doi=10.1145/3448016.3457552 |chapter-url=https://webdl.archiveacm.org/webdoi/20200903081722abs/https:10.1145//medium3448016.com/@ericmelz/pinot-factual-6118ebaf64b33457552 |archivedatepublisher=Association 3for SeptemberComputing 2020Machinery|arxiv=2104.00087 |languageisbn=en9781450383431 |dates2cid=17232478317 May 2020}}</ref> [[Microsoft]],<ref name="pinot-joins-apache-foundation" /> and [[Factual]].

== History ==

Pinot was started as an internal project at LinkedIn in 2013 to power a variety of user-facing and business-facing products. The first analytics product at LinkedIn to use Pinot was a redesign of the social networking site's feature that allows members to see who has viewed their profile in real-time. The project was open-sourced in June 2015 under an Apache 2.0 license and was donated to the Apache Software Foundation by LinkedIn in June 2019.<ref name="open-sourcing-pinot" /><ref name="pinot-joins-apache-foundation" />

== Architecture ==

[[File:Pinot Architecture.png|520x520px|thumb|alt=Architecture of Apache Pinot|Architecture diagram of Apache Pinot]]

Pinot uses [[Apache Helix]] for cluster management. Helix is embedded as an agent within the different components and uses [[Apache ZooKeeper]] for coordination and maintaining the overall cluster state and health. All Pinot servers and brokers are managed by Helix. Helix is a generic cluster management framework to manage partitions and replicas in a distributed system.

<br>

=== Query management ===

Line 40 ⟶ 39:

== Features ==

Pinot shares similar features with comparable OLAP datastores, such as [[Apache Druid]].<ref>{{cite book |last1=Ordonez |first1=Carlos |last2=Song |first2=Il-Yeol |last3=Anderst-Kotsis |first3=Gabriele |last4=Tjoa |first4=A. Min |last5=Khalil |first5=Ismail |title=Big Data Analytics and Knowledge Discovery: 21st International Conference, DaWaK 2019, Linz, Austria, August 26–29, 2019, Proceedings |date=2 October 2019 |publisher=Springer |isbn=978-3-030-27520-4 |page=170 |url=https://books.google.com/books?id=sf-pDwAAQBAJ&dq=Pinot+(data+store)+-wikipedia&pg=PA170 |language=en}}</ref><ref>{{cite book |last1=Uttamchandani |first1=Sandeep |title=The Self-Service Data Roadmap |date=10 September 2020 |publisher="O'Reilly Media, Inc." |isbn=978-1-4920-7520-2 |url=https://books.google.com/books?id=pEn8DwAAQBAJ&dq=Pinot+(data+store)+-wikipedia&pg=PT72 |language=en}}</ref> Like Druid, Pinot is a column-oriented database with various compression schemes such as [[Runrun-length encoding|Run Length]] and [[Variablevariable-length encodingcode|Fixed -Bit Length]]. Pinot supports pluggable [[Database index|indexing technologies]] - Sorted Index, [[Bitmap Index]], [[Inverted index|Inverted Index]], Star-Tree Index, and Range Index, which are what primarily differentiates Pinot from other OLAP datastores.

Pinot supports near real-time ingestion from streams such as [[Apache Kafka|Kafka]], [[AWS]] Kinesis and [[Batch processing|batch]] ingestion from sources such as [[Hadoop]], [[Amazon S3|S3]], [[Microsoft Azure|Azure]], [[Google Cloud Storage|GCS]]. Like mostly, allmost other [[Online analytical processing|OLAP]] datastores and [[data warehousing]] solutions, Pinot supports a [[SQL]]-like query language that supports selection, aggregation, filtering, group by, order by, distinct queries on data.

== See also ==

{{Portal|Free and open-source software}}

* [[List of column-oriented DBMSes]]

* [[Comparison of OLAP servers]]

== References ==

{{Reflist|30em}}

== External links ==

Line 59 ⟶ 60:

[[Category:Structured storage]]

[[Category:Free database management systems]]

[[Category:Free software programmed in Java (programming language)]]

[[Category:Database engines]]

[[Category:Big data products]]