Wikidata/Development/RDF: Difference between revisions

Revision as of 11:08, 26 June 2014

Latest revision as of 04:31, 12 December 2022

(2 intermediate revisions by 2 users not shown)

Line 1:

{{MovedToMediaWiki|Wikibase/Indexing/RDF_Dump_Format}}

{{WikidataEditNote}}

<div style="background: #FFC; padding: 1ex; border: #B00 solid 1px; ">

This document is a draft, and should not be assumed to represent the ultimate structure.

</div>

This historic note explains part of the '''RDF export of Wikidata''' and explains some of the considerations that led to this design .

A more recent description of the RDF format used in Wikidata is found in the report '''[http://korrekt.org/papers/Wikidata-RDF-export-2014.pdf Introducing Wikidata to the Linked Data Web]''', which supersedes most of the material on this page. Nevertheless, the new work is based on the old, and is compatible in most aspects. There is also a web page with periodic '''[http://tools.wmflabs.org/wikidata-exports/rdf/ RDF exports of Wikidata]''' based on the most recent format.

== General notes ==

The more you are familiar with the following topics the easier will it be for you to understand this note. If you cannot follow a part, feel free to use the references given here.

* [[:en:Resource Description Framework|RDF]] as the basic export model

* [[:en:Web Ontology Language|OWL]] for some constructs

* [[:en:Turtle (syntax)|Turtle]] as a syntax

* [[:en:Linked data|Linked data principles]]

* [[Wikidata/Data model|Wikidata data model]] or at least the [[Wikidata/Notes/Data model primer|Wikidata data model primer]]

Furthermore, throughout this note we will not use the URIs that will be used in the system later on, but rather suggestive and convenient URIs that help to follow the argumentation in this note. There exists a separate [[Wikidata/Notes/URI scheme|note on the Wikidata URI scheme]]. The namespaces used in this note are listed at the [[#Namespaces|bottom of the page]].

== Motivation ==

=== Example ===

For the following discussion, we introduce two statements with the same item and property, the first with two, the second with one qualifier:

<div style="padding: 2ex; border:#444 solid 1px; ">

{{Wikidata statement|item=Berlin|property=Population|value=3,499,879|qualifier1=as of|value1=November 30, 2012|qualifier2=Method|value2=Extrapolation}}

{{Wikidata statement|value=8,000|qualifier1=as of|value1=15th century|numberofsources=1}}

</div>

=== Ground triples ===

The simplest possible way to represent the first statement would be the following triple:

w:Berlin p:Population "3499879"^^xsd:integer .

We will call this triple the ground triple of the first example statement. The ground triple completely omits the qualifiers, which in case of the first example does not seem too bad. But the second statement is giving the population of Berlin as 8,000 in the 15th century. This would lead to the following ground triple:

w:Berlin p:Population "8000"^^xsd:integer .

If this ground triple was given too, Berlin would be in the result set when querying for cities with less than 10,000 inhabitants, which does not sound desirable.

=== Statements with qualifiers ===

Instead, we need to represent the statement including the qualifiers. Recommendations on how to represent this kind of data is given by the Semantic Web Best Practice note on [http://www.w3.org/TR/swbp-n-aryRelations/ Defining N-ary Relations on the Semantic Web]. In the following we follow their recommendation (pattern 1, use case 1).

We have to introduce an intermediary node to represent the statement as such, and connect it to the item and the value. The following triples represent the claim from the first statement.

w:Berlin s:Population Berlin:Statement1 .

Berlin:Statement1 rdf:type o:Statement .

Berlin:Statement1 v:Population "3499879"^^xsd:integer .

Berlin:Statement1 q:As_of "2011-11-30"^^xsd:date .

Berlin:Statement1 q:Method w:Extrapolation .

Berlin:Statement1 rdfs:label "3,499,879 (As of Nov 30, 2011, Method Extrapolation)"^en .

Note that we introduced two new properties, <code>s:Population</code> (<code>s:</code> like statement) and <code>v:Population</code> (<code>v:</code> like value) instead of the original <code>p:Population</code> property (mind the namespaces). The original property, in the <code>p:</code> namespace, is a datatype property that connects items directly with an integer value (i.e. the ___domain is item). The <code>s:</code> property, on the other hand, is an object property connecting the item with the statement about it.

For the same reason we introduced the <code>q:</code> properties for qualifiers, connecting a statement node with the value of the qualifier.

A user of the data could easily get the ground triple, either by adding an [http://www.w3.org/TR/2009/WD-owl2-primer-20090421/ OWL2 axiom] stating that <code>p:Population</code> is derived from the [http://www.w3.org/TR/2009/WD-owl2-primer-20090421/#Property_Chains property chain] of <code>s:Population</code> and <code>v:Population</code>, or by using a [http://www.w3.org/TR/sparql11-query/#construct SPARQL construct query] with the same effect.

One additional feature in order to support Semantic Web browsers is to give the statement a label that corresponds to a human-readable form of the value including the qualifiers. The label would be exported in all available languages. This enables Semantic Web browsers to display the first triple in the above serialization in a useful way for the viewer of the data. We tested it with [http://marbles.sourceforge.net/ Marbles] and the [http://uriburner.com/ode/ OpenLink Date Explorer].

In order to see how ''none'' and ''unknown'' are represented, please refer to the specification below.

=== Statements with references ===

Now that we have a node for the statement, it becomes trivial to add a reference.

Berlin:Statement1 prov:wasDerivedFrom Berlin:Statement1-Reference1 .

The [http://www.w3.org/TR/prov-o/ provenance ontology] is still a moving target, and it might be necessary to adapt this property later.

Also we will further expand on how the references are modeled as soon as this is progressed further.

=== Representation of values ===

Note that the values might and usually are also further structured themselves. We will detail how the values are exported as we progress with specifying them further within Wikidata. Since values might often have a unit or an accuracy associated with them, they will often be represented as yet another intermediate node with the respective values attached to it. For our examples we only deal with the datatype ''entity''.

== Specification ==

The specification is defined as a mapping of [//meta.wikimedia.org/wiki/Wikidata/Data_model#Wikidata_Object_Notation Wikidata statements written in the Wikidata Object Notation] to [http://www.w3.org/TR/owl2-syntax/ OWL2 axioms written in Functional-style syntax]. The [http://www.w3.org/TR/owl2-mapping-to-rdf/ transformation of OWL2 axioms to RDF triples in turn is defined by the OWL2 standard], the result is given here for convenience as well. The current specification is slightly simplified for now as it omits <code>PropertyIntervalSnak</code>, <code>PropertySomeIntervalSnak</code>, <code>PropertyInstanceOfSnak</code>, <code>PropertySubclassOfSnak</code>, and <code>Rank</code>.

The following is a reiteration of the relevant part of the Wikidata Object Notation.

{{User:Markus Krötzsch/BNFtable|

{{User:Markus Krötzsch/BNFdef|ItemDescription|'ItemDescription(' '''Item''' {'''Statement'''} ')'}}

{{User:Markus Krötzsch/BNFdef|Statement|'Statement(' '''MainSnak''' {'''Qualifier'''} {'''ReferenceRecord'''} ')'}}

}}

Every statement is translated into a number of OWL2 axioms as described below. Every statement is identified by a <code>StatementID</code> which is a <code>IRI</code>.

=== MainSnak a PropertyValueSnak ===

If the <code>MainSnak</code> is a <code>PropertyValueSnak</code>, then it is translated as follows:

ObjectPropertyAssertion( s:'''Property''' '''Item''' '''StatementID''' )

ClassAssertion( o:Statement '''StatementID''' )

ObjectPropertyAssertion( v:'''Property''' '''StatementID''' '''Value''' )

Annotation( '''StatementID''' rdfs:label ValueLabel('''Statement''') )

{{Collapse|

'''Item''' s:'''Property''' '''StatementID''' .

'''StatementID''' rdf:type o:Statement .

'''StatementID''' v:'''Property''' '''Value''' .

'''StatementID''' rdfs:label ValueLabel('''Statement''') .

|title=Serialization in RDF}}

<code>s:'''Property'''</code> is an <code>IRI</code> that has the same local name as '''Property''' but replaces the <code>p:</code> namespace with the <code>s:</code> namespace. The same is defined for <code>v:'''Property'''</code> and <code>q:'''Property'''</code> respectively. Whenever one of the properties is used, an annotation property to connect them to the base property <code>p:'''Property'''</code> will also be given. The function <code>ValueLabel</code> returns an appropriate label describing the value of the statement including the qualifiers (not defined here).

=== MainSnak a PropertySomeValueSnak ===

If the <code>MainSnak</code> is a <code>PropertySomeValueSnak</code>, then it is translated as follows:

ObjectPropertyAssertion( s:'''Property''' '''Item''' '''StatementID''' )

ClassAssertion( o:Statement '''StatementID''' )

ClassAssertion( ObjectSomeValuesFrom( v:'''Property''' owl:Thing ) '''StatementID''' )

Annotation( '''StatementID''' rdfs:label ValueLabel('''Statement''') )

{{Collapse|

'''Item''' s:'''Property''' '''StatementID''' .

'''StatementID''' rdf:type o:Statement .

'''StatementID''' rdf:type _:1 .

_:1 rdf:type owl:Restriction .

_:1 owl:onProperty v:'''Property''' .

_:1 owl:someValuesFrom owl:Thing .

'''StatementID''' rdfs:label ValueLabel('''Statement''') .

|title=Serialization in RDF}}

=== MainSnak a PropertyNoValueSnak ===

If the <code>MainSnak</code> is a <code>PropertyNoValueSnak</code>, then it is translated as follows:

ObjectPropertyAssertion( s:'''Property''' '''Item''' '''StatementID''' )

ClassAssertion( o:Statement '''StatementID''' )

ClassAssertion( ObjectAllValuesFrom( v:'''Property''' owl:Nothing ) '''StatementID''' )

Annotation( '''StatementID''' rdfs:label ValueLabel('''Statement''') )

{{Collapse|

'''Item''' s:'''Property''' '''StatementID''' .

'''StatementID''' rdf:type o:Statement .

'''StatementID''' rdf:type _:1 .

_:1 rdf:type owl:Restriction .

_:1 owl:onProperty v:'''Property''' .

_:1 owl:allValuesFrom owl:Nothing .

'''StatementID''' rdfs:label ValueLabel('''Statement''') .

|title=Serialization in RDF}}

=== Qualifier ===

Each <code>Qualifier</code> is translated as follows. If the <code>Qualifier</code> is a <code>PropertyValueSnak</code>, then it is translated as follows:

ObjectPropertyAssertion( q:'''Property''' '''StatementID''' '''Value''' )

{{Collapse|

'''StatementID''' q:'''Property''' '''Value''' .

|title=Serialization in RDF}}

If the <code>Qualifier</code> is a <code>PropertySomeValueSnak</code>, then it is translated as follows:

ClassAssertion( ObjectSomeValuesFrom( q:'''Property''' owl:Thing ) '''StatementID''' )

{{Collapse|

'''StatementID''' rdf:type _:1 .

_:1 rdf:type owl:Restriction .

_:1 owl:onProperty q:'''Property''' .

_:1 owl:someValuesFrom owl:Thing .

|title=Serialization in RDF}}

If the <code>Qualifier</code> is a <code>PropertyNoValueSnak</code>, then it is translated as follows:

ClassAssertion( ObjectAllValuesFrom( q:'''Property''' owl:Nothing ) '''StatementID''' )

{{Collapse|

'''StatementID''' rdf:type _:1 .

_:1 rdf:type owl:Restriction .

_:1 owl:onProperty q:'''Property''' .

_:1 owl:allValuesFrom owl:Nothing .

|title=Serialization in RDF}}

=== ReferenceRecord ===

Every <code>ReferenceRecord</code> is given a <code>ReferenceID</code> which is an <code>IRI</code>. Every <code>ReferenceRecord</code> is translated as follows:

ObjectPropertyAssertion( prov:wasDerivedFrom '''StatementID''' '''ReferenceID''' )

{{Collapse|

'''StatementID''' prov:wasDerivedFrom '''ReferenceID''' .

|title=Serialization in RDF}}

== Discussion of alternatives ==

The following discussion is for giving rationales to the design decisions in this note and can be skipped if the reader is not interested in them.

=== Punning for the property names ===

Instead of having different properties in the <code>p:</code>, <code>s:</code>, <code>v:</code> and <code>q:</code> namespaces, turning every property in Wikidata to four in the RDF export, we could have used a single property and just use it in all the four use cases. In almost all cases it would still be clear which property is actually used:

<code>p:</code> connects the item with the value, <code>s:</code> the item with the statement, <code>v:</code> the statement with the value, and <code>q:</code> is used for qualifier, connecting the statement with the qualifier value. The only case of ambiguity would be between the <code>v:</code> and <code>q:</code> as they both connect the statement with a value, and it leads to an ambiguity if a qualifier with the same property is used as in the ground triple.

Punning is in general frowned upon on the Semantic Web, but it is not unheard of, e.g. in OWL2 individuals and classes can be punned in general. Also a recent proposal for the [http://www.jenitennison.com/blog/node/170 notorious HTTP-range14 discussion proposes to use punning] as a solution.

We decided to not use punning in our case, but rather to accept the proliferation of properties. The reason for that is that it is not only best practice, but also necessary for the

OWL2 DL serialization to be valid OWL as we need to make a clear difference between object and datatype properties.

We also wanted to ensure that the vocabulary developed within Wikidata will be reusable outside of Wikidata. As in most cases we expect external data publishers to use the direct representation of data with triples—i.e. just the ground triple—we also wanted to ensure that such a property is available for external reuse. Ironically this property is not the one we use in the Wikidata export itself, but there are well defined relationships between them, as described above.

=== Named graphs ===

Adding a reference to a claim can be done in three ways:

# put every claim in one file and then add provenance metadata about that file

# put every claim in a named graph and then add provenance metadata about that graph

# reify every claim so that we can add provenance metadata directly to the claim

Ad 1: Having one file for every claim would lead to many files. Even if you are conservative and expect about only ten statements per item, resolving that item would require ten or more HTTP requests. This is prohibitively slow, especially considering that they are all very small files that we are requesting. It also would lead to an unreasonable amount of load on the server, something MediaWiki is not very well equipped for (a server based on [http://nodejs.org/ Node.js] or [http://twistedmatrix.com/trac/ Twisted] would probably be much better equipped for that, though).

Ad 2: The file holding all statements about an item could also contain a named graph for every single claim and then add metdata about these graphs, like the references. Whereas this could be a solution, there is, as of time of writing, no standard for the serialization of named graphs in files. There are a number of contenders (TriX, TriG, NQuads, etc.), but none of them is even on the way of becoming a standard.

Ad 3: See the section on reification below.

=== Quads ===

Quads are a serialization for RDF that add a name (IRI) for every triple (the fourth value), or alternatively a context (i.e. it groups triples into sets). There is again no standard serialization for quads, and also no standard semantics.

=== Reification ===

[http://www.w3.org/TR/rdf-mt/#Reif Reification as per RDF standard] is widely regarded as bloated and disliked. RDF introduces its own reification syntax, which has never really caught on. Due to its widely negative reputation, and due to discussions about deprecating reification from RDF, we decided against using this mechanism.

=== Publish the ground triple ===

One alternative decision regards the publication of the ground triple. We decided not to publish it, in order to be more consistent through the RDF serialization of our data model. It avoids publishing a triple like stating the population of Berlin at 8,000 as per above example.

One might say to publish unqualified statements with the ground triple at least, and not to do so for qualified statements. Again we decided against it: first, we would need to represent the statement anyhow in order to publish the reference. Second, this would mean that we would publish potentially conflicting ground triples in the same file - if there are two different sources for two different statements. By publishing everything on the level of statements only, we can remain consistent throughout the dataset by always remaining on the level of statements, taking the role of Wikidata as a secondary database serious.

== To do ==

* Representation of labels, descriptions and sitelinks

* Representation of data values

* Representation of references

* Representation of rank

* Representation of the following Snaks: InstanceOf, SubclassOf, PropertyInterval, and PropertySomeInterval

== Namespaces ==

* <code>w:</code> for Wikidata items

* <code>o:</code> for the Wikidata ontology (a fixed and small set of terms)

* <code>p:</code> for Wikidata properties

* <code>q:</code> for properties used as qualifiers

* <code>s:</code> for properties used to connect items and statements

* <code>v:</code> for properties used to connect statements and values

* <code>Berlin:</code> used as a shortcut for <code>w:Berlin</code> but defined as a prefix

* <code>xsd:</code>, <code>rdf:</code>, <code>rdfs:</code>, <code>owl:</code> and <code>prov:</code> with their usual meanings

== Acknowledgements ==

The data model and its serialization in RDF has been created with input from the whole Wikidata team, especially Markus Krötzsch and Daniel Kinzler. It further was deeply informed by the RENDER project and ongoing discussions there on how to represent a diversity of knowledge, special thanks to Elena Simperl, Andreas Thalhammer, and Ioan Toma. It was also strongly influenced by discussions during the ESWC Summer School 2012, special thanks go to Aidan Hogan, Barry Norton, and Dan Brickley. Discussions during the time at KIT helped further sharpen it, special thanks to Andreas Harth, Steffen Stadtmüller, Daniel Herzig and Günter Ladwig. It was also based on discussions concerning Shortipedia, special thanks to Varun Rathnakar and Yolanda Gil. --[[User:Denny Vrandečić (WMDE)|Denny Vrandečić (WMDE)]] ([[User talk:Denny Vrandečić (WMDE)|talk]]) 09:50, 6 August 2012 (UTC)

== See also ==

* '''[http://korrekt.org/papers/Wikidata-RDF-export-2014.pdf Introducing Wikidata to the Linked Data Web]''' is a more recent report that supersedes this document

* Periodic RDF exports of Wikidata are found at http://tools.wmflabs.org/wikidata-exports/rdf/

* Examples of the official URI scheme for Wikidata entities: http://www.wikidata.org/entity/Q80 and http://www.wikidata.org/entity/P31

* Example of live export RDF generated by Wikibase: http://www.wikidata.org/wiki/Special:EntityData/Q42.nt

** As of June 2014, this does not yet include all the data that is found in the dumps.

** The same data can be retrieved from the official URIs using content negotiation; the URLs under Special:EntityData are not URIs for the entities in Wikidata.

[[Category:Wikidata]]