Wikipedia:Authority control integration proposal: Difference between revisions

Revision as of 23:03, 24 June 2012

Latest revision as of 03:21, 25 April 2021

(40 intermediate revisions by 6 users not shown)

Line 1:

{{Superseded|[[Wikipedia:Authority control integration proposal/FAQ|the project FAQ]], as '''the project has now concluded'''}}

This proposed project intends to extend and systematise the use of [[authority control]] identifiers, using the {{tl|Authority control}} template, on English Wikipedia articles.

{{Notice|If you've '''found an error''' with one of the VIAF codes added, please [[Wikipedia:VIAF/errors|list it here]].}}

The current proposal focuses on biographies, although this may be extended in future to cover other topics, and on the use of identifiers from [[VIAF]], a composite system bringing together several major authority files. VIAF algorithmically matches and clusters entries from the individual authority files, and uses data scraped from Wikipedia to aid the process; as a result, there have already been a large number of Wikipedia-VIAF matched pairs identified.

==Video Summary of the proposal==

The proposal was originally written up here, and discussed on the Village Pump [http://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(proposals)&oldid=499166049#Authority_Control_Integration here]. It has since been updated to include some of the feedback and commentary received during the discussions. While the Village Pump discussion was broadly favourable, it will soon be formally listed as an RFC in order to ensure clear support from the community before implementation later in 2012.

On [http://www.youtube.com/watch?v=uwwTNmJUQ8w youtube].

==Introduction==

This plan is being coordinated by [[User:Maximiliankleinoclc|Max Klein]], the Wikipedian in Residence at [[OCLC]], and [[User:Andrew Gray|Andrew Gray]], the Wikipedian in Residence at the British Library. OCLC are the central operating group for VIAF, and have offered to provide technical support for the matching process. If you would like to help work on it, please [[Wikipedia talk:Authority control integration project|let us know]].

This proposed project intends to extend and systematise the use of [[authority control]] identifiers, using the {{tl|Authority control}} template, on English Wikipedia articles. ''Authority control'' is the [[term-of-art]] in librarianship, archival practice and related fields for [[unique identifiers]] to [[Wikipedia:Disambiguation|disambiguate]] objects (people, places, academic subjects, etc.). These fields of study have different conceptualisations of unique identifiers form some other fields because many systems in place are backwards-compatible to pre-computerisation systems. This project aims to connect the English Wikipedia to this [[long tail]] of identifiers.

The current proposal focuses on biographies, although this may be extended in future to cover other topics, and onis built around the use of identifiersdata from [[VIAF]], a composite system bringing together several major authority files. VIAF algorithmically matches and clusters entries from the individual authority files, and uses data scraped from Wikipedia to aid the process; as a result, there have already been a large number of Wikipedia-VIAF matched pairs identified and this provides a very effective springboard to work from.

The proposal was originally written up here, and [[Wikipedia:Village pump (proposals)/Archive 89#section Authority Control Integration|discussed on the Village Pump [http://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(proposals)&oldid=499166049#Authority_Control_Integration here]]. It has since been updated to include some of the feedback and commentary received during the discussions. While the Village Pump discussion was broadly favourable, it willhas soon bebeen formally listed as an RFC in order to ensure clear support from the community before implementation later in 2012.

This plan is being coordinated by [[User:Maximiliankleinoclc|Max Klein]], the Wikipedian in Residence at [[OCLC]], and [[User:Andrew Gray|Andrew Gray]], the Wikipedian in Residence at the British Library. OCLC are the central operating group for VIAF, and have offered to provide technical support for the matching process. If you would like to help work on it, please [[Wikipedia talk:Authority control integration projectproposal|let us know]].

==Background==

Line 11 ⟶ 19:

{{main|Wikipedia:Authority control}}

[[Authority control]] is a system primarily used in libraries and other metadata services, where a single entity is given a canonical unique identifier. This allows clear disambiguation between different entities with similar names, while also allowing the use of a single identifier for those with multiple variant names. On Wikipedia, this is handled with the {{tl|authority control}} template, which places the identifiers at the end of the article and links out to library catalogues and central authority databases.

As well as these reader-visible links, the embedded data helps build infrastructure for future work, such as:

Currently, around 4,000 articles on the English Wikipedia have some form of embedded authority control identifier. On the German Wikipedia, by comparison, around 220,000 articles have embedded identifiers.

*'''Reliable linking from external services''' - we can build lookup services, such as this tool for the German Wikipedia's PND files: http://toolserver.org/~apper/pd/person/pnd-redirect/de/118768581 - which takes you to the article represented by that PND. Such tools allow people to automatically generate links to Wikipedia without guessing at article titles, use the API to pull out leads from articles for reuse in other sites, etc.

The practical uses for these identifiers are varied. Among other things, they can:

*'''Extending the scope for checking metadata''' - we already have methods, such as the [[Wikipedia:Death anomalies project|Death anomalies project]], for comparing the metadata between Wikipedia language editions and spotting inconsistencies. Including identifiers which tie into external services, with reliable APIs, give us a lot of additional data for cross-checking.

* help provide access to material written by and about the subject of an article, by linking directly into catalogues;

*'''Returning metadata to the outside world''' - working backwards from this, once we have embedded identifiers, the curators of this metadata will find it a lot easier to incorporate information from Wikipedia, taking advantage of our fairly fast update cycle for things like death dates.

* identify alternate names for which we can create redirects;

*'''Identifying alternate names''' - particularly for non-standard transliterations, the alternate headings in authority files give us an extensive and curated collection of variants of names. The linkage will help the creation of redirects.

* allow direct linking to, and reuse of, Wikipedia articles by external services, without needing to check page titles;

*'''Content creation support''' - the presence of the identifiers allows future work on tools to, e.g., develop scripts to generate author's bibliographies for articles.

* support the development of better metadata services, by helping information flow from Wikipedia back to the central data stores;

* help tie together articles on specific individuals, supporting the development of [[:meta:Wikidata|Wikidata]]

Currently, around 4,000 articles on the English Wikipedia have some form of embedded authority control identifier, and on Commons, around 45,000 articles contain authority control. On the German Wikipedia, by comparison, [[:de:Wikipedia:Normdaten|around 220,000 articles]] have embedded identifiers.

This project is not the first effort to merge authority control with Wikipedia, but rather aims to build on previous projects; its main goal is to prepare infrastructure for use in Wikidata and future interoperability with external authority files, and to support the opportunities for future innovation with linked data.

==The proposal==

This specificinitial proposal focuses on including identifiers fromin thebiographies; [[VIAF|Virtualhowever, Internationalit Authorityis File]]not (VIAF), an international projectintended to merge multiple national authority files into a single master system. In this phase, identifiers will be included in biographiesexclusive, butand the system can be extended in future to other articles if there is community support for it.

It is built around use of the [[VIAF|Virtual International Authority File]] (VIAF), an international project to merge multiple national authority files into a single master system. VIAF identifiers correspond to identifiers in other systems, and can be used in parallel with, or instead of, these other identifiers.

It will involve identifying an appropriate VIAF identifier number for as many articles as possible, using a number of different methods ranked by probable accuracy. Following this, and testing of the data to ensure it is consistent and accurate, the identifier will be added to these articles by a bot, using an extended version of the {{tl|Authority control}} template.

ItThe process will involve identifying an appropriate VIAF identifier numberto formatch to as many articles as possible, using a number of different methods ranked by probable accuracy. Following this, and testing of the data to ensure it is consistent and accurate, thea VIAF identifier will be added to these articles by a bot, using an extended version of the {{tl|Authority control}} template. This tool can later be reused to include other identifiers, such as [[LCCN]], if desired.

===Data sources===

There are fourthree available sources of data:

#'''Articles already using {{tl|Authority control}}'''. - someSome of these will have VIAF numbers. Where they do not, we can use the LCCN/GND numbers to match a VIAF number and include it in the existing template.

#'''Interwikied articles with identifiers'''. - aroundAround 220,000 articles in the German Wikipedia have identifiers; some include VIAF, some do not. Where an interwiki to the German Wikipedia exists, we can pull the identifier from the linked page, doing some basic metadata checks to ensure the interwiki linkage is accurate.

#:''Around [[:de:Vorlage:NORMDATENCOUNT|145,000 articles]] on the German Wikipedia currently have VIAF identifiers; the rest use other identities, but it may be practical to match them to VIAF.''
#'''VIAF authority file links'''. - asAs part of the matching process, Wikipedia is used as a source of information to help bring VIAF "clusters" together. OCLC have provided an extracted list of over 250,000 English Wikipedia articles with corresponding VIAF numbers, though these may have to be checked to ensure that pages have not been moved since the matching was carried out.

#:''(The matching is done with this [http://dl.dropbox.com/u/10997393/wikipedia2auth3.py python code] written by OCLC Research Scientists Thom Hickey and Jenny Toves. During the algorithmic creation of the VIAF file if a Wikipedia link is matched with ~98% accuracy then it is included in the entry. Right now there are 266,202 links from VIAF to Wikipedia. Those links are available [http://dl.dropbox.com/u/10997393/wikilinks.out as a tab-delimited text file].)''

#Automated matching - for articles not covered by any of the above, it ''may'' be possible to generate matches to VIAF using metadata such as variant names or birth and death dates. This will need a degree of error-checking and review to determine the accuracy of the matching process, and should not be used if another source is available.

===Implementation===

The implementation will be done in four stages.

# Create a listlists of page titles and associated VIAF cluster IDs. Thisfrom willthe beenwiki done off-wikidataset, andthe willdewiki usedataset, multipleand datathe sourcesVIAF - those noted as #1-3 abovedataset. These canwill then be checkedsampled againstto each other, and any with discrepancies can be examined manually. (This database will be made availablecheck for reuse and to help with ongoing maintenance.) A sample of these will be selected and manually compared to ensure accuracy.

# Prior to the bot run, {{tl|Authority control}} will be redeveloped to ensure it scales effectively to the new usage, creating sub-templates for specific identifiers. The documentation for this template, along with [[Wikipedia:Authority control]], will be checked and updated or overhauled where necessary.

# A bot will be developed and tested, then approved through [[WP:BRFA|the standard bot approval process]] to ensure there are no technical problems and that it is compliant with this proposal.

# Finally, thisThis bot will add {{tl|authority control}} along with the VIAF codes from this list, once testing is complete.

# Finally, this bot will run periodic reports in conjunction with the VIAF update schedule, to reflect any reshuffling that occurs in the file.

Initially, this will be done using data sources #1-3 above - ie, those where we have some form of match between VIAF and Wikipedia to begin with. ''If'' the data in source #4 turns out to be usable and accurate, then a second wave of matches will be done using these identifiers.

{{:User:Maximiliankleinoclc/VIAF graphical timeline}}

Line 62 ⟶ 71:

===Template details===

The template currently used to handle authority control data is {{tl|Authority control}}; it is placed at the extreme end of the article, just above the categories, and displays a narrow box with the identifiers. These link to an external service. For an example, see [[AlbertFyodor EinsteinDostoyevsky#External links|AlbertFyodor EinsteinDostoyevsky]] - this uses GND, LCCN, and VIAF codes, and is nested under four collapseda navigational templatestemplate following the external links. It will only be used on "main" articles, and not on subpages or related bibliographies - no two articles should share an identifier.

As part of this project, we will need to rewrite {{tl|authority control}} to form a wrapper for a number of subsidiary templates, each handling a specific identifier. This will make it easier to maintain as well as easier to develop support for other identifiers, without the need for experimentation on a template used on several hundred thousand pages. Documentation on {{tl|authority control}}, [[Wikipedia:Authority control]], and related pages will be updated accordingly.

It will only be used on "main" articles, and not on subpages or related bibliographies - no two articles should share an identifier.

{{Wikipedia:Authority_control_integration_proposal/FAQ}}

As part of this project, we will need to rewrite {{tl|authority control}} to form a wrapper for a number of subsidiary templates, each handling a specific identifier. This will make it easier to maintain as well as easier to develop support for other identifiers, without the need for experimentation on a template used on several hundred thousand pages.

- [[User:Maximiliankleinoclc|Max Klein]], OCLC Wikipedian in Residence, and [[User:Andrew Gray|Andrew Gray]], British Library Wikipedian in Residence.

===Frequently asked questions===

==Progress==

# Why use VIAF and not another identifier?

#:[[VIAF]] is a composite of several existing authority control databases, and so includes all the content from many of the other systems. Any entity with, for example, a LCCN should have a corresponding VIAF number as well, but not every entity with a VIAF number will have an LCCN. Adding VIAF now does not preclude the inclusion of other identifiers in future; this isn't aiming to impose a sole standard.

# Why only people?

#:The authority control system does cover other things, but for the moment we are only planning to cover people - this is to simplify the initial program, as well as target the articles where the template is most likely to be useful.

# What about errors in VIAF?

#:We plan to set up a method for reporting apparent errors in VIAF (or its constituent catalogues) back to the relevant managing body. For the German equivalent noticeboard, see [[:de:WP:PND/F]].

# What about licensing?

#: VIAF is licensed as [http://opendatacommons.org/licenses/by/ ODC-BY], which is compatible with Wikipedia licensing; the use of a VIAF URI is sufficient attribution for the terms of the license.

# Will this give any control over Wikipedia content to third parties?

#: '''No'''. While we will be including VIAF ''identifiers'', the content of Wikipedia and VIAF will remain entirely separate. No metadata will be imported automatically from VIAF, nor will Wikipedia need to follow VIAF naming conventions.

Now that [[Wikipedia:Authority_control_integration_proposal/RFC|RFC]] has passed, the work of the bot is underway. Code can be viewed at [https://github.com/notconfusing/VIAFbot github].

If the community RFC decides not to go ahead with the project, we'll still be able to pass the data generated so far to the Wikidata team, and hopefully it can be used there.

Any comments, criticisms, etc. gratefully received.

- [[User:Maximiliankleinoclc|Max Klein]], OCLC Wikipedian in Residence, and [[User:Andrew Gray|Andrew Gray]], British Library Wikipedian in Residence.