Objective Revision Evaluation Service

The Objective Revision Evaluation Service (ORES) is a web service running in Wikimedia Labs that provides machine learning as a service for Wikimedia Projects. The system is designed to help automate critical wiki-work -- for example, vandalism detection and removal. This service is developed as part of the R:Revision scoring as a service project.

The ORES Logo.

By keeping contribution open, but being good at quality control, open knowledge projects maximize productivity and quality -- and this works for large wikis that are well supported by quality control tools (e.g. English and German Wikipedia), but remain a burden for small wikis. ORES is intended to provide a generalized service to support quality control and curation work in all wikis.

Models

The primary way that a user of ORES will interact with the system is by asking ORES to apply a 'scorer model' to a particular revision.

Support table

This table provides a summary overview of which models are supported in which wikis. The sections below discuss these models in more detail.

context	edit quality			article quality
context	`damaging`	`goodfaith`	`reverted`	`wp10`	`draftquality`
`arwiki` Arabic Wikipedia
`cawiki` Catalan Wikipedia
`cswiki` Czech Wikipedia
`dewiki` German Wikipedia
`enwiki` English Wikipedia
`enwiktionary` English Wiktionary
`eswiki` Spanish Wikipedia
`eswikibooks` Spanish Wikibooks
`etwiki` Estonian Wikipedia
`fawiki` Persian Wikipedia
`fiwiki` Finnish Wikipedia
`frwiki` French Wikipedia
`hewiki` Hebrew Wikipedia
`huwiki` Hungarian Wikipedia
`idwiki` Indonesian Wikipedia
`itwiki` Italian Wikipedia
`kowiki` Korean Wikipedia
`nlwiki` Dutch Wikipedia
`nowiki` Norwegian Wikipedia
`plwiki` Polish Wikipedia
`ptwiki` Portuguese Wikipedia
`rowiki` Romanian Wikipedia
`ruwiki` Russian Wikipedia
`svwiki` Swedish Wikipedia
`trwiki` Turkish Wikipedia
`ukwiki` Ukrainian Wikipedia
`viwiki` Vietnamese Wikipedia
`wikidatawiki` Wikidata

Edit quality models

ORES edit quality flow. A descriptive diagram of edits flowing from "The Internet" to Wikipedia depicts the "unknown" quality of edits before ORES and the "good", "needs review", "damaging" labeling that is possible after ORES is made available.

One of the most critical concerns about Wikimedia's open projects is the review of potentially damaging contributions. There's also the need to identify good-faith contributors (who may be inadvertently causing damage) and offering them support. These models intended to make the work of filtering through the recentchanges feed easier. #ORES edit quality flow shows how the stream of edits can be labeled as "good", "needs review" and "damaging" by the machine learning models.

damaging -- predicts whether or not an edit causes damage.
goodfaith -- predicts whether an edit was saved in good-faith
reverted -- predicts whether an edit will eventually be reverted

Article quality models

English Wikipedia assessment table. A screenshot of the English Wikipedia assessment table generated by WP 1.0 bot is presented. This screenshot was taken on Dec 23rd, 2014.

The quality of encyclopedia articles is a core concern for Wikipedians. Currently, some of the large Wikipedia's roughly follow the Wikipedia 1.0 assessment rating scale when evaluating the quality of articles. Having these assessments is very useful, because it helps us gauge our progress and identify missed opportunities (e.g. popular articles that are low quality). However, keeping these assessments up to date is challenging, so coverage is inconsistent. This is where the wp10 machine learning model comes in handy. By training a model to replicate the article quality assessments that humans perform, we can automatically assess every article and every revision with a computer. This model has been used to help Wikiprojects triage re-assessment work and to explore the editing dynamics that lead to article quality improvements.

wp10 -- predicts the Wikipedia 1.0 assessment class of an article or draft

The service

ORES service is available as a en:RESTful API at http://ores.wmflabs.org. This section describes the structure of paths that can be requested.

Example query: /scores/enwiki/?models=reverted|wp10&revids=34854345

{
  "34854345": {
    "reverted": {
      "prediction": true,
      "probability": {
        "false": 0.24530635297335074,
        "true": 0.7546936470266493
      }
    },
    "wp10": {
      "prediction": "B",
      "probability": {
        "A": 0.0,
        "B": 0.4,
        "C": 0.2,
        "FA": 0.3,
        "GA": 0.1,
        "Start": 0.0,
        "Stub": 0.0
      }
    }
  }
}

If you're querying the service about a large number of revisions, it's recommended to batch 50 revisions in each request as described below. It's acceptable to use up to four parallel requests.

Machine-readable paths

Objective Revision Evaluation Service/Machine paths

Licensing

ORES, revscoring and related software that we develop is freely available under an MIT license. All scores produced by the service are licensed to the public ___domain (CC0.)