Objective Revision Evaluation Service: Difference between revisions

Revision as of 19:04, 18 November 2015

Latest revision as of 07:06, 22 March 2024

(45 intermediate revisions by 25 users not shown)

Line 1:

{{MovedToMediaWiki}}

<div style="float: right;">

{{inline figure

| filename = Objective Revision Evaluation Service logo.svg|

| size=250px

| id=The ORES Logo

}}

</div>

The '''Objective Revision Evaluation Service''' (ORES) is a web service running in [[Wikimedia Labs]] that provides [[:en:machine learning|machine learning]] [[:en:as a service|as a service]] for Wikimedia Projects. The system is designed to help automate critical wiki-work -- for example, vandalism detection and removal. This service is developed as part of the [[R:Revision scoring as a service]] project.

{{list subpages}}

By keeping contribution ''open'', but being good at quality control, open knowledge projects maximize productivity and quality -- and this works for large wikis that are well supported by quality control tools (e.g. English and German Wikipedia), but remain a burden for small wikis. ORES is intended to provide a generalized service to support quality control and curation work in ''all wikis''.

== Models ==

The primary way that a user of ORES will interact with the system is by asking ORES to apply a 'scorer model' to a particular revision.

=== Support table ===

This table provides a summary overview of which models are supported in which wikis. The sections below discuss these models in more detail.

<center>

{{/Support table}}

</center>

=== Edit quality models ===

<div style="float: right;">

{{inline figure

| filename = ORES edit quality flow.svg

| size = 351px

| id = ORES edit quality flow

| caption = A descriptive diagram of edits flowing from "The Internet" to Wikipedia depicts the "unknown" quality of edits before ORES and the "good", "needs review", "damaging" labeling that is possible after ORES is made available.

}}

</div>

One of the most critical concerns about Wikimedia's open projects is the review of potentially damaging contributions. There's also the need to identify good-faith contributors (who may be inadvertently causing damage) and offering them support. These models intended to make the work of filtering through the recentchanges feed easier. [[#ORES edit quality flow]] shows how the stream of edits can be labeled as "good", "needs review" and "damaging" by the machine learning models.

*<code>[[ORES/damaging|damaging]]</code> -- predicts whether or not an edit causes damage.

*<code>[[ORES/goodfaith|goodfaith]]</code> -- predicts whether an edit was saved in good-faith

*<code>[[ORES/reverted|reverted]]</code> -- predicts whether an edit will eventually be reverted

<br style="clear: right;" />

=== Article quality models ===

<div style="float: right;">

{{inline figure

| filename = Article quality and importance.wp10bot.enwiki.png

| size = 350px

| id = English Wikipedia assessment table

| caption = A screenshot of the English Wikipedia assessment table generated by WP 1.0 bot is presented. This screenshot was taken on Dec 23rd, 2014.

}}

</div>

The quality of encyclopedia articles is a core concern for Wikipedians. Currently, some of the large Wikipedia's roughly follow the [[:en:WP:Wikipedia 1.0|Wikipedia 1.0]] assessment rating scale when evaluating the quality of articles. Having these assessments is very useful, because it helps us gadge our progress and identify missed opportunities (e.g. popular articles that are low quality). However, keeping these assessments up to date is challenging, so coverage is inconsistent. This is where the <code>wp10</code> machine learning model comes in handy. By training a model to replicate the article quality assessments that humans perform, we can automatically assess every article and every revision with a computer. This model has been used to help Wikiprojects triage re-assessment work and to explore the editing dynamics that lead to article quality improvements.

*<code>[[ORES/wp10|wp10]]</code> -- predicts the Wikipedia 1.0 assessment class of an article or draft

<br style="clear: right;"/>

== The service ==

ORES service is available as a [[:en:RESTful]] API at <code>http://ores.wmflabs.org</code>. This section describes the structure of paths that can be requested.

{{hidden|'''Example query:''' [http://ores.wmflabs.org/scores/enwiki/?models{{equal}}reverted{{!}}wp10&revids{{equal}}34854345 /scores/enwiki/?models{{equal}}reverted{{!}}wp10&revids{{equal}}34854345]

|ta1=left|fw1=normal|bg1=white|

<source lang="javascript">

{

"34854345": {

"reverted": {

"prediction": true,

"probability": {

"false": 0.24530635297335074,

"true": 0.7546936470266493

}

},

"wp10": {

"prediction": "B",

"probability": {

"A": 0.0,

"B": 0.4,

"C": 0.2,

"FA": 0.3,

"GA": 0.1,

"Start": 0.0,

"Stub": 0.0

}

}

}

}

</source>

}}

== Machine-readable paths ==

{{/Machine_paths}}

== See also ==

* [[Research:Revision scoring as a service]]

* [[Wiki labels]]

* [https://github.com/wiki-ai/ores ores] repository on GitHub

* [https://github.com/wiki-ai/ores-wikimedia-config ores-wikimedia-config] repository on GitHub

[[Category:Revision scoring as a service]]