Form processing
editWhen an URL gets submitted via the WikiSignals form in Wikipedia:
- Fetch URL data (HTTP status, Wayback, Wikimedia) from APIs.
- Parse ___domain from URL.
- Lookup the ___domain in WikiSignals' JSON dataset (a local Wikipedia file).
If ___domain is found in the JSON:
- Display the URL/___domain data in the form.
- (Record a count of times users submitted this ___domain?)
If ___domain is not found:
- Fetch new-___domain data from APIs.
- Some API-calls are fast, some slow. Make the calls in two groups: fast and slow.
- (Display the fast data as soon as we get it, and not wait for the slow?)
- (Should form ask user for website type: e.g., news outlet, science journal?)
- Display the URL/new-___domain data in the form.
- Add the new-___domain data (fast and slow) to the JSON.
Crude code (JavaScript) illustrating the above:
const usrUrl = {URL_from_User}; // Submitted via WS form.
const usrDomain = getDomain(usrUrl); // Parse URL.
const WS_JSON = getDataset(); // Get local JSON file.
// Get ___domain data from WS_JSON if found, else var is undefined.
let domainData = WS_JSON.find(item => item.___domain === usrDomain);
// Get data about domains not in JSON.
if (domainData === 'undefined') {
fastApiData = fetchFastDomainData(usrDomain);
domainData = fastApiData;
}
// Get URL data (whether ___domain is in WS_JSON or not).
fetchUrlData();
// Display URL and ___domain data below form, print more form fields.
printUrlData();
printDomainData(domainData);
// Get more data about ___domain not in JSON, then save to JSON.
if (fastApiData) {
slowApiData = fetchSlowDomainData(usrDomain);
saveDomainData(fastApiData.concat(slowApiData)); // Write to WS_JSON.
}
MVP
editFor WCNA2025 October talk, do we use the existing mockup or build something with real data, like pulled from a JSON dataset of info we already have for about 20K USA news domains?
If we decide to build it out even further by making API calls for domains not in our JSON, what are the minimum indicators we'll fetch?