Wikimedia Meta-Wiki

Learning and Evaluation/Archive/Connect/Questions

< Learning and Evaluation | Archive
This is an archived version of this page, as edited by Visdaviva (talk | contribs) at 17:22, 8 January 2014 (→CIS-A2K Evaluation Forms: re). It may differ significantly from the current version.

Evaluation/Header

Have you created or collected surveys for your program implementations?

Program leaders have been inquiring about survey design since the creation of the Program Evaluation and Design team here at the Wikimedia Foundation.

We’re excited to let you know that we are proceeding with exploring how we can support your efforts to survey participants in your programs. In order to proceed, we need your help! We would like you to share any surveys that you have sent and collected with our team.

This will allow us to learn more about what types of information you are collecting through surveying, and the types of surveys you are producing. This will also allow us to work with you to build tools for better evaluation and surveying!

 
Surveys are important tools in evaluation!
  1. You can email me your surveys in email or as attachments (sarah wikimedia org)
  2. You can email me a link to your survey (sarah wikimedia org)
  3. You can post a link to you survey on wiki in response to this post.
  4. It can be in any language

We’ve already identified a number of survey priorities, and while survey development will be ongoing, we want to be sure to include the interests of everyone in our planning process. As we identify, prioritize, and begin to take steps to meet the survey needs of you and other program leaders, the more you are able to share the better we can learn from what you’ve done, and the better we can respond to everyone’s needs. If you have developed or used surveys to help tell the story of your program, please send them our way so we are sure to review and consider everyone's interests.

We look forward to your sharing and thank you for your time and amazing work. -- SarahStierch (talk) 23:57, 11 December 2013 (UTC)[reply]

Hi, the main feedback form for training events/editathons that we use is this: wmuk:File:Event_feedback_form_template.pdf. We have it set up in Survey Monkey so that data can be input and we are able to run summary reports to analyse the results. The form was seen previously by the PE&D team and the feedback was very useful! Daria Cybulska (WMUK) (talk) 09:03, 12 December 2013 (UTC)[reply]
A copy of the only survey TWG has done so far can be found here. The results have been used in two pieces of research we have done to identify costs related to our work- --LauraHale (talk) 18:30, 15 December 2013 (UTC)[reply]
Great, thanks User:LauraHale and User:Daria Cybulska (WMUK)! SarahStierch (talk) 19:26, 16 December 2013 (UTC)[reply]

CIS-A2K Evaluation Forms

Hi, we at CIS-A2K have recently recruited a person into the team. As part of this I have used this evaluation form to seek feedback from the community members. Will share some more soon :) --Visdaviva (talk) 06:54, 21 December 2013 (UTC)[reply]

Wikimedia Train the Trainer Program participant feedback form —unsigned by Visdaviva (talk)
Great! Thanks User:Visdaviva! Two questions: Would you like feedback for improvement (if needed) from our team? Second, if so, do you have a deadline for one or both of the surveys that you would need that feedback by? Thanks again for sharing! SarahStierch (talk) 17:15, 23 December 2013 (UTC)[reply]
Sorry for this delayed revert SarahStierch. Been swamped :( Would appreciate feedback. No hurry, do take your time.--Visdaviva (talk) 17:22, 8 January 2014 (UTC)[reply]

Dutch Evaluation Forms

To respond to the question about surveys: We are quite busy with streamlining our entire workflow from idea to evaluation for events. It all starts with a checklist, more of a grocery list with the forms that are needed for an activity. These depend on:

  • Is there financial support needed for the activity?
    • If yes: you need a google spreadsheet for the costs
  • Is support from the office needed in organising the activity?
    • If yes: We need to reserve and monitor the hours of staff involved
  • Will there be a training for new editors?
    • If yes: you need two surveys and two project pages (work in progress). The first survey should be held during event and the second survey after a couple of months. The project pages are on our chapter wiki and on the Dutch Wikipedia. The project page on our chapter wiki consists of all the basic information (type of activity, date, ___location, etc.). The project page on Wikipedia consists of the Wikipedia related information (articles to write, articles that need improvement, who's working on the article, etc.)
  • Do people have to register to attend?
    • If yes: you need a registration form (currently a google form, this will be a form linked to our CRM in the near future)

Once someone has filled in the grocery list we help them setting up the documents. The forms for the costs, surveys and registration just have to duplicated by a staff member and shared with the volunteers that help with the activity. The reasons that a staff member duplicates these forms are:

  1. Surveys can not be made public available (downloadable) like spreadsheets and text documents.
  2. This way we - the staff members- always can get to the information and fill in the information fields that we don't expect volunteers to do (for example: every event will get a unique identifier that needs to filled in on all documents).

All the forms and documents mentioned and linked above are in Dutch, please let me know if you want anything translated. Ter-burg (talk) 13:30, 12 December 2013 (UTC)[reply]

Thanks User:Ter-burg, we will let you know if we have any questions. Are you seeking any feedback for your survey questions? SarahStierch (talk) 19:27, 16 December 2013 (UTC)[reply]
Yes, Sarah. If you have any tips for our surveys that would be great. I'm especially interested if there is specific data that would make our events internationally comparable. I've made a lot of progress since this post. We now have a "manual" on events that leads a volunteer through all necessary steps of setting up an event with a lot of templates and standard forms. These surveys are part of this process.


Oh, and I do have a question. Can you translate the following questions from survey one for me? Thank you.
  1. Was vooraf duidelijk wat het evenement in zou houden? informatie via website, facebook, email
Translation: Did you know up front what the event would be like? Sometimes visitors come to our events with different expectations. With this question we hope to determine if we've managed their expectations in a good way.
  1. "MEE BEZIG gebruikt bij artikelen" (from Wat heeft u tijdens de edit-a-thon kunnen doen?)
Translation: I've used the template that indicates I'm still working on the article. We have a template that (new) users can add to a page to indicatie that they're working on it at that exact moment to hold off more experienced and faster editors that might delete their edits before they even have the chance to finish their work. We ask all the participants at an edit-a-thin to use this template, but sometimes they forget to use it.
  1. "Een afbeelding geplaatst" (from Wat heeft u tijdens de edit-a-thon kunnen doen?)
Translation: I've added an image to an article
SarahStierch (talk) 01:03, 17 December 2013 (UTC)[reply]
Ter-burg (talk) 10:05, 17 December 2013 (UTC)[reply]

Rationale for the privacy form

I can not understand the rationale for having participants in a course/event/edit-a-thon to fill a form to opt-in in Wikimetrics. In principle I could get any set of random user names from the history of any page, create a CSV file, upload it on Wikimetrics and analyse the users' activity. Without even meeting anybody. So why for events do we need an opt-in form? Again, in principle I could do this kind of analysis downloading the complete dump of a given edition of Wikipedia and write some tools to calculate the some things that Wikimetrics does. All that without asking anybody. Given this premise, with Wikimedia Italia we organized recently a course for librarians (see the project page) where every participant created a personal account. We then asked everybody to put their usernames in that page so that we could monitor their progress with Wikimetrics. We did not know about this opt-in form until today and so they have not signed it, even if they have been explicitly warned during the course. What can we do? Thank you. -- CristianCantoro (talk) 11:37, 10 December 2013 (UTC)[reply]

Thank you Cristian for your comment and chance for clarification. You are correct that, in many cases, participant user names are a matter of public record, in that case it may not be necessary, but a matter of operating in a completely open manner in which participants actually approve of their being tracked and monitored via Wikimetrics which make it rather simple to do so. Importantly, Wikimetrics increases accessibility to metric data in a way that user names are transferred to Wikimedia Foundation (internationally to the United States) and to best cover everyones privacy interests and protections we need to allow them to opt in, as well as out, of this monitoring when we know we are collecting usernames explicitly for this purpose. The particular language and guidance for disclosing the who, what, where, and why that have been thoughtfully crafted by the WMF legal team to be sure privacy interests are protected and transparency is upheld. This is a shift in practice that we are working to assist program leaders in coordinating so that the future collection of usernames will best protect all parties interested. The usernames you already have public record of are still useable, we just recommend that you also allow people to opt out should they become aware of this usage of their usernames and wish to be excluded from monitoring (you may want to post the language and possibly instructions for "opting out" on the page they added their names to). I hope this helps, please let me know if I can answer any further. JAnstee (WMF) (talk) 21:58, 10 December 2013 (UTC)[reply]
Hi User:CristianCantoro and JAnstee (WMF). I'm going to give a quick "TL:DR" about this:
  • It's so we can be more transparent with the participants so they know what we're doing and in case they have concerns we can cover our butts with opt-in form documentation and terms of service (if they sign up on wiki).
  • Perhaps it's different with your events, but, for me, sometimes people come to the event and don't sign up or even participate on the event page (but they edit at the event). Also, sometimes you'll have people sign up online but NOT attend/participate (online or at the event). So having an opt-in sign in sheet is an additional way to make sure everyone who attended is accounted for.
  • And yes, like Jaime said - perhaps someone realizes that they signed up on the wiki event page, and they'd rather remove their name and opt out of being in the cohort. While sure, it's available and you're savvy enough you could dig and not tell them - transparency is the key. Some people - especially newbies - might not be aware of what happens after they sign their name on that event page, and perhaps they don't want to be a part of the research. Even if it's there in the history, it's important that we give people the option of being used in research or not. It's about being honest, transparency and respectful of participants.
Hope this helps :) SarahStierch (talk) 22:30, 10 December 2013 (UTC)[reply]
Also, User:Harej has started using the opt-in form at the Wikimedia DC events, which we share on the opt-in documentation page. He might even have some additional insight into why it's important for program leaders to use. Or he might not :) SarahStierch (talk) 22:31, 10 December 2013 (UTC)[reply]
Sarah and Jaime covered most of it. Part of it is that the on-wiki list of users is not comprehensive, since some people create accounts on the spot and others attend without signing up ahead of time. The main benefit, however, is that it makes our work easier. I would rather have a form generate a list of usernames for me than have to scour Wikipedia to put together a list. This helps save time, which is great because we're all volunteers and want to do quantitative analysis with as little effort as possible ;-). It also allows me to conduct surveys of attendees; I'm especially interested in measuring what percentage of attendees come from the organizations we co-host events with, but you could also ask other questions. harej (talk) 00:48, 11 December 2013 (UTC)[reply]
Hi Nemo! Great to see you here. We're not responsible for maintaining the mediawiki documentation. I think that is a task for the analytics team. We actually do discuss it in our module, however, that was built and developed by the Program Evaluation and Design team, not by analytics. I'll ping them and let them know your concerns. I can't really comment about Wikimetrics and Wiki Loves Monuments, I'll defer to others to respond about that. SarahStierch (talk) 18:01, 11 December 2013 (UTC)[reply]
As mentioned in my response above, usernames that already exist as public record are a grey area when it comes to this guidance for program leaders conducting in-person events and their collection of user names for use in tracking and monitoring contributions via Wikimetrics. Technically everything on wiki is public already, however, WikiMetrics introduces an ease at tracking and monitoring online activity and changes the game a bit. Importantly, once we are interfacing with community members in-person and collecting names of people who participate in those events we introduce direct data collection for which there may or may not have been a public record. There are some important precautions that we should then take to protect any participation information that may be sensitive. When we do this systematically, with intention to track and monitor, as we are guiding program leaders toward, we should disclose that we are doing so and give people the option to opt in or out. Also important, it is my understanding that this "opt in" is not a requirement of all users of Wikimetrics, but guidance for program leaders collecting usernames of participants of their in-person events. JAnstee (WMF) (talk) 21:01, 11 December 2013 (UTC)[reply]
Ah, good to know, then I can ignore the Meta-Wiki page and refer to the official docs. Thanks, Nemo 22:39, 13 December 2013 (UTC)[reply]

Hi JAnstee (WMF), SarahStierch, harej. Thanks for your answers (I am sorry for my late answer). I think that we can evaluate from case to case what to do, in general we never collect data such to be able to relate a given user name to a given person (i.e. a name and surname) and, in general, we have (so far) always used only publicly available data. I understand that in the case in which we would collate public data with non-public data (e.g. name, surname, age, profession) we would need some kind of opt-in and privacy statement. Wikimedia Italia already has its own privacy policy following the Italian law. Also, some of our members noted that such a statement could scary (unjustifiably) some people if we present it when just using public data and also add a distorted impression of "bureaucracy" in the Wikimedia projects. So we should use it only when needed (i.e. when requesting data which are not already public). -- CristianCantoro (talk) 14:32, 16 December 2013 (UTC)[reply]

Hi User:CristianCantoro! I do want to clarify, just so we are all on the same page:
  • If you or your chapter intend on collecting usernames of people who participate in "in person" events (edit-a-thons, workshops, wiki takes your city, etc), then you have to use the opt-in forms. We know that some people might get scared - that is one reason why this opt-in form exists. We have to disclose to them. If they express unease or fear about data collection, we suggest that program implementers (like you, the person who plans and implements the event) explain more to them if needed, and if that works, they can opt in, and if it doesn't, they opt out. Giving a choice at in person events is important.
Hope that helps just to clarify again. SarahStierch (talk) 19:51, 16 December 2013 (UTC)[reply]
[citation needed] Nemo 07:13, 18 December 2013 (UTC)[reply]
Can you perhaps expand on the point you're implying by adding this citation needed tag, Nemo? It's not very helpful on its own. Jtmorgan (talk) 21:43, 18 December 2013 (UTC)[reply]
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Learning_and_Evaluation/Archive/Connect/Questions&oldid=7001429"