Date: 2021-02-18
Facilitator: Bruno
Participants: Vasilis, Katie, Bruno
Agenda
check in
reviewing proposals on voting mechanism
question: choose one of the proposals or make a prototype for each of them?
Katie’s feedback:
is it necessary for only editors to create proposals? Makes sense for technical purposes, but not for community purposes. But maybe not an issue given new people won’t be so engaged
Likes the timescale
Was on board for the temp check on the beginning, but maybe too complex now. Having score voting may achieve what temp check proposed (given they can give neutral “0” voting).
Anybody can vote! Definitely
Low participation threshold for approval of proposal
Wonders if people would use the neutral vote, but likes the weighted system
Make visible proposal and voting so people can engage
Vasilis: trying to picture different scenarios for using agreements and voting, combining in-person and digital
how to take the in-person agreements and discussions and move them to the digital? Maybe some will never end up them
to give a voice to people who were not present at meetings, maybe better bringing the discussion/decisions to the digital. In Katie’s group: the admin people could be responsible for bringing that in
Katie: there is currently the question of were to put the agreements, so maybe it can be useful. Relying on the wall where things get lost.
Bruno: very important to keep in mind the combination of in-person meetings and the digital
Vasilis: let’s ask the groups! Maybe a questionnaire, interview or something…
Bruno: there’s a scenario of board meetings and decisions that might not fit well, but that’s fine, we want to have more ample democratic participation in the group
Katie: should probably not design for our cases, and maintain the values and culture of Karrot
Another scenario: agreement proposals being created before/during each meeting.
Bruno: maybe it’s useful to set a specific date related to in-person meetings?
Which proposals/ideals are we going to choose?
maybe we can test some different ideas on different prototypes
try both a more complicated and less complicated one and see what ideas come from the people
let’s think about other criteria as well that we didn’t include in the previous proposal: like anononymity, feedback on negative votes…
Proposal to go into the protoype (simplest)
only editors can submit proposals (katie doesnt morally agree, me neither, but it wont be a problem–> its easy to get karrots thus its ok)
timescale: minimum 1 week + suggestions + custom → pick a specific date
any editor can change the proposal.
What happens when there’s a change? Votes reset? Suggestion: people get a notification of the change and are asked to review their vote
no temperature check
everybody can vote
score voting with negative weighting (2, 1, 0, -1, -2)
anonymous voting
negative votes require a reason and they’re kept anonymous (“explain why you don’t like it”)
At least 5% of members should vote for an agreement to get approved
main screenshot from foodsharing stockholm (its in english)
most probably there is not gonna be a problem to use it
Vasilis and Bruno landing page notes:
/welcome → landing page
access to the about page on the landing page (bofore log in) and accordingly when you are logged in
about page: a seperate page with texts and resources and links + some text existing on the current version of the landing page
what should we put into the about page:
texts…
resources → organize them by ways to participate (e.g. willing to participate as an activist, programmer, foodsaver etc)
mastodon
community forums
github
foodsaving.world (maybe not?)
sceenshots (slideshow) u have to click or slide:
- we got some screenshots
- groups on karrot
- activities
- various types of activities (party, pickup,costum activity)
- history
- offers
Nick governance process Karrot Prototyping <–link
- when proposing a new agreement → calendar for now when setting up for how long the agreement would be debated. If you select the current day → til midnight the same day
- the prototype looks good so far. we suggested that we do not make it look like a finish design but keep it in a way that shows that there are possiblities
might take some of the stuff from the welcome/landing page
for people who might want to get involved, or know what kind of people/structure behind karrot
could also include values and vision, maybe towards the end of the page
could add a team page, but maybe later
how to describe getting involved / onboarding process
including the information directly in the about page makes it harder to change (and more likely it gets out of date)
getting in contact with one of us
maybe different profiles of people (e.g. devs, designers, community, academic, …)
idea/proposal: create a wiki forum post that has the definitive information
can link from the about page, or maybe even pull in the content via the discouse API
Nick will show progress on the protoyping
Solutions for saving data can be explored
Editing boxes seperatlely
Value tags can show the context
if proposal is changed would it reset vote?
what about submit button?
put voting at the end? to encourage them to read?
+1 for emoji voting
does the person who created the proposal get to vote?
split the proposal voting / chat page from the editing page
we didn’t think of how to get rid of an agreement…
paper/workshop (katie)
position paper for a small workshop around politics and technology in conjunction with the Annual Political Science Days in May 10th-12th at University of Helsinki
~200 words
due by 14th March
Katie happen to write it, but not available for presenting it, Vasaleios and Bruno “WE CAN DO IT!!!”
maybe some content relevent from the NordiCHI
Working breakout
Just do it and report back next time
checkout
Actions and outcomes
create a how to get involved / onboarding wiki forum post (Nick to set up, others can jump in to edit)
Katie will share the next draft of the welcome page to get approval for the FS-STHLM Screenshots
Vasileios feedback: Bravo to Nick, wondering about the chat function. Suggestion for editing an existing agreement its nice to gave the option to add tags. It’s not so easy but should it be?
Nick wonders if we need to add some more descriptive explanations?
Make a manual
Who will we trust it with and how
“reason for proposal” still probably a bit confusing
add values
include the language a bit more instead of just boring “add values”, maybe work in the “this agreement will help to support… language”
can also include the language in the dialog, “To support …” “we encourage…”
it looks nice when they are added, so nice to include some of that style before any have been added
adding more radical values?
educative/performative role for karrot too
e.g. queerness, anarchism, degrowth
maybe groups then start discussing the topic?
a session to rethink the categories and the values
we do want to think more about the values and categories at some point, but could still proceed with testing
Agenda:
from previous call
refine prototype (Nick)
improve mobile/small screen behaviour (including separate button/whatever for the chat) (done)
incorporate more of the values language in the interface, as disussed above (done)
plan a session for refining the values and categories (to do)
schedule a co-working session for user testing (to do)
start working on user testing script (Katie & Vas & Bruno)
I have started a testing working plan (Katie)
plan and logistics of testing the prototype (Vas & … )
Some notes/thoughts from Vasilis
Karrot prototype logistics
Finalize the prototype
One prototype (?)
Instructions text
Scenarios (bruno & katie got from the ground experience)
Mobile or not? or both?
Tasks for the users that going to test it?
Single user testing
Multiple users testing?
Single user testing (2 users from a group on the same time)
Which groups? Consider different organizing structures and culture of the groups and include some question for the background info
Mentioned: Warsaw, Luxembourg, DLC in France, Robin Foods Austria
Stockholm ← Katie
Gothenburg ← bruno
Less ‘traditional’ food sharing groups (e.g. community fridge kopenhagen) ← who and how we approach this group? (Katie said knows some people there)
How do we do the testing?
call and we record their screen
They do it on their own time and then they give us feedback
Do we need sth like a consent form?
How data are going to be used?
Katie? Vasilis? Bruno? Nick? Will we use those date for writing sth?
Expectations?
Timeline?
new agenda points:
Katie’s notes on the prototype testing to be found here:
2 versions of prototype testing
have an empty canvas
bring a sample rule and plug it in to the system
propose an agreement
reviewing/voting/discussing/participating
who we recruit?
questionnaire: like a survey demographic
age: how comf people feel with technology (you get away from ageism)
the easiest to recruit → they can give us the less active
contact active and ask them to find less active (pyramid sampling)
stockholm
gotemburg
big groups active on the forum
warsaw? lux? efa? french groups (nantes) robin food (austria). Fællesskabet (KBH)
more top down groups
debriefing interviews after the prototype testing
what do we ask?
what have been looking for?
e.g. bruno: top down groups views on a more participatory approach
outcomes
katie will work on the scenarios
pilot test for the testing (vas, bruno…)
checkout
Actions and outcomes
refine prototype (Nick)
write sample agreements/proposals
maybe split the luxembourg one up to use as multiple sample agreements
(optional) include the section for vision/organisation boxes
work on the questionnaire for before the testing and the debriefing (Vas and Katie)
do a pilot test (Katie, Bruno)
How we move on?
Katie’s notes testing the prototype
Participants:
Users should be english speaking, active members (with editing permissions) of a foodsharing group on Karrot.
We could aim for a sample of 10-15 participants for the first round of testing.
We may also run A/B testing on different versions of the prototype e.g alternative voting methods text or slider
Testing will be conducted remotely on Jitsi or BigBlue Button where test participants will be provided with the link to the prototype and will share their screen while conducting the tasks.
We may also run A/B testing on different versions of the prototype e.g alternative voting methods text or slider
Test Setup:
Welcome the user to the test and give them some background information and a brief explanation of what will be involved.
Have the user fill out a background questionnaire (approx. 5 questions, e.g. demographics, experience with Karrot, role in their foodsharing group, group culture etc)
Have the user sign consent for participating in the study and how you will manage their data.
Have users conduct each task while using the “Think aloud” protocol (audio/video record the session)
Observe the user as they conduct the tasks and take notes on what you see
Conduct a short debriefing interview with the participant.
Features to test:
Agreements
Proposals
Group vision and decision making?
Scenario 1: “out of the box”
What kind of scenario do we want to test? Should we begin with a bank page experience where we assume that we have just implemented this feature to Karrot and users are exploring it for the first time?
Scenario 2: Working with e
Scenario 3: Revewing, Proposing, Discussing
Use Cases:
Create a proposal from a sample (Participants own agreement)
From the home screen, navigate to the proposals page
Bring a rule from your group (Have a sample rule just in case)
From the home screen, navigate to the proposals page
Proposal you rule in the system
Participate in an exisiting proposal (Our sample proposals)
From the home screen, navigate to the proposals page
Select an existing proposal (from 2, one boring and one provocative)
Read the proposal
Make a comment
Cast your vote
Change an existing agreement (Our sample agreement)
From the home screen, navigate to the agreements page
Select an existing agreement
Read the agreenent
Propose a change
Background questionnaire Questions
approx. 5 questions
demographics,
experience with Karrot,
role in their foodsharing group,
group culture etc
Debriefing Interview Questions
Language
Values
Documenting and analysing results
Present the results of the test detailing any successes and/or failures the user has had with conducting the tasks and their overall experience.
Write an analysis of the results detailing any usability issues detected and why they may have occurred
Date: 2021-08-15
Faciliator: Nick
Participants: Vasilis, Nick, Kristin, Bruno
duration (max 1h for Kristin, 1h30 maybe for the rest)
checkin
introduction with/from Kristin
Comments from Kristin: high-fidelity prototype. Never done a prototype like this. Complex issue and wants to know how we tested and then tell how she’d test it. Seems clear how we want to proceed.
Nick’s open to big changes if that’s the case, depending on the feedback
Recruitment of people already using Karrot
Comments about the duration of our design process (diffrences between business-setting and open-source voluntary)
includes the first draft of the user testing script
initial questions, “talk aloud” protocol, tasks to do
feedback from Kristin
in general, a common way to do it
starting with a familiar scenario
not being too specific with instructions
asking questions in the beginning
probably not worth testing with people who are not familiar with karrot already
live?
good to have two people, one to lead the talk, and then visitors to observe. after that the two people can discuss.
the moderator can then ask the visitors if they have more questions during it
can do it with just one person, recording video/audio, audio is actually sufficiently mostly, especially with immediate debrief
usually testing 3 people, literature says 5, but 3 ok for “quick and dirty”
afterwards come together as a group, then from all notes select 3 very good things to make these very clear, then 3-5 most important issues (could be technical/design/etc) that were troublesome
the recording can be nice, but prefer to do more low-effort tests more than record more, then can change things and test again…
question: more meta conversations to have afterwards?
e.g. would it make the group more democratic
a good idea! but problem with asking vs showing, observing vs listening. people sometimes say things are fine, when observing can reveal more information.
people are biased and say “yes it’s very nice!” but not clear if they will use it
question: we do collect some usage data, but struggle to make use of it
need more clear definitions of what we’re trying to achieve
e.g. using it every day (maybe for whatsapp, but not for all apps)
feedback from test we run with a member of Solikyl (Gothenburg group)
- difficulties with quantitive vs qualitative
- finding out if they are happy using it can only really be collected qualitatively
- question/reflection: recruiting different types of people, two profiles: “active” people (who would write proposals), and “less active” (we want to make it easier for these people to participate more), idea to ask active people to also bring in someone less active to do user testing?
- usually use incentives, e.g. a lunch, some yummy jam
- in this kind of project, seems nice idea to get people to ask someone else
- would not ask beginners to use the prototype as it is, due to complexity of the feature, perhaps trying out the whole app
- maybe can find a nice incentive for a short as possible user test, usually people like being asked and people enjoy expressing their opinion
- question: about having different types of sample data?
- this is more about the content though, not the feature, so not so important
- if we have multiple versions, “tinder testing”, “swipe left/right” concept for getting feedback from quick questions that can be put to people
- risk to try and have too much control where we don’t really have it in our prototype… some things might work better inside the app later…
- our prototype is very advanced right now, so hard to do small incremental adoption
- would not make any singificant changes now before next user test, maybe just minor ones
- writing notes after each user test right away, even if debrief happens later
- creating summary at the top of the notes with the main points, including things like quotes from the person for personal touch
- trying to include the whole team too, e.g. including developers
bruno was faciliator, vasileios was visitor, and did a chat later
was nice to get over the technical hurdles that came up
was interesting with Joakim, when he wanted to make a proposal, he used the edit text like a chat box rather than just editing it
reflections from chat with Kristin
not making big changes on prototype, just small bits
should lower the number of participants we aim for, perhaps 5, not 15
had noticed from a previous user testing how much we got from so few
perhaps background perspective of making things “fast and easy” for users different in karrot (democratizing processes in karrot)
keeping a balance of usable, but not necesarily “easy” (democracy is not easy)
perhaps don’t need incentives, as karrot is built on voluntary open contributions
could think more about this scenario usage (to “provoke”), but also realising the real testing will be in real life when it actually happens, but could try something within the prototype, making the test subjects into people with their own thoughts
already got some feedback that he might be using it differently as it’s not a real scenario, not getting into the context, only the feature
perhaps general principle that the people are not just there to be tested, political framing of user testing
aligns with Karrot’s ideas on how to bring forward the social and political context, a challenge to traditional user testing
user testing as user participation, it can still make sense to a bigger complex feature
could have we broken down the feature to smaller parts?
putting everything on forum
wondering about refining our interview script
especially the feedback and discussions from today
checkout
Next facilitator: Vasileios
Next meeting: 2021-04-22
Actions and outcomes
start another thread “Stage 5: Testing” on the forum and put the interview script there (Vasileios/Bruno, race!)
include notes from tests on the forum as well (all? for the future)
rework agreements page to have “approved/proposals” tabs inside the agreements page + change new proposal button to say “propose new agreement” (and perhaps put it at the top) + fix the width issue with the chat (Nick)
refine the interview script based on meeting content (?)
consider conducting another user interview, e.g. Daniel (?)
Short recap from the test we run (Bruno & Nick) with a member of Gotenburg group
observation notes and some reflections from this session can be read here: Stage 5: Testing the prototype - #2 by bruno
Preparation for the test we run with a member from Stockholm group
Katie is the facilitator
Nick is the observer
Vasilis and Bruno also take notes if they stay in the call
We ask the participant if it’s OK for them, all 4 of us to stay in the call
If so, we can all ask questions to the participant at the very end based on our notes
All 4 for us stayed in the call finally
confusion between approved and proposed agreements
one idea from Karolina: put the proposed on the left, maybe some notification (badge)
maybe get rid of the summary, or making it optional
make the box smaller, both for summary and reason to change
reason to change only when proposing a change
one way to solve the question about authorship and people not feeling comfortable changing each other’s texts
make it possible to write only on reason to change and not edit the content of the agreement itself
how to discuss an approved agreement?
maybe the model of issues and pull request on Github
interesting when something happens (issue) and the outcome is not clear. Outcome could be a conflict resolution, a new agreement, etc.
more general purpose discussion + decision making
an “action button” that you can take an existing discussion whereever it is happening (e.g. on the wall), to spawn a new “proper” discussion to then do something, with the ability to then see where it came from
how do we proceed? do some small changes and implement, or do another round of user testing?
Nick is fine with both options
Katie agrees. Do some small adjustment, get rid of the values. And then do one or two testings
Vasilis agrees with Katie. Do a couple of more interviews. Do we set a deadline? Do we have the energy to keep working on it? We do have time to work more on changes, maybe even the “github scenario”
Bruno. A bit more wanting to get stuff done and out there, but can see perspectives from others, so could be useful!
redesign a little
make proposals more prominent
maybe with “notification icon”, or at the top of the list
removing the “reason” for new proposals?
remove values? (maybe one day incorporate it more in thte context of the whole group)
a simpler one level version of it
maybe make it clear that they are the values of the group (faked in the prototype of course)
include some fun/anti-values?
adding (optional) bits to optional things… (reason, summary)
making summary smaller?
making it clear/possible that people don’t need to actual make changes to the text initially… could just include their “comment” in the reason field
maybe explicit choose for whether to make concrete changes?
if no changes proposed, then make that clear on the proposal edit, and not possible to vote until a change has been made…
anarchist cybernetics?
pick next facilitator
checkout
Next date: 2021-05-20
Next faciliator:
Actions and outcomes
refine prototype based on ideas mentioned above (nick)
… depending on progress of that, schedule some more user tests! (anyone)
Confusion between approved/existing agreements and proposals and where to find them (1, 2, 3, 4, 5, 6)
needs clarity with terms
both the approved and the proposal/discussion are visible on different tabs. Confusion between what is the agreements that needs to followed in a group at the time (bruno)
who gets to edit and social norms about it (1, 3)
only editors? (6)
timeline and showing the author of a change would be better (6)
notifications, how would it work? (2)
notification that sb voted (1)
negative voting
explanation should be clearer (4)
does not preserve anonymity (6)
where would discussions take place? On the agreements proposals, wall, or somewhere else outside Karrot? (2, 3, 4)
Voting vs having a say (2)
reminder to re-evaluate proposals periodically (5)
how many should vote for an agreement to pass (3)
propose changes (can change title and summary?) (Vas)
should we have a discussion under approved agreements? (4)
“because I made it, I cannot do resistence voting” (didn’t notice warning) (5) → a hover message over the disabled butons
not clear: propose change → choice between “Start the discussion first” and “Propose Changes first” (Bruno)
instead of “start a discussion”, could be “open for discussion” (5)
wondering how the summary will be displayed (6) <–could there be a display option before posting the changes/suggestions?
“depends how I want to work” when thinking about discussion first, or proposing changes first (6) → propose a change does not exclude starting up a discussion as well
Guidelines for creating an issue on github for the development of the agreements feature
Solutions/changes to be made
should be changed
- “Proposals/Approved”
- get rid of the tabs
- all agreemetns and proposals on the same page, as a list
- proposals should appear on top
- add a filter “All, Only proposals, Only approved”
- highlight approved as ‘stamped’ and maybe add a “check” icon
- maybe use an icon, a visual cue, for the proposals which indicates that they are ‘open’ for discussion/edits
Propose change to approved agreement
- Instead of “Open Details” put “Show More”?
- Add button “Start a discussion” next to “Propose Change”, instead of clicking first on “Propose Change” and then “Start a discussion”.
- When interacting with a proposal use “Propose change” instead of “edit proposal”
Make it clearer why people cannot vote negatively without commenting
- maybe a hover message over the disabled butons?
- maybe a warning message when trying to vote (resistance/strong resistance)?
Clearer vote confirmation
maybe a toast “Your vote has been casted”
Show somewhere information about the votes needed to approve an agreement/change
percentage: 5% of members should vote for an agreement to be approved
show the score of each voting option (e.g. neutral=0, strong resistance = -2 etc) and provide an explanation/information on how votes are counted
take a look at the conflict resolution feature and make it consistent. Maybe improve explanations/information on the conflict resolution feature too
how the feature will appear on Karrot
- On the sidenav menu, an item called agreements
- We should probably discuss a redesign of the whole sidenav menu, which items appear
Notifications
we probably never discussed how it should work but we suggest:
- Bell notifications for everybody whenever an agreement is created, opened for discussion/changed and approved (or not?)
- Discussions/Conversations about an agreement: only to the person who created/changed an agreement and those who participate in the conversation either on chat via editing
- Should there be a reminder (x days left to decide the agreement)?
would be nice to have/change
when sb applies to a group would be nice to be informed about the agreements/rules a group has approved
Other notes/thoughts
what happens with existing very simple agreements? (1)
first move all the rules to Karrot which is good (everyone has Karrot, not Facebook) (3)
its sort of a confirmation
how would this new feature adjust to karrot?
when somebody joins, something they should read? (2)
who can vote (2) only trusted? not karrot trusted can leave a comment?
everyone can vote
Actions and outcomes
make a post on karrot group activity on Thursday: we discuss about the design process in general, feedback, reflections, thoughts blabhabla
post notes on forum (monday and last thursday)
issue on github
pick next facilitator
checkout
Next date: 2021-13-09
Next facilitator: ??
and people enjoy expressing their opinion