Initial Design Process: Stage 5: Testing the prototype

bruno · April 22, 2021, 12:55pm

Here we collect notes from testing and reflections.

Script for testing

Preamble/context…

background information
- governance design process trial!
- user/group participation
- involving people from groups in Gothenberg and Stockholm, academics, and me…
- improving governance features of karrot
- lots of notes on the forum…
brief explanation
- prototype, not the real thing, doesn’t connect to other users, only in their browser/computer
- it’s not a test of them, but of the feature
- 3 scenarios to go through
- one person to lead the session, the others are observers
- afterwards some chat!
instructions to think out loud as they go through it

Propose an existing document

Start with a written document that you want to use within your group
On the home page press “Start with empty data”
Make a proposal for the group to accept the document

Participate in an existing discussion

Go back to the home page
On the home page press “Start with sample data”
Select an existing proposal
Make a comment in the discussion
Cast your vote

Change an existing agreement

Select an existing agreement
Read the agreement
Propose a change

bruno · April 27, 2021, 8:25pm

Notes from the test of the testing of the prototype with a participant from Solikyl (gotemburg)

How do we record? Dropbox account? Jitsi?
We need sth that records both voice and screen

General information
The little tasks
Talk outloud as you do stuff
it took us an hour and a half

Suggest an agreement/make a proposal

Go back to check how other agreements look like as an example → example agreement

How do tags work? Is it for a better search?
tags do not always read good

We missed one bit → timeline for agreement
Make it more clear that is a proposal for an agreement
Become a proposal → surprising…
Date doesn’t come back when your editing a proposal
Plus button can be misleading
Could be different → established agreement

Participate in an existing proposal
Negative voting can’t see the rest of the text
He didnt get the connection with the text box (chat box)
Write a message doesnt sound ok → does not promt you to make a comment

Propose a change to an existing agreement

Text with values doesn’t read good actually
Edit proposal vs suggest changes → can you change the title?
Istead of making an edit participant wrote my proposal is
I wouldnt overwrite someones text?
Who has the ownership of a proposal?
Voting control?
Not participating in the chat? →
Question using a proposal? Should solikyl take money?

notes

jitsi recording works good (both voice and sharescreen)
recs are save in a dropbox account (can we have a shared dropbox?)
agreements tab: can we improve it with some text when it’s an empty screen? make it more intuitive, a better explanation
proposals and agreements are the same thing when there are no data…so maybe we only need the proposals tab and option
established agreements? → no discourse how this works? how we support it with the software?

Reflections

importing initial established agreements, not clear that it is just making a proposal
- hence didn’t notice the time period part, it was just meant to be an initial “import"
wondering the purpose of the value tags?
- searching…?
- for us the text is clear explaining what the reason is…
when making a proposal, wrote a comment inside the text instead of editing it
- “who am I to change someones view” >(e.g. why/how is how allowed to edit someone else’s text)
- wondering why/if the comment could go in the chatbox… didn’t make so much sense
  - could work, but not intuitive
  - maybe just a issue with our framing? editing vs commenting?
    - google has editing, viewing, commenting modes (I think)
chat box covers some of the screen
- should be fixed!
confusion between agreements and proposals
- creating agreement and creating a proposal is actually the same
- if there are no agreements, have a button to propose one
- maybe making it that proposals are kind of a “sub part” of agreements and inside the agreements list UI
- maybe colour coding, and making clear there are “approved” agreements and “proposed” ones
- maybe mirrors members + applications, processes that lead to things (application → member, proposal → agreement)

Vasilis_Ntouros · April 29, 2021, 10:12am

Notes from test with a member of Stockholm group

22/04

#Task 1: Create a proposal from a sample

Instructions are better given spoken, not written
instructions weren’t updated since there is not a “proposals” page now
“I wonder if I change the time period”
icons in datepicker took a while to load
Some confusion regarding the task itself (make up a sample, or based on real-life examples?)
rules vs agreements - “where do rules fit into this page/form”
“saw the values button and just checking it out”, “I wonder if I can click these” … possibly wondered why clicking one value highlighted it multiple times? (or maybe just me…)
“I get I’m supposed to submit a proposal and getting feedback from the group”, but not sure about rules vs agreement part
“I think it might be really useful to have, it keeps on happening on karrot, there is something to be discussed, and lots of threads, but not easy to come to some sort of conclusion or agreement, so makes sense”

#Task 2: Participate in an existing proposal

test setup: maybe clearer language for setting scenario data
imaginging the list of values is things that the group has already agreed on, or whether they are from karrot itself (made more sense to him that they would have come from the group)
skipped the step to propose a change… then confused where to make a comment … ah it’s the difference between an existing proposal vs an approved agreement
found the proposals eventually “just a matter of getting used to it, if somebody tried it once it should be clear, wasn’t initially sure on the difference between the tabs” (approved/agreements)
double checked that he had done the voting, checking the title of the voting section

#Task 3: Change an existing agreement

found the “propose a change” quickly here, and it made more sense from the previous task confusion
wasn’t clear that it went into the “proposals” list… looked in the approved agreements section

Overall / general impressions

based on current stuff in the group, they need to come to a decision about how to deal with a particular supermarket, it’s currently very unclear in many threads, no time limit… can immediately see a direct use for it. decisions that need to be taken. currently have conversations within karrot and seperate groups.
rules vs agreements? not clear on what rules would be in the context. a standard set of rules, or something over time? when somebody joins, something they should read? softer word than rules could be guidelines
good idea for people to join can have a say, but maybe for new people they could spend some time to get familiar with the group first. maybe the karrot trust system is good at filtering people who are active, so better for people who know a bit know about the situation to have a say, of course comments are welcome. for voting it would be nice and inclusive to have newcomers participate in discussion and voting. good to make people feel included.
to get people involved, making it clear how it works is important, maybe with a notification about the feature, and to invite them to have their opinions heard, from experience, in general people respond to polls much quicker than discussions which are more unclear. but yeah, understanding how it works is important.
never ending discussions that have one thread, and then maybe another post/thread, seems nice idea to handle the discussions separately that need to reach some sort of agrement. currently, if people aren’t subscribed to a thread they won’t see the discussion happening.
values feature? was searching for rules at that time, so maybe got confused there. ponders about values. wondered where the set of values come from, the group? do they set them up, then after that can link them into the agreements. or a standard exhaustive set that karrot provides that people can then connect to that specific agreement. don’t have a strong opinion for giving feedback on it. ponder ponder. was quite clear what it meant, clearly under two headlines, what causes is this relevant for. lack of clarity was about trying to find the rules … and confusion about who decides the list. seems useful to link discussions to the values the group has. maybe could be more flexible having custom ones.
confusion of agreements/proposals. saw the approved agreements that looked like they could be proposals, but maybe just because trying to do a task. but understand the two terms afterwards. at the beginning didn’t have clarity around the proposals vs approved agreements (maybe we can put explanation in the ui). “propose a new agreement” → wouldn’t that be a proposal? although didn’t take too much time to get used to the meanings.

bruno · April 29, 2021, 5:19pm

Notes from user testing session with participant from Foodsharing Warsaw

Date: 2021-04-29

other feedback…
- yesterday their meeting they have every 5 weeks
- they discussed for half an hour why people post on facebook instead of karrot
  - unfair, as everyone is on karrot, but not facebook
  - it’s quicker on facebook, better for reactions, in a hurry…
  - maybe if there would be features on the wall? better? more developed
  - communication features more refined it might help, eaier, more effecient, clearer, more organised
  - could switch to use karrot for more
  - sometime end up having the situation where they agree things on facebook
  - easier to discuss on facebook as people are used to the functions
  - can you delete entries on the wall in karrot
- idea to invite more feedback from groups, multiple people from groups, have discuss, get feedback, etc…

Propose agreement from existing document

started reflection on the discussion they had in their group and that they wanted to propose a new rule
“Looks weird that approved is on the left and proposals on the right”. Was expected the other way around
term could be cleared
looked at the time scale (7 days). Talked about the conflict resolution, 7 days is too long.
- for a proposal is ok
- for a conflict is too long, too much conversation and wait long for a decision
- chose 5 days
- used the reason for proposal (short version, normally would write more)
“values? ok… what values?”
- “I wouldn’t use it” see what you mean
- “it would be difficult, I’d spend a really long time to decide what it is and not sure if it’d be useful. Maybe it would.”
Not sure whether reason and summary would be separate. One would be enough
- start the summary with the same thing “we discussed and agreed, etc”
voting
- not clear, assumes voting will happen
- sees the propose page and the voting voting
- what is enough people, regarding number of votes
  - is it % of users? how does it relate to trust system?
  - Read the chat bubble about voting negatively
    - “ah ok, seems cool/intersting”
    - Wrote something on the chat
    - maybe people would write whatever just to vote negatively
    - got no feedback from voting like “Are you really sure?”
      - Would like to have a confirmation
      - She sees however that it can be changed
      - Wait for others to vote and see what happens after 5 days

Participate in existing discussion

looked at tabs
- not sure what approved is…
- Seems like archive of old stuff, but also “coop with companies” that is under proposals
- approved through vote or just put there?
- maybe active ones should be first
looking at discussion
- opened the full document: Woaaah, long thing
show changes
- “ok, I don’t know… hmmm, ok…”"
- not sure what it was
edit
- can edit even though did not create it
prone to errors
- Karrot is not hierarchical, works most of the time
- peopple could do a mistake when editing
- not clear whether people can vote and edit
forgot to make a comment
- commented the text on the chat
edits the proposal to test it (would not do it otherwise, but ok in the context of the test)
- checked for changes and saw that her change was included
- “this is not clear to me”
- I discovered something. Normally would not like to make an error/edit someone’s work
  - I’d like to comment, not like to edit

Change existing agreement

You said “propose a change”…
- oh this has values, I thought the other one did not have
- confused about agreement on approved and proposal with the same name “cooperating with stores”
Reading the agreement, clicked propose a change
- Maybe someone could have abused it and set for one day
- 7 days default is too long
Reason for proposal is provbably reason for change. Or is it reason for rules on meetings?
- “I want to change point 6 because of…”. Assumed reason for change
- I’m confused, repeated the above
Change point 6
Checked changes
- So there is a place where people can see changes
Already I can vote on my own suggestion
- voted “of course I support it, my own suggestion :)”
- probably commented (screen froze again)

Chat after testing

Can it be useful for her group?
- Needs to think
- first move all the rules to Karrot which is good (everyone has Karrot, not Facebook)
- now they have a link on both Karrot and Facebook to google docs
- difficult to know how people would be using it
  - maybe everyone would be using it a lot proposing a lot of changes
  - but from experience, that would rarely be the case, a few people more into organizing and coordinating
  - 2 kinds of meeting: everyone and org. group
- experience from yesterday:
  - they had the big meeting and had a clear idea of the rule, but it would take time to do it together, but the smaller group would take care of it
  - expects that it would work similar in practice with the feature
  - in the end it’s like a meeting, but online
  - everybody can take part, would be an advantage
- could it increase participation for less active users?
  - hard to say… only have experience from the conflict resolution feature, don’t use it for all, as the only option is to throw them out or not… but they do start them in that case, 100 comments in 3 days!
  - not sure if non-coordinate people would use it, at least how it looks, as it’s quite complex looking, people might be intimidated, people who things a point should be changed, would rather write a comment on the wall, they would put it as topic for next meeting
- something in karrot, softer sanctions
- already have processes for making agreements,

Jitsi froze screenshare 3 times

Our reflection

confusion about the approved/proposals left/right thing
confusion when there is a proposal for changing existing one as they have the same title
fields for “reason for proposal”
- similar to summary?
- reason for the agreement itself or reason for change?
- she had the reason, maybe just changing the name would make it clear
- and remove “reason for proposal” for new proposals?
- putting “(optional)” might make people for more comfortable to leave it
- clearer distinction between meta content (duration, reason) and the actual proposal (or “agreement”)
social norm of “changing someone elses work” echoing joakim
- in context of modifying an existing proposal
- prefer to make a comment
- respecting the persons authorship
- should we design for a collaborative process or adapt to these social norms?
question about active/less active people in the group
- her remark: people less active would probably feel more comfortable to write on the wall instead of on the agreement
- where do discussions take place?
- should we have a discussion under approved agreements?
- similar to Github’s issues and PRs
pulling in existing agreements without re-debating/re-voting
they already use google doc, do they use the commenting feature already?
- she didn’t mention it
values!
- phew, would take a lot of time to think, and maybe not too useful
- if we had this idea where the group has already select it’s values, then each proposal is a subset of the group values
- maybe keeping values to the group vision/governance stuff makes sense now?
- group description (to be renamed to information) is also related, as that’s where people put things about the group, could include visions, etc…

Facilitator for next meeting: Bruno

nicksellen · July 1, 2021, 1:57pm

governance design process - user testing with member of Robinfoods

Date: 2021-07-01
Lead: Bruno
Observers: Nick, Vasilis
User: Dave (Robinfoods, they use telegram a lot)

Prototype URL: https://karrot-prototyping.netlify.app/
Notes go over to: Stage 5: Testing the prototype

bruno does the faciltation
nick and vasilis take notes
intro by bruno
20min user testing and then discussion

1st task notes (comment outloud)
participant should share their screen

make a proposal to the rest of the group to accept it or not
very short agreement, the title/summary/text is basically the same
noticed the time afterwards, seemed to make sense
wondered if it’s the conflict resolution feature (because the voting systme was the same)
- “I think it’s great the 5 points” <— the voting levels
- can edit it,
“because I made it, I cannot do resistence voting” (didn’t notice warning)
after voting, is it saved automatically? was not clear, but figured it out (only sure after going back to the main page and back in again)
could find the proposal in the interface
overview, seems quite clear
was eager to start the discussion on the new one he had created

2nd task notes
participate in a discussion over an agreement
start with sample data

was supposed to open existing proposal, but opened an approved agreement first, was supposed to open an existing proposal
proposed a change, starting a discussion first, used the “reason” box as intended
could see that voting was not possible until a change is made, ande correctly made a change
found the diff view
confused the existing approved agreement was still there, not showing any changes (it is hiding over on the proposals page) (we should show a hint that there is an open proposal for it)
… confusing trying to find an existing discussion to participate in
… hard to find proposals when they are connected to existing approved agreements
had to nudge to look over to the “proposals” tab (still not clear of course!)
3 proposals open for the money agreement! assumed they were created by the system (I think maybe 1 was? buttwo were created by him) (we should maybe limit it so you can only have 1 open change proposal at a time)
but also, “for me it’s totally fine” (aside from confusing duplication of proposals)

concete idea:

have only 1 list of proposals + approved
- for proposals to approved agreements, show it on top somehow
- have filters for what to see (e.g. only proposals)

Discussion after the test

confusing where the proposals go, should probably go back to the page after creating a proposal
should reference the proposals from the approved agreements
would the feature be useful? would be very useful if … it was more open, as some people don’t have time, or have karrot accounts. currently they don’t have so many decisions or agreements. mostly need people that can rescue food.
- for open use case, discussion, voting, or both? could be ok if they can just discuss things… they have 400 person telegram group, 5-8 would probably discuss the topics
- when the chats get too big on telegram, would be better to have it over on karrot - having another place to have a more focused discussion about the topic
- have telegram, signal, and karrot
explained about negative voting (which hadn’t been seen) - could be nice to explain that in context (maybe just reordering the message would be enough, or a hover message over the disabled butons)
what happens if there is a long chat, would be nice to open the disussion at the same time as the edit boxes
very confident user clicking around the interface
“I like it that it’s simple”
how they use telegram and signal
- a few different telegram groups
  - guerlla gardening
  - upcycling
  - food waste heros (short notice calls for help)
  - robin foods main one
  - robin foods “office”
- use channels to advertise events that need a bunch of people, and ask to receive private messages to co-ordinate them (maybe could be an activity with multiple roles, or something)
- also could use polls to find dates for one-time pickups
- use karrot for reoccuring pickups, but also have to post it in the telegram groups for less active ones… (maybe we can reactivate the telegram bridge idea)
uses the “disable activity” feature quie a bit, if somebody is doing the activity via a telegram message
uses number prefixes in the place names e.g. “(1040) Place Name”, the number is the district, so people can use it to find how close they are
too many places on the side, would be nice to filter! (nick: I’d like a whole page which shows the places nicely), could also show distance from user…
if people on telegram could sign into activity via telegram link automatically, would be really nice
- could have short link that people can register with one click, and also see karrot, simple landing page, with the activities
great having people show us how they use karrot, could be good idea for community sessions
should add translation links to our onboarding page
- How to get involved! Onboarding into the Karrot team
we can write a short description for a call for developers and share it with the member from robinfoods, and on mastodon, must include emojis

nicksellen · August 3, 2021, 12:02pm

user testing with person from Luxembourg group

Date: 2021-08-03
Lead: Bruno
Observers: Nick

proposing new agreement

wondering whether to make something up, or not.
notices default time period of 7 days.
assuming summary is a summary of the main text, and noticing it’s optional.
fetches text from existing group agreement. selects one subpoint from their long agreement, about context resolution.
wondering how the summary will be displayed, compared to the full text, so wondering how detailed to make the summary. settles on one sentence for the summary.
proposal created!
“reason for proposal” is shown, but we had that field removed, and they actually always use a reason … so maybe we could add an optional reason back for new agreements in…

participate in an existing discussion (proposal)

opens an approved agreement first, rather than a proposal. so there is no discussion.
notices that it is already approved (with the green text in the top right)
looks in propoals tab, and finds the open proposal
see’s reason too!
notices that it references an existing approved agreement too
wondering how to see the change to the agreement?
not sure about the pop-up view, whether it’s needed. would be fine with scrolling on the page.
wanting to see the changes more clearly.
checks the instructions, easily makes a message in the chat
is able to vote ok.
does not see “show hanges to existing agreement” button

change an existing agreement (creating a change proposal)

selects the money agreement.
notices similarities to creating new proposal.
uses the reason field kind of like the change itself.
“depends how I want to work” when thinking about discussion first, or proposing changes first.
expects, correctly, to be able to edit the existing text if selecting “propose changes first”
is able to propose a change easily
does not see “show hanges to existing agreement” button

notes

making the change view the main view somehow? to make it clearer
still some confusion between approved and proposals (tabs a bit smaller)
instead than “start a discussion”, could be “open for discussion”
maybe add back in the “reason for proposal” for new agreements, using principle from sociocracy to make clear that everybody knows what the problem is, or what needs to be solved. what are you trying to solve?
another idea to add: evaluating the proposals, reminder to check the policies periodically (e.g. after 1 year notify people to review, or automatically date for review)
again from sociocracy, start with good proposal than perfect one, then review it
preventing agreements become stale
could it be useful for foodsharing luxembourg? maybe, but hesistation, as they take some decisions at meetings, not sure. it could be in their meetings, they add the proposal in karrot. not everybody goes to meetings, so could be good if they can give input in karrot. some people that don’t come to meetings, but don’t give their feedback usually.

Notes on new feature (roles)

Open slots vs guest

open for non-approved

confusion because everything is on one line
Daniel likes the idea of a preview too
first design is clearer, Daniel really likes the slider
Nick will start working on how the activity will look like. Then a preview when creating the activity…

role is related to trust
make specific what to trust for

contact Dominic for UI ideas
should make clear what the sliders are for

nicksellen · August 19, 2021, 2:34pm

notes from user test with user from Lund

More experienced user, board member.

Had already had some issues with users with rules. Some fake accounts, and people doing too many pickups, so they wrote some rules because of that.

propose a new agreement

making up an agreement, so avoid having to go and fetch a document.
mostly some dummy data. very quick
wondering about the voting options.
wondering if it’s a good option to force people to write in the chat to vote negative.
wondering about if that makes it really anonymous if you have to write in the chat. maybe pushes the vote to the positive side.
in lund they had some votings 3 months ago. 3rd one was for removing the fake account. discussed online a lot, but heard a lot of lies, and some admissions. they got voted out. wouldn’t feel comfortable to state reasons.

participate in an existing discussion for ongoing proposal

didn’t find the existing proposal, initially, opening an approved agreement.
only see’s option to propose a change, says it looks quite ok!
divert to thinking about how the feature would be useful in the group, or not.
nudges to look for existing proposal. finds it back on the main page again.
does “edit proposal” and adds some text.
finds the “changes” view.
trying out how the feature is.
wonders if every member would be able to change the proposal? answers: editors.
feel problem having made a change to the proposal, and want to see previous version of text. so a timeline of edits would be desirable in that case.
comparison to ms word (or google doc) comments I guess. and where one person is there to accept it.
6 member board in lund, the rules went back and forth with disussions. concerned about how it would be if each person directly changed the proposal text, without the history of changes. because people might disagree with the change.
nudges to write text in the chat, but focuses on proposal changes.
in their group, voting on the rules was just by the board. noticing here that every user gets to vote. not sure how it would work out in the group. as perhaps the people who are less responsible in the group.
nudges to find where to give a comment.
didn’t find the chat by default. can be more clear, perhaps more integrated into the design, or where it’s referenced in the text, make a connection.
ah, noticing in the diff view that we have the original old text, but not all the intermediate text changes.
could be interesting to show the author of the change.
on anonymity, conflict resolution should be anonymous for sure.
a lot of people only interested in the food pickups, but maybe not the rules, etc.
some people stepping back just before the pickup.
could be interesting to have everyone discussing the rules.
issue with people touching bread that they don’t then take.
some rules that might be good to discuss, and others that the board might not want to discuss. so perhaps two systems.
would be interested in softer conflict sanctions, e.g. block for some time.

other chat

could make the “vote” tab clearer in the conflict resolution
feeling of hidden features in karrot, and people writing in different places, hard to find particular messages