Last night (Friday, September 8, 2017) a script ran against Github to close, and comment on, all unaccepted applicants for the Mozilla Festival 2017 (MozFest). This post is an attempt to bring to light the process of proposal selection in the months beforehand.
Please note: this is speaking only for the Privacy and Security space and through my own eyes. Others will have done things differently.
I've been rather active on Twitter, IRC, Telegram, Slack, email, and of course Github, which has meant I became a first point of contact for many. Even if I had nothing to do with their proposal. Approaching 900 sessions were proposed by last night, and we got a notification - for - each - and - every - one -of - them. I've been contacted by people seeking clarity at nearly every step of the way. Despite our best efforts to work in the open and transparently curate each space, we have not been immune to conflict. In keeping with my own desire to further open my own methods of working, I decided to transparently discuss my part in the selection process this year. Here goes..
I've been volunteering at Mozfest since 2014. It was my first real contribution to the Mozilla Foundation, and I haven't looked back. Sometimes it's been really tough to connect with people, but I've tried to keep picking up the pieces and persisting in my own way.
Somehow, after sustained efforts, it's culminated this year by being selected as a Space Wrangler in the inaugural Privacy and Security (P&S) space. I knew someone in the team and that was due to face to face contact at Mozfest. The tech industry might rock it out online, often with team members at opposite ends of the globe (and with subsequent conference call timezone clashes), but never underestimate the power of offline.
So back in May, Jonathan Kingston from Mozilla approached me about being a Wrangler. Ooh yeah! Affirmative! Absolutely! Yes! I had to temporarily but quickly shutdown some personal activities with 'Linuxing in London' and 'Missing Maps', but heck this was an opportunity not to be missed. Mozfest has almost 2,000 attendees and I'm very keen on understanding the logistics required to create events of this scale. It's still a volunteer role. I don't care, BRING IT!!
I got up to speed on previous design sprints at MozRetreat in Tallin. Everyone did a great job of planning and it helped defined the narrative for the Privacy and Security space:
Do you feel safe online?
Sometimes, online security can feel unattainable. State-sponsored surveillance, theft of personal information, trolls arrogantly abusing others - this has become normal. And every day, many of us trade personal information for convenience.
It's time to flip the script. Not everyone knows how to install a VPN or use encryption - but everyone should feel safe online. Can we imagine a future where manufacturers respect our safety by default when they build products? Where governments take steps to protect us? Can we find fun, new ways to engage our friends and neighbours around these issues?
At Mozfest, we'll build, teach and play with these new approaches to Privacy & Security. Let's put safety first.
That was also the month we had our very first team conference call with Mozfest Production. I remember it clearly as I was having to use my new dumbphone (non-smartphone) on a walk past roadworks and traffic rushing to Facebook for a Hacks & Hackers meetup. Everyone says it makes me look a spy. Like spies all use dumbphones. Who cares, the battery lasts almost 2 weeks. It was hot that day too. They wanted me to talk right when I was near a pneumatic drill. I couldn't work out how to unmute quick enough. That was just the start of my tech troubles. (H&H was awesome btw. Elliot from Bellingcat spoke. We had cocktails and finger food. Journalism seems like it can be quite sexy and fun. I'm betting it's often not like that at all. Elliot's story was an incredibly harrowing experience.)
In June we continued to have a Mozfest Wranglers call every fortnight. I really struggled with my old laptop. There were a couple of Privacy and Security team calls too but Vidyo was as patchy as a worn our pair of socks. But that laptop - it's all I have. I can't do video calls but I can dial in on my dumbphone. At least we have etherpads. One call I lasted, at most, 10 minutes. I found it highly embarassing and felt I came across as totally unprofessional. I know other people in the community experience this, so I was quite vocal with nearly every Mozilla staff member I could speak to around that time. When I finally got connected, we were discussing at how people did things previous years, what to expect from the process, what was expected of us. Something else important…
Oh yeah.. OPEN CALL FOR PROPOSALS. Almost overnight we had nearly 30 submissions. Draft website went live so my name was up there looking all official and stuff.
This is where I started building out the spreadsheets. Robert Friedman jumped on a Wrangler call and spoke of some of the tools he used in 2016. He shared a spreadsheet and I kicked into overdrive. Data! I built a Google docs script to pull in live data from the Github master spreadsheet. As each source was a separate column I had to have two separate sheets, one where Privacy and Security was the preferred space (milestone in Github) or secondary space (secondary space label in Github).
Our first round of scoring against a rubric was done blind. This meant we had no idea of who had submitted, their sex, their location, their affiliated organisation. Any information that could have biased our decision. I was manually merging the information from the scraped data, as you can't merge in Google Sheets, plus there was some dummy 'test' proposals thrown in here and there to help teams automate some processes. I went manual as time was flying by this point and I could be assured I knew where I was up to. That was down to marking a line in the scraped sheets to designate where I was. This was ok on my own, then when other wranglers started to jump I had to start documenting the process on the fly.
I didn't want to be waiting until the closing date to get started on the first round of scoring. We started needing to around 10 each in this month. By the close of the month we needed to do 50 each. Marking the space alignment, participant focus, and a gut check.
Around this time Github went down. Oh yes people. Shit got real. We lost proposals though the data scraping. Some had to be removed as there were duplications everywhere. You know that line I'd marked? Yeah stuff that. That is irrelevant now. I couldn't sleep.
I still had quite a few further tech problems getting on the conference calls. I really wanted to share what I was doing with the spreadsheets, but my audio kept cutting out. I'd have to restart the app, log back in, for the same thing to happen a few minutes later. I was rather excited to see Emma Irwin write about this exact problem communicating with the diverse community. I was in tears that someone even recognised the issue. What can I say? The pain was real.
Note for future Wranglers. Unsubscribe from issues that are not yours. As they come in. It will become a major source of pain if you don't.
August saw the start of weekly Wrangler calls. Greater information about the week beforehand at the RSA started coming in [details coming], as did ticket availability, volunteer open call, artists assigned to each space (P&S space has the cheeky and mischievous Paolo Cirio) But most importantly August 1 was the closing date for proposal submission. At one point that morning I calculated 1 coming in every 3 minutes. Notifications - ping ping ping!
<img src"/blog/img/nonewnotifications.png">
Around 175 proposals were accepted for our P&S space. I wanted each proposal to have been action at least 3 times before making a decision. This would reduce any bias, and give everyone a fair chance.
By a week later we'd marked all the proposals for our space the first time. We used a traffic light system: high score was a green, through to next round, mid-score was a yellow, needed some more oomph or further clarification, low score was a red, a potential refusal. I then built out another spreadsheet, hid the previous score, added more fields (affiliated organisation, name, github issue number, stipend requests, etc). And we started scoring again.
We needed to get more information on stipends. They are a request by the applicant for Mozilla to cover their travel and accommodation. An estimated 80% of all proposals asked for this. We simply don't have the capacity as a not for profit to cover every request, so we asked as early as possible for people to consider all options. The intention was to make sure there was enough among the community. No point in us bringing people who come time and again. We wanted new contacts, people new to the network who'd never experienced Mozfest, people from across the globe, people who had no other way to get here. I knew this was going to get sensitive real fast, so I pinged people as quickly as possible when I saw their proposal had real possibility being accepted. Can your business, or university help support you? We also took the decision in our team to limit the size of the stipend. At first I'd estimated only one proposal, to one stipend. A couple of weeks later I started noticing some sessions wanted four people to come. It got tough, but I had to be fair. We wanted this to spread as much as possible, not a big chunk going to one group.
When we finished the second phase of selection each proposal had received two scores. We were still reluctant to start refusing. Other teams were starting to accept some sessions and we felt quite behind that yet. We jumped on a conference call for a third selection round, this time I sat in the London office. Again I nearly cried at how amazing the conferencing system is when it actually works. It felt like I was in the same room as everyone else. Sorry, little things like that feel incredible to me. I know plenty of you will be rather blase about such tech. The rest of the world doesn't work like that. This call lasted almost three hours. We quickly discussed each of the agreed red, in turn. We then proceeded to those where there was disagreement. Some were a close call. Others were polar opposite. That's how people are, we don't always think the same! Can be frustrating as hell when you're trying to reach consensus.
We then went into a fourth selection phase. What were the types of content? Were they community? Were they an expert? Was this an important message? Did they fall on an axes of diversity? Would their voluntary contributions to the Mozilla benefit purely by access to this event? I can tell you at this point we still had around 130 proposals in selection. These were all good at this phase. And that was really frustrating. We had to cut 50% immediately. Our deadline was September 6. It was 2 weeks away.
We connected in Slack, on etherpads, in email, on conference calls. I had to say no to several dear friends, I'm sure it was the same for the others judging by how passionate they got too. The cut got down to 65 proposals. It was at this point people really started pinging us. "Can I have an update?", "I don't think you gave our session enough thought", "I don't think you know how much this means to me." I truly did my best to communicate to each and every person that needed further information. Some requests for information were on proposals I'd not scored, but heck I was so glad we did it the way we did. I could go into the spreadsheet and immediately see what the transparent review process fed back. We were not tasked to do that I must add. I made the personal decision to do that. We have an event to put on, yes. But I'm also empathetic enough to realise how much many people cared about this event. Still after doing that some people moaned. They had a lot of disappointment to get out. I couldn't fix that.
By phase five we brought in the stipend information as a considered factor, not just a thing to chase, as seen in round two. I won't go too much into this process as it's sensitive budgetary information. Is this session 'free'? If we pay for this session to come, will our community benefit in a unique way? Those kind of decisions. Everyone deserved to be there in consideration but we had to find a balance. This was somewhat easier again by our trail of notes. I was at 12 spreadsheets by this point.
The sixth and final selection process was a dummy run of the schedule. Added another spreadsheet. We didn't want too many similar sessions. We really wanted it to be as fun as possible. Privacy and Security are often really dull to people. It was our intention from the start to have a vibrant mix.
Each space this year is tasked with filling: Shed - for hands on hacking, doing, and prototyping, Learning Forum - for sitting back a little but still interactively learning, Gallery - installations and demos. We also deeply embedded: Advocacy, OpenNews, Youth Leaders, and Open Leaders.
Massive thanks to Sarah, Erika, Marc, and Esme. They all been patiently helpful, during what's a busy time for them too, but they still manage to jump on those calls and help us through some often complex scenarios. And a big shout out to all the Wranglers this year for putting in some solid time and effort selecting an eclectic mix of final sessions. And to my co-wranglers in the P&S space. This was a team effort, and every single one of them were there helping shape the final selection to the best of their availability.
The Mozfest 2017 Privacy and Security space is happening for the very first time. We feel proud of what we have achieved. We've still got a long way to go before doors open at London's Ravensbourne October 27–29. Just in our space we have people coming from Brazil, Mexico, India, West Africa, Canada, USA, Europe, and of course the UK. And more (I'm sure I missed some).
Seriously you should come. I can't wait!