JORDANA GLOBERMAN

User-led Rapid Impact Evaluation

Problem

The ESDC Innovation Lab was hired to conduct a Rapid Impact Assessment (RIA) for the New Horizons for Seniors Program, a federal program supporting Seniors, and to incorporate user-feedback throughout the evaluation.

Role - Design Lead & Project Manager

While working at the ESDC Lab, I created the end-to-end design strategy for the project, established and maintained relationship with the client and stakeholders, designed research activities and led analysis.

Team

Me + 2 designers, and 3 facilitators.

Project findings were reported to two Deputy Ministers in Employment and Social Development Canada and included in the program's summative evaluation. I presented the project methodology at the World Conference on Qualitative Research in Porto, Portugal in 2019. Highlights from the project were also published by the Observatory of Public Sector Innovation.

Context & Approach

 

A counterfactual was codesigned through 4 workshops with government, organizations serving Seniors, and diverse Seniors (the largest including over 70 participants). Workshop activities used techniques such as polarity thinking, gamification, visualization, appreciative inquiry, systems mapping, personas, disruptive scenarios, and backcasting to build off of program gaps, and measure the impact these might have on users.

 

Comprised of designers with experience in experimentation, service design and systems analysis, we focused on results-driven projects that improved the lives of Canadians. From November 2019 until March 2020, we worked directly with representatives from every federally regulated industry, accessibility experts, and users with diverse accessibility needs to co-create prototypes for accessibility plans, accessible workshops, and supports within industry for greater accessibility. We conducted a literary review, interviewed 22 specialists and users, and facilitated accessible design workshops in 3 cities across Canada.

pexels-kampus-production-7983590_edited.jpg

Tools

 

  • Focus Groups & Co-design

  • Cardsorting

  • Personas & Scenario Development

  • Systems mapping

  • Appreciative Inquiry

  • Backcasting

  • Polarity Thinking

  • Generative Evaluation

  • Quantitative Experimentation

  • Collaborative Analysis

  • Counterfactual Testing

Prototypes and program alternatives were produced, such
as a digital service hub to connect users with organizations, and a collective impact model. These prototypes informed the program’s delivery model in the following year.

This new methodology, which fused Design Thinking and RIA, was later profiled by the OECD’s Observatory for Public Sector Innovation and presented at the 2019 World Conference on Qualitative Research.

I led 3 sets of 2-day codesign workshops with persons with disabilities and executives from federally regulated industries in February and March of 2020. Workshops took place in Toronto, Montreal and Vancouver. I had a GIS map of created, and based workshop locations areas with the densest and most diverse areas of pertinent industry. C-Suite executives from across the country and persons with visual, auditory, cognitive, intellectual and/or motor impairments, as well as neuro-divergence, attended as participants.

 

Traditional co-design activities aren't accessible; we needed to adapt ours and test them extensively with persons with disabilities before the workshops. Our Montreal workshop was bilingual, with two sets of interpreters. Activities like scenario development and journey mapping became facilitated conversations with the support of interpreters and assistive technology. We doubled the time required for each exercise and practiced them extensively. We iterated on designs several times before the workshops and it paid off.

A photo of seven people sitting around a table in discussion. One person is interpreting. Another man walks by in the background.

Participants told us these were the most inclusive events that they had been a part of, and were moved. It went a long way to build trust in government.

Recruiting participants challenging. My colleagues and I spent days on the phone trying to convince industry reps to take two days from work to attend sessions. We also recruited at a local conference. We found users with disabilities by looking for regional service providers and using a snowball sampling approach. Procuring services to support the workshops was sometimes challenging. In Vancouver, we spent weeks looking for support personnel with skills in AAC. In the end, we managed to reach our target participation of around 40 people per workshop.

My intention with these workshops was to provide as accessible and inclusive an experience as possible in order for capacity building to be part of the experience. Through these workshops we were not only learning what persons with disabilities expected from federally regulated industry; we were showing industry reps what persons with disability expected from them. It was very effective.

I also wrote a survey for those who couldn't attend the workshops. TD's Innovation Lab participated in our research through a focus group too.

 

As project lead, I designed the strategy and activities for the project as whole, and led the delivery of each project milestone. I also managed client relationships. This project involved a lot of risk and, especially in a government context, this never goes down easy. I spent late nights working with clients to explain the design process, protect its integrity, and strategize risk mitigation. In addition, I also managed relationships with other departments and regulatory agencies whose work overlapped with ours.

A graphic recording illustration of a conversation. Thematic sessions include Planning Process, Feedback Loops, Accessibility Plans and Prioritization Process, as well as a the pillars of the Accessible Canada Act.
Graphic recording from our Toronto workshop

Result

 

The output from our workshops, interviews and surveys was iterated into several paper prototypes, including:

  • Model accessibility plans

  • Frameworks for leading accessible workshops (including resources for budgeting and recruiting participants, and the plans for the workshops we developed)

  • A proposal for a repository of accessibility experts, individuals with lived experience, and organizations representing persons with disabilities

  • A requirements list for an online hub to support accessibility among federally regulated industry

  • A journey mapping tool to support federally regulated industry

These prototypes were given to our client, the Accessibility Secretariat at ESDC, to inform their regulatory guidance and future standards. The process of co-creation with users with disabilities ensured these prototypes went beyond compliance, to be more in line with the desires of users with accessibility needs. Participants praised the workshops for being the most accessible approaches they had experienced; as a result, the workshop design was also shared with federally regulated agencies as a prototype of an engagement format that they could use to be compliant with the new Act’s demand to engage persons with disabilities on all Accessibility planning.