JORDANA GLOBERMAN DESIGN
JORDANA GLOBERMAN
User-led Rapid Impact Assessment
Problem
The ESDC Innovation Lab was hired to conduct their department's first Rapid Impact Assessment (RIA) on a critical program, and to incorporate user-feedback at every step.
Role - Design Lead & Project Manager
While working at the ESDC Lab, I created the end-to-end design strategy for the project, established and maintained relationship with the client and stakeholders, designed research activities and led analysis.
Team
Me + 2 designers, and 3 facilitators.
Project findings were reported to two Deputy Ministers in Employment and Social Development Canada and included in the program's summative evaluation. I presented the project methodology at the World Conference on Qualitative Research in Porto, Portugal in 2019. Highlights from the project were also published by the Observatory of Public Sector Innovation.
Context & Approach
Rapid Impact Assessment (RIA) is a form of evaluation that uses experimentation to assess the effectiveness of a program. In RIA, a program is measured against a counterfactual, an alternative version of the program. This alternative can be real or hypothetical. The method can be a quick and effective way of gathering data. It had been used in other areas of government before the department of Employment and Social Development Canada (ESDC) decided to pilot it.
The New Horizons for Seniors project is a national funding project for programming and services that support seniors. The program itself is worth over 8 million dollars. The ESDC Lab was hired by ESDC's Evaluation Directorate in 2018 to conduct the department's first RIA, and to do so for the NHSP. The Evaluation Directorate wanted a robust qualitative approach to this RIA, and hired my team for our expertise in this area. They also hoped co-design would form part of our research approach.
Tools
-
Focus Groups & Co-design
-
Cardsorting
-
Personas & Scenario Development
-
Systems mapping
-
Appreciative Inquiry
-
Backcasting
-
Polarity Thinking
-
Generative Evaluation
-
Quantitative Experimentation
-
Collaborative Analysis
-
Counterfactual Testing
Prototypes and program alternatives were produced, such
as a digital service hub to connect users with organizations, and a collective impact model. These prototypes informed the program’s delivery model in the following year.
This new methodology, which fused Design Thinking and RIA, was later profiled by the OECD’s Observatory for Public Sector Innovation and presented at the 2019 World Conference on Qualitative Research.
A counterfactual was codesigned through 4 workshops with government, organizations serving Seniors, and diverse Seniors (the largest including over 70 participants). Workshop activities used techniques such as polarity thinking, gamification, visualization, appreciative inquiry, systems mapping, personas, disruptive scenarios, and backcasting to build off of program gaps, and measure the impact these might have on users.
Traditional co-design activities aren't accessible; we needed to adapt ours and test them extensively with persons with disabilities before the workshops. Our Montreal workshop was bilingual, with two sets of interpreters. Activities like scenario development and journey mapping became facilitated conversations with the support of interpreters and assistive technology. We doubled the time required for each exercise and practiced them extensively. We iterated on designs several times before the workshops and it paid off.
Participants told us these were the most inclusive events that they had been a part of, and were moved. It went a long way to build trust in government.
Recruiting participants challenging. My colleagues and I spent days on the phone trying to convince industry reps to take two days from work to attend sessions. We also recruited at a local conference. We found users with disabilities by looking for regional service providers and using a snowball sampling approach. Procuring services to support the workshops was sometimes challenging. In Vancouver, we spent weeks looking for support personnel with skills in AAC. In the end, we managed to reach our target participation of around 40 people per workshop.
My intention with these workshops was to provide as accessible and inclusive an experience as possible in order for capacity building to be part of the experience. Through these workshops we were not only learning what persons with disabilities expected from federally regulated industry; we were showing industry reps what persons with disability expected from them. It was very effective.
I also wrote a survey for those who couldn't attend the workshops. TD's Innovation Lab participated in our research through a focus group too.
As project lead, I designed the strategy and activities for the project as whole, and led the delivery of each project milestone. I also managed client relationships. This project involved a lot of risk and, especially in a government context, this never goes down easy. I spent late nights working with clients to explain the design process, protect its integrity, and strategize risk mitigation. In addition, I also managed relationships with other departments and regulatory agencies whose work overlapped with ours.
Graphic recording from our Toronto workshop
Result
The output from our workshops, interviews and surveys was iterated into several paper prototypes, including:
-
Model accessibility plans
-
Frameworks for leading accessible workshops (including resources for budgeting and recruiting participants, and the plans for the workshops we developed)
-
A proposal for a repository of accessibility experts, individuals with lived experience, and organizations representing persons with disabilities
-
A requirements list for an online hub to support accessibility among federally regulated industry
-
A journey mapping tool to support federally regulated industry
These prototypes were given to our client, the Accessibility Secretariat at ESDC, to inform their regulatory guidance and future standards. The process of co-creation with users with disabilities ensured these prototypes went beyond compliance, to be more in line with the desires of users with accessibility needs. Participants praised the workshops for being the most accessible approaches they had experienced; as a result, the workshop design was also shared with federally regulated agencies as a prototype of an engagement format that they could use to be compliant with the new Act’s demand to engage persons with disabilities on all Accessibility planning.