Jordana Globerman Design
Role - Design Lead & Project Manager
While working at the ESDC Lab, I created the end-to-end design strategy for the project, established and maintained relationship with the client and stakeholders, designed research activities and led analysis.
Team
Me + 2 designers, and 3 facilitators.
Project findings were reported to two Deputy Ministers in Employment and Social Development Canada and included in the program's summative evaluation. I presented the project methodology at the World Conference on Qualitative Research in Porto, Portugal in 2019. Highlights from the project were also published by the Observatory of Public Sector Innovation.
Context & Approach
Rapid Impact Assessment (RIA) is a form of evaluation that uses experimentation to assess the effectiveness of a program. In RIA, a program is measured against a counterfactual, an alternative version of the program. This alternative can be real or hypothetical. The method can be a quick and effective way of gathering data. It had been used in other areas of government before the department of Employment and Social Development Canada (ESDC) decided to pilot it.
The New Horizons for Seniors project is a national funding project for programming and services that support seniors. The program itself is worth over 8 million dollars. The ESDC Lab was hired by ESDC's Evaluation Directorate in 2018 to conduct the department's first RIA, and to do so for the NHSP. The Evaluation Directorate wanted a robust qualitative approach to this RIA, and hired my team for our expertise in this area. They also hoped co-design would form part of our research approach.
Tools
-
Focus Groups & Co-design
-
Cardsorting
-
Personas & Scenario Development
-
Systems mapping
-
Appreciative Inquiry
-
Backcasting
-
Polarity Thinking
-
Generative Evaluation
-
Quantitative Experimentation
-
Collaborative Analysis
-
Counterfactual Testing
We began research through a literature review and interviews. We gathered as much information as we could about the future of aging and consolidated this into 6 thematic areas that we wanted to focus on. These were then presented to our key stakeholders and approved.
We began by bringing together internal government stakeholders in a series of workshops to design our counterfactual. We created personas and scenarios to help stress test ideas around the current program, the needs of users, and any assumptions about their pain points. We took this a step forward into systems mapping, so that we could consider all the interdependent parts of the program and its related ecosystem.
Next, it was time to bring the users into counterfactual development. We organized workshops in Ottawa and in Toronto that brought together the perspectives of government, organizations serving Seniors, and diverse Seniors (the largest workshop including over 70 participants). Workshop activities used techniques such as polarity thinking, gamification, visualization, appreciative inquiry, systems mapping, personas, disruptive scenarios, and backcasting to build off of program gaps, and measure the impact these might have on users. Our final counterfactual allowed us to both evaluate the current program and dream of future program possibilities. These possibilities included prototypes of program alternatives, such as a digital service hub to connect users with organizations, and a collective impact model for serving seniors. An evaluation team went on to compare the current program with these alternative.
Result
​
Our department's evaluation used our counterfactual to test the existing program and provide recommendations to the program leads. Based on our findings, several additions were made to the program's delivery model in subsequent years, including the development of digital supports for seniors.
​
The methodology that we developed for this project fused Design Thinking and Rapid Impact Assessment. It was the first of its kind to do so and was later profiled by the OECD’s Observatory for Public Sector Innovation and presented at the 2019 World Conference on Qualitative Research.