USER RESEARCH & EVALUATION
A design-led evaluation method for an $8M national seniors program.
Leading the first user-centred Rapid Impact Assessment in ESDC's history; a hybrid methodology later recognized by the OECD and presented internationally.
PROBLEM
A rigorous evaluation method, lacking the user's voice.
ESDC's Evaluation Directorate needed to pilot Rapid Impact Assessment (RIA) for the first time in the department's history, on the New Horizons for Seniors Program (NHSP), an $8M+ national funding program supporting services for seniors.
They wanted genuine user voice built into a process that, historically, evaluators alone had controlled.
ROLE
Design & Research Lead
I owned the design strategy end-to-end. I designed the research approach, built and held the relationship with ESDC's Evaluation Directorate, and led analysis and synthesis.
No design-led RIA precedent existed inside government, so a meaningful part of my role was inventing the methodology itself. My methodology fused Design Thinking with a quantitative evaluation method that, until then, had been run by economists and evaluators with no user research built in at all.
TEAM & SCALE
6
Core team — me, 1 designer, 2 researchers, 3 facilitators
3
Workshops in 2 cities
2
Assistant Deputy Ministers reported to
Intl.
Presented at WCQR, Porto 2019
A core team of me, two designers, and three facilitators. Findings were reported directly to two Assistant Deputy Ministers at Employment and Social Development Canada and folded into the program's summative evaluation. I later presented the methodology at the World Conference on Qualitative Research in Porto, Portugal (2019), and it was profiled by the OECD's Observatory of Public Sector Innovation.
Context & approach
01
Framing the problem
Rapid Impact Assessment evaluates a program against a counterfactual — a real or hypothetical alternative version of the program — to measure effectiveness quickly. RIA had been used elsewhere in government, but never on a program of this profile at ESDC, and never with genuine co-design input.
The Directorate came to my team specifically because they wanted a credible qualitative practice that integrated user feedback directly. I started with a literature review and exploratory interviews on the future of aging, then worked with my team to consolidate everything into six thematic areas. I brought those to our government stakeholders for sign-off before designing a single workshop:
→
RIA's credibility with evaluators depended on the counterfactual standing on solid footing — which meant getting buy-in early rather than asking forgiveness later.
02
Building the counterfactual
I ran a first set of workshops with internal government stakeholders to construct the counterfactual itself, developing personas and scenarios as inputs to stress-test assumptions about user needs and program gaps. Next, I engaged participants in systems mapping to account for the program's full ecosystem of interdependent parts. This is typically where evaluation work stops. I pushed it further.
Building the counterfactual — mapping program achievements on a bilingual workshop wall.
03
Bringing users into the evaluation itself
I organized two larger workshops — in Ottawa and Toronto, the largest drawing more than 70 participants — that brought together government staff, organizations serving seniors, and seniors themselves. I used polarity thinking, gamification, appreciative inquiry, systems mapping, disruptive scenarios, and backcasting to pressure-test where the current program fell short and what it might look like instead.
The result was a counterfactual that did two jobs at once. It gave evaluators a rigorous basis for assessing the existing program, and it produced concrete alternative-program prototypes, including a digital service hub connecting seniors to organizations, and a collective-impact model for senior services.
4
Why users mattered
Most RIA work in government runs as a closed quantitative exercise between evaluators and economists, with users encountered only as data points. I built a version where seniors and the organizations serving them helped define what "impact" should mean, before a single number was measured.
That distinction is why the Evaluation Directorate brought design in at all — and it's the part of the methodology that ended up generating outside interest.
Trying on an unfamiliar collaboration role was the real skill I was there to build.
METHODS & TOOLS
Card sorting
Personas & scenario development
Focus groups & co-design
Polarity Thinking
Systems Mapping
Backcasting
Appreciative Inquiry
Qualitative experimentation
Collaborative Analysis
Counterfactual testing
RESULT
A method that became the outcome.
The Evaluation Directorate used our counterfactual to test the existing program and ground its recommendations to program leadership. Several additions to NHSP's delivery model followed directly from our findings in subsequent years, including new digital supports for seniors.
01
Tested the existing program
Our counterfactual gave evaluators a rigorous basis to assess NHSP and ground recommendations to program leadership.
02
Changed the delivery model
Several additions to NHSP followed directly from our findings, including new digital supports for seniors.
03
Became a reusable method
The first fusion of Design Thinking and RIA — profiled by the OECD and presented internationally.
Beyond NHSP, the methodology itself became the outcome: it was the first fusion of design thinking and Rapid Impact Assessment of its kind, later profiled by the OECD's Observatory of Public Sector Innovation and presented at the 2019 World Conference on Qualitative Research in Porto — establishing a reusable model for bringing genuine user research into government evaluation practice.