How we measure the impact of our grants
This is a challenging question, one we can't answer in a single sentence
Estimated reading time: 3 minutes, 40 seconds. Contains 736 words
What we want to achieve is difficult to measure: improved employability across a working lifetime resulting in sufficient earned income for our grantees to provide adequately for themselves and any dependants and the ability to contribute to the common good.
Here we explain how we measure the ‘impact’ of our grants. We use the terms outputs (meaning activity) and outcomes (what results from that activity). We look for realistic, achievable results that are, on balance, likely to help move our grantees along the paths we speak about in much of our literature and which are shown on in the graphic on the right, below.
We are willing – and able - to take risks by trying new approaches. Not all these approaches will work or achieve their planned outcomes. We don't think of these as failures but as steps in learning more about how to go about our historic aim of relieving poverty.
Grants to organisations, projects and schools
Our grants to organisations and schools are solely a means of reaching our target beneficiaries: low income Lambeth citizens.
We measure the impact of our grants by assessing grantee reports against the originally agreed outputs, outcomes and post-project outcomes. An important part of the original grant assessment process is working with the applicant to ensure that the outputs and outcomes are as clear, specific and measurable as possible.
Even when a project has an emphasis on more qualitative impact (for example building confidence or changing attitudes) we have a strong preference for being able to use quantitative measures as proxies for these qualitative changes. For instance, where a project is working in a school to improve confidence, we wish to see outcomes that show how this has followed through into changed behaviour (such as improved attendance or behaviour). Where this is not possible, projects should use recognised consistent qualitative measuring tools (for example the Warwick-Edinburgh Mental Well-being scale).
We analyse reports from grantees (we have a 95% compliance rate). Each output and outcome is scored against original proposals (exceed target; 100%; 75%-99%, 50%-74%, under 50%) and then an overall score is given to the report. We class as satisfactory a grant that has met 75% or more of its agreed targets.
We report this data to the Grants Committee. Of the committee’s six annual meetings three are focused on considering grant proposals and three concentrate on impact findings, policy and strategic matters.
The scale of our work does not make it feasible to fund control groups or counter-factual studies. We cannot categorically state what would have happened if our grant had not been made. Neither is it possible to prove that it is one particular intervention that was the main cause of positive change. We avoid taking easy or unevidenced credit: there are many variables in the lives of our grantees and often they are receiving help or services from other, even multiple, sources.
Our aim with every application is to reach the best judgement we can by looking closely at the reasoning behind a proposal and taking account of what we already know is likely to work (through research findings, reported evaluations and our experience).
Grants made directly to individual grantees
We measure the impact of our individual grants through an annual impact survey in which we ask for details of their current situation (for example whether they are continuing in education or have found employment). This is undertaken nine months after the end of each academic year. It enables us to gauge what impact the training or qualification we funded has had on the grantee’s circumstances. The online survey is sent out by email and then followed up by ‘phone calls to non-respondents. Our response rate is around 50% (with confidence level of 95% and margin of error of 8%).
We humans are notoriously difficult to predict and each of us has been shaped by an infinite number of influences, events and circumstances. Our aim is simple: to help Lambeth citizens adversely affected by household poverty gain the opportunities most likely to allow them to develop the skills and aptitudes needed to gain decently paid work across their working lifetime. Measuring whether what we fund leads to that aim being realised – and to what degree – is not simple or always possible. With all this in mind, we think our approach is proportionate and pragmatic.
"Our aim is simple: to help Lambeth citizens adversely affected by household poverty gain the opportunities most likely to allow them to develop the skills and aptitudes needed to gain decently paid work across their working lifetime. Measuring whether what we fund leads to that aim being realised – and to what degree – is not simple or always possible".