How to measure the impact of a Digital Champion project
General evaluation principles & resources
When we measure the impact of a digital inclusion project it’s good to consider all stages of the project including:
- Why you are evaluating – to demonstrate impact? To understand what works and why?
- Who the audience for the evaluation will be: internal/external (for instance a funder), general public or beneficiaries?
- Researching what stakeholders and beneficiaries would find most effective and have maximum impact.
- Developing a Theory of Change or a Logic Model to facilitate structured planning and evaluation of the proposed activities. Information to help develop these can be found in the resources section.
Identify the project stakeholders and incorporate their views– clearly identifying what’s important to them to make the project successful. Stakeholders could include the beneficiaries of your project, as well others such as the funder of the project. Consider what questions to ask and what the areas for discussion might be. For funders this could be, why have they funded you and what impact do they expect to see? For beneficiaries this could be what the barriers they face are, what are potential digital hooks to develop their interest in support and training?
Decide what you want to measure and how you will collect the information and data you need. Are you interested in counting just the number of participants, or do you want to evidence how gaining new skills has had an impact on the individual? It’s good to start to collect data at the start of the project so you can track progress over time.
Once the project is up and running it can be valuable to re-visit the assumptions that underpinned the evaluation model and adjust them to reflect operational experience.
Output and outcome information enables projects to assess the impact of the work they are doing. The National Lottery Community Fund provides some useful definitions for the terms involved.
Outputs are the products, services or facilities that result from an organisation’s or project’s activities. For example in a programme to improve well-being amongst older people, outputs might include the different types of interventions being offered by projects, or the numbers of people overall participating in activities under the programme.
Outcomes are the changes, benefits, learning or other effects that result from what the project or organisation makes, offers or provides. For example, for the same well-being programme, outcomes might be improvements in clients’ physical or emotional health, or projects’ improved ability to extend their reach to different client groups.
Impact is the broader or longer-term effects of a project’s or organisation’s outputs, outcomes and activities. For example, in addition to an understanding of the extent to which projects funded by the well-being programme have achieved their outcomes, there might be a longer-term change in the way some projects work with their clients, new partnerships may have developed, or policy may have been influenced at a local or wider level.
One Digital event – Measuring the impact of digital inclusion projects
In November 2018 the Community of Practice held a learning event on ‘Measuring Impact’ – the keynote opening presentations are available as well as information from the different workshops. There are also some useful background resources on evaluation.
Measuring the impact on learners, Digital Champions & partnerships
A useful summary of How to measure the impact of a Digital Champion project on learners can be accessed here.
A discussion paper on How to measure the impact of working in partnership can be accessed here.
Some useful tips on How to measure the impact on Digital Champions can be accessed here.
Recording digital inclusion outputs
Projects will record outputs such as numbers of learners reached and how many sessions they have attended.
The Digital Unite Digital Champions Network (DCN) supports One Digital projects to use a tally facility to record the numbers of people accessing digital training and support. The DCN also has a web app enabling Champions to record their learner interactions quickly and easily.
Outcomes for the individual
One option for digital inclusion projects wanting to develop measures for the impact on the individual would be to capture the developing skills of learners in more detail. This could be linked to the Essential Digital Skills Framework which provides a set of skills for life and work. SCVO have developed a toolkit for projects to use with their learners to measure their confidence levels before and after a learning intervention.
There is also an international framework, DQ (Digital Intelligence) Framework, which is a set of global standards for digital literacy, skills and readiness.
Learner progress in developing these skills can be used to measure support provided by digital inclusion projects. The Framework also includes Digital Foundation skills which are needed by a new learner and cover the basics of setting up and using a device(s).
Some projects will aim to develop digital skills in all areas of the Framework, others will focus on particular outcomes which arise once digital skills are being confidently used. Some projects may capture learning after a specific period of training and intervention. Others may be interested in looking at longitudinal impact over a period of time on the individual. Here are some examples to consider.
A training centre providing a 6-8 week structured programme of learning supported by a professional Digital Champion would be likely to use all elements of the Framework to review skills developed. The centre would aim to provide learners with a full range of digital skills.
An employment project might measure Foundation skills, confidence and motivation plus specific skills needed to move into work such as CV building or searching for jobs online.
A mental health project supporting people transitioning from a residential setting to successfully living in the local community, might look at how digital skills and confidence develop over a period of two years.
Wider digital inclusion project outcomes
Possible outcomes that you might want to consider are outlined in the table. The majority of digital skills projects are set up to achieve improved digital skills for the individual beneficiary. Sometimes organisations establishing digital skills projects have multiple outcomes they want to achieve.
Some digital inclusion projects are set up alongside a particular organisational digital transformation (a change to a digital delivery of a service) and carry out projects alongside that change.
A social housing provider wanting to support tenants to access services such as booking repairs online sets up a project which develops a new repairs software system, staff receive training in using the new system. Once launched, staff promote the advantages of using the new system and identify individual tenants who need support to use the new online repairs booking system.
A young people’s project working with 16-24 year old’s who are not in work, education or training sets up a digital skills group to support their gaining IT skills that will help them find work, at the same time they run some sessions on managing your digital footprint and staying safe online.
A partnership project which includes charities, the libraries service, a local supermarket and local authority frontline staff runs a project designed to improve public knowledge about where local people can go to access free Wi-Fi and 1-1 support.
The table below provides some example outcomes that could be used in measuring the impact of a project. The examples are illustrative of an approach that looks at the impact on the individual, for an organisation and for the wider community.
Linked outcomes and other barriers to digital skills development
Digital skills development is one element of digital inclusion support and overcoming the barriers people face could include measuring support to improve –
- Literacy and confidence in using English
- Financial capability
- Access to appropriate devices to get online (such as recycled laptop schemes, or schemes with IT providers to access cheaper tablets)
- Access to social tariffs to reduce costs of having broadband at home for people on income based welfare benefits
Digital Inclusion Evaluation Toolkit
There is a useful Digital Inclusion Evaluation Toolkit published by the Department for Digital Culture Media and Sport (DCMS) and developed with a range of partner contributions – the outcome categories and suggested measures for each outcome are fully identified in the Toolkit. More about the background to the development of the Toolkit and the case for using it can be found in this blog by Douglas White, Head of Advocacy at Carnegie UK Trust.
If you are interested in learning more about how organisations have evaluated the impact of their digital inclusion projects here are some examples.
One Digital – Citizens Online
Get IT Together Longitudinal Study 2014
Just Economics – The Value of Digital Inclusion
Northern Housing Consortium – The Social Case for Digital Inclusion
Carnegie Trust – #NotWithoutMe: A Digital World for All?
Good Things Foundation – Libraries Delivering Digital Inclusion