Statement by the Evaluation Collective on the OfS Guidance on Access and Participation Plans (April 2023)

Statement by the Evaluation Collective on the OfS Guidance on Access and Participation Plans, April 2023.

Since the appointment of the new Director of Fair Access and Participation in late 2021, evaluation has been much higher on the regulatory agenda. In April 2022, the Evaluation Collective came together to publish a Manifesto which challenged aspects of the new agenda which were perceived by some as unhelpful for the HE sector.

At the end of March 2023, the Office for Students published their response to their sector consultation on revisions to the Access and Participation Plan. Evaluation is still front and centre; some aspects and expectations have changed, and others remain the same. The Evaluation Collective have now published their response to these APP revisions based on their Manifesto principles.

1. Evaluation should be a collective and transformative action

In line with the principles in the Evaluation Manifesto, the Evaluation Collective (EC) welcome the OfS instruction to evaluate the ‘effectiveness and impact of the activity’ in the APP and the focus on institutional learning from their ‘own and other providers’ work on equality of opportunity’. Although this is likely to happen through existing sector networks, progress may be slow, and we would welcome any opportunity to accelerate this process through increased regulatory support and the explicit prioritisation of collaborative practices.

Furthermore, we continue to advocate against performative evaluation and note the risk that the new reporting requirements could lead to evaluation being treated as a time bound, box ticking exercise which discourages providers from ensuring that evaluation is embedded in a longer-term approach to practice development.

2. Evaluation advocacy can be an empowering tool for social justice

We are pleased to see some development of the intervention strategies and specifically how each strategy may address multiple risks to equality of opportunity. Hopefully, this will enable providers to discuss interventions holistically and recognise the complexity of intervention design and implementation in HE spaces. 

The continued emphasis on theories of change is also an important inclusion. However, while we acknowledge the importance of identifying clear quantitative targets in APP intervention strategies, we also recognise that the worth and value of interventions will often be broader than can be captured by a single measure. We therefore encourage institutions to treat their headline measures as indicators of change, but equally consider broader or more difficult to measure outcomes in their theories of change. This should include a recognition of the value of provisional, partial, and productive forms of knowledge/ data, including qualitative understanding.

3. Evaluation will evolve in visible, inclusive, and supportive spaces

We understand the pressure facing colleagues in institutions that have chosen to submit their APPs in Wave 1 and appreciate any learning that can be shared with the sector on evaluative expectations. We hope that the support and guidance offered during this timeframe will recognise the different levels of experience of institutional teams and take steps to address any barriers to evaluation design and implementation, specifically by encouraging cross-provider learning.

4. Evaluation can be demystified by rejecting labels and gatekeepers

We acknowledge and appreciate the change in language regarding evaluation and a refocus on inclusive and accessible communications.

5. Building confidence and capacity is key to developing evaluation practices

We recognise the continued need to develop confidence and capacity building within institutional teams to rise to these new expectations. The EC will continue to be led by the sector and make supportive connections to enable colleagues to submit an APP response. 

6. Evaluations should be transparently grounded in criticality and context

We appreciate the softening of the emphasis on independent evaluation and recognition that practitioner-led evaluation has a place in HE. We hope this includes notions of participatory evaluation at intervention level, which moves beyond the OfS expectation that providers encourage student involvement in ‘the design, implementation and evaluation of the plan’. We note, however, this may be epistemologically at odds with the desire for ‘a robust, objective and credible evaluation plan’ which suggests there could be judgements made about suitability.

7. All types of evidence used in evaluation can have value

We continue to believe that the OfS standards of evidence provide a useful starting point for discussions about evaluation purpose and proportionality. Careful interrogation of the implicit assumptions underpinning these standards should still be conducted to ensure they do not discourage innovative and purposeful evaluation. 

We also hope that the OfS continue to recognise ‘that each provider will be at a different stage of developing its evaluation expertise’ rather than using the standards of evidence as judgement criteria for intervention strategies in the APP.

8. Hierarchical assumptions about evaluation methods should be disrupted

We welcome the method-neutral stance of the APP guidance, whilst acknowledging the continued specialist remit of TASO for generating causal evidence. We continue to believe that there should be no hierarchy to evaluative evidence, and that pragmatic evaluators can and should make use of a range of methods to enable learning. We would like regulatory language to reflect this.

9. An inclusive approach to evaluation funding should be adopted

We encourage providers to include funding for evaluation capacity building in the reported spend on evaluation and for the OfS to acknowledge this as integral to APP returns. In addition, the increased regulatory emphasis on evaluation should now be used as a further mandate within institutions to secure the resources to enable confidence and capacity building in teams across the student lifecycle. 

10. Higher education requires a cultural shift to ensure evaluations lead to learning

We have some concerns that the timebound publishing requirement (note the regulatory impact of a ‘reportable event’) may inhibit meaningful reporting and learning. From experience, evaluations can be ‘derailed’ by many events in the complex world of HE. We would ask OfS/TASO to review this aspect of their guidance as APPs begin to be rolled out.

The Evaluation Collective is a cross-sector group of like-minded evaluation advocates working to enhance higher education student outcomes.