What works in enhancing social and emotional skills development during childhood and adolescence? A review of the evidence on the effectiveness of school-based and out-of-school programmes in the UK: Programme templates.

Executive Summary Introduction This review sought to determine the current evidence on the effectiveness of programmes available in the UK that aim to enhance the social and emotional skills development of children and young people aged 4-20 years. The review was commissioned by the Early Intervention Foundation (EIF), the Cabinet Office and the Social Mobility and Child Poverty Commission as part of wider efforts to encourage evidence-based commissioning and delivery of services for young people. Based on a systematic search of the literature, this report presents a narrative synthesis (i.e. a qualitative summary of findings as opposed to a statistical meta-analysis) of the review findings from evaluations of programmes implemented in the UK in both the school and out-of-school settings. This review addresses the question of what works in enhancing children and young peoples social and emotional skills and the quality of existing provision in the UK. Extensive developmental research indicates that the effective mastery of social and emotional skills supports the achievement of positive life outcomes, including good health and social wellbeing, educational attainment and employment and the avoidance of behavioural and social difficulties. There is also a substantive international evidence base which shows that these skills can be enhanced and positive outcomes achieved through the implementation of effective interventions for young people. There are a number of ways of defining social and emotional skills. CASEL (2005) defined social and emotional skills as relating to the development of five interrelated sets of cognitive, affective and behavioural competencies: self-awareness, self-management, social awareness, relationship skills and responsible decision making. The Young Foundation (McNeil et al., 2012) identified a core set of social and emotional capabilities that are of value to young people. These capabilities have been grouped into seven clusters, each of which is supported by an evidence base that demonstrates their association with positive life outcomes. These capabilities include; managing feelings, communication, confidence and agency, planning and problem solving, relationships and leadership, creativity, resilience and determination. Drawing on existing models and frameworks, a list of these core social and emotional skills were included in this review. The key objective of this review was to systematically review the peer review and grey literature (2004- 2014) examining evidence on the effectiveness of school and out-of-school interventions implemented in the UK that are aimed at enhancing children and young peoples social and emotional skills. In reviewing the evidence, specific questions were addressed: what programmes are effective in enhancing social and emotional skills in the (i) school setting and (ii) out-of-school setting? what is the strength of the evidence? what programmes strategies are ineffective in enhancing social and emotional skills? what are the key characteristics of effective programme? what are the implementation requirements for these programmes what implementation factors are important in achieving programme outcomes? what interventions are effective according to age gender ethnicity socio-economic background and level of vulnerability what is the evidence on the costs and cost-benefits of these interventions? 3 Methods An electronic search of relevant databases and the grey literature was undertaken, including; a systematic search of five academic databases, international databases of school and out-of-school evidence-based programmes, public health databases, a search of the grey literature and a Call for Evidence distributed to UK organisations that work in this area. The findings were, therefore, dependent on organisations that had either published evaluations or proactively submitted evaluation data to the researchers. The primary outcomes of interest were social and emotional skills. In addition, the review provides information (where available) on the impact of interventions on broader educational, health and social outcomes, including educational attainment, employment, productivity, social inclusion, health, violence, substance misuse, delinquency and crime. In order to be included in the review, programmes must have met the following criteria: Address one or more social and emotional skills as outlined by SEAL, the Young Foundation, Cabinet Office and Education Endowment Foundation (See Appendix 3 for a full list of the social and emotional skills used in search process) Implemented in the UK Implemented in the school or out-of-school setting Involve children and young people aged 4-20 years Involve children and youth in the general population or those identified at risk of developing problems (individuals whose risk is higher than average as evidenced by biological, psychological or social risk factors). Children or young people in need of treatment (individuals identified as having minimal but detectable signs or symptoms of a mental, emotional, behaviour or physical disorder) were not included in this review. Treatment programmes for delinquency, drug-abuse and mental health problems were excluded while prevention programmes in these areas were included. In the case of parenting interventions, the intervention must contain a child youth component. In addition to these programme criteria, the programmes evaluation had to meet the following criteria to be included in the review: Published between 2004 and 2014 Adequate study design, using the Early Intervention Foundations (EIF) Standard of Evidence as a guide Adequate description of the research methodologies Description of the sample population Description of the intervention and its theoretical foundation Description of programme implementation including training, resources and workforce costs Include measures of at least one social or emotional outcome. Following the initial screening for inclusion, the intervention studies were reviewed according to the availability of evidence: School interventions were selected for review if a reasonably robust evaluation of the intervention (randomised control trial, quasi-experimental, pre-post design) was carried out in the UK and or the intervention had an established evidence base. Out-of-school interventions were selected for review if the intervention had a theory of change, had been evaluated in the UK (quantitative or qualitative evaluation) and or had an established evidence base. The use of less stringent inclusion criteria for out-of-school interventions was as a result of the scarcity of evidence-based interventions robust evaluations of out-of-school interventions. 4 Assessing Quality of Evidence All studies meeting the inclusion criteria underwent an assessment by the research team of the strength of the evidence using the Early Intervention Foundations Standard of Evidence (http: guidebook.eif.org.uk the-eif-standards-of-evidence). These standards of evidence differentiate between interventions that have established evidence, those with formative evidence and interventions which have non-existent evidence or where the evidence has been shown to be ineffective or harmful. Table 1 provides a description of the EIFs Standards of Evidence. Table 1: The EIF Standards of Evidence Evidence or rationale for programme Description of evidence Description of programme EIF rating A consistently effective programme with established evidence of improving child outcomes from two or more rigorous evaluations (RCT QED) Established Consistently effective 4 An effective intervention with initial evidence of improving child outcomes from high quality evaluation (RCT QED) Initial Effective 3 A potentially effective intervention with formative evidence of improving child outcomes. Lower quality evaluation (not RCT QED) Formative Potentially effective 2 An intervention has a logic model and programme blueprint but has not yet established any evaluation evidence Non-existent Theory-based 1 The programme has not yet developed a coherent or consistent logic model, nor has it undergone any evaluation Non existent Unspecified 0 Evidence from at least one high-quality evaluation of being ineffective or resulting in harm Negative Ineffective Harmful -1 For this report, assessment of the quality of evidence was undertaken by the research team. Interventions received a pre-rating of Level, 4, 3, 2, 1. In grading the evidence, particular attention was paid to the quality of the research design and the use of standardised outcome measures (i.e. objective and reliable measures that have been independently validated). Interventions that received a pre-rating of 4 or 3 were classified as well evidenced i.e. a number of rigorous evaluation studies available (Level 4) or where there is at least one good quality study (Level 3). Interventions that received a pre-rating of 2 or 1 were classified as having limited evidence i.e. evidence from lower quality evaluation available (Level 2) or programme has an evidence-based logic model but has not yet established evaluation evidence (Level 1). Subsequent work will be undertaken by the EIF and a formal assessment of the programmes for inclusion in the EIF online Guidebook will be undertaken by an evidence review panel. 5 Key Findings Searching the academic databases, health and education databases and the grey literature, a total of 9,472 school articles and 12,329 out-of-school articles were screened. Out of this, 113 school interventions and 222 out-of-school interventions were identified. A total of 39 school-based interventions fulfilled the review criteria (implemented in the UK with a robust UK evaluation and or an international evidence-base) and were selected for review. Of the out-of-school interventions, 55 interventions fulfilled the review criteria (implemented in the UK with a quantitative or qualitative UK evaluation and or an international evidence base) and were selected for review. Interventions were classified as (i) interventions with a competence enhancement focus and (ii) interventions aimed at reducing problem behaviour through the development of social and emotional skills. Results for School Programmes Of the 39 school programmes, 24 were adopted from international evidence-based programmes. Almost three quarters of the interventions were evaluated in the last five years (71.8%). The majority of studies employed a randomised control trial or quasi-experimental design (84.6%) and were sourced from published articles (84.6%). The majority of interventions were short term in duration (less than one year). Just under half of all interventions (46.2%) were implemented in primary school, 20.5% were implemented across primary school and secondary school and 33.3% of interventions were implemented with young people in secondary school. Drawing on existing classifications, school programmes were classified as follows: 1. Interventions with a competence enhancement focus a. Universal social and emotional skills interventions b. Small group social and emotional skills interventions c. Mentoring and social action interventions 2. Interventions aimed at reducing problem behaviours a. Aggression and violence prevention interventions b. Bullying prevention interventions c. Substance misuse prevention interventions Findings within each group were as follows: Interventions with a competence enhancement focus Universal social and emotional skills interventions Sixteen universal social and emotional skills-based interventions implemented in the UK were identified. The majority of these interventions (N = 13) are well evidenced in terms of their effectiveness on children and young peoples social and emotional skills. Of these programmes implemented in the UK, the strongest evidence is apparent for programmes with an established evidence base either from international and or UK studies (PATHS, Friends, Zippys Friends, UK Resilience, Lions Quest, Positive Action). These programmes were shown to have a significant positive impact on children and young peoples social and emotional skills including coping skills, self esteem, resilience, problem solving skills, empathy, reduced symptoms of depression and anxiety. Broader outcomes from secondary school interventions that adopted a whole school approach include reduced behaviour problems, enhanced academic performance, and improved family relations (Lions Quest, Positive Action). 6 There is promising emerging evidence in relation to UK developed interventions including Circle Time, Lessons for Living, Strengths Gym, Rtime .b Mindfulness Programme. The Australian developed online cognitive behavioural skills intervention MoodGYM, is well evidenced, and is currently being implemented and evaluated as part of the Healthy Minds in Teenagers curriculum in the UK. Results from evaluations of the primary and secondary Social and Emotional Aspects of Learning (SEAL), which adopt a whole school approach to implementation, provide limited evidence of improvements in primary school childrens social and emotional skills. No programme impact was reported in an evaluation of secondary SEAL. Quality of implementation was identified as essential in producing programme outcomes including enhancing the school environment, pupils social experiences, school attainment and reducing persistent absence. Small group social and emotional skills interventions Three small group classroom-based interventions implemented as part of primary SEAL and two afterschool interventions were identified for students at higher risk of developing social and emotional problems. Two of the small group classroom-based interventions identified are well evidenced in terms of having at least one good quality study that reported a positive impact (self- and teacher reported) on childrens social and emotional skills, reducing emotional problems and enhancing peer relationships (Going for Goals, New Beginnings). Similar findings were evident for the after-school small group intervention, Pyramid Project, for children identified as withdrawn and emotionally vulnerable. Mentoring and social action interventions Two mentoring and one social action intervention were identified. There are too few studies to draw strong conclusions regarding the effectiveness of these types of interventions when implemented in the school setting. In addition, the quality of the studies reviewed was compromised as a result of weak study design and use of non-standardised outcome measures. Further testing of these interventions using more robust methods is warranted. Interventions aimed at reducing problem behaviours Aggression and violence prevention interventions Four violence prevention interventions were identified. All four interventions are well evidenced as a result of multiple rigorous international evaluations indicating their impact on reducing social and emotional problems and aggressive and disruptive behaviour. These interventions, which are implemented in primary school, differ in terms of their approach including (i) classroom management strategies: Incredible Years Classroom Management Curriculum, Good Behaviour Game (ii) curriculum-based violence prevention intervention: Second Step (iii) whole school approach: Peacebuilders The Good Behaviour Game, which is currently being trialled in the UK, shows consistent evidence of its effectiveness, including sustained social, emotional, behavioural and academic findings at 14 year follow up. 7 Bullying prevention interventions Six bullying prevention interventions were identified. Three interventions are well evidenced in terms of their impact on social and emotional skills including social relations, prosocial behaviour and reduced bullying and victimisation. These interventions adopt a whole school approach to bullying prevention providing curriculum resources, whole staff training, parent guides and addressing school environment and ethos (Olweus, Kiva, Steps to Respect). There is evidence from the studies reviewed to indicate that bullying prevention peer mentoring interventions are ineffective in improving children and young peoples social and emotional skills and in some cases can have a negative impact on rates of bullying. Substance misuse prevention interventions Five substance misuse prevention interventions that teach personal and social skills for self-management and resilience were identified. These interventions have an established international evidence base indicating their positive impact on the use of skills and strategies to resist risk-taking behaviour and reduced alcohol, cigarette and drug use (LifeSkills Training, Keepin It Real, All Stars and Project Star, SHAHRP). Lifeskills Training intervention has a well established evidence base with sustained findings reported at six years follow up. Characteristics of programme effectiveness for school interventions Effective school-based programmes identified in this review shared a number of common characteristics and these include: Focus on teaching skills, in particular the cognitive, affective and behavioural skills and competencies as outlined by CASEL Use of competence enhancement and empowering approaches Use of interactive teaching methods including role play, games and group work to teach skills Well-defined goals and use of a coordinated set of activities to achieve objectives Provision of explicit teacher guidelines through teacher training and programme manuals. Impact on Equity and Cost-Benefit Results There was a paucity of studies that reported on subgroup differences. There is, however, some evidence to indicate that interventions aimed at increasing social and emotional skills and reducing problem behaviours are particularly effective with children and young people who are most at risk of developing problems. This is demonstrated by the findings from universal social and emotional interventions, aggression and violence prevention and substance misuse prevention interventions. There is a paucity of information regarding the cost-benefit ratio of school-based social and emotional skills programmes as implemented in UK schools. Cost-benefit analysis information, provided by Dartingtons Investing in Children database and Blueprints for Positive Youth Development database, is available for five interventions: PATHS (1:11.6), UK Resilience (1:7.1), Good Behaviour Game (1:26.9), Lifeskills Training (1:10.7) and Project STAR (1:1.2). These cost-benefit ratio results show a positive return on investment for these evidence-based interventions. 8 Results for Out-of-School Programmes The majority of interventions identified were developed in the UK (83.6%) and evaluated in the UK in the last five years (81.8%). A total of 35 interventions were located in unpublished reports (63.6%). Interventions were predominantly evaluated using a pre-post study design with no control group (78%). The majority of interventions were short term in duration (less than one year) and implemented with socially excluded and disadvantaged young people (aged 13-20) determined to be at risk of developing social and emotional problems engagement in risk-taking behaviour. Drawing on existing classifications, these programmes were classified into the following groups: 1. Interventions with a competence enhancement focus a. Youth arts and sports interventions b. Family-based interventions c. Mentoring interventions d. Education, work, career interventions e. Cultural awareness interventions 2. Interventions aimed at reducing problem behaviours a. Crime prevention interventions a. Substance misuse prevention interventions Interventions with a competence enhancement focus Youth arts and sports interventions Eight sports, music and drama-based interventions were identified. There is limited evidence of their effectiveness due to weak study designs and the use of non-standardised outcome measures. There is evidence from three interventions which used standardised outcome measures to indicate significant improvements in young peoples self esteem, confidence, emotional regulation, organisation and leadership skills (Hindleap Warren Outdoor Education Centre which provides outdoor activities for young people; Girls on the Move Leadership Programme provides training for females in dance and sports activities; Mini-Mac, a peer led music project) The quality of the evaluation studies on the remaining five interventions was too weak to determine programme impact. Family-based interventions Five family-based interventions were identified, all of which focused on enhancing family functioning, promoting positive parenting, enhancing child and adolescent social and emotional skills and reducing problem behaviours. Four of the interventions, which were adopted from the US and implemented in the UK, are well evidenced in terms of their impact on children and young peoples social skills including self concept, self efficacy, internalising and externalising behaviour and peer and family relations (Incredible Years, Families and Schools Together, Strengthening Families Programme; Social Skills Group Intervention- Adolescent). Broader outcomes include improved academic performance and attachment to school, improved parental engagement and reduced rates of parental substance misuse. 9 Mentoring interventions Eleven mentoring interventions were identified, however, the quality of the evidence from the majority of studies is inadequate to determine programme impact as a result of weak study design (no control group) and use of non-standardised outcome measures to determine programme impact. One intervention is well evidenced. The Big Brothers Big Sisters mentoring programme has an established international evidence base in terms of positive long-term impacts of matching adult volunteer mentors with young people aged 6-18 to support them in reaching their potential over the course of a year. Outcomes include improved self worth, relationships with peers and parents, reduced substance misuse and improved academic outcomes. The Teens and Toddlers programme, which is aimed at reducing teenage pregnancy through training adolescent participants to become mentors in early childcare, reported improvements in girls self esteem, self efficacy and decision making, although no positive impact was found in relation to use of contraception or expectation of teenage parenthood. Education, work, career interventions Five UK developed interventions were identified that aim to increase young peoples personal and social skills so that they are able to make gains in employment, education and training. The quality of the evidence from these studies was insufficient to determine impact as a result of weak study design and use of non-standardised outcome measures. Qualitative results suggest a potential positive impact on young peoples confidence, self esteem, aspirations, social relations and on broader outcomes including progression into education, training, volunteering or employment and reduced truancy. Social action interventions Twelve social action interventions were identified, eleven of which were developed in the UK. National Citizen Service was the only intervention to utilise a quasi-experimental design and some standardised outcome measures to determine programme impact. This intervention produced promising evidence in terms of its significant impact on young peoples confidence, happiness, sense of worth, anxiety levels, interest in education and attitude towards mixing in the local area. Additional selfreported improvements included social competence, resilience, communication, leadership, decision making and teamwork skills. Another four interventions which used a pre-post design produced limited evidence in terms of their effectiveness on young peoples self confidence, self esteem, social skills, leadership skills, problem solving, organisational skills, communication skills and motivation. (vInspired Team V, vInspired 24 24, vInspired Cashpoint, Youth Voice UK). Broader outcomes from these four interventions and National Citizen Service include increased community engagement, enhanced career ambition, improved attitudes about future employment, increased intention to engage in voluntary activities in the future. The quality of the evidence, however, needs to be strengthened using more robust evaluation designs with standardised outcome measures. 10 Cultural awareness interventions Two cultural awareness interventions were identified. Both interventions were developed in the UK and were designed to challenge negative attitudes and racism (Think Project), and support the cultural heritage and a sense of identity among ethnic minorities (Sheffield Multiple Heritage Service). Results from these studies indicate a positive impact on young peoples self esteem, wellbeing and their understanding and respect for other cultures. Further testing of these interventions using more robust methods and standardised outcome measures would assist in determining the immediate and long term impact of these interventions and mechanisms of change. Interventions aimed at reducing problem behaviours Crime prevention interventions Nine crime prevention interventions were identified, six of which were developed in the UK. A number of these multi-component interventions were grounded in a mentoring approach. Evidence regarding the effectiveness of these interventions is currently limited as a result of weak study designs and the use of non-standardised outcome measures to evaluate programme effectiveness. One intervention (Coaching for Communities, a five day residential intervention in combination with nine months mentoring) , which utilised a strong study design and standardised measures reported significant improvements in young peoples self esteem and prosocial behaviour and a significant reduction in antisocial behaviour. While there is promising evidence from the remaining interventions, use of more robust study designs and evaluation measures is required to determine programme impact. Substance misuse prevention interventions Three substance misuse prevention interventions, which were developed in the UK, were identified. There is limited evidence regarding the effectiveness of the RisKit multi-component personal and social skills interventions in terms of enhancing peer resistance strategies and reducing adolescent engagement in risk behaviour including use of alcohol and smoking. Evaluations of the remaining two interventions were too weak to determine programme impact. Characteristics of programme effectiveness for out-of-school interventions Effective out-of-school programmes identified in this review shared a number of programme characteristics. These programme adopted a structured approach to delivery including: having specific and well-defined goals direct and explicit focus on desired outcomes provision of structured activities training of facilitators and use of a structured manual implementation over longer period of time. 11 Impact on Equity and Cost-Benefit Results The majority of out-of-school interventions were delivered with young people identified as being at risk of developing social, emotional, behavioural problems, engaging in risky behaviour, and being socially excluded. However, only a small number of evaluation studies reported on the equity impact of these interventions for different subgroups of young people. A greater focus on assessing the equity impact of the out-of-school interventions is needed in order to determine the benefits for different groups of young people over time and to ensure that these interventions reach those young people with the greatest need while also addressing the social gradient. In terms of cost-benefit results, the majority of interventions (N = 37) provided information on the costs related to delivering the programme in the UK. Information on cost-benefits was available for three familybased and four social action interventions. The results from the family-based interventions were particularly positive for FAST (1:3.3). The cost-benefit ratio for the Incredible Years was reported by Dartington to be 1:1.4, whilst the results from the Strengthening Families programme were less positive (1:0.65 with a 93% risk of loss as reported by Dartington). Four UK developed social action interventions reported promising findings in relation to their cost-benefit analysis: vInspired Cashpoint (1:1.4.8), National Citizen Service (1:1.39-4.8), vInspired Team V (1:1.5), Millennium Volunteers (1:1.6).