You are here

Leading early childhood experts warn against basing policy on flawed data

Monday, 20 March, 2017

A group of leading  experts in early childhood education have warned against basing policy on a recent paper which claims to find only a weak link between the presence of graduates in preschool provision and children's outcomes.

The paper in question, from economists Jo Blanden, Kirstine Hansen and Sandra McNally, published by the Centre for Economic Performance at the London School of Economics, had received widespread coverage for its claims that their analysis of large scale datasets showed little correlation between children's outcomes in assessments at the end of Reception and Key Stage 1, and quality as measured by Ofsted ratings or the presence of graduate staff in preschools. As the authors themselves noted, this appeared to contradict most previous research which had shown that graduate and teacher-led provision significantly improved children's outcomes, especially for the most disadvantaged children.

In a paper published this week by Early Education, Pam Sammons and a team of colleagues from the Universities of Oxford and Exeter and UCL Institute of Education have critiqued the approach taken by Blanden and colleagues.  They challenge the findings on the basis that the measures of quality of provision and of children’s outcomes are too crude, and fail to take into account limitations inherent in the data used.  Key weaknesses in the study’s design included:

  • The measures of quality used were access to a graduate and Ofsted ratings.  Although previous research has shown a link between staff qualifications and quality, the simple presence or absence of a graduate is too crude a measure to give an accurate picture of setting quality.  Ofsted ratings have also been shown to correlate very poorly with observational measures of quality.  Even allowing for this, the Ofsted scores could have been obtained up to six years before or one year after the child’s attendance.  It is very likely that the quality of provision could have changed during this period, especially as settings rated inadequate or requires improvement receive intensive support to improve.
  • The measures of child outcomes used were the total score from the Early Years Foundation Stage Profile (EYFSP), and a single conflated score in Key Stage 1 assessments.  Both of these are very broad-brush, and also based on teacher assessment, which is known to be less reliable than the kind of externally administered standardised assessments used in other studies. 
  • By using the EYFSP scores obtained at the end of reception, it becomes impossible to separate the effect of preschool from the effect of the year of full-time reception provision in schools, particularly as it included no way of measuring “value added” from children’s individual starting points.
  • The measures included to take account of social background were also very limited: eligibility for Free School Meals and a postcode measure of neighbourhood disadvantage.  Other studies have additionally looked at, for example, parents’ education, family income levels and the home learning environment.
  • The statistical methods used did not sufficiently take account of the effects of clustering.


The reports authors say “For these reasons we believe it would be dangerous for the LSE study to be used in drawing policy conclusions about the importance of pre-school quality and the role of qualified teachers in supporting young children’s development.”

Professor Pam Sammons commented:

"We need to use the best ways to measure the quality of young children's pre-school experiences to investigate how such experiences can support their development as they move on into primary school. Observations of children and staff in pre- schools by trained researchers  provide important evidence about quality and the role of qualified teachers."

“Establishing the Effects of Quality in Early Childhood:  Comparing evidence from England” by Professor Pam Sammons, Professor Kathy Sylva, Dr James Hall, Professor  Iram Siraj,  Professor Edward Melhuish, Brenda Taggart, and Sandra Mathers is published as an Occasional Paper by Early Education and can be downloaded from the Early Education website.

Notes to editors: 

Early Education (The British Association for Early Childhood Education) is the leading independent national charity for early years practitioners and parents, campaigning for the right of all children to education of the highest quality. Founded in 1923, it has members in England, Northern Ireland, Scotland and Wales and provides a national voice on matters that relate to effective early childhood education and care of young children from birth to eight. The organisation supports the professional development of practitioners through publications, training, conferences, seminars and access to a national and regional branch network. For more information on the work of Early Education visit www.early-education.org.uk