The limitations of School Exclusion Statistics

The Department for Education publishes an annual report on exclusions, the latest edition published in August 2018 being ‘Permanent and fixed-period exclusions in England: 2016 to 2017.

The 2018 report shows that the overall rate of permanent exclusions was 0.1 per cent of pupil enrolments in 2016/17. The number of exclusions was 7,720.

exlusion statistics.png

The report also goes into more detail, for example….

  • The vast majority of exclusions were from secondary schools >85% of exclusions.
  • The three main reasons for permanent exclusions (not counting ‘other’) were
    • Persistent disruptive behaviour
    • Physical assault against a pupil
    • Physical assault against an adult.

Certain groups of students are far more likely to be permanently excluded:

  • Free School Meals (FSM) pupils had a permanent exclusion rate four times higher than non-FSM pupils
  • FSM pupils accounted for 40.0% of all permanent exclusions
  • The permanent exclusion rate for boys was over three times higher than that for girls
  • Over half of all permanent exclusions occur in national curriculum year 9 or above. A quarter of all permanent exclusions were for pupils aged 14
  • Black Caribbean pupils had a permanent exclusion rate nearly three times higher than the school population as a whole.
  • Pupils with identified special educational needs (SEN) accounted for around half of all permanent exclusions

The ‘reasons why’ and ‘types of pupil’ data probably hold no surprises, but NB there are quite a few limitations with the above data, and so these stats should be treated with caution!

Limitations of data on permanent exclusions

Validity problems…

According to this Guardian article, the figures do not take into account ‘informal exclusions’ or ‘off-rolling’ – where schools convince parents to withdraw their children without making a formal exclusion order – technically it’s then down to the parents to enrol their child at another institution or home-educate them, but in many cases this doesn’t happen.

According to research conducted by FFT Education Datalab up to 7, 700 students go missing from the school role between year 7 and year 11 when they are  supposed to sit their GCSEs…. Equivalent to a 1.4% drop out rate across from first enrolment at secondary school to GCSEs.

Datalabs took their figures from the annual school census and the DfE’s national pupil database. The cohort’s numbers were traced from year seven, the first year of secondary school, up until taking their GCSEs in 2017.

The entire cohort enrolled in year 7 in state schools in England in 2013 was 550,000 children

However, by time of sitting GCSEs:

  • 8,700 pupils were in alternative provision or pupil referral units,
  • nearly 2,500 had moved to special schools
  • 22,000 had left the state sector (an increase from 20,000 in 2014) Of the 22,000,
    • 3,000 had moved to mainstream private schools
    • Just under 4,000 were enrolled or sat their GCSEs at a variety of other education institutions.
    • 60% of the remaining 15,000 children were likely to have moved away from England, in some case to other parts of the UK such as Wales (used emigration data by age and internal migration data to estimate that around)
    • Leaves between 6,000 to 7,700 former pupils unaccounted for, who appear not to have sat any GCSE or equivalent qualifications or been counted in school data.

Working out the percentages this means that by GCSEs, the following percentages of the original year 7 cohort had been ‘moved on’ to other schools. 

  • 6% or 32, 000 students in all, 10, 00 of which were moved to ‘state funded alternative provision, e.g. Pupil Referral Units.
  • 4%, or 22K left the mainstream state sector altogether (presumably due to exclusion or ‘coerced withdrawal’ (i.e. off rolling), of which
  • 4%, or 7, 700 cannot be found in any educational records!

This Guardian article provides a decent summary of the research.

Further limitations of data on school exclusions

  • There is very little detail on why pupils were excluded, other than the ‘main reason’ formally recorded by the head teacher in all school. There is no information at all about the specific act or the broader context. Labelling theorists might have something to say about this!
  • There is a significant time gap between recording and publication of the data. This data was published in summer 2018 and covers exclusions in the academic year 2016-2017. Given that you might be looking at this in 2019 (data is published annually) and that there is probably a ‘long history’ behind many exclusions (i.e. pupils probably get more than one second chance), this data refers to events that happened 2 or more years ago.

Relevance of this to A-level sociology

This is of obvious relevance to the education module… it might be something of a wake up call that 4% of students leave mainstream secondary education before making it to GCSEs, and than 1.4% seem to end up out of education and not sitting GCSEs!

It’s also a good example of why independent longitudinal studies provide a more valid figure of exclusions (and ‘informal’ exclusions) than the official government statistics on this.

 

I’ll be producing more posts on why students get excluded, and on what happens to them when they do and the consequences for society in coming weeks.

 

This is a topic that interests me, shame it’s not a direct part of the A level sociology education spec!

Advertisements

Do 25% of children really have their own mobiles? Invalid Research Example #01

This is a ‘new thread’ idea… posting up examples of naff research. I figure there are two advantages to this…

  1. It’s useful for students to have good examples of naff research, to show them the meaning of ‘invalid data’ or ‘unrepresentative samples’, or in this case, just plain unreferenced material which may as well be ‘Fake News’.
  2. At least I get some kind of pay back (in the form of the odd daily post) for having wasted my time wading through this drivel.

My first example is from The Independent, the ex-newspaper turned click-bait website.

I’ve been doing a bit of research on smart phone usage statistics this week and I came across this 2018 article in the Independent: Quarter of Children under 6 have a smartphone, study finds.

invalid research.png

The article provides the following statistics

  • 25% of children under 6 now have their own mobile
  • 12% of children under 6 spend more than 24 hours a week on their mobile
  • 80% parents admit to not limiting the amount of time their children spend on games

Eventually it references a company called MusicMagpie (which is an online store) but fails to provide a link to the research,  and provides no information at all about the sampling methods used or other details of the survey (i.e. the actual questions, or how it’s administered.). I dug around for a few minutes, but couldn’t find the original survey either.

The above figures just didn’t sound believable to me, and they don’t tie in with OFCOM’s 2017 findings which say that only 5% of 5-7 year olds and 1% of 3-4 year olds have their own mobiles.

As it stands, because of the simple fact that I can’t find details of the survey, these research findings from musicMagpie are totally invalid.

I’m actually quite suspicious that the two companies have colluded to generate some misleading click-bait statistics to drive people to their websites to increase advertising and sales revenue.

If you cannot validate your sources, then do not use the data!

Validity in Social Research

Validity refers to the extent to which an indicator (or set of indicators) really measure the concept under investigation. This post outlines five ways in which sociologists and psychologists might determine how valid their indicators are: face validity, concurrent validity, convergent validity, construct validity, and predictive validity. 

Validity refers to the extent to which an indicator (or set of indicators) really measure the concept under investigation. This post outlines five ways in which sociologists and psychologists might determine how valid their indicators are: face validity, concurrent validity, convergent validity, construct validity, and predictive validity.

As with many things in sociology, it makes sense to start with an example to illustrate the general meaning of the concept of validity:

When universities question whether or not BTECs really provide a measure of academic intelligence, they are questioning the validity of BTECs to accurately measure the concept of ‘academic intelligence’.

When academics question the validity of BTECs in this way, they might be suspicious that that BTECs are actually measuring something other than a student’s academic intelligence; rather BTECs might instead actually be measuring a student’s ability to cut and paste and modify just enough to avoid being caught out by plagiarism software.

If this is the case, then we can say that BTECs are not a valid measurement of a student’s academic intelligence.

How can sociologists assess the validity of measures and indicators?

what is validity.png

There are number of ways testing measurement validity in social research:

  • Face validity – on the face of it, does the measure fit the concept? Face validity is simply achieved by asking others with experience in the field whether they think the measure seems to be measuring the concept. This is essentially an intuitive process.
  • Concurrent validity – to establish the concurrent validity of a measure, the researchers simply compare the results of one measure to another which is known to be valid (known as a ‘criterion measure). For example with gamblers, betting accounts give us a valid indication of how much they actually win or lose, but wording of questions designed to measure ‘how much they win or lose in a given period’ can yield vastly different results. Some questions provide results which are closer to the hard-financial statistics, and these can be said to have the highest degree of concurrent validity.
  • Predictive validity – here a researcher uses a future criterion measure to assess the validity of existing measures. For example we might assess the validity of BTECs as measurement of academic intelligence by looking at how well BTEC students do at university compared to A-level students with equivalent grades.
  • Construct validity – here the researcher is encouraged to deduce hypotheses from a theory that is relevant to the concept. However, there are problems with this approach as the theory and the process of deduction might be misguided!
  • Convergent validity – here the researcher compares her measures to measures of the same concept developed through other methods. Probably the most obvious example of this is the British Crime Survey as a test of the ‘validity’ of Police Crime Statistics’. The BCS shows us that different crimes, as measured by PCR have different levels of construct validity – Vehicle Theft is relatively high, vandalism is relatively low, for example.

Source 

Bryman (2016) Social Research Methods

 

 

Why Do Voting Opinion Polls Get it Wrong So Often?

Surveys which ask how people intend to vote in major elections seem to get it wrong more often than not, but why is this?

Taking the averages of all nine first and then final polls for the UK general election 2017, the predictions for the Conservatives show them down from 46% to 44%; and Labour up from 26% to 36%.

voting intention 2017 general election

The actual vote share following the result of the general election shows the Conservatives at 42% and Labour at 40% share of the vote.

2017 election result share of vote UK

Writing in The Guardian, David Lipsey notes that ‘The polls’ results in British general elections recently have not been impressive. They were rightish (in the sense of picking the right winner) in 1997, 2001, 2005 and 2010. They were catastrophically wrong in 1992 and 2015. As they would pick the right winner by chance one time in two, an actual success rate of 67%, against success by pin of 50%, is not impressive.’

So why do the pollsters get it wrong so often?

Firstly, there is a plus or minus 2 or 3% statistical margin of error in a poll – so if a poll shows the Tories on 40% and Labour on 34%, this could mean that the real situation is Tory 43%, Labour 31% – a 12 point lead. Or it could mean both Tory and Labour are on 37%, neck and neck.

This is demonstrated by these handy diagrams from YouGov’s polling data on voting intentions during the run up to the 2017 UK general election…

Voting Intention 2017 Election 

Statistics Margin Error.png

Seat estimates 2017 General Election

Seat Estimates

Based on the above, taking into account margin for error, it is impossible to predict who would have won a higher proportion of the votes and more seats out of Labour and the Tories.

Secondly, the pollsters have no way of knowing whether they are interviewing a representative sample.

When approached by a pollster most voters refuse to answer and the pollster has very little idea whether these non-respondents are or are not differently inclined from those who do respond. In the trade, this is referred to as polling’s “dirty little secret”.

Thirdly, the link between demographic data and voting patterns is less clear today – it used to be possible to triangulate polling data with demographic data from previous election results, but voter de-alignment now means that such data is now less reliable as a source of triangulating the opinion polls survey data, meaning pollsters are more in the dark than ever.

Fourthly, a whole load of other factors affected people’s actual voting behaviour in this 2017 election and maybe the polls  failed to capture this?

David Cowley from the BBC notes that…. ‘it seems that whether people voted Leave or Remain in 2016’s European referendum played a significant part in whether they voted Conservative or Labour this time…. Did the 2017 campaign polls factor this sufficiently into the modelling of their data? If younger voters came out in bigger numbers, were the polls equipped to capture this, when all experience for many years has shown this age group recording the lowest turnout?’

So it would seem that voting-intention surveys have always had limited validity, and that, if anything, this validity problem is getting worse…. after years of over-estimating the number of Labour votes, they’ve now swung right back the other way to underestimating the popularity of Labour.

Having said that these polls are not entirely useless, they did still manage to predict that the Tories would win more votes and seats than Labour, but they just got the difference between them oh so very wrong.

The problem of obtaining representative samples (these days)

According to The Week (July 2017) – the main problem with polling these days is that finding representative samples is getting harder… When Gallup was polling, the response rate was 90%, in 2015, ICM had to call up 30 000 numbers just to get 2000 responses. And those who do respond are often too politically engaged to be representative.

 

Qualitative Data – Strengths and Limitations

Qualitative data includes….

  • Open question questionnaires
  • Unstructured interviews
  • Participant observation
  • Public and private documents such as newspapers and letters.

Theoretical strengths

  • Better validity than for quantitative data
  • More insight (Verstehen)
  • More in-depth data
  • More respondent-led, avoids the imposition problem.
  • Good for exploring issues the researcher knows little about.
  • Preferred by Interpretivists

Practical strengths

  • A useful way of accessing groups who don’t like formal methods/ authority

Ethical strengths

  • Useful for sensitive topics
  • Allows respondents to ‘speak for themselves’
  • Treats respondents as equals

Theoretical limitations

  • Difficult to make comparisons
  • No useful for finding trends, finding correlations.
  • Typically small samples, low representativeness
  • Low reliability as difficult to repeat the exact context of research.
  • Subjective bias of researcher may influence data (interviewer bias)
  • Disliked by Positivists

Practical limitations

  • Time consuming
  • Expensive per person researched compared to qualitative data
  • Difficult to gain access (PO)
  • Analyzing data can be difficult

Ethical limitations

  • Close contact means more potential for harm
  • Close contact means more difficult to guarantee anonymity and confidentiality
  • Informed consent can be an issue with PO.

Nature of Topic – When would you use it, when would you avoid using it?

  • Useful for complex topics you know little about
  • Not necessary for simple topics.

Crunch Paragraph/ Conclusion – Generally, how important is qualitative research to Sociology?

  • More hassle practically, lacks objectivity, but good for validity which is probably the most important factor required when doing research.

 

 

Evaluate the View that Theoretical Factors are the most Important Factor Influencing Choice of Research Method (30)

Just a few thoughts on how you might answer this in the exam. 

Introduction – A variety of factors influence a Sociologist’s decision as to what research method they use: the nature of topic, theoretical, practical and ethical factors.

Theoretical factors – Positivism vs Interpretivism – Positivists are interested in uncovering the underlying general laws that lie behind human action. They thus prefer quantitative methods because these enable large samples to be drawn and allow for the possibility of findings being generalised to the wider population.

They also prefer quantitative methods because the data can be put into graphs and charts, allowing for easy comparisons to be made at a glance.

Another method that is linked to the positivist tradition is the experiment – laboratory experiments allow researchers to examine human behaviour in controlled environments and so allow researchers to accurately measure the effects of one specific variable on another

Interpretivists generally prefer qualitative methods which are regarded as having high validity. Validity is the extent to which research provides a true and accurate picture of the aspect of social life that is being studied. Most sociologists would agree that there is little point doing sociological research if it is invalid.

Theoretical factors – Validity – Qualitative methods should be more valid because they are suitable for gaining an in depth and empathetic understanding of the respondent’s views of life. Qualitative methods are flexible, and allow for the respondents to speak for themselves, which avoids the imposition problem as they set the research agenda. Qualitative methods also allow for rapport to be built up between the respondent and the researcher which should encourage more truthful and in depth information to flow from the respondents.

The final reason why qualitative methods such as Participant Observation should yield valid data is that it allows for the researcher to see the respondents in their natural environment.

Theoretical factors – Reliability – Is the extent to which research can be repeated and the same results achieved. Positivists point out that it is more difficult for someone else to replicate the exact same conditions of a qualitative research project because the researcher is involved in sustained, contact with the respondents and the characteristics and values of the researcher may influence the reactions of respondents.

Moreover, because the researcher is not ‘detached’ from the respondents, this may detract from his or her objectivity. Participant Observers such as Willis and Venkatesh have, for example, been accused of going native – where they become overly sympathetic with the respondents.

Interpretivists would react to this by pointing out that human beings are not machines and there are some topics that require close human contact to get to the truth – sensitive issues such as abuse and crime may well require sympathetic researchers that share characteristics in common with the respondents. Interpretivists are happy to forgo reliability if they gain in more valid and in depth data.

 Representativeness – Obviously if one wants large samples one should use quantitative methods – as with the UK National Census. However, one may not need a large sample depending on the research topic.

 Practical Factors – Practical issues also have an important influence on choices of research method. As a general rule quantitative methods cost less and are quicker to carry out compared to more qualitative methods, and the data is easier to analyse once collected, especially with pre-coded questionnaires which can simply be fed into a computer. It is also easier to get government funding for quantitative research because this is regarded as more scientific and objective and easier to generalise to the population as a whole. Finally, researchers might find respondents more willing to participate in the research if it is less invasive – questionnaires over PO.

However, qualitative methods, although less practical, may be the only sensible way of gaining valid data, or any data at all for certain topics – as mentioned above UI are best for sensitive topics while participant observation may be the only way to gain access to deviant and criminal groups.

Ethical Factors – Ethical factors also influence the choice of research methods. In order for research to gain funding it will need to meet the ethical guidelines of the British Sociological Association. How ethical a research method is depends on the researcher’s efforts to ensure that informed consent is achieved and that data is kept confidential and not used for purposes other than the research.

Real ethical dilemmas can occur with covert participant observation. However, sometimes the ethical benefits gained from a study may outweigh the ethical problems. McIntyre, for example, may have deceived the hooligans he researched but at least he exposed their behaviour.

Howard Becker also argued that there is an ethical imperative to doing qualitative research – these should be used to research the underdog, giving a voice to the marginalised whose opinions are often not heard in society.

Nature of topic – There are certain topics which lend themselves naturally to certain modes of research. Measuring how people intend to vote naturally lends itself to phone surveys for example while researching sensitive and emotive topics would be better approached through UI.

Conclusion – In conclusion there are a number of different factors that interrelate to determine a sociologist’s choice of research method – practical, ethical, theoretical and the nature of the topic under investigation. In addition, sociologists will evaluate these factors depending on their own individual values. Furthermore it is too simplistic to suggest that sociologists simply fall into two separate camps, Positivists or Interpretivists.  Many researchers use triangulation, combining different types of method so that the advantages of one will compensate for the disadvantages of another.

Theory and Methods A Level Sociology Revision Bundle 

If you like this sort of thing, then you might like my Theory and Methods Revision Bundle – specifically designed to get students through the theory and methods sections of  A level sociology papers 1 and 3.

Contents include:

  • 74 pages of revision notes
  • 15 mind maps on various topics within theory and methods
  • Five theory and methods essays
  • ‘How to write methods in context essays’.

Participant Observation in Social Research

Participant Observation is where the researcher joins in with the group being studied and observes their behaviour. This post covers the theoretical, practical and ethical strengths and limitations of using overt and covert participant observation in social research.

participant-observationParticipant observation is closely related to the ethnographic method (or ‘ethnography’), which consists of an in-depth study of the way of life of a group of people.

Ethnography is traditionally associated with anthropology, wherein the anthropologist visits a (usually) foreign land, gains access to a group (for example a tribe or village), and spends several years living with them with the aim of uncovering their culture. The ethnographic method involves watching what participants do, listening to them, engaging in probing conversations, and joining them in day to day tasks as necessary; it also involves investigating any cultural artefacts such as art work and any written work if it exists, as well as analysing what religious rituals and popular stories can tell us about the culture. Ethnographic research has traditionally involved taking copious field notes, and the resulting ‘monographs’ which are produced can take several months, if not a year or more to write up.

To cut a long winded definition short, ethnography is basically the same as participant observation, but includes the writing up of a detailed account of one’s findings:

Ethnography = participant observation + a detailed written account of one’s findings.

Participant Observation and the use of other methods

Most participant observers (or ‘ethnographers’) will combine their observations with other methods – most obviously unstructured interviews, and some will combine them with more formal questionnaire based research, normally towards the end of their study period, meaning many of these studies are actually mixed-methods studies. Nonetheless, Participant Observation is still technically classified, for the purposes of A-level sociology as a ‘qualitative’ method.

Overt and Covert Observation

An important distinction in Participation/ Ethnography is between covert and over observation.

  • Overt Observation – this is where the group being studied know they are being observed.
  • Covert Observation – this where the group being studied does not know they are being observed, or where the research goes ‘undercover’.

These both have their strengths and limitations – overt research is obviously more ethical because of the lack of deception, and it allows the researcher to ask probing questions and use other research methods. Covert research may be the only way to gain access to deviant groups, it may enable you to gain fuller ‘immersion’ into the host culture and avoids the ‘Hawthorne Effect’. However, ethically it involves deception and can be very stressful for the researcher.

The Strengths of Participant Observation

Theoretical Advantages

The most significant strength of both types of participant observation is the high degree of validity the method achieves. There are at least five reasons for this:

participant observation anthropologyYou can observe what people do, not what they say they do – In contrast to most other methods, participant observation allows the researcher to see what people do rather than what people say they do.

 

Participant Observation takes place in natural settings – this should mean respondents act more naturally than in a laboratory, or during a more formal interview. This should mean the Hawthorne effect will be less, especially with covert research. You also get more of a feel for respondents’ actions in context, which might otherwise seem out of place if in an artificial research environment.

Digging deep and gaining insight – the length of time ethnographers spend with a community means that close bonds that can be established, thus enabling the researcher to dig deeper than with other methods and find out things which may be hidden to all other means of enquiry.

Verstehen/empathetic understanding– participant observation allows the researcher to fully join the group and to see things through the eyes (and actions) of the people in group. Joining in allows the researcher to gain empathy through personal experiences. This closeness to people’s reality means that participant observation can give uniquely personal, authentic data.

Flexibility and generating new ideas – when completing questionnaires researchers begin with pre-set questions. Even before starting to collect the data, therefore, the researchers have decided what’s important. The problem with this is what if the questions the researcher thinks are important are not the same as the ones the subject thinks are important. By contrast, participant observation is much more flexible. It allows the researcher to enter the situation with an open mind and as new situations are encountered they can be followed up.

Practical Advantages

There are few practical advantages with this method, but participant observation might be the only methods for gaining access to certain groups. For example, a researcher using questionnaires to research street gangs is likely to be seen as an authority figure and unlikely to be accepted.

Ethical Advantages

Interpretivists prefer this method because it is respondent led – it allows respondents to speak for themselves and thus avoids a master-client relationship which you get with more quantitative methods.

The Limitations of Participant Observation

Theoretical Disadvantages

One theoretical disadvantage is the low degree of reliability. It would be almost impossible for another researcher to repeat given that a participant observation study relies on the personal skills and characteristics of the lone researcher.

Another theoretical disadvantage is the low degree of representativeness. Sociologists who use quantitative research methods study large, carefully selected, representative samples that provide a sound basis for making generalisations, In contrast, the groups used in participant observation studies are usually unrepresentative, because they are accessed through snowball sampling and thus haphazardly selected.

Critics also question how valid participant observation really is. They argue the method lacks objectivity. It can be very difficult for the researcher to avoid subjectivity and forming biased views of the group being studied. Also researchers decide what is significant and worth recording and what’s not, therefore, it depends on the values of the researcher. In extreme cases, researchers might ‘go native’, where they become sympathatic with the respondents and omit any negative analysis of their way of life.

A further threat to validity is the Hawthorne Effect, where people act differently because they know they are being observed, although participant observers would counter this by saying that people can’t keep up an act over long time periods: they will eventually relax and be themselves.

Also, the methods lack a concept of social structures such as class, gender or ethnicity. By focussing on the participants own interpretation of events, the researcher tends to ignore the wider social structures, which means giving only a partial explanation.

Practical Disadvantages

Firstly, this method tends to be time consuming and expensive in relation to the relatively small amount of respondents. It can take time to gain trust and build rapport, and so for this reason, it may take several days, weeks or even months, before the respondents really start to relax in the presence of the researcher.

Participant Observation also requires observational and interpersonal skills that not everyone possesses – you have to be able to get on with people and understand when to take a back seat and when to probe for information.

Gaining access can also be a problem – many people will not want to be researched this way, and where covert research is concerned, researchers are limited by their own characteristics. Not everyone can pass as a Hells Angel if covert observation is being used!

Ethical Disadvantages

Ethical problems are mainly limited to Covert Participant Observation, in which respondents are deceived and thus cannot give informed consent to participate in the research.

Legality can also be an issue in covert research where researchers working with deviant groups may have to do illegal acts to maintain their cover.

Some advantages of Overt compared to Covert Observation

Students often think that Covert Observation is superior to Over Observation, however there are five reasons why Overt might be a better choice of research method:

1. You can ask awkward, probing questions

2. You can combine it with other methods

3. You can take on the role of the ‘professional stranger’ – respondents might tell you things because they know you are not ‘one of them’

4. It is less stressful and risky for the researcher

5. It is easier to do follow up studies.

Related Posts

Learning to Labour by Paul Willis – A Summary

https://revisesociology.com/2016/01/25/learning-to-labour-by-paul-willis-summary-and-evaluation-of-research-methods/

 

Sources:

Bryman (2016) Social Research Methods

Chapman et al (2016) Sociology AQA A-level Year 1 and AS Student Book