Tag: research methods

Secondary Data on Academic Progress

What are the strengths and limitations of using secondary data to research the academic progress of students in schools?

This challenging question came up in the methods in Context section of the November 2021 AQA A-Level Sociology exam, and students found it difficult according the Examiners Report, with significant numbers focussing only on quantitative secondary data, rather than both quantitative and qualitative, and many answers making generalisation and failing to pick up on the specifics of different types of data, let alone APPLY these to the topic at hand which was student progress.

So this applied research methods topic is probably worth going over in some depth! (Remember, even though this came up relatively recently it can still come up this year, especially since the examiners know it’s a challenging topic for many students!).

The Question and Item

Applying material from Item C and your knowledge of research methods, evaluate the strengths and limitations of using secondary data to investigate the
academic progress of pupils in schools.

Notes towards an answer

The item suggest that you should focus on both quantitive and qualitative forms of secondary data.

And with methods in context questions you need to at least try and apply the strengths and limitations of the data to the actual topic in the question: academic progress!

Secondary quantitative data to research academic progress

This topic is partly dealt with in this post: Official Statistics on Education: Strengths and Limitations

Official Statistics include exam results and SATs. They have excellent representativeness and usually these are easy to compare, but with education statistics, there are several different versions to measure progress and this can get confusing, also GSCE results changed from A-C to numerical form which makes comparing more difficult over time.

However, official stats do not tell us WHY students achieve at different rates, also for Gypsy and Roma children, many don’t sit formal exams so there is missing data here.

Schools may also record their own quantitative data in the form of internal tests (not official statistics) which provide more insight than official statistics but there are access issues.

Secondary qualitative data to research academic progress

Secondary qualitative data will give you more insight into WHY students achieve at different rates, and such data includes OFSTED reports, school progress reports, the written work of students and even personal documents such as diaries.

Written work in particular can give you an insight into the quality of feedback students get and also how much effort they are making, while personal documents can tell you what is going on in students’ lives outside of education.

The main problem with both of these sources is access.

This topic is covered in depth in this post: Assessing the usefulness of secondary data for researching education. NB this post is broader than this topic, and some of the sources mentioned in it may not be useful for measuring academic progress.

Sources

The AQA’s mark scheme for the November 2021 Sociology A-level Education with Theory and Methods exam paper.

For more information on exams see my exams and essay writing page.

Bias in Presenting Quantitative Data

Newspapers can ‘bias’ the presentation of quantitative data by stretching out the scale of the data they present, making differences between bars seem larger than they actually are (or vice versa!).

Quantitative research methods are usually regarded as being more objective than qualitative research methods as there is less room for the subjective biases and interpretations of researchers to influence the data collection process in quantitative research.

However, bias can still creep into quantitative research and one way this can happen is over the decision in how to present the data in even a basic visualisation.

Specifically, one can take the same data and stretch out the scale of a graph displaying that data and give the impression that the differences between the subjects under investigation are wider than in the original presentation.

Bias in scaling graphs

A recent example of what I’m going to call ‘bias in scaling graphs‘ can be found in how an article by The Guardian displays recent data on how much GDP (Gross Domestic Product) has grown in different European Countries between 2019 to 2022.

the same data from the Office for National Statistics in a more ‘stretched out’ scale which

The Guardian article (September 2022) in question is this one: UK is only G7 country with smaller economy than before Covid-19 which displays the following graphical data to show how the UK’s GDP is falling compared to other G8 Nations.

Source: The Guardian, 2022

Now you might think ‘this is quantitative data so it’s objective’ and on that basis no one can argue with what it’s telling us – the U.S. economy is doing VERY WELL compared to most Euro nations, growing more than TWICE as fast is the impression we get.

And after all, this is fair enough – a 2.6% growth rate is more than twice as fast as a 1% or less growth rate!

Same data different scale…

However you might think differently about the above when you see the same data (almost) displayed by the UK Government in this publication: GDP International Comparisons: Key Economic Indicators which features the graph below:

Source: Commons Library 2022

Note that the data is ALMOST the same – except for Britain’s data being different at 0.6% positive rather than negative – the Guardian article was written after the UK Gov report on the basis of the UK Economic growth forecast being downgraded, but everything else is the same.

My point here is that the data above is (almost) the same and yet the graph has been ‘squashed’ compared to the graph showing the same data in The Guardian article – note the scaling is the same – if you look above you can see that the US Bar is twice as high as the EU bars, but the difference APPEARS smaller because it’s not as stretched.

The Guardian achieves its stretched out scale by displaying the bars horizontally rather than vertically – that way there is more room to stretch them out and make the differences appear larger in a visual sense.

And with the UK now in an economic downturn it makes Britain seem further behind compared to other countries than what would have been the case with the more squished presentation in the Government’s version.

But aren’t they both biased…?

In a word yes – someone has to decided the format in which to present the data which is going to skew what people see.

But the reason I’m calling out The Guardian on this is for two reasons:

  1. it’s unusual to display bars horizontally, the standard is vertically, but there’s not way you can stretch out the visualisation vertically without it looking very odd.
  2. The differences are quite small – we are talking 1-2% points of change so having a more squished scale to represent the small differences seems appropriate, The Guardian has chosen to exaggerate these from the original display possible to make them seem larger than they actually are.

Signposting and Related Posts

This material should be of interest to anyone studying Research Methods.

It’s also a useful example of Left Wing bias in the media, most sociologists focus on right wing bias!

Please click here to return to the homepage – ReviseSociology.com

The Mass Shooter Database…

Mass shootings per year in America are increasing, and some recent research from the Violence Project aims to help us understand why this is.

For students of A-level Sociology this is a useful case study relevant to both research methods and crime and deviance.

The project has interviewed hundreds of people convicted of mass shootings and their family members to better understand their life histories (nice link to secondary qualitative data here!) and then fed this information into a database in oder to quantify it and to see what the main characteristics of mass shooters are.

Interestingly the data shows that there is a broad difference between people who do mass shootings in restaurants, bars and retail establishments compared to people who shoot up workplaces, religious institutions or schools and colleges. In the former, the victims tended to be strangers to the shooters, in the later type the shooters were much more likely to have known their victims.

The main characteristics of mass shooters in America….

  • Out of 172 cases only four were women, two of these acted with a man.
  • 50% are white, 50% from other ethnic backgrounds
  • 65% of shooters had a criminal record, 63% had a history of violence
  • The most common ‘motivation’ was a history of psychosis (30% of shooters) where the shooter was loosing their grip on reality.
  • Half the shooters acquired their guns legally.

You can explore the database for yourself at the link below.

These seem to be a very ‘postmodern’ set of findings…

The researchers note that the data reveals that there is ‘no one type of shooter’ – mass shooters in America come from a diverse array of backgrounds and have diverse motives for what they are doing.

Although personally i can see one clear trend from the data which is the huge bias towards to males – as is the case with many other crimes!

And another is the recent shift to grocery store shootings – the first of these wasn’t until 2018, and since then there have been ‘copycat’ cases following it – Shooters tend to take lessons from other shooters who have done the same before!

Controlling Gun Crime…

The project suggests two main solutions to bring down the number of mass shootings…..

  1. Monitoring people with high risk characteristics and restricting gun sales to these people (nice link to Actuarialism here within crime and deviance).
  2. Stopping giving attention to mass shooters – which should help stop the copycat spreading of such hideous acts!

Sources

Teaching Resources for A-level Sociology: Research Methods

teaching resources for A-level sociology AQA focus 2020

I’ve just released the latest research methods teaching resources for sale as part of my sociology teaching resources subscription package, available for only £9.99 a month!

This image has an empty alt attribute; its file name is sociology-teaching-resources-724x1024.png

Research Methods teaching resources

The November download includes the following lesson materials:

  1. Unstructured interviews            
  2. Participant Observation lesson
  3. Participant observation lesson 2         
  4. Non-Participant observation     
  5. Official Statistics
  6. Cross National Comparisons
  7. Secondary Qualitative data      
  8. Content analysis
  9. Bringing it all together: stages of the research process

NB the October release contained all of the preceding methods (intro, surveys and experiments) and the next release in December will include Methods in Context material – the monthly subscription will give you access to all the methods material, and all material to date (education and families and households), so that’s almost the entire first year of A-level sociology teaching!)

Resources in the bundle include:

  • 5 workbooks covering  the methods above
  • 5 Power Points covering most of the above lessons
  • 9 lesson plans covering all of the above lessons.
  • Various supplementary hand-outs for some of the above lessons as necessary.

Fully modifiable resources

Every teacher likes to make resources their own by adding some things in and cutting other things out – and you can do this with both the work pack and the PowerPoints because I’m selling them in Word and PPT, rather than as PDFs, so you can modify them!

NB – I have had to remove most the pictures I use personally, for copyright reasons, but I’m sure you can find your own to fit in. It’s obvious where I’ve taken them out!

More resources to come…

I’m making resources available every month as part of this teacher resource subscription package. The schedule of release of resources is as below:

  • March – June 2020 – Education Resources
  • July – September 2020 – Research Methods, including methods applied to education 
  • October – December 2020 – Families and Households
  • January – April 2021 – Global Development 
  • May – August 2021 – Crime and Deviance 
  • September – October 2021 – Theory and Methods 
  • November 2021 – January 2022 – Revision Material
  • February 2022 – Intro material. 

Issues surrounding researching in schools

There are tens of thousands of schools in the United Kingdom, which means that observational research which focuses on just one, or a handful of schools will be unrepresentative. This is also a  problem with any of the popular documentary programmes which focus on just one school – they are very interesting as they focus on the stories of the school, and some (but only some) of the pupils and teachers, but they are never going to be representative of all schools!

There are a lot of official statistics available on schools, much of it freely available on the DFES website – information on results, attendance, exclusions are all available, as are the latest OFSTED reports, so using a mixture of secondary qualitative and quantitative data may be a good choice for researchers given that schools are ‘data rich’ institutions.

A researcher could also use official statistics to easily select a sample of schools which represent all the regions in the UK, different OFSTED grades, and/ or different school types.

However, official statistics on education can be misleading – exam results may not reflect the underlying ethos of a school, or show us the difficulties a particular school faces, and schools can manipulate their data to an extent – for example, they can reduce their exclusion statistics by ‘off-rolling pupils’ – getting parents to agree to withdraw them before they exclude them.

Schools are potentially very convenient places to conduct research – because the law requires pupils to attend and teachers/ managers need to attend to keep their jobs, you can be reasonably certain that most people you want to research are going to be in attendance! You have a captive audience!

However, school gatekeepers (i.e. head teachers) may be reluctant to allow researchers into schools: they may see research as disruptive, fearing it may interfere with their duty to educate students.

Schools are also highly organised, ‘busy’ institutions – researchers may find it difficult to find the time to ask questions of pupils and teachers during the day, meaning interviews could be a problem, limiting the researcher to less representative observational research.

The researcher will also need to ensure they blend-in, otherwise they may be seen as an outsider by teachers and students alike, which would not be conducive to getting respondents to open up and provide valid information.  

Researching Parents

Home factors have more of an influence on pupil performance than school factors, and parents are certainly the biggest influencers of pupils at home, especially in their early years.

Parents can influence a child’s attitude towards education in various ways:

  • The amount of time they spend reading with their children in early years
  • How they play with their children more generally, and how educational that play is.
  • How strict they enforce rules.
  • The importance they attribute to education themselves
  • The amount of interest they show in their child’s education

As a result of early socialisation, children end up being either culturally deprived or having cultural capital (or somewhere in between), which means they are either ill-prepared for school or very well prepared, which will make an enormous difference in how well they adapt to school life when they first start.

If you wish to research pupils, you may well need the consent of parents, so some minimal contact may well be necessary even if it’s not them you are actually researching.

Problems of researching parents

Validity issues

Middle class, pro-school parents are more likely to want to engage with research about education, as they will be more interested and will probably be able to use it as an opportunity for self-validation – they can show off how much they care about their children’s education. They will also be more familiar with filling in questionnaires, and engaging in social research, which are quite ‘middle class’ pursuits.

Working class parents, who themselves maybe didn’t have such a positive experience of schooling, might be more reluctant to take part in research, feeling less comfortable engaging with the middle class researchers.

Parents may also try to ‘impression manage’ with researchers, exaggerating their involvement in their children’s education for example, because this paints them in a more positive light.

Practical Problems

Gaining access to parents could be difficult – if you don’t want to hang around the school gates or at parents evenings then you would have to approach them either at home or via phone/ email.

Gaining access to parent’s private addresses is going to be difficult because schools will not share that data with you because of GPDR, thus if you wanted a representative sample by postcode then you wouldn’t be able to get it.

Schools might agree to send out questionnaires or letters asking for interviewees to parents on your behalf, but then you’ve got the problem of getting a self-selecting biased sample back. The chances are only those parents who are pro-education would want to take part in your research.

It could be very difficult to gain access to the parents of traditionally underachieving groups – white working class parents or traveler parents for example.

When it comes to researching, it would be more difficult to get parents into a group to research them (also, this might be pointless anyway), so you’d probably have to do one on one research which could be more time consuming.

Researching Teachers in Education

Teachers are the ‘front line’ of education, with the primary day to day responsibility students’ education and well-being.

If you want to understand the impacts that education policies are having on different types of student, then teachers are probably best placed to be able to tell you.

However, there are a number of potential problems when researching teachers:

Teachers have hectic working lives

Teachers work very long hours and often suffer with high stress levels, and they may not be willing or able to spend more time to engage with researchers.

For this reason questionnaires may be a better choice of method than interviews and observations may also be a good choice as these don’t really take up any time, but they could add to teacher stress, so it might be difficult to get teachers to agree to being observed.

Teacher professionalism

The validity of information you get from teachers may be compromised because of their professional status.

Teachers are bound by the GDPR and have a duty of care towards their students and so probably will not share data about their students with researchers from outside of the school.

Teachers could also be concerned about ‘impression management’ – they may want to present themselves in the best light possible and some may feel duty bound to present their school in a good light, because to do so is good for marketing and student recruitment, which could limit the critical views you get from teachers as a researcher.

On the other hand, there are also ‘jaded’ teachers that are fed up with their jobs, and are just time-serving their way to retirement – if you got a group of these together in a group interview, you might just get unrepresentative biased moaning about how bad life is as a teacher.

Line managers

If you want to gain access to teachers in a school you will have to approach the senior management team, and these may limit your access to the teachers you can research, possibly directing you towards the better and more compliant teachers to pain their school and the management in a positive light.

Even if you had unlimited access to teachers, they may not wish to be critical of the school for fear of this getting back to their superiors. In some schools there may well be very few critical teachers, and if research findings showed negative views of the school, in such cases it would probably be obvious which teachers were responsible for such negative comments, even if data was anonymous.

Home may be the best place to research teachers?

You don’t have to research teachers in their school setting don’t forget, you may get more valid information if you interview them in their homes, away from the school setting, away from the ‘front stage’ where they are performing their teacher role.

Starters for An A-level Sociology Non-Participant Observation Lesson

Non-Participant Observation involves the researcher observing respondents, but keeping their distance, and not engaging with those respondents.

As with many of the ‘minor’ research methods in A-level sociology, this one can be a bit of a struggle to make interesting, but here are three starter activities to get your students in the mood for making observations…

Starter 1: How many passes does the team in white make?

I won’t give too much away, but it does show one of the limitations of doing narrowly focused structured observations where you are only looking for one thing!

Starter 2: Whodunnit?

Again, I don’t want to give too much away, but this demonstrates how difficult observation can be, in terms of the amount of things you might miss if you’re not paying close attention!

Starter 3: Street Life:

This is a bit of an old video, but it introduces students to some of the strengths and limitations problems of qualitative, unstructured observations.

You might like to think about showing this in contrast to a video of street life in a very underdeveloped country, and comparing the differences.

How to use these starters

I use these three starters one after the other before I get students to go out and perform their own structured and unstructured observations of street-life in the local high street.

The first two are really just a bit of fun, but they do drive home the fact that you might miss a lot if you are just focusing on a few factors when doing structured observations.

If you like this sort of thing and want to see how these starters blend into the rest of my A-level sociology lesson on non-participant observation you might like to subscribe to my A-level sociology teacher resources.

Non Participant Observation material is scheduled for release in October 2020.

A-Level Sociology Teaching Resources

NB – you get All of these starters and more as part of my A-level sociology teaching resources, available as a monthly subscription, for only £9.99 a month! The subscription includes lesson plans and modifiable student hand-outs and PPTs. Activities such as these starters are embedded into the student learning materials.

I hope you find these resources useful, and happy teaching,

Karl, September 2020.

Please click here to return to the homepage – ReviseSociology.com

Experiments within schools

What are the strengths and limitations of using laboratory and field experiments to research education?

As a student you probably would have been the subject of an experiment within your school at some point during your 13 years of formal education.

Experiments conducted by schools themselves are a lot more common than educational researchers conducting their own field experiments within schools, and so such experiments are a rich source of data for students studying Methods in Context for A-Level sociology.

It is quite usual for schools to conduct small scale field experiments to try out new teaching techniques or to evaluate the effectiveness of banding of streaming.

If there are several classes of students of similar abilities doing the same subject, it is relatively easy to keep some classes being taught in the same way as usual but to change one aspect of teaching of other classes, and to measure the effect this has on student behaviour or learning.

Some years ago we conducted the following experiment in my institution, designed to measure the effectiveness of splitting students into ability bands:

In five subjects we deliberately created one class of all higher ability students (those predicted to get Bs or above at A-level based on their GCSE grades), but kept some high ability students in all of the regular mixed ability classes.

So in A-level sociology we had 7 classes at AS, and we ended up with one ‘high ability class’ and then 6 mixed ability classes, each with 2-3 A/B students rather than 3/5 A/B students.

The hypothesis of the Senior Management was that grouping all the higher ability students into one group would lead to improved results.

The control students are the high ability students still in the regular mixed classes.

We let this run for a year and compared the AS exam results at the end (it was a long time ago, when we still had AS exams.

The results in the end were inconclusive when we looked at the results of the ‘top band’ classes across all subjects – there was no signficant evidence that putting them in once class led to them getting better results.

As far as I am aware the students involved to this day have no idea they were subject to this experiment!

This kind of experiment within education is probably more common than you think!

A lot of schools put students into ability bands, and it makes sense that they review their results from time to time and ‘experiment’ to see if mixing up the bands give better results – for example, if you’ve got 6 maths groups in one school year, you could either have 6 discrete bands, or 2 more general bands, so you end up with a wider range of abilities split across 3 classes.

It kind of makes sense for schools to play around with how the split groups up to see if they can improve behaviour our outcome.

If you think about all of the things that schools can do differently, there are a lot of potential variables schools can change in one class, say, but not in others, just to see if changing that one variable makes a difference after a year in one class, rather than risking rolling out a change across the whole year group.

Variables you might change (all possible in-school experiments):

  • The gender mix of classes.
  • Seating lay-outs within classes
  • The length of lessons/ number of lessons in a week
  • The timing of ‘support lessons’ – before or after school, weekends, holidays.
  • How support staff are used in classrooms
  • Wider school policies on uniform, discipline and punishment
  • The teaching techniques used in lessons.
  • The ratio of face to face and online learning.

The practical, theoretical and ethical strengths of schools conducting experiments

Don’t kid yourselves, this kind of micro-experiment goes on all the time, but there some good reasons:

  • Ethically teachers and schools have to provide students with the best education they can – educational theory about what the most effective teaching techniques changes, technology changes, so teachers and schools have to adapt. Doing an experiment for a year with one class can be a useful way of finding out how to implement changes more effectively across the whole school the following year. Or if some experiments don’t work out, at least it’s not all students who suffer.
  • Practically, the students are there, the school is there, it’s relatively easy for teachers and schools to do experiments, rather than having them done externally.
  • Theoretically – validity should be very high because one typically doesn’t inform students they are part of an experiment. Reliability should also be good because the conditions are relatively stable over time in most schools.

The practical, theoretical and ethical weaknesses of schools conducting experiments

  • There is the ethical problem of deception and some students getting treated differently for the period of the experiment, which goes against equal opportunities.
  • Teaching one class differently to the rest can be stressful and demanding for the teacher.
  • In very small classes and schools with few classes, it’s hard to get a large enough sample for good representativeness.

Related Posts

For a great example of a really extreme experiment in a school, see ‘Chinese School‘.

For some of the general strengths and limitations of field experiments please see this post.

Autobiographies in social research

An autobiography is an account of the life of an individual, written by that individual, sometimes with the assistance of a professional biographer.

One of the most popular UK autobiographies of 2020 was Harry and Meghan’s ‘Finding Freedom’, and it is supposed to ‘dispel rumors about their relationship from both sides of the pond’.

The Amazon critics, however, disagree. The comments ranked at 2 and 3 (accessed 18 August 2020)  in order of usefulness both give the book 1 star out of five and comment thus:

Dela – 1.0 out of 5 stars Pure fantasy

“… the reader can only assume a good proportion of [this book is] made up… the reader is left with a very poor impression of the couple. As someone else said – this is very much an ‘own goal’.”

600 people found this helpful

hellsbells123 – 1.0 out of 5 stars Dross of the highest order – all time low for Harry

“Dreadful book full of ridiculous unnecessary detail from a couple who profess to want privacy. This is a book masquerading as a love story but full of bile, hatred and bitterness. “

578 people found this helpful

Source

The strengths and limitations of autobiographies as a source of data

Whether they have a readership of millions or tens, autobiographies are selective in the information they provide about the life of the author.

They thus tell you what the author wants you to know about themselves and their life history.  

However, you have no way of knowing whether the events outlined in an autobiography are actually and I wouldn’t even trust an autobiography to give me an accurate view of the authors’ own interpretation of what the most significant events in their life history were.

The author may exaggerate certain events, either because they mis-remember them, or because they want their book to sell, thus they are selecting what they think their audience will want to read.

In some cases, events may even be fabricated altogether.

As a rule, I’d say that the more famous someone is, then the less valid its contents are.  An exception to this would be less famous ‘positive thinking’ lifestyle gurus, whose income maybe depends more on their book sales than really famous people, who could possibly afford to be honest in the biographies!

Either way, there are so many reasons why an autobiography might lack validity, I wouldn’t trust the content of any of them – think about it, how honest would you be in your autobiography, if you knew anyone could read it?

Using autobiography sales data may be more useful…

IMO the value of autobiographies lies in telling us what people want to hear, not necessarily in getting to the truth of people’s personal lives.

If want to know what people want to hear, look a the sales volumes – there are really no surprises…..

Top selling autobiographies of all time (source)

Relevance to A-level Sociology?

Twitter data is a source of secondary qualitative data (public rather than private data) and so is relevant to the research methods part of the course.

%d bloggers like this: