The impact of watching Earthlings is smaller than you think

by Bobbie

Every vegetarian is regularly faced with the question, “what made you go vegetarian?” Overwhelmingly, the response I hear follows more or less the same logic: “I watched/read X and just couldn’t imagine eating meat ever again” (where X is a documentary or book such as Earthlings, Food Inc., Speciesism, The Ethics of What We Eat, …).¹

This “critical juncture” — or “conversion experience” — explanation feels true to many vegetarians. After all, their meat consumption is dramatically lower today than it was before watching Earthlings, and they also eat much less meat than the large proportion of the general population that has never seen the documentary.

However, these simple “before/after” and “between group” comparisons wildly over-estimate the true causal effect of watching Earthlings, which is defined as the difference between your meat consumption today and what it would have been if you had not watched Earthlings (while everything else remained unchanged). The fact that we cannot directly observe this counterfactual is the fundamental problem of causal inference. Randomized controlled trials (RCTs) solve this problem by randomly allocating subjects to watch Earthlings or some “placebo” documentary, which ensures that in large enough samples the treatment and control groups are identical on all characteristics except for the fact that one group watched Earthlings and one did not.

In this post, I argue that the true effect of watching factory farming documentaries is much smaller than many animal advocates think. Nevertheless, these documentaries may still be an effective advocacy tool, especially if they help vegetarians have larger spillover effects on friends and colleagues. Ultimately, what is needed is much better evidence on the effects of these documentaries across different sub-populations and evidence on whether such spillover effects exist.

 

Would you have still become vegetarian if you had not watched Earthlings? Probably.

In the real world where watching Earthlings is not randomly assigned, the kind of people who choose to watch Earthlings are likely to be very different than the people who do not. In particular, people who watch Earthlings are likely to already be much more sympathetic towards animal welfare and may already be experiencing a downward trend in meat consumption. As a result, simple comparisons between the people who have watched Earthlings and those who have not will result in biased estimates of the true causal effect. Similarly, simple before/after comparisons of a person’s meat consumption before and after they watched Earthlings will be biased if they were already on a downward trend in meat consumption, or if something else happened around the same time that led them to change their meat consumption (e.g. change in their social network, change in accessibility of vegan products).

These biased estimates of the true causal effect are illustrated in the figure below, which divides the population into two groups: “sympathizers” who are already sympathetic to animal suffering on factory farms, and the remainder of the general population who are not.

The purple and green lines show the trends in meat consumption over time for sympathizers and the general population if they did not watch Earthlings. Conversely, the turquoise line represents what would have happened to the meat consumption of sympathizers if they had watched Earthlings, and the red line represents what would have happened to the meat consumption of the general population if they had watched Earthlings.

This figure is meant to communicate three testable hypotheses. [Note: this figure is based entirely off of my own priors, rather than real data.]

1. Short-term effects fade over time.

First, if we just look at the short-term effects of watching Earthlings, there seems to be a substantial effect on meat consumption among both sympathizers and the general population. For sympathizers, this can be seen in the sudden dip in the turquoise line after watching Earthlings, such that the causal effect is the gap that this creates between the purple and turquoise lines. For the general population, this can be seen in the sudden dip in the red line relative to the green line.

However, for both of these populations, the effects fade over time. In the general population, the meat consumption of people who watched Earthlings increases back to its original level as the salience of the documentary fades over time. Among sympathizers, the short-term effect of watching Earthlings fades as the long-term trend in meat consumption catches up. In other words, watching Earthlings merely accelerates the transition to vegetarianism among sympathizers (without any lasting long-term effects), while in the general population there are no long term effects.

2. Before/after and between group comparisons over-estimate the true effect.

Second, both the before/after comparison and between group comparison greatly over-estimate the true causal effect of watching Earthlings. In the general population, the true long-term effect of watching Earthlings is zero, since there is no gap between the red and green lines. Among sympathizers, the effect of watching Earthlings also converges to zero over time as the turquoise line converges with the long-term trend in meat consumption (purple line). However, both the before/after and between group comparisons would result in much larger (and therefore biased) estimates of treatment effects, as shown in the figure.

3. The short-term effect is larger among sympathizers than among the general population.

Third, the short-term dip in meat consumption is larger and lasts longer among sympathizers than in the general population. This hypothesis isn’t very consequential if the long-term effect of watching Earthlings is zero in each group anyways, yet it is nevertheless a testable hypothesis and has implications for the generalizability of studies conducted on sympathizers. Specifically, any RCT conducted on a sample of sympathizers that finds encouraging short-term effects on meat consumption must be taken with a grain of salt, since it seems likely that even these short-term effects would be smaller (and fade faster) if the study were replicated in the general population.

 

Assumptions

These three hypotheses rest on at least three important assumptions.

Assumption 1: I’ve assumed that vegetarians have no spillover effects on non-vegetarians. However, in reality, a sympathizer who watches Earthlings could have spillover effects on the meat consumption of friends/colleagues through at least two different mechanisms.

First, by accelerating the transition to vegetarianism among sympathizers, watching Earthlings creates more time for vegetarians to have spillover effects. Assuming that non-vegetarians are more likely to reduce their meat consumption the longer they are exposed to vegetarians (rather than having backlash effects), then watching Earthlings will have a greater overall effect on meat consumption than I’ve argued in the figure above.²

Second, watching factory farming documentaries could have spillover effects by strengthening the conviction and ability of sympathizers to expand the animal advocacy movement. For instance, the documentary Speciesism walks viewers through the problems with different justifications for eating certain animals but not others (e.g. pigs vs. dogs). Even if this has no long-term causal effect on the viewer’s own trend in meat consumption, it may empower them to spread the animal welfare message more effectively.

Assumption 2: I’ve ignored the possibility that there exists some small fraction of the population that really was dramatically affected by watching Earthlings, and that their perceptions of this causal effect are not biased by the flawed before/after or between group comparisons I’ve described above.

I’m willing to accept that there are some people in the general population who were not very sympathetic to animal suffering before watching Earthlings (i.e. they had a flat pre-documentary trend in meat consumption), but were somehow randomly exposed to Earthlings and then dramatically reduced their meat consumption over a sustained period. For these people, the true effect of watching Earthlings is large. However, even if this is true for some people, I don’t think there are many others out there who are either willing to or capable of dramatically changing their behavior (and remaining committed to this change) after watching a single documentary. Hence, if you are one of these people for which watching Earthlings changed your life, please keep in mind that most people aren’t like you.

Assumption 3: I’ve assumed that changing behavior is costly. This seems reasonable in the domain of meat consumption and dietary change, since most people really enjoy eating meat and don’t like learning new recipes or figuring out how to replace animal products in their diet.

However, this assumption of costly behavior change means that the hypotheses I’ve described above may not carry over to other domains. For instance, the documentary Blackfish has been enormously successful at galvanizing public protest against the treatment of orcas at SeaWorld and other locations. It seems unlikely that this eruption of public anger would have happened if it were not for Blackfish, implying that the documentary has had a large causal effect. However, changing your opinion on this issue is much less costly than changing your everyday meat consumption. As a result, there is a limited amount that the anti-factory farming movement can learn from the success of Blackfish.

 

Summing up

So should animal advocates continue using videos, documentaries, and books to promote vegetarianism/veganism?

I think the answer is still “yes”, but I have no idea what combination of messaging strategies and sub-group targeting is likely to be most effective. My prior is that vegetarian documentaries may accelerate the meat reduction trend of sympathizers and strengthen their spillover effects, but direct effects on the general population are likely to be small. This suggests that an effective strategy would be to target sympathizers with “empowering” documentaries (like Speciesism) in order to enhance their spillover effects. Or maybe animal advocates should turn their attention towards the general population, since sympathizers are likely to decrease their meat consumption anyways and may turn out to have negligible spillover effects. Until I see strong evidence in support of one alternative over the other, I have no idea what the optimal strategy is.

Two closing recommendations:

(1) Animal advocates need to think carefully about their theory of change (e.g. which sub-groups in the population do you expect to be affected by the documentary and why? Are these people likely going to reduce their meat consumption anyways? Will these people have spillover effects on their friends/colleagues?)

(2) We need rigorous evidence on the long-term effects of vegetarian documentaries across different sub-populations. Capturing spillover effects poses more of a challenge, but it wouldn’t be difficult to conduct rigorous RCTs on the long-term effects of vegetarian documentaries. It’s just a matter of time, resources, and adequate knowledge of research design. Animal advocates are increasingly recognizing this need for evidence, though the movement has been slow to find the necessary expertise and resources to conduct these studies.

If you’re an animal advocacy group and are interested in conducting these kinds of studies, feel free to reach out to us at bmacdon@stanford.edu or gboese@sfu.ca and we are happy to talk about potential ways of collaborating.

 

^{1. My own journey into veganism started around the time I read The Ethics of What We Eat in early 2013, after which I begin avoiding products associated with what I thought were the worst forms of animal cruelty (i.e. caged eggs and beef/chicken/pork sold in major grocery stores). Since then, I’ve progressed slowly towards veganism. After moving back to Canada in September 2013 (from the UK), I began to purchase meat and eggs only from local free range farms that seemed trustworthy and I sought organic dairy products in grocery stores (under the belief that conditions were at least marginally better for animals on organic farms), though I still rarely checked product labels for dairy/egg content (e.g. salad dressing, desserts) and often made mistakes when eating out. Since moving to the US in September 2014 to start my PhD, I’ve stopped buying all meat/eggs (in part because I haven’t taken the time to search for trustworthy free range farms in the area), have cut out all dairy, and have gotten much better at checking product labels. Today, I eat 100% vegan at home, though continue to mess up from time to time when eating out (e.g. not realizing that a restaurant’s vegetarian burger comes with cheese). Moving forward, my next goal is finding ways to effectively encourage friends and colleagues to reduce their impacts on animal suffering without coming off as “holier than thou” or ostracizing them. What led me down this path towards veganism? It’s tempting to point to The Ethics of What We Eat as the causal force that got me started, but this simply raises the question of why I chose to read the book in the first place. Was I already becoming more sympathetic towards veganism and was likely going to head down this path whether or not I read the book? I have no idea.}↩

^{2. It’s not clear whether most vegetarians have positive or negative spillover effects on others. Minson and Monin (ungated) illustrate this in a study of “do-gooder derogation” where they show that non-vegetarians rate vegetarians more negatively when anticipating that vegetarians see themselves as more moral than non-vegetarians.↩}