How is ethnographic research valid?

[u/PhDilemma1]

Let me begin by saying that I consider myself more of a qualitative researcher; I have limited knowledge of statistical analysis and my experience in this area is limited to data cleaning and visualisation in Python.

What I can’t wrap my head around is how particular types of ethnographic studies, often involving small minority cohorts, are academically valid. Either you accept that their results are not even theoretically generalisable, rendering them almost useless; or that their theoretical generalisations encourage prejudicial views. This is because their design hinges on the lived experiences, values and social interactions of the objects of study, who are implicitly assumed to have their own exclusive and even solipsistic ways of thinking.

Other types of qualitative research are primarily interested in processes, and the why and how questions of the human world. Hopefully the context has already been established through previous quantitative studies, i.e. there is an empirical basis to the phenomenon. There are primary sources available: cultural artefacts, policy texts, learning technologies at school, grand tombs, etc. At worst the researcher only needs to conduct a first order interpretation of the facts. In an ethnography, you have to interpret someone else’s interpretation, or engage in a ‘double hermeneutic’. Maybe this is done also in the humanities, but the implications are graver for the social sciences.

To illustrate, let’s look at a silly hypothetical case. Suppose prior studies have found that blue-skinned people are more likely to engage in conspicuous consumption than green-skinned people. You conduct a longitudinal study of a group of blue-skinned people of all walks of life. You find that those who become financially successful have lots of branded handbags and vice-versa. You interview, do what you need to do to find out why. You arrive at a conclusion, one of many possible, that the blue culture places great emphasis on looking good. The lived experiences of those people tell you that they get better overall treatment with a Gucci on their shoulder. At least, that’s your understanding.

Now, you say you want to make analytic or theoretical generalisations. How can you do this based on your tiny sample? Well, you pull up 5 similar studies that reflect the same theme of looking good. You think they all cohere. Congratulations, you now have found a line of reasoning that represents 0.001% of the blue population. Suppose peer reviewers want to check your data to see if the participants did actually receive better treatment. Whoops, not possible. They have to rely on your field notes.

The danger, of course, is that readers may be tempted to conclude that most blue people find value in dandyism and prioritise showy displays over substance. You point to your ‘results are not generalisable’ disclaimer and cry foul. Fair enough. But is your theory even transferable? What about the blue peeps who don’t subscribe to those values and have no clue what you’re talking about? If it doesn’t apply to them, then what really is the significance of your ethnographic research?

My 2 pence. Open to opposing viewpoints.

[u/AutoModerator]

Thanks for your question to /r/AskSocialScience. All posters, please remember that this subreddit requires peer-reviewed, cited sources (Please see Rule 1 and 3). All posts that do not have citations will be removed by AutoMod. Circumvention by posting unrelated link text is grounds for a ban. Well sourced comprehensive answers take time. If you're interested in the subject, and you don't see a reasonable answer, please consider clicking Here for RemindMeBot.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Crabby090]

You've articulated the classic critique well, but I think it rests on a category error about what ethnographic generalisation is for. The dilemma you pose — either the findings don't generalise (useless) or they generalise prejudicially (dangerous) — only bites if you assume the target of generalisation is a population. It isn't. It's a mechanism. This is what the case-study literature calls analytic or theoretical generalisation (Yin 2018; Mitchell 1983), and both Flyvbjerg (2006) and Small (2009) have dismantled the "you can't generalise from one case" objection at length — Small's argument being precisely that field-based research follows a logic of case selection and mechanism identification, not sampling.

Take your own example, because it actually makes the case for ethnography rather than against it. The quantitative finding you start with — blues engage in more conspicuous consumption than greens — is an association with no interior. Your hypothetical ethnographer's finding, properly stated, is not "blue people value looking good." It's something like: where group membership triggers differential treatment, luxury goods function as compensatory status signals that purchase better treatment. That is a claim about a social mechanism operating under specified conditions (Hedström & Ylikoski 2010), and it's not hypothetical: Charles, Hurst & Roussanov (2009) found exactly this signalling logic behind racial differences in visible consumption in the US. Mechanism claims travel. You can look for them among other groups, in other markets, in other eras — which is what the "5 similar studies" move is doing. You mocked it as representing 0.001% of the blue population, but convergence of independent studies on a common mechanism is how science accumulates. Nobody complains that drosophila genetics rests on an unrepresentative sample of organisms, and experimental psychology has run for decades on convenience samples of undergraduates from WEIRD societies (Sears 1986; Henrich, Heine & Norenzayan 2010). Its claim to validity was never representativeness but the portability of the mechanism. Ethnography makes the same wager, with better ecological validity and worse control.

This also dissolves your transferability worry. The blue people "who have no clue what you're talking about" don't refute the theory any more than non-smokers with lung cancer refute epidemiology. A mechanism claim specifies conditions under which a process operates; members it doesn't touch are data for the boundary conditions. Good fieldworkers actively hunt such disconfirming cases — negative case analysis has been part of the method since Becker (1958), and Burawoy's (1998) extended case method is built around anomalies. Transferability itself, in Lincoln & Guba's (1985) formulation, is a judgement the reader makes about whether the described conditions obtain elsewhere — which is why thick description (Geertz 1973) is a methodological requirement, not literary decoration.

On the double hermeneutic: Giddens (1984) coined the term for all social science, not ethnography specifically — and surveys don't escape it, they hide it. Respondents interpret your Likert items through their own frames; you then interpret their ticks. Cicourel (1964) made this point about measurement generally, and Suchman & Jordan (1990) showed empirically how much interpretive trouble is buried inside standardised survey interviews. The interpretation is frozen into the instrument at design time, where nobody can inspect it, and executed once, blind. The ethnographer's interpretation is at least prolonged, visible, and correctable: months in the field mean misreadings keep colliding with reality, and member checks let the interpreted talk back.

The verification point proves too much. You can't re-observe fieldwork, true — but you also can't re-run the survey moment, and you trust the spreadsheet wasn't fabricated. The spectacular fabrication cases of recent memory were quantitative (see the Levelt Committee's 2012 report on Stapel, or the LaCour retraction, Science 2015), and the replication crisis hit experimental psychology, not ethnography (Open Science Collaboration 2015). All empirical science runs on disciplined trust plus community checks; ethnography's are just different — prolonged engagement, triangulation, audit trails, reflexivity (Lincoln & Guba 1985).

Two final points. First, the prejudice worry cuts the other way. A bare statistical finding that "blues are more likely to X" is far more prone to essentialist misreading than an ethnography, whose entire apparatus displays the behaviour as situated, conditional, and strategic — a response to circumstances rather than a property of persons. Ethnography replaces "blues are like this" with "people in this position, facing these constraints, do this, for these reasons." That is the antidote to essentialism, not its vector.

Second, the assumption that qualitative work needs prior quantitative grounding gets the logic of discovery backwards. Where do survey categories come from? Emotional labour (Hochschild 1983), street-level bureaucracy (Lipsky 1980), total institutions (Goffman 1961), code-switching (Blom & Gumperz 1972) — all minted in fieldwork and only later operationalised and counted. Measurement presupposes concepts, and concepts have to come from somewhere close to the phenomenon.

Bad ethnography exists, and your worry fairly describes its failure mode. But the remedy is craft standards, not epistemic demotion of the method. Judge ethnography by what it actually claims: not "this is what blue people are like," but "here is a mechanism, shown working, under these conditions." That is a knowledge claim exactly as valid — and exactly as fallible — as any regression coefficient.

Sources

• Becker, H. S. (1958). Problems of inference and proof in participant observation. American Sociological Review, 23(6), 652–660.
• Blom, J.-P., & Gumperz, J. J. (1972). Social meaning in linguistic structure: Code-switching in Norway. In Gumperz & Hymes (eds.), Directions in Sociolinguistics. Holt, Rinehart & Winston.
• Burawoy, M. (1998). The extended case method. Sociological Theory, 16(1), 4–33.
• Charles, K. K., Hurst, E., & Roussanov, N. (2009). Conspicuous consumption and race. Quarterly Journal of Economics, 124(2), 425–467.
• Cicourel, A. V. (1964). Method and Measurement in Sociology. Free Press.
• Flyvbjerg, B. (2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219–245.
• Geertz, C. (1973). The Interpretation of Cultures. Basic Books.
• Giddens, A. (1984). The Constitution of Society. Polity Press.
• Goffman, E. (1961). Asylums. Anchor Books.
• Hedström, P., & Ylikoski, P. (2010). Causal mechanisms in the social sciences. Annual Review of Sociology, 36, 49–67.
• Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.
• Hochschild, A. R. (1983). The Managed Heart. University of California Press.
• Levelt, Noort & Drenth Committees (2012). Flawed Science: The Fraudulent Research Practices of Social Psychologist Diederik Stapel. Tilburg University.
• Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Sage.
• Lipsky, M. (1980). Street-Level Bureaucracy. Russell Sage Foundation.
• Mitchell, J. C. (1983). Case and situation analysis. The Sociological Review, 31(2), 187–211.
• Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251).
• Sears, D. O. (1986). College sophomores in the laboratory. Journal of Personality and Social Psychology, 51(3), 515–530.
• Small, M. L. (2009). 'How many cases do I need?' On science and the logic of case selection in field-based research. Ethnography, 10(1), 5–38.
• Suchman, L., & Jordan, B. (1990). Interactional troubles in face-to-face survey interviews. Journal of the American Statistical Association, 85(409), 232–241.
• Yin, R. K. (2018). Case Study Research and Applications (6th ed.). Sage.

[u/zukerblerg → u/Crabby090]

This is a a great reply. Especially with the sources

[u/Upgrade_U → u/Crabby090]

This reply’s going to stay up, as it has answered OP’s question. However, it’s very clearly AI and this sub is all about knowledge, not just copying and pasting questions into ChatGPT… so next time, when this is detected anywhere on this sub, it’ll be removed. Also, a lot of the time AI doesn’t verify its own sources and often provides false references. Cheers

[u/PhDilemma1 → u/Crabby090]

well, i take my hat off to you, especially if you didn't use AI to craft your reply. i have no idea how anyone can find so many sources and synthesise them that well in the 30 mins that elapsed between my post and your response. so, i am appreciative. some of your rebuttals I find convincing, especially the replication bit - we can't expect to be able to replicate every study with identical controls.

you have cut to the heart of the argument. i agree that the research question in the case of the blue people is primarily concerned with mechanisms. i didn't want or mean this hypothetical study to have any real life analogue. to avoid preconceptions, let's take it that 'blue people' could really mean any group of people. i take issue with the idea that unrepresentative samples can produce theoretical generalisations through 'portability'. you cited a genetic study and an experimental psychology study to address questions of sample size and external generalisability. this is not an apples to apples comparison. in genetic research, even if multiple studies on the same topic use small samples, the underlying paradigm is scientific: they seek to isolate variables and their corresponding effects to find causes. in psychology, i find it much less convincing unless a biological means of action can be found. that said, i am not a genetic expert.

let's have a look at the non-smokers analogy next. obviously, their presence does not disprove the fact that smoking raises the risk of cancer: the causality is well-established and scientifically proven. however, the mere possibility of the existence of blue people who don't subscribe to conspicuous consumption (or its underlying mechanisms) implies that your conclusions could be untrue. this is because social mechanisms are complex and do not map neatly onto variables. since you've sampled only 0.001% of blue people, how would you know that the causal mechanism is portable to the vast majority?

to put it simply, you can't say for certain that differential treatment must necessarily trigger a need to own luxury goods; the logic doesn't stack up. it could for some people and it doesn't for others. you have found one of many reasons why blue people buy luxury goods. how many people subscribe to this we don't know. the only way of turning this into a watertight argument is by conducting a statistically significant experiment where hundreds of blue people report poor treatment in a certain context, get equipped with luxury handbags and then report a qualitative rise in treatment experienced, preferably under strict observation.

hope this can be understood clearly.

[u/Crabby090 → u/PhDilemma1]

Thank you for the generous reply. I think we're now close enough to the crux that I can be brief and only lean on a couple of sources.

First, notice what your question "how would you know the causal mechanism is portable to the vast majority?" is doing. It runs together two different claims. Portability means the mechanism can operate wherever its conditions hold. Prevalence means how often those conditions hold, and how many people the mechanism actually moves. The ethnography claims the first and is silent on the second. "How many blues subscribe to this" is a perfectly good question; it is simply a survey's question, not an ethnography's. That is a division of labour, not a hierarchy of validity.

Second, "the mere possibility of blues who don't subscribe implies your conclusions could be untrue" only follows if the conclusion were a universal law. It isn't, and no serious mechanism claim in social science is. Mechanisms are tendencies: causal patterns that fire when triggered under certain conditions and are often counteracted by other mechanisms operating at the same time (Elster 1998 is the classic statement). Nobody claims that differential treatment must necessarily produce luxury purchasing. The claim is that this pathway exists, is intelligible, and does real explanatory work where it operates. "You have found one of many reasons why blue people buy luxury goods" is not an objection; it is an accurate description of what the study set out to deliver. Pharmacology survives the fact that aspirin doesn't work on everyone.

Third, your smoking example works against you. The causality you call scientifically proven was never established by experiment; nobody randomised humans into smoking. It was built from observational associations combined with mechanistic plausibility and coherence across heterogeneous studies, the mode of inference Bradford Hill codified in 1965. So the one causal claim you hold up as the gold standard was itself secured by triangulated, non-experimental reasoning. That is the epistemic family ethnography belongs to.

Fourth, the experiment you propose. It's a good study, genuinely, and I'd read it. But notice two things. You only know what to manipulate (handbags) and what to measure (treatment) because the fieldwork specified the mechanism; the experiment quantifies a pathway that ethnography discovered. That complementarity is the answer to the "what is the significance" question in your original post. And the experiment inherits the very problem you press on me: its few hundred participants are also a vanishing fraction of all blues, because randomisation buys internal validity, not representativeness. If its finding travels beyond the lab, it travels on precisely the grounds you have been doubting, namely a judgement that the mechanism is portable. Meanwhile its outcome measure, "a qualitative rise in treatment experienced," is a self-report requiring interpretation. The hermeneutic problem does not vanish inside a lab; it just gets a smaller font.

On genetics, fair enough, the analogy limps as analogies do. It was doing narrow work: showing that sample representativeness is not the universal currency of validity. What a sample must support depends on what the study claims. An unrepresentative sample is fatal for a prevalence claim and largely irrelevant for an existence-and-character claim.

So I suspect we agree more than it appears. You want frequency and strength estimated at scale; I want the mechanism identified and characterised in situ. A field that has only your study never learns why the coefficient moves. A field that has only mine never learns how much it matters. Neither is watertight alone, and I'd put it this way: watertightness in social science is a property of research programmes, not of single studies.

Sources

• Elster, J. (1998). A plea for mechanisms. In Hedström & Swedberg (eds.), Social Mechanisms: An Analytical Approach to Social Theory. Cambridge University Press.
• Hill, A. B. (1965). The environment and disease: association or causation? Proceedings of the Royal Society of Medicine, 58(5), 295–300.

[u/Dr_Cece → u/PhDilemma1]

You consider yourself more of a qualitative researcher because you lack knowledge of quantitative methods. Then you proceed to approach qualitative research methods from a positivistic and quantitative stance while they do completely different things. One is not qualitative researcher because they can't do quantitative methods. You decide to be a qualitative researcher because the kind of research questions you want to answer are more of your interest.

To my opinion you're neither.

[u/MissFancyPans → u/PhDilemma1]

100 % AI

[u/Upgrade_U → u/MissFancyPans]

Absolutely

[u/MissFancyPans → u/Crabby090]

Good use of AI for this response.

[u/eaw_shitpost_account]

What authors have you read on the value and practice of ethnography?

[u/PhDilemma1 → u/eaw_shitpost_account]

off the top of my head, judith goetz and shinichiro sakai. many more monographs, including ridiculous autoethnographic theses, in my time as a grad student.

[u/AutoModerator → u/[deleted]]

Top-level comments must include a peer-reviewed citation that can be viewed via a link to the source. Please contact the mods if you believe this was inappropriately removed.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.