Reality Checks

Possibly the most difficult part of academic research for me is the falsifying. We are taught about Popperian hypothesis testing, and then generally proceed to ignore it once the initial research design for our dissertation work has been approved. This isn't to say I don't test hypotheses. It's just that it is somewhat less formal, and messy.

Instead, I'll be working on crunching my data, and something extraordinary will appear. This happened this week. For the previous month and some, I ran dozens of statistical tests on the distribution of different kinds of forms of ceramic vessels and other evidence for likely uses of those vessels for frying foods, household storage, transporting liquids, etc. As a result, I have uncovered what I believe to be a commercial food serving location (either a food stall or perhaps part of an tavern or inn) in what may be an open "shopping mall" style area to the south of the main plaza of old San Salvador. Other details have emerged, including the importance of liquid storage and transport in one apparently rich household that might be evidence of tribute collection and bulking, and minimal differences in the activities associated with what may be households led by a Spanish male, and what appears to be a Mesoamerican household.

To get that far, I found myself making test after test. Initial tests to give me a feel for what I might be looking at. Then, after coming up with some hypotheses (this signature might be excess food production, and in this case it might be for commercial reasons or perhaps to satisfy Spanish tastes), I'd have a two step process.

First, I'd do additional literature review. For example, if I thought I was dealing with a commercial food vendor, what sort of businesses existed in Europe or the Spanish Empire in America in the 16th century? What kinds of food might they make? What tools would they use? Is there anything in the published history on early San Salvador or Central America that might suggest a change in diet by this time? What methods have other archaeologists used to examine kitchens and other food preparation areas? What effect would a house fire have on such an assemblage?

Then comes more testing. Any or all of the research mentioned above could spawn a new statistical test or review of my data. I might run an earlier test, but remove all burnt vessels. I might apply a new mathematical transformation. I might map out the spatial distribution of different vessel classes. All to satisfy for myself that if someone else asked one of those questions, I'd have a reasonable and satisfactory answer that did its best not to violate Occam's razor. This sort of thing can go on forever, since a hypothesis is never proven, it just survives testing or is modified/discarded. But the real-world constraints of time and imagination eventually factor in, and an answer holds, at least for this stage of research. In the case of the food vendor, the concept held up to a number of taphonomic and historical questions I posed, though one initial hypothesis (that this vendor did much more frying than is typical in prehispanic Mesoamerican cuisine, because of pig lard and because of Spanish tastes) largely fell apart. The location and the site are slightly more likely to have low simmering or frying vessels than typical Mesoamerican cookware assemblages, but the difference isn't terribly striking.

So this week, I am going through the process anew. In working on my last data analysis chapter, I have found that some tests wholly unrelated to the vessel function tests came up with the same basic groupings as the functional tests, with one exception that makes a lot of sense due to other evidence. And furthermore, the general theoretical principle I am using for understanding cultural shift could explain this correlation. I won't get into it just yet, because I need to do more testing (and began yesterday, applying a statistical test I've never used before, but that I like, on measuring the richness and diversity of variation).

This is the big struggle for me, fighting the desire to run with my initial hypothesis that is based on the data. The ideas I can imagine from the initial examination of the data or early testing will get me very excited, and I'll want to tell people and start thinking about the rammifications. In fact, I've done this a couple of times this year, talked about hypotheses which have since turned out to be incorrect. On the other hand, a number of my major ones that have emerged over the last year and some of analysis have survived or been enhanced by the reality checks. It is just getting over that initial excitement of making a mental leap from recognizing a pattern, and settling down to the methodical skeptical testing.

This is of course at the heart of the scientific endeavor. I want to say that scientists emphasize the skepticism too much at the expense of the imaginative element. Not because proper ideas are crushed by too much skepticism, but because science is often popularly described and presented as having little to do, or even being at odds, with imagination. Imagination is seen as the province of the artist or the lunatic, while the scientist is supposed to truck only with technical skill and math. However, it is clear that the skepticism still needs to be emphasized, not just to get the notion across to laypeople, but to remind researchers that a press release or other opportunity for fame and fortune is not a good enough excuse to forget about the self debunking. Far too often, the pressures of presenting progress, ego, or even just sheer excitement at discovery will conspire to create the tempation to count all the data before they have been properly vetted.