Saturday, November 10, 2012

empiricism is inextricable from theory, data is meaningless outside of norms, science exists within philosophy

This post will be quite long. Please consider that before deciding whether to read it.

I'm writing this from my office. I came here at an ungodly early hour on a Saturday morning because I've been trying to work out a problem. The problem lies in a particular search string. I'm currently engaged in a research project investigating the composition processes of second language learners, in this case Chinese Mandarin and Hindi L1s. I am utilizing corpus linguistics software to mine a vast archive of student essays for certain patterns of argumentative and rhetorical structures. The software reports back to me about frequency and position, and through these outputs I can statistically compare the use of such structures between demographics-- language of origin, years of education in English, etc. And since a key to pragmatic results from second language studies is often reference to native speakers, I'm creating a baseline through reference to an equivalently-sized corpus of L1 English subjects.

Much of corpus linguistics has focused on the level of morphosyntax, for the simple reason that the software is better equipped to look for certain word-level constructions or word pairings than it is to examine the larger, more complex, and more variable argumentative plane. English is notoriously morphologically inert; that is, our use of inflections such as affixes is quite limited in comparison to other languages. (Compare, for instance, to a language like Spanish.) For this reason, searching for particular syntactic structures with computers can be quite tricky. It's also for this reason that formalist poets in other languages often have an easier go of it than in English-- it's much harder to write a villanelle or in terza rima when words lack consistent inflectional endings. In a language like Latin, word order is vastly more malleable because the inflections carry so much of the information necessary for meaning. In English, word order is quite mutable in an absolute sense but quite restricted in comparison to many languages. (There are exceptions, such as floating quantifiers, eg all-- "All the soldiers will eat," "The soldiers all will eat," "The soldiers will all eat," etc.)

But recently, researchers in composition have had some success in looking for certain idiomatic constructions as a clue to the kind of arguments that students are making. Some of these are obvious, such as the use of formal hedges ("to be sure") or boosters ("without question"), and those are types of features for which I'm searching. Some are more complicated and require a little more finesse to search for effectively.

Code glosses, for example. A code gloss is an attempt by a writer to explain to readers what a particular word or term in his or her text is meant to convey in the context of the particular writing. Code glosses are not or not merely definitions; a definition provides denotative information that is accurate or inaccurate regardless of context. A code gloss, in contrast, has to provide the information necessary for a reader to follow the writer's argument, and so a code gloss could fail as a general definition but succeed in its specific purpose. (This paragraph itself amounts to a code gloss.) The study of these kinds of features in writing, if you're feeling fancy, is referred to as metadiscourse. Many types of metadiscourse have certain formal clues that can be used to search for them in large corpora.

Unfortunately, false positives are common. The further you get from a restricted set of idiomatic phrases, the more likely it becomes that the computer will return a morphologically identical but argumentatively distinct feature-- so a search for "to be sure" as a formal hedge will also return "I looked it up in a dictionary to be sure that I got it right," which is not a hedge. The flexibility of language, one of our great strengths as a species, makes this sort of thing inevitable to a certain extent. The recourse is often just to sift through the returned results, looking for false positives. (Or, if you're lucky enough to have one, making a research assistant do it!) You might ask why to bother with the computer at all, if you have to perform a reality check yourself with most strings. The answer is just that it's possible to look through the, say, 600 returned examples from a given search string and eliminate the false positives but not to look through the 500,000-2,000,000 words in a given corpus looking for what you want to find.

Beyond that, your only recourse is to building effective search strings given the interface of the particular corpus linguistics software you are using. This requires carefully calibrating wildcards, places in the search string where the software can include any result. You can restrict these wildcards in a variety of ways-- for example, you can allow the wildcard to return any particular letter or one of a certain number of letters. Or you can bind the wildcard in terms of immediacy of surrounding letters or words; that is, the wildcard can be formatted so that the software will look a certain distance in characters from a particular search term. The more open-ended you make your search strings, the more likely you are to have false positives that have to be laboriously culled for accurate data; the more restrictive you are, the more likely you are to exclude relevant examples and thus jeopardize the quality of your research.

And that's why I'm here on a Saturday morning: I'm poking around with a particular search string, looking at the results it returns, and trying to fine tune it in order to better approach the results that I want. All of this is in the service of coming up with research that can express certain qualification- and caveat-filled conclusions, responsibly presented, in order to provide some small amount of progress in our understanding of second language literacy acquisition, which is one of my primary research interests. It's what I love to do.


This is all an exceptionally long-winded windup. (If you are looking to make the accurate criticism that my posts are way to long, this may be the best proof yet.) I mention it all as context for how I feel when I read this excellent post from Shawn Gude, on Ezra Klein and a certain brand of liberal commentator, commonly referred to as a wonk.

Gude quotes Klein as saying that he doesn't think of himself as a liberal anymore, just an empiricist. Gude points out, correctly, that this is a mistake on any number of levels. He shows an old Bloggingheads video where Will Wilkinson and Klein debate the necessity of first principles. I find Klein to be agonizingly frustrating in the video. Wilkinson keeps asking him to accept a simple fact: that whatever the empirical reality of health care, Klein is embracing that empiricism as a means to advance a particular normative end. And as Wilkinson accurately points out, that normative end must itself be justified and argued. It does not exist a priori. Klein repeatedly and doggedly evades having to make that justification, and in so doing makes his own arguments weaker and his own credibility suspect.

Gude is right to view all of this as a profound mistake on Klein's part. But it would be a mistake to view this as a conflict between empiricism and theory. Rather, it is a failure to understand what empiricism is, both in ideal and real-world terms. I referred to my own research here because I want to discuss how empirical work exists in a theoretical and normative framework. I think that, in addition to the theoretical, political, and moral failings in Klein's worldview, there are internal contradictions within Klein's worldview that leaves him bordering on incoherence. And as he is now one of the five or ten most influential journalists in the world, those failings have profound consequences.

Empiricism exists within a framework of theory, and theory cannot be derived empirically. The fact-value distinction is real. (This argument of mine is illustrative of its own point: I take it as an empirical truth, not a normative statement, but its empirical claims are necessarily grounded in theoretical assumptions.) And fact-value problems exist for both the commission of empirical projects and the evaluation of empirical results.

Conducting empiricism requires making a seemingly ceaseless number of choices, choices that cannot be resolved through reference to other people's empiricism. Sadly for all of us, a guidebook for empiricism has never been handed down to us from the heavens. Arguments and complaints about research methods and methodology are vast, and an enormous literature devoted to adjudicating these arguments has been written. These disagreements stretch from the most limited and quantitatively-oriented questions (when is it appropriate to use a p=.05 level of statistical significance? When is it necessary to use a .001 level? What aspects of a given research project determines the use of one or the other?) to the broadest questions of purpose and justification (why research? Towards what end? For what purpose and for whose good?). None of them can be answered empirically-- not that they shouldn't be but that they can't be. We lack even an idea about how such an empirical investigation of questions of value could be undertaken.

In the example from the Bloggingheads video, Wilkinson is trying to get Klein to acknowledge that before we assess empirically the data he says is in his favor, they have to determine what would constitute empirical success and even what questions they are trying to answer. If Klein prefers the language of social sciences, Wilkinson is asking him to consider what their construct is, how it is operationalized, and what results must be returned through the assessment of that operationalization in order to suggest success at a certain degree of confidence. Klein cannot answer those questions empirically. I think he knows that, but he is so stuck on this monolithic and transcendent vision of what empiricism is that he seemingly can't confront those necessary and prerequisite questions.

The real shame is that he would never make this mistake if he ever attempted to do social science research himself.

This semester, I'm taking a seminar in quantitative language testing. Among many other methods, one tool we've looked at is multiple choice tests. Some find such tests to be inherently clumsy or reductive instruments, and indeed there are a host of issues with them. But the science of multiple choice testing is extraordinarily sophisticated. Writing a good multiple choice item involves decisions having to do with deciding what construct to test, formatting of the language in which the question is expressed, formatting of the key, what kind of distractors should be included, and many more-- all before you even begin situating it within the context of a larger test. There are certainly statistical and quantitative ways to assess multiple choice tests, and any responsible test administration would have to utilize them. But the stats themselves are meaningless outside of a theoretical understanding of what they mean in application to a particular question. You can get great stats from your multiple choice test with worthless questions.

You might, for example, get a point-biserial correlation that suggests that a MC test item is perfectly discriminating between high- and low-scoring test takers, in a case where the test item is useless. A point-biserial correlation compares continuous variables to dichotomous variables; it is often used to show how well a test item (which can be scored dichotomously, ie, right or wrong) discriminates between higher and lower scorers. This is a clue to the validity of the item; if more low scorers (based on the rest of the test) are getting the question right than high scorers, the suggestion is that something has gone wrong with the item. But even with perfect point-biserial coefficients, you can have flawed, even lousy tests. Construct-irrelevant variance happens. The reason your high scorers are getting the question right and low scorers are getting the question wrong might have nothing to do with the given construct your items are meant to test. We have empirical tools that can help us find this kind of error, but we can never farm out our interpretation of such problems to empiricism. They have to be understood theoretically; theory is the only guide to pragmatic solutions to such problems.

I think of someone like Dr. Glenn Fulcher, whose credentials and reputation are beyond reproach. His wonderful website is a testament to a decades-long pursuit of better, fairer, more accurate tests of language skill. Dr. Fulcher is an empiricist, as I am, and he believes as I do in the positive power of responsibly-generated social science. And yet his work is filled with caveats, provisos, and discussions of limitation. In his indispensable book Practical Language Testing, he expresses this kind of self-skepticism again and again, declaring repeatedly that any effective empirical inquiry into human behavior requires locating the meaning of data in a theoretical framework. Speaking towards Gude's point explicitly, he says of the task of crafting test specifications, "Specifications therefore reflect the theoretical and practical beliefs and judgments of their creators." The meaning of the test is always a reflection of the pre-empirical beliefs of the people who wrote. In a world where tests like the SAT or standardized mastery tests in public schools have profound impact on human lives, understanding this empirical-theoretical interchange is essential, and that understanding is what Klein is at risk of obscuring.

Nothing could be more discouraging of the "I only deal in empiricism" mindset than a lifetime spent performing empirical research.

That skepticism of experience is why wonks frequently worry me, because in their (necessary and well-intentioned) policy generalism they fail to acquire the experience through which a broad understanding of a given field of human inquiry is derived. I don't question the dedication and responsibility of the wonk class. But I do believe that really becoming fluent in the ins-and-outs of messy, contingent research requires that one performs research, and that one spends the endless frustrating hours of reading and writing necessary to become credentialed in a given area of the social sciences. Our system of development of expertise, flawed though it may be, creates the indispensable conditions of time, thoroughness, and review. All three of those are frequently impossible for journalists and bloggers. Smarts are necessary but not sufficient. I have no doubt at all that Dylan Matthews, wonk-of-the-future and Klein's frequent research assistant, is extremely smart. But smarts are not nearly enough. If they were, we'd have fewer problems.

I said before, and will say again, that wonkery is necessary in today's democracy. But it can never be enough, and the sad fact is that in the world of liberal media these days, many people flatter themselves to believe that wonkery is the only way to advance the progressive cause. That's a ruinous and self-defeating notion. It is also one which is applied inconsistently: wonks, for example, tend to be supportive of conventional school reform efforts, despite the incredible empirical failures of those same efforts. (It seems that behavior does not descend so cleanly or inerrantly from empirical results after all.) Liberalism no doubt needs wonks. But the growing resistance to academics, theorists, and philosophers that Gude identifies within the liberal media is a profound mistake.

This all precedes the political question, which to my mind has been answered dispositively by global warming: the facts are not enough. We flatter ourselves to self-identify as the party of facts and science, but the human mind does not operate by facts alone. Last week, I saw a lecture by a rhetoric professor who works in a center for global sustainability. He told stories about how often climatologists and environmental scientists lament to him that the science is sound and yet not convincing for the people who need to be convinced. His job is to help them better craft their arguments in order to make convincing easier, and he often has to remind these scientists that conviction does not develop from science alone. Aristotle was one of the fathers of empiricism, and yet he wrote extensively on the dangers of trying to convince people based on facts alone. The human animal doesn't work that way, not 2,300 years ago and not now. The facts are not enough.

The urge towards empiricism as a tool is a necessary response to an ideology which has rejected plain facts again and again in the pursuit of its political ends. The urge to a transcendent or totalizing empiricism, however, is a deeply human, deeply understandable, and deeply flawed one: it seeks to remove messy moral questions from the scramble of everyday life. But no such refuge can be found, not in numbers or science or anywhere else. We are in a period of great liberal self-satisfaction. I get it, and I'm taking part in it. But these periods never last long, and they don't in large part because we mistake political victory for the triumph of the facts. This misunderstands both our strengths and our weakness, both the American character and the human character. The facts are only the beginning. To fail to understand that is to fail not only philosophically but practically and politically as well.


redscott said...

Well put. We've lived on and off for 40 years with earnest liberals saying that if we just put the facts out there, we'll prevail. But human beings don't get out of bed in the morning to look at the facts, as you note, they need something of value to drive them on. Love, hate, anger, greed, lust, justice, truth, transcendence, ambition - all values and none of them dispositively dealt with by an appeal to facts. I've heard people in the last few days criticize the self-delusion of conservatives about polls and other things by calling it "motivated reasoning," like it's an unanswerable put-down. But all reasoning is motivated. Reasoning and empiricism are methods in search of goals, but they don't prescribe the goals. So, yeah, we do have to argue facts with the other side and point out how they're wrong on the facts, but we also have to argue with them about our values. Liberals are great on facts and have been for a long time. How far has that gotten us? When will we think it's worthwhile to criticize stuff like attempts to harm Social Security not because it's "extreme" or because "it's actuarially unnecessary" but because it's fucking wrong to make old people scrabble for their existence at the end of their lives. It's not like the fact/value distinction was thought of yesterday, it's been around and around and around, but sometimes I feel like we've never heard of it. It's time we started grappling with the value part of the argument and telling stories to actual humans about why the values we have are ones they can support. Westen and Lakoff have been arguing for this approach for more than a decade. When will we really begin doing it?

K-Pop said...

All observation is theory-laden.

PrajK said...

Great post. But methinks the problem doesn't start with Klein as much as it does with academics. The delusion that empiricism alone can solve our problems started with us, and we continue to promote it. The Ezra Kleins out there are simply following our lead.

While I sympathize with your efforts...there are sound historical, institutional, and cultural reasons why social scientists are not the standard-bearers for empiricism and its connection to theory.

For better and for worse, it's usually the natural and physical scientists who speak on these matters. Even more specifically, it's often physicists. And trust me, they are not going to accept that "science exists within philosophy." I have tried to raise this very point before, and I can't begin to describe the ruckus it caused.

If your goal was to pick a fight, you couldn't have chosen a better title. I can predict my friends' response: "What? Are you trying to say that the mass of the electron depends on philosophy? That there are norms about the existence of gravity?"

These discussions always get reduced to something like gravity. Again, trust me because I've been there.

You're trying to raise a very complicated idea. One that requires a ~1,000 word example to even get started. It's a bit much to expect wonks to descend into this territory.

It's especially harder when much public outreach about science and empiricism by academics stress that data alone can set us free, that empiricism is a way to free ourselves from our pre-conceived philosophies, and that thinking otherwise is the first step to destroying the Enlightenment.

If you want to change Ezra Klein, start with the physics professor down the hall from you.

PrajK said...
This comment has been removed by the author.
PrajK said...

Not to beat a dead horse, but I think the goal should be to get scientists to accept 2nd half of this statement:

"And fact-value problems exist for both the commission of empirical projects and the evaluation of empirical results."

IMHO, that's where the battle should be fought. The first part is fairly uncontroversial. The second is where tempers can get flared.

Freddie said...

I agree, Praj. I think one important point to make is that even statements like "Science is the best method do adjudicate disputes about facts that can be referenced in the material world" is itself philosophy. "Science is good" is also philosophy.

For this post in particular, I tried to avoid a lot of the deeper epistemological controversies by focusing on issues that are likely to come up in politics-- which are, generally speaking, social science problems. I didn't want to avoid those issues because I'm not interested but just the opposite-- it's really easy for me to get too focused on them and lose the immediate concerns.

Joseph said...

Fascinating post.

I'd be interested to see your take on Jim Manzi's project and specifically his book Uncontrolled. He seems to suffer from the same pathologies as a liberal wonk on this score but at the same time he does seem to evince a self-critical understanding of what science actually is. And nothing demonstrates that more than the fact that he actually talks about philosophy of science in his book.

PrajK said...

Thanks for the response. I agree that w.r.t politics, social science problems are most relevant. But then it's even more problematic that the (to pick on a field) particle physics epistemology is dominant in public discourse. I've even heard geophysicists complain about how their research methods (largely observational b/c you can't really experiment with the sun, e.g.) are neglected when we talk about "the scientific method."

Given that you know I largely agree with you, I will push back a little on this:
"statements like "Science is the best method do adjudicate disputes about facts that can be referenced in the material world" is itself philosophy. "Science is good" is also philosophy."

I can agree with these statements because I know what you mean. Most physicists do not, and will tend to interpret such statements in the worst possible way. More often than not, they'll assume you are trying to deny objective reality or something like that.

I like the phrase Londa Schiebinger used in "Has Feminism Changed Science": we need a "translation project" from people like you to people like me. And part of that project has to involve the boring, messy work of semantics and term definition. What exactly do you mean by "is also philosophy?" What about "material world?"

Anyway...this has gone on long enough. Thanks again for the post. I'm probably the only one of your readers who gets more excited about epistemology than drones. Although I assure you I also get excited about drones.

Anonymous said...

You forgot about engineering.

Not all facts are created equal.

There are word facts, which social scientists love to point out are embedded in this and that and the other thing.

Then there are object facts, which create things like the internet and books and paper and pens, which allow social scientists to point out that facts are embedded in this and that and the other thing.

Take gravity, for example. You can argue with someone claiming that it is a universal force affecting things that have mass that causes them to move toward each other at a rate proportional to their weights and location relative to each other, and say that's embedded in philosophy.

This is all fine and good for someone standing on a stable surface. Go to the roof of a tall building, step off of it, and make your argument about gravity being embedded in theory and you will soon be embedded in the ground.

So that's the thing: You don't really change people's minds with word facts, for the most part. There are many situations where a person is motivated to be influenced by word facts. One such situation is referred to as teaching. This is different from marketing and arguing wherein your person will be less motivated.

But you can change minds with engineering facts. Or you can make those who won't change their minds lose their minds. But even though the person shooting has a politico-philosophic attitude that influences their behavior, the bullet doesn't.

sizegenetics discount said...

a good number of outstanding writer templates

Anonymous said...

fwiz [url=]stylo mont blanc[/url] pqyncz opms [url=]mont blanc[/url] acrhcs dwzm [url=]cheap mont blanc[/url] skfmvz rohy [url=]mont blanc[/url] iclbur jgzk [url=]mont blanc sale[/url] fzkuob wybv [url=]mont blanc pens[/url] jfywan hulu

Anonymous said...

hzsrq [url=]dr dre beats[/url] zzlmr tpzex [url=]cheap beats by dre[/url] tjkyn ommlx [url=]cheap beats by dre[/url] nunkt paxal [url=]beats by dre[/url] depam vsnlv [url=]beats by dre[/url] rhemy ruajp [url=]cheap beats by dre[/url] xhyrc xtfc

Anonymous said...

cdnfz [url=]cheap jordan shoes[/url] gliad ojwqb [url=]jordan retro 11[/url] ytzxv roire [url=]jordan 11[/url] pwwnr hmlqm [url=]air jordan[/url] xpafd jogmr [url=]jordan shoes[/url] rxxve hxbth [url=]air jordans[/url] ktasp yhvn

Anonymous said...

itneh [url=]cheap jordan shoes[/url] khhbf jhrdf [url=]jordan shoes[/url] tmfjl bokez [url=]jordan concords[/url] eupih qgshz [url=]cheap jordan 11[/url] catqq javix [url=]jordan concords[/url] qjqqn hmgba [url=]jordan shoes[/url] nxekf hyrw

Anonymous said...

xyqvx [url=]red bottom heels[/url] aakyfq swhvy [url=]Red Bottom Shoes[/url] vwsbbf aypta [url=]christian louboutin[/url] nqnof puxzj [url=]christian louboutin sale[/url] yuyji lorgv [url=]cheap christian louboutin[/url] iftml rsyqg [url=]christian louboutin sale[/url] tizix aporp

Anonymous said...

jqpsu [url=]authentic louis vuitton[/url] wjaxbm sepgj [url=]cheap louis vuitton[/url] ujwtkq hwguy [url=]louis vuitton outlet[/url] xavsju lubs [url=]louis vuitton sale[/url] ubishx ofktu [url=]discount louis vuitton[/url] cjzytx eesgl [url=]louis vuitton outlet[/url] xtdknx wnaa

ryan said...

But you can change minds with engineering facts.

No, you can't. There is no such thing as a fact without a theory. Go to the top of the building and release a ball. Does it fall, compelled by the force of gravity? Well, that's one theory. Another would be that it's fulfilling its natural desire to achieve the lowest state of elevation possible. Same empirical results, two wildly divergent theories.

ryan said...

Freddie, I think what you've touched on here is an issue related to the frustration experienced by so many religious persons in their conversations with secular persons. The secular person will frequently say "I'm just working from the facts," disguising or ignorant of their smuggling enormous ideological assumptions into the making of that very statement. Most religious persons don't tend to view theory--or life in general--as easily divisible between "religious" and "non-religious," so this attempt to impose such a division while denying the imposition is downright maddening.

One thing I've always appreciated about your writing is your willingness to own these sorts of ideological commitments (speaking in general terms, not about religion in particular) rather than pretending to operate from some position of objective detachment.

But unlike you, I'm willing to grant wonks their dedication, but not their responsibility. The problem you describe here, to me, is an indication of a vast and powerful lack of self-awareness which I find extraordinarily troubling in persons who wish to be taken seriously.