Simon Cullen's recent 'Ok kids, let Daddy talk for a moment' take-down of experimental philosophy is not a simple rejection of xphi. His point is that it might be important, but not yet. Cullen's argues that X-Phiers have been blind to the pragmatics of survey interpretation, and therefore, are discovering less about the untutored intuitions of the folk and more about latent biases built into the surveys they administer.
I have little to say about that, other than that Cullen's argument seem persuasive to me and I'd like to hear an x-phier respond. But it did get me thinking about a related, if more amateurish point:
The assumption behind x-phi is that there are rules for basic but philosophically relevant concepts like knowledge, intention, value and so forth. They like to draw an analogy to linguistics: any English speaker effectively knows a whole lot about English grammar, even though most hard-pressed to tell you much about it. So too, reason the x-phiers, the average knower knows a lot about knowing, as the average valuer knows a lot about value, but they are unable to say a whole lot about it explicitly. That's why we have surveys, in the same way that linguists proffer tests to learn how a particular linguistic community deals with this word, or that transformation, or this sort of sentence, etc.
More importantly, there is an explicitly normative component to experimental philosophy. Lots of philosophical arguments seems to come down to the tutored intuitions of trained philosophers. But we philosophers are a weird bunch. So we likely have abnormal and biased intuitions. We should--argue the x-phiers--at the very least weigh our intuitions against the folk before deciding whether to trust them. A similar dynamic of course plays out in linguistics, where there is a strong feed-back relation between the normative and descriptive elements.
In any case, my thought is this: why disparage our training? Joe Folk probably knows what he needs to know about knowing, and valuing, and intention-ascribing, and no more. Put Joe Folk in a weird, Gettier type situation, a situation he's never had to confront before, and he breaks down. X-Phiers will argue that giving Joe Gettier, or Old vs. Young Mary, or Truetemp situations is the philosophical equivalent of a wug-test. But it's not. Let me illustrate with the following analogy: Joe Folk also has intuitions about gravity. He knows that what goes up must come down. He knows that the farther things fall, the faster they go. Now physicists have some pretty strange intuitions about gravity and the shape of the universe. From what I can gather, lots of them think that the universe might well be a flat, and shaped like a donut, because, you know, donuts are flat. Suppose we ask Joe if he thinks that space is flat and shaped like a donut?Or suppose, because we want to wug-test him, we ask a question like 'Suppose that Jane starts out in a space ship in one direction from the Earth at light speed and travels for an infinite amount of time. At some point, continuing exactly in that direction, will she wind up where she started?' intending to get at Joe's implicit knowledge about space, time and gravity? I hardly need point out that Joe's answers to such questions, while interesting for folk-theories of space and time, are hardly important for physics.
My point is that, in some sense, Joe's theory of gravity and the physicists theory of gravity are the same thing and aimed at the same object: in some sense, they are both thinking about objects like rocks falling to the ground. But this hardly means that we should give equal weight to Joe's implicit beliefs about gravity. A better assumption to make is that Joe doesn't really have a 'theory' of gravity at all...he has whatever idea of gravity is necessary for him to get around in life, and no more. So too with Joe and philosophers. In some sense, sure, we share a basic understanding about knowledge, and value and intentions with Joe. But we are experts in the field, and so our intuitions are special. I would finally add that the field of mathematics relies upon intuition-based arguments quite often, and while this causes problems about the nature and possibility of proof, hardly any mathematician thinks that the right answer might be to go survey Joe and Jane Folk about Incompleteness and transinfinite numbers.
Let me briefly address a counter argument: one might argue that physicists' intuitions about space, time, etc., don't matter until they are testable and subject to public and verifiable scrutiny. The x-phier argument is that, in many philosophical disputes, intuitions themselves are the 'test' and 'verification', and since these are potentially biased, we should look to correct that bias. My reply is to iterate, what benefit do we get from extending our 'testing' to the folk. Many issues are insoluble and untestable in physics as well. Does anyone think that including the Folk in these disputes is going to clarify matters at all?