By the end of July, Ren and Wren have become coosters: worn-out libertines. So they take a few days break from their debauches to give their livers and their sore edea a rest. Wren with a W goes to visit some friends. Ren without a W goes to see Ken and Halberd in Ken’s flat to discuss progress on the Lucius project.
Ken with a K, and Halberd without a K, have been getting on with attempting to replicate some of Lucius’s work. Lucius thinks they’ve given up on doing that – he thinks they’ve gone back to doing their own studies – so he’s paying them no attention. He’s no longer using them to do analysis on his data.
Ken tells Ren what he has learned at a recent conference.
‘I was talking to this psychologist who’s now in clinical who told me that he left social psychology because the whole field is bullshit. When he was a graduate student he was trying to replicate all these studies that had impressive results, and he never could, no matter how careful he was. So he writes to all the people who did these experiments, asking for more details of what they did, and they all sent back detailed information. Very detailed. In most cases the experiment only works in certain conditions.’
‘By “works”, you mean getting the desired sexy, political result?’ says Ren.
‘Yeah,’ says Ken. ‘So there were instructions like, don’t do this test on a computer, it only works if the participants are using paper and a pencil. Or it doesn’t work if they use a pencil, it has to be a pen. Or the participants have to be given something else to do after reading the materials, for precisely two and a half minutes. If you make it any more than that, or any less, then you don’t get the result you want. And if you more or less people than some specific number in the group then it doesn’t work either. And there was… I’m trying to think of another one he mentioned…’
‘Didn’t you say there was something to do with the words used?’ says Halberd.
‘Oh yeah,’ says Ken. ‘On one experiment he was told that he had to use certain exact words when talking to the subjects. If you used other synonyms, even really close ones, then it didn’t happen.’
‘My favourite was the clothes,’ says Halberd. ‘On one experiment concerning behaviour in contexts where you are reflecting on your eventual death you only got the result you wanted if the experimenters wore black clothes and had a gloomy manner.’
‘Jesus, those sort of factors should discredit any experiment,’ says Ren. ‘Don’t the researchers understand that?’
‘Well, this guy did,’ said Ken, ‘that’s why he changed fields. But I don’t know if other researchers do.’
‘They don’t know much about science then,’ says Ren. ‘Or else they’re deliberately ignoring these issues because it that allows them to publish their propaganda in good journals and make a name for themselves.’
‘That’s a harsh interpretation,’ says Ken.
‘Is it any worse than the alternative?’ says Ren. ‘That these supposed top scientists don’t understand the basics of confirmation? Either way things are bad. Frankly, it’s just dishonest to claim that such-and-such is the case, when in fact that such-and-such only happens in certain carefully-contrived circumstances, which the experimenters have searched hard to find. “People are more X in an Y environment, but only if the environment is Y in the precise ways A, B, C, D and E. And it’s 3:37 on a Tuesday afternoon and it’s exactly twenty-seven degrees. At the very least you should make all this clear in your paper.’
‘But the journals don’t want that much detail,’ says Halberd. ‘And they don’t want such qualified conclusions. They want strong conclusions that are supposed to hold up generally. So if you put all that in your paper then it will probably get rejected. And then your career suffers.’
‘That’s basically an admission that for the sake of your career advancement you’re willing to publish highly misleading research. That doesn’t constitute any sort of defence.’
‘We know,’ says Ken. ‘You don’t have to tell us that it’s dishonest. We’re putting our careers on the line for the sake of honesty.’
‘I have to admire you guys for that,’ says Ren. ‘It’s very admirable. That sort of bravery isn’t very common in academia. Look at how Miles has dropped out of sight. He’s worried that he’ll get caught up in the blast. Or the fallout.’
‘If it all goes well then he’ll say he was part of it all along, won’t he?’ says Ken.
‘Yes. Mind you, he was part of it all along, to some extent. But it’s you guys doing most of the work.’
‘Might be too late for your field’s credibility, though. I mean, if Professor Plum thinks it’s okay to publish a paper saying X on the basis of experiments he has done, even though X only appears in some particular circumstances, whereas not X appears in lots of other circumstances, then he’s just as entitled to publish a paper saying not X as he is saying X. But it’s X that he wants to support, because it supports some more general theory he likes, or it’s more PC, so he publishes the former paper, not the latter.’
‘It’s the same in principle,’ says Ken, ‘as those researchers who do ten studies, and nine don’t support X, but one does, so they publish that one.’
‘The “bottom drawer effect”,’ says Halberd.
‘And few people try to replicate these studies,’ says Ken. ‘No-one other than a few grad students starting our with some practise studies is much interested in replicating other people’s work. Especially if you try one and it doesn’t work. Then you just say, I’m not wasting my time trying to replicate someone else’s work. As soon as you get a license to do your own studies, then you want to make a name for yourself with your own stuff.’
‘And journals aren’t very interested in publishing replications,’ says Halberd. ‘Especially negative ones. So X stays as the accepted wisdom.’
‘Yeah, even if the author of the paper supporting X has sucked off the entire editorial board of World Science to get published,’ says Ren.
‘That’s, ah, putting it rather more colourfully than I would have,’ says Ken.
‘All the section editors want to see their sort of stuff confirmed,’ says Ren. ‘They might get tough with the stats, and so on, but are they asking the really important questions, like, how robust is this result? Do you have data which doesn’t support this conclusion?’
‘And if so, why isn’t that included in the overall data set?’ says Ken.
‘Does the supposed phenomena only appear if the circumstances are just right?’ says Ren. ‘We’re not doing physics or chemistry. If you can show, say, that the gravitational effect doesn’t hold in some circumstance, then that’s big news, regardless of how circumscribed that circumstance is. But this is research designed to show that the human mind works, for the most part, in some particular way. If it often doesn’t work in that way then your thesis is blown, or at least greatly weakened. Your conclusion may be something like “Men found to consistently undervalue woman’s contributions”, say, but really your conclusion should be something more like “In a certain limited, artificial set of circumstances, and only when they are suitably primed, men undervalued women’s contributions. Assuming, that is, that our somewhat tendentious interpretation of the data is correct”.’
‘Well, all that sort of thing is a different issue to whether Lucius is faking his data,’ says Ken. ‘That’s what we’ve got to concentrate on at the moment.’
‘But all of what you’ve just told me means that your attempts at replication are a bit pointless. They’re bound to fail, because even if Lucifer isn’t faking anything, it’s probably the case that you can’t replicate his work because you don’t know the exact right circumstances you need to have. These experimenters put an awful lot of time into finding just the right circumstances. You aren’t likely to stumble across those first time. As you’ve just said, people who want to replicate these sorts of results need to be told the secret combination of circumstances for it to work. Whether to use one word, and not a similar one, for example. So these replications are likely to fail even if Lucius has been honest all along.’
‘I see your point,’ says Ken. ‘You’re right. But it’s still worth doing them, it will help our case. If Lucius hasn’t specified the circumstances sufficiently then that’s his failing, not ours.’
‘Well,’ says Ren. ‘Maybe it’s worth finishing what you’ve started, but I wouldn’t spend much more time on the replication side of things. It’s the analysis that’s more important.’
But he knows as he says it that Ken and Halberd will waste their time pointlessly trying to replicate Lucius’s work. The replications are going to fail regardless of whether Lucius’s data is made up or real.