Today I was reviewing a paper that reported recording EEG with a Biosemi Active-Two system and adjusting the impedances to below 5kΩ. This mistake (explained below) sent me down a bit of a rabbit hole.
For those of you who don’t know, Biosemi systems are designed such that impedance (measured in Ohms [Ω], or more commonly kΩ) is more or less irrelevant (more info from Biosemi here: https://www.biosemi.nl/forum/viewtopic.php?t=486 and here: https://www.biosemi.nl/forum/viewtopic.php?p=4239). For this reason, the software that comes with the Biosemi system does not report impedance. The thing that we adjust when we are setting up a Biosemi recording is the offset of each electrode (measured in Volts, or more commonly mV). It is the mV difference in the DC offset between each electrode and the reference (the CMS). This is different from all other EEG systems that I am aware of, which universally use impedance adjustments.
Now, this reference to impedance might be a typo in the article I am reviewing, or it might reflect the author assuming offsets and impedances are the same thing. However, I suspect this is not the case, because offsets are typically adjusted to be within ±20-40mV (the Biosemi website suggests 25-40mV is acceptable, but we tend to go for better in this lab), while papers adjusting impedance typically aim for ±5-25kΩ. This difference in range (20 vs 5) suggests a different possibility; That the authors are writing things in their Methods section that they didn’t do and don’t know the meaning of, presumably just because they have seen it in lots of other papers.
This is far from the first time I have come across this mistake. In fact, I suspect it is fairly widespread. But good scientists shouldn’t form judgements based on vibes. So, to check this suspicion, while I worked on other things, I had ChatGPT 5.6 Sol gather some data for me.
I asked ChatGPT to gather every EEG paper published since 2017 that had reported collecting data with a Biosemi ActiveTwo system and to count how many reported adjusting offsets, how many reported adjusting impedances, how many reported both, and how many reported neither. I told it to only count papers for which it had access to the full text (no snippets); Only papers from task-based or resting state cognitive neuroscience studies (no biofeedback or BCIs, because these fields are huge and I wanted to constrain the scope); And only use data from experiments in which the offset/impedance report is from the same experiment as the declaration of using a Biosemi (to weed out papers that used different systems in different experiments and might adjust impedance for a different system).
It found 1,872 papers using a Biosemi for cognitive neuroscience research since 2017 and to which ChatGPT had access to the full text. The results were… worse than I thought. Of these 1,872 papers, 264 (14.10%) reported adjusting offsets, 210 (11.21%) reported adjusting impedances, 6 (0.32%) reported both, and 1392 (74.35%) reported neither.
To highlight the most important thing here: leaving aside the both and neither cases, almost as many studies reported adjusting an aspect of the data that does not matter, is not given by the software, and cannot be true (210), as those that reported the thing they presumably actually did (264). This difference was significant, for whatever that’s worth: χ2(1,N=474)=6.152, p=.0131 (excluding those that did both), which suggests that correct practice is at least a little more common than incorrect reporting, which I suppose is a good thing.
But, as I alluded to earlier, this alone does not show people are adding things to their papers just because they have seen them in other papers. They might have the two measures confused. So I also asked ChatGPT to extract the impedance and offset values reported in the respective categories. If the 11% of papers that report adjusting impedance are simply using the wrong word, then we would expect no significant difference between the values they report adjusting to. However, if they are reporting impedance values they have read in other papers, ones that have nothing to do with actual practice of using a Biosemi, then we would expect the mean of those reported values to be significantly lower than for papers that report adjusting offsets. The findings:
Papers reporting adjusting Offsets: N = 250*, mean offset criterion = 30.06, SD = 11.03
Papers reporting adjusting Impedance: N = 186*, mean impedance criterion = 17.42, SD = 12.44
* these numbers differ from those reported above because not all papers reported their signal adjustment criteria.
Independent groups t-test comparing these: t(434) = -11.21, p = 9.05 x10^-26
This result is highly consistent with the suggestion that authors are putting things in their Methods sections just because they have read them before.
To drive this home, the criterion of using a threshold of 5kΩ or 10kΩ comes from the need to have data this clean to reduce line noise when using passive electrodes. Active EEG systems that allow the experimenter to adjust impedance (non-Biosemi systems) typically recommend an impedance threshold of around 25-30kΩ, as they are better able to exclude line noise due to their use of active electrodes. So if people were reporting offsets under a different name, or even if they had found some workaround so they could actually see the impedances, there would be very little reason for them to report using criteria of 5-10[kΩ or mV]. And, indeed, for the papers that report using offsets, this is true. Only 1 of the 250 offset-reporting papers reported a criterion of 5mV, and only 4 reported a criterion of 10mV. But this was not the case for those reporting adjusting impedances. Forty (21.51%) of the 186 impedance-reporting papers reported using a criterion of 5kΩ and 48 (25.81%) reported using a criterion of 10kΩ. These values simply cannot be reflecting anything that the authors actually did in practice.
So, in conclusion, reporting of a metric that is impossible to access (impedance) occurs almost as often as reporting the correct metric (offset). What’s more, we cannot assume all those authors simply used the wrong word, because we have a strong indication that over 47% of them are just writing things in their Methods sections because it sounds good. Does this matter? I don’t know. But it troubles me. It is important that science and scientific writing be precise, to enable replication through complete and honest reporting of one’s methods. To err is human, and I would never hold it against an author for accidentally making a mistake in their Methods. I certainly have. But writing things in your methods that you don’t know the meaning of, and that cannot in fact be true, is not a mistake. For this many papers to include spurious (impossible!) information just because they think it sounds right… like I said, it troubles me. What does it suggest about the rest of the information they include?
PS: I have not checked ChatGPTs findings, so take them with a grain of salt I suppose. I am aware of the hypocrisy of saying this in light of the sentiments I expressed in the final paragraph.
