On the design of experiments

Okay, well a lot has happened today and I’m not even sure where to start! I guess the main thing I want to write about today, despite being exhausted, is planning. I plan everything, I probably write more about my plans than anything else now that I’m done taking my required classes. It’s no secret, I love planning because having a plan means knowing what to do when that plan goes sideways. It means knowing what to look for before a plan goes sideways. Most importantly, it lets you know you’re on the right track.

Designing experiments is a whole series of classes you can take as an engineer/researcher/etc. I’ve taken several and they are almost always about how to find your sample size, best design an experiment to remove systematic biases, and just generally how to make sure your experiment answers the question that you are asking. Sometimes experiments we think will answer the question we want to answer end up not actually telling us anything useful. In short, design is important and a badly designed study will only end up in wasting time and effort.

Today I had a conversation with a colleague about some experiments they were doing. The people running the study have no real clear goal, no plan, and it’s one of those “see what we find” type of studies that make my head hurt because how does something like that get funded to begin with? Because of my experience I was asked for some advice about some of the data they had collected and was shocked to see just how bad it was.

It turns out they recruited some grad students from a local university (since there are several and the one I attend just happens to be the furthest away from work…. boo). Unfortunately they either overstated there experience working with EEG data or they had no idea what they were getting into because from looking at the filtered data (as in the data “ready to be used” for analysis), they had done almost nothing with it. The data looked awful and I gave several opinions on where they went wrong and how to fix it. I even offered to help in my “free time” give advice/suggestions to the students because as far as I’m concerned any dataset that doesn’t have the huge artifacts I had to deal with for “last paper” is a easy dataset to work with.

This means we have two problems right out of the gate (1) the group doesn’t know what types of analysis they need to run or how to draw comparisons between groups and (2) they don’t have people experienced with working with the data they collected. The problem is one informs the other, knowing what type of analyses you want to run tells you what kind of people you need and the expertise they need to have. Having a clear picture of what you’re looking for, even if it’s some wild ass guess, means that you have at least some vague end point.

There are numerous ways you can “interrogate” a dataset and while the experience problem can be solved relatively easy, the fact that there is no clear outcome measures means that you don’t know how many people you need to have sufficient statistical power, you can’t even guess. You just pick a number and pray to the statistics gods that it’s enough when you finally settle on an analysis. You also can’t ensure that the data you are getting is good data.

What makes good data? Well data that is high signal to noise ratio for one. Signal to noise ratio is just as it sounds, it’s the ratio between the thing you are measuring and the noise that inevitably finds its way into the measurement. It’s literally signal/noise and if signal is much larger than noise then you’re good, heck even low signal to noise (EEG for example) data can be useful as long as it’s collected properly and the proper steps are done to ensure you maximize the signal to noise ratio. Quick example of how signal to noise ratio works, say you have some data and you know your signal is 10 made up units and your noise is 5 made up units your signal to noise ratio is just 10/5 or 2 and that’s really good in terms of EEG, but ideally you would have that number be much larger.

There are two ways to fix that problem, one is to get more invasive. The closer you are to the thing generating the signal the better, so electrodes directly inserted in the brain (e.g., neuralink) has a high signal to noise ratio compared to EEG because EEG we’re measuring much, much further away. Since we can’t just hammer some electrodes into people’s brains (as much as I really want to!) we have to do the other thing to improve our signal to noise ratio and that’s reduce the noise.

In the EEG world we do that by making sure our sensors have low impedance. Impedance is just how difficult it is for an electrical charge to move across something and the higher the impedance the more noise we will end up with because the signal can’t reach the sensor. We use a conductive gel to bridge the gap between the senor and the scalp, but adding it is very much a learned skill and while you can check impedance, if you’re bad at applying the gel (we use a blunt tipped needle and syringe to add gel to each and every sensor individually) then you will be stuck with high impedance and left wondering why or blaming the equipment for a faulty reading.

So you can see the cascade effect here, because the experiment was designed poorly, the experience needed for the study wasn’t well defined. Thus impacting the quality of your data and the subsequent analyses that follow. And let me be painfully clear here, there’s no substitute for good data, so even if they forced their way through the experiment and got “some sort of data” even if they recruited the best people to do the data processing the world has ever seen, you’re more than likely going to end up with absolutely nothing. You can’t get data from noise, it just doesn’t work.

So friendly reminder to make sure you take the time to design your experiments properly.