The mysterious data
I sometimes miss the days when the answers were in the back of the book. At this point I would take even just having answers that are semi related to the questions I’m trying to work out. Being a researcher is a double edged sword. On one side, for a brief moment in history you will know something that no other person in the world knows. On the other, how do you know that it’s correct? Questioning your results is an important part of research and right now, there’s more questions than answers.
You learn as you go to school, it’s true, but some of the things you learn aren’t exactly the things you’re taught. I was taught that you do an experiment, you get a result, you add to the body of knowledge that exists. The idea is that others try to do what you did and it verifies that you got the correct (or close to the correct) answer. So we build a consensus in the direction of the correct answer while never actually hitting it exactly.
Sure certain things we can say with overwhelming certainty, man-made climate change is real, smoking dramatically increases your risk of cancer, black liquorice is disgusting, but some things are more limited. For example, do sugar substitutes harm the body? I tend to think no, but only because there’s no clear consensus one way or the other. This is partly because of confounding factors and the effect size is probably incredibly small, which is why I don’t worry about it. After all if the effect size is that small then there are other health issues that I should be worrying about!
That was what I was taught, but what I’ve learned is to question my results. I think that’s important to talk about and we don’t do it enough. Big idea for example (here) will need a lot of checking and rechecking before we’re sure of our result. That’s why we replicate the experiment with a large number of people (for our work anywhere from 10 – 20 people, but for things like fMRI it’s even higher). We’re building a mini-consensus within our little lab, but that doesn’t preclude systematic errors, code errors, or just general processing errors. We’re all only human after all, so mistakes happen even when we don’t want them to. I’m sure there are dozens of spelling errors on my blog despite having a built in spell check in my browser for example.
So having healthy skepticism about your result is important and over the years of doing research I’ve found I would rather be on the “too skeptical” side, which compared to people like hospital-PI is still not skeptical enough, but compared to school-PI I am far too strict. It really depends on the person and their aversion to risk, I certainly wouldn’t want to put big idea out into the world only to find out that it was a big flop after the fact. That doesn’t mean “little ideas” don’t deserve the same reverence and caution, but at the same time, it’s important to acknowledge that the stakes scale with the potential outcomes.
For the past few days I’ve been working with some of the data we’ve collected for “big idea” and hospital-PI yesterday let me know that he wants to show off our result at the beginning of next week. If you missed yesterday’s post, I’m running under the assumption that there are potential sources of funding for the project involved or we wouldn’t be showing results so preliminary. Hospital-PI is even more cautious than I am with claims so I assume he wants to basically demonstrate feasibility since that’s the level we’re at right now.
For the past two days I’ve been processing the data. Then reprocessing the data. Then reprocessing the data. You get the idea. Since the recordings are incredibly valuable (see: rare and hard to get), we need to be very thorough about how we process the data and work with it. With the rise of computer power we can use super complex algorithms to process the data and remove noise. For instance, I had a ton of line noise, 60 Hz (in the US) electrical noise that gets picked up by the sensors. It’s a common issue, we see it all the time and there are dozens of ways to remove it. For this project I’ve found a better way to do it than my school lab does it, so I’m going to be passing along that info soon so they can adopt it to the processing pipeline we use.
Most of these algorithms have variables that need to be tuned to get the best result. That’s not an automated process because it can’t be, at least not that I’m aware of, so there’s a lot of data processing, then manual inspection of the data after the fact. In our lab, before you run a step in MATLAB (our preferred software) you make a copy of your data in your workspace and then you can compare the filtered data to the previous step. You end up with about a dozen copies of the same data that are at various steps of the processing pipeline you use, but it makes life easier because you can go back one step, multiple steps, or back to the beginning without having to reload data.
Normally our pre-processing pipeline is 90% fixed. We have step 1, then step 2, then step 3, etc. and those steps aren’t changing. We do it that way because hundreds (if not thousands) of people have come before us and found the best way to clean the data we’re working with. Now while the pipeline is fixed, we still have to adjust those variables, but there are somewhat tight tolerances and “best” ranges that we know. One filter for example we use in our lab I barely ever have to adjust the values for because the values to filter the artifacts it filters are the same across subjects (typically).
With this new dataset, I not only have all those variables that I need to adjust, but I also need to see what happens when I mix up the pipeline to see if I get a better result. That was a realization I had this morning when I realized I had assumed the pipeline we use is the best for this new type of data. As someone who hates making assumptions, when I realized this I decided to switch two steps and sure enough it yielded a better result. I also dropped two of the steps and that solved some issues I was having as well (sometimes filters can be unstable and actually add in noise to your data, fun fact).
This is why it pays to be skeptical. If I hadn’t been, I would’ve gone through blindly, filtered my data until it looked correct, and moved on. But after looking at the power spectral density of the data, they did not look correct. That was the first hint that I had a problem. Now the data look somewhat better. Mostly better… still weird, but better if that makes sense. It’s closer to what I would’ve expected to see which is a relief, but now I’m questioning if that data look weird because that is just how they look (as in it’s correct and I’m just seeing something new and confusing to me).
There’s still a long road ahead for this project and I won’t be publishing my results anytime soon, even if I get more data, there’s still too many checks to do and things to look at. That’s the problem with working in a new space, there are no answered questions so every question you could think of could potentially be answered. So the question becomes, which question do you start with? I have a select few I would love to start out with that I think will make for an interesting splash into the field, but there are a lot of other questions we could answer too and I think that’s part of the fun of it. While I’m still a bit skeptical about what we have, I know we have something and that’s already a really big hurdle to overcome.
One thing’s for sure though, it’s been one hell of a week and the week hasn’t even started yet!