The curious case of something, or nothing?

It was such an obvious gap that I didn’t understand why it was there. Four years ago, almost exactly, I realized that when it came to the research our lab was doing, brain-machine interfaces mostly, that gap existed, I was confused. Surely I was mistaken, but in-depth searching turned up nothing. Coming in I thought the field was mature, everything that could be discovered was and now it was a slow slog of incremental improvements. Yet there it was staring me in the face like a red car in a black car lot. And thus super secret technique (SST) was born.

It’s been a slow crawl from the first days of the realization. I quickly set out to try to determine if this would actually work or if I was just wasting my time. In record speed I did my qualifying exam (QE defined now, since I use the acronym later) and while it was an n = 1 study (see, no one would care about it no matter how great the result), I demonstrated feasibility of SST. The results were stark because it worked well in one case, but not another. That was most likely my fault and I know why it failed, but that’s good news because if it had worked then I would have other issues.

Still, finding funding was an uphill battle and it wasn’t until last year (a year, again almost exactly) that I managed to secure funding for the project (here). It’s been a whirlwind of interviews, awards, and all sorts of things I never imagined since that day. Heck, even DARPA took interest (here), which is probably the most terrifying part in this whole ordeal. SST was finally getting noticed, but the problem remains an n = 1 study isn’t proof of anything, it’s barely good enough for feasibility. I’ve been torn between feeling like I’m going to do something amazing and preparing for the epic failure that could come out of this. I can sum it up with this meme…

SST was always high risk and high reward. If it succeeds we can do some really cool stuff across multiple different fields. It’s very exciting in my opinion because it opens the doors for a lot of discoveries that I think will lead to a huge change to the way we do certain things. If it succeeds it will also be a good reminder that you’ll never find something if you don’t bother to look. I’ve already been told what I want to do is impossible, but that’s just because no one has tried it. It’s a case of, when I talk about it, it really does seem so obvious that it would be odd for no one to have already tried, so people assume someone has, failed, and since there’s no journal of null results (aka, journal-o-failures) there’s no publications.

I’m approaching it from a different angle. I think that SST was overlooked because other more interesting stuff was there and people had a set of misconceptions about how the central nervous system worked so SST wasn’t of interest. Plus computers for brain-machine interfaces (BMI) have only recently (let’s say past 20 years) become common so we could do this type of research. In my mind, it’s a case of assumptions built on assumptions. Everyone in the room assumes the other person already tried it, so no one tried it.

I LOVE challenging assumptions, because that’s where the fun science lives. I’m doing something similar at work with hospital-PI and surgeon-PI and while we’ve only got one dataset (sound familiar? this is not big idea, something totally different), we had found something that was the “surprising” result, so I’m excited to continue that course of exploration. But that’s a whole other post after more data has been collected. Point being, I like the high risk/high reward projects because the risk is artificial. The risk is simply finding nothing surprising and that’s confirming the science we’re building on. While not publishable, it’s important and I think poking the foundations of science must be done if we’re going to build anything of value.

So SST has been low stakes frankly. The rewards to the people this could benefit are huge, but the cost of failure is what, my PhD? Oh no, please don’t give me that thing I’ve worked so hard for… see what I’m saying? Of course that was before DARPA got involved and labeled me as someone to watch, while failure doesn’t normally scare me, failure like that would be embarrassing. So the risk went from no cool publication attached to my PhD to abject embarrassment in front of a group of people who wouldn’t normally look at me, much less my work. I’ll refer everyone to the meme above for those wondering how I feel about this.

Which is a long winded way to bring us to the topic of the day, but here we are. The curious case of something, or maybe what I have is nothing?

Since those early days, I’ve collected all the dissertation data! Okay, the first phase of dissertation data. It was a lot, but I did it and in record speed. So fast that even school-PI was shocked. He emailed me a few weeks ago suggesting that I stop at the halfway point just to check the data and I responded with that would’ve been smart, but I collected all the data already). So now I have the data, I have (and it pains me to admit this) the skills to work with it, and while I don’t have a lot of time, I don’t need to have it all done. So I set out to process just a single participant’s dataset last weekend.

There was a ton of leapfrogging over errors to get it done before last Monday. I mean I was coming up with solutions as I was going to problems that I really needed to stop and fix. It was a compounding issue that just built up as I went, but damn it I did it and I created “the plot,” which is the thing that would at least tell me if my QE was correct because the thing we (I) found there was so obvious that if it were true I couldn’t miss it in another dataset (assuming SST is generalizable). And ended up with what I thought was nothing. Sad day. So I checked again, and again, and again, nothing.

Okay, I made one plot and I thought I had something. When I saw it I cried tears of happiness until I realized it was an obvious error which I caught really fast in my code, then there were tears of terror. It was an error and the result looked so similar to the original data from my QE it scared me to the point of being wide awake and this was at roughly midnight. Thankfully, I checked the original data to make sure I didn’t make that error when I processed the QE data. I hadn’t, again thankfully, and upon closer inspection the two plots (the new data and my QE data) looked similar, but not similar enough to be the same thing. I was safe, at least in that regard. My QE data still holds up and I cannot find the error (if one exists), so I was at a loss to explain the clear result of that data and the miserable outcome I had now.

So yesterday I went through all the code and fixed the mistakes I had made. I’m still doing a very rough analysis, meaning nothing that would be published for sure and nothing that I’m proud of showing, but it’s good enough for the presentation at least. And to verify that I had done the correct analysis over the weekend, I produced the same plot… and got the same result. Bummer.

But that was just one of many conditions, so I checked a few more of the “easy” ones. Still nothing. Extra bummer. Then I checked the one that would be the most contentious out of all the data I had collected for that first set of conditions. Bingo, found something, of course it had to be THAT condition, the one that everyone would chalk up to something else. I know the argument coming and it will happen, that I’m sure of. But keeping a skeptical (but open) mind I pressed ahead and tried to build a case for my result.

Which I’ve (mostly) done. I’m working now to finalize some things, but it looks like I have something for at least one set of conditions for this particular participant anyway. Now I need to build my case for why we have something here, but nothing for the other cases and that’s where we get into a bit of a bind. I have a good explanation for it, but it’s not perfect. So instead of trying to fight why we didn’t find anything in that particular case, I want to spend my time arguing why what I found wasn’t the “obvious” thing that people would assume it to be.

That’s the goal for today and frankly I’ve done most of it yesterday, so today I need to finalize the figures (even if they are ugly) and put together my powerpoint, because tomorrow is the big day. I wish I could get a second dataset processed, but at this point I think it’s not going to be possible. There’s just not enough time, if there was I would’ve preferred to go back and reprocess the current dataset to see if I could clean up some of the noise that I have with the other conditions that “didn’t show anything.”

So yeah, for now I think I have something, or maybe it’s nothing. And that’s why n = 1 studies are so hard to deal with, I’ve gone from one n = 1 situation to another n = 1 situation and I make my way though this data. The good news is that now it’s technically an n = 2, but I would prefer to keep them seperate in my head since they are two different experiments at two very different points in my life. However, the fact that I found the same thing in both is promising, even though I got nothing for some of the conditions.

A more detailed analysis will help resolve some of that, but it will take time and effort to get to that point and I really don’t have that in me right now, there’s just no time. Tomorrow is presentation day ready or not. I’m ready of course, but I don’t have everything exactly the way I want it quite yet. At least when surgeon-PI comes to visit I’ll have a more cohesive story and this bodes well for the DARPA presentation, so fingers crossed.

This weekend I’m going to pause the analysis of this dataset to get started on at least a few more so I can have something to compare my results will. Some of the data was collected better than others, so that’s the only scary part about the whole situation, you get better as you go and oh did I have a rocky start. For now though, I need to just focus on getting things ready for tomorrow, so once again I refer you to the meme above.

I did explain that bringing ideas to life was a violent thing.