To catch an error

Well it’s been an adventure! First we collected some one of a kind OR data, then because of technical difficulties couldn’t get it to open. The support team for the software has been amazingly responsive and we were able to recover the file! Or so we thought, saving the file caused the error to reoccur, so we needed more help. Thankfully because we could open the file, I could debug and now I look like a data wizard. So this is the story of how I attacked and dethroned the data gods.

For months now we’ve been trying to arrange a Frankenstein’s monster of experiments for our trips to the OR. This was a multi-team collaboration that almost ended in disaster on our first successful attempt (here), which was actually our second try. We literally had people who shouldn’t even know we exist trying to figure out what the hell we were doing. We became somewhat infamous from that occurrence, but since then things (on the hospital side anyway) were getting to be smoother.

Our third attempt also ended in failure, when the surgery was called off for things that were outside our control. However, every time we rolled our equipment into the OR we had another chance to practice and smooth things over. Our fourth attempt just happened recently (here) and we successfully collected SOME of the data we were after. It wasn’t a perfect run, but it was by far our most successful trial to date.

Then the bottom fell out so to speak. After the data were saved and we tried to open the file, we got an error. It wouldn’t open, the file size was correct so we knew the data were there waiting for us, but it was out of reach and I wrote a funny, but frustrating tale of that (here). Thankfully the company who made the software we used had outstanding support staff because within days we had sent our data off for repair.

Not too long afterwards the data was returned to us and the file opened, problem solved… or so we thought. Several hours later hospital-PI sent another email to the team letting them know that once he tried to save the file, the error returned and we could no longer open the file. Thankfully I had my own copy of the data that I had not saved over so I could open it and I did a deep dive into determining what was causing the problem.

From the team we found out that the software thought we were using more than 32-channels. Each channel is one data stream we were recording from and we weren’t using 32, much less more than 32. Whatever was going on must have been related to this so I started trying to remove channels, add channels, basically just throwing things at the data and seeing if anything resolved the issue.

Finally I found something happening that was concerning. We had set one of our channels as a marker channel. It was a way to literally mark what was going on in the data when the channel met a certain requirement that we could set. The catch was, we didn’t use that marker channel and no matter what I did I couldn’t get rid of the reference to it. Literally I had five marker channels in the data and six were showing up in the analyzer. This was the source of our problem, now I just needed to figure out how to fix it.

After an hour or two (it was a late, late night) I emailed hospital-PI and let him know what I found, then reached out to the support team to let them know what the problem was. Not too long afterwards I tried something that solved the problem, I exported just the channel data and nothing else. While this was good, we needed the rest of the stuff that wasn’t being saved so I knew the data were good, just something about the way we had the file was causing the problem. So I went back to work.

Somehow after some tinkering I solved the problem… but only once. I saved the file, opened the file and everything was good. As I was taking a victory lap, I uploaded the file to email off to hospital-PI and before I hit the send I double checked the file to make sure it worked and the error reappeared. Sometimes being an anxious person does help!

Fast forward a bit and I finally found the actual solution to the problem! Turns out that somehow we had added a channel 35 during the experiments and set that channel as a marker channel. We must have corrected the mistake by removing the channel while we were doing the experiment, but the software doesn’t delete the data from that channel, so it was just sitting there hidden from us. Once I removed the designation as a marker channel and deleted the first portion of the dataset where we had some other connectivity issues the file could be saved and reopened as many times as I wanted.

I emailed this off to hospital-PI close to midnight last night (again late night!) and got a response early this morning. I’ve officially been given the title of “better than he is at using the software” which is mine until he decides to claim the title back for himself.

Now that we have the data playing nicely, we can get to the interesting part! Looking at the data to see what we can find! It’s a little early, but we now have a fun little gift to play with. YAY!