Forward progress

I don’t want to say I’ve learned a lot in the last few years, but I have to say, I’ve learned a lot in the last few years. Sometimes you don’t realize the progress you make until you find yourself in a situation that would’ve been overwhelming or even impossible before and you are (not so) suddenly able to handle it without too much of a problem. Take for instance my dissertation work. When we last discussed (yesterday) I wanted to process ALL THE DATA!!!! But realistically I thought I would get a handful of datasets done, 3 to 5 maximum. Well…

It looks like I should have all the data processed before this weekend, when I plan to dedicate some time to make sense of it all. Yeah, you read that correctly, all of it. I’m not exactly done just yet, but after the last two days I’ve made some serious progress, enough that I feel confident I’ll be done before the weekend hits for sure. But I guess I’m getting ahead of myself, so let’s back up just a bit and talk about what I’ve done so far and what remains.

So I tend to break my pre-processing steps up into smaller steps. You don’t need to do this, if anyone is interested, but I like having bite sized steps that I can work through. It makes things easier for me to follow along with. It’s especially helpful when I want to use the assembly line approach like I’ve done in the past, everything goes through one step, then the next, until it’s all done and ready for analysis. It’s just what works best for me.

When I started in the lab, I processed my qualifying exam data in about six months, maybe (definitely) longer. The paper I recently had published on TSS and EEG (this one) took much, much longer because I went from an n = 1 to an n = 5 x 2, as in 2 datasets per person, so 10 datasets total (I should’ve gotten my PhD for that paper!!!! ugh!!!! Okay, I’m just being a bit dramatic). I spent a good year or so just trying to figure out the best way to process all that data and make sense of it. It was years (with an s) long efforts to get that paper done, most of which was spent working on processing the data. Now I can do things much quicker, how quick you ask?

I’ve got all ten of the datasets processed to the last two steps, ICA (which takes a long time) and aligning the EEG and EMG together (which takes a certain amount of time, but it’s not too bad). I got all that done just yesterday. Today I’ve managed to finish three of the datasets completely and I should be able to finish three or four more tomorrow. I started with the early datasets because those were going to be the hardest to get done, especially the alignment part.

It’s not that aligning data gets easier as you go, it’s that you have “flags” you add to the data, each flag is a comment with a latency (time) and they are separate from the data, but with it you know at t = whatever you had that flag happen. Sometimes we pre-define them, sometimes you add them as you’re going. I tend to “over add” because data loses context after its collected. You have a huge stream of numbers and if you don’t have exact recollection of what happened or what went wrong (specifically what went wrong) at particular times in the data, then you may be confused because something doesn’t make sense or something looks wrong. When really say the reference electrode for the EEG came off (an actual example of something that happened in my dataset, and yes I made a comment in my data for it). Being consistent about how you comment your data speeds things up in the aligning step, but consistency happens with practice, so the earlier datasets aren’t as organized as the latter.

Anyway, I’ve managed to get the last three pre-processed despite the differences in the way I put the comments in and how I ran things. It was some work (a lot of work), but it’s done and now I can finish the rest. Technically speaking one dataset (the pre-processed one I used for my presentation last Friday) isn’t even counted in this, but it could be I guess since the EMG is already aligned and finished. I’m going back through it one more time to re-process it since I think I could do a bit better with it (it was a rush job anyway), but that doesn’t change anything on the EMG side and so technically it’s finished as well.

The past two days have been a good reminder of the progress I’ve made since I started in the lab. Regardless of what happens with the data (success or failure) I have managed to learn a lot and sometimes it takes things like the last couple of days to remind me that I’m not the beginner I once was. Now if you excuse me, it’s been a long day and I would like to sleep.