The long data processing road

Okay I’m taking another break from the “in statistics” posts, at least for today since the next topic is going to be a long multi-day post worth of stuff and I need an easier post for today. I have way too much stuff going on at the moment to organize a long post! In any case, I figure I could give an update to the data I’ve been working with and how that whole mess is going. It’s mind numbing and exhausting work, but not in the way you would think.

Since I’m trying to be better about this a quick introduction. I’m a third year PhD candidate in neuroengineering and I’m studying how the brain and spinal cord communicate. More specifically how that communication changes after spinal cord injury. I have a BS and MS in mechanical engineering. If you think that the transition from mechanical to neuro- was easy because they are both engineering fields, you would be wrong! It was a painful transition, but one I was happy to make and now I get to do cool stuff while building cool stuff.

This has been a long story, first I collected some data back in September-ish (read more). Then of course I had to get everything ready to give my funding source (the people who paid me to do the experiment) an update at the conference back in December (read more). Since the beginning of the year I’ve been doing basically nothing but writing (my R21 grant). However, now that it’s submitted, I’m somewhat forced (here) back to work on this project. Oh and I’m still angry about how my main-PI handled that, but I don’t have any control here.

Every week we have a lab meeting. For the past two weeks, and for this upcoming week at least, I am to present the work I’ve done on the dataset. I wouldn’t call it a punishment persay, but it definitely isn’t me being celebrated! The last first week I gave an overview of what I did with the data, which I was told to make several complex changes to some code that I had. I didn’t write the code, which made it hard, but I did the damn thing (more).

After last week’s meeting I was given a long list of suggested changes to make to the code (yet again) and these changes are more drastic and certainly not easily implemented. However, I finally was told that I could finish the data cleaning … again… for the third time now. The process takes days of manual work, the first few steps are automated and take just as long, but I can set the computer up and forget about it. ICA cleaning (read more) can also be automated, but it isn’t great so we do it manually in our lab (for the most part anyway).

I spent all of yesterday working on the dataset I collected and I got roughly a quarter of the way through, that was roughly 9 hours worth of work. So a total of ~36 hours to do the full dataset… again… for the third time. Can you tell I’m really looking forward to doing all this? Well I’m doing it to put off trying to edit the code, which I am hoping if I get that done my main-PI won’t be too upset for putting that off. I’ll give editing it a shot, maybe it will be easier than I think, but I really don’t see how that’s possible unfortunately and frankly there’s a good chance I won’t ever use these images for anything other than to show him.

Anyway it’s time to get back to working on that data. I hope to get about half way done today or more if I can help it. I only have a few days to get this all done so I’m once again working against the clock.