We're a little crazy, about science!

Dataset mining

We use heavy equipment to mine our data around these parts…

I did an experiment! That’s old(ish) news, but now I have some data to play with… lots of data. So now I get to do something somewhat enjoyable and that’s try to get cool and interesting stuff to fall out of it. All you have to do is shake it really hard and hope for the best.

I’ve only got four subjects worth of data, but that’s still a whole lot of stuff to dig through. Even just a single subject has 10 different conditions and the type of analysis you can run for each of those conditions varies. In fact, you can do a whole lot of different kinds of analysis for a single condition and a single subject.

For now I’ve been doing some of the more basic analysis. Things that don’t take a lot of time, just some code and comparing the response between different sensors (we have 64 sensors remember and there’s a lot we can do with them). We can also look at the big picture and see all of the responses together, but more importantly we can see how those responses change over time.

Long story short I have so many different things I can compare WITHIN a single condition, I feel like I could spend a whole year just toying with this dataset and still not find all the cool things I could pull out of it. Unfortunately there isn’t a whole lot of research done in the area I’m working in. This means I’m stuck fumbling in the dark and guessing at what the more interesting findings would look like and where the may live in this vast dataset of mine.

We’ve already had some interesting things pop out, but that’s for a single subject, single condition, and just a handful of sensors (so many sensors!) and while that’s a great first step, I really want to see what else we can pull from this dataset. In a perfect world we would find something new and strange that would make us scratch our head and question if it was something or if it was just noise.

Right now, we’re basically on the same page that my technique works (finally). I think my Co-PI is still slightly skeptical and that’s how it should be, but I’ve also seen him get very excited about the results I’m sharing with him, which means I’m close to having enough evidence to support that this works.

Since this was a pilot for my pilot (seriously, this was just a small test for the real, hopefully larger, test haha), we’re already planning things we want to change, things we can simplify, and things we can just throw out all together. We’re planning on streamlining a lot of what we’re doing now, the main thing is to prove this works beyond a shadow of a doubt. Or as my Co-PI says, have to “present convincing data proving whatever we see.” In short, we need a whole hell of a lot of data all showing the same thing.

Thankfully we’re getting there with this dataset we’ve already taken the first step to show that we can do this across subjects, an important milestone since my QE was with a single subject. There was no guarantee that we could do this with others, there may have been something unique about him or the experiment could’ve had some systematic flaw that would’ve come out if we had repeated it with others.

For now, I’m going to keep trying to pull every last bit of useful information out of this dataset while keeping my eye on the prize. This could do a whole lot of good if it works (in my opinion anyway). Now we “just” need to show that it does what we claim and find the limits of the technique before we share it. I expect that in the next few months we will be ready to share this with the world (if all goes well) and it’s a little anxiety inducing, but it’s also exciting.

These past few weeks have been… well let’s just say this is exactly what I wanted when I started my PhD.

But enough about us, what about you?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.