It came from the fifth dimension!

Programming, you either love it or you hate it. Okay, let’s be honest, you either hate it or you tolerate it. There’s just so much to keep track of and as you start tacking on more and more, it can quickly become overwhelming. What’s worse is when you don’t have a whole lot of time to make sense of it all, you just push ahead as quickly as you can. Something has to give, it’s just a matter of what.

It’s been an interesting couple of days. I’ve been chipping away at some data processing and once again I find myself having to work in higher dimensions. I hate having to do that, but here we are. While this is a story about working in higher dimensions (and the pain that comes with it), it’s also a story about something called technical debt and how we have to come up with ways of paying that debt when the time comes.

Recently I collected some data for an idea I had, my “big idea.” It’s been a very exciting time and there’s been some interesting developments, but a lot of the progress I’ve made came to a halt after hospital-PI left for vacation. That’s because I crammed several weeks/months worth of work into a few days. It was a few sleepless nights there, but I got a rough answer to one of the simpler questions we could answer with our data, and I don’t regret it because it was a very exciting result.

As a consequence my code and a lot of the stuff I was churning out wasn’t good. It was enough, it was correct, but it wasn’t polished. It wasn’t something we could publish to say the least, but it was enough to use internally so we could figure out what we had and the exciting possibilities for future work. That’s where technical debt comes in.

Because I was speed running my code I didn’t give myself a lot of time to think about the best way to do something, instead I opted for the first way I could come up with. Hospital-PI has been out for almost two weeks now and there’s been minimal forward progress because I’ve been paying back that technical debt. I don’t like to let it build up because it always needs to be paid one way or another. Technical debt is the corners you cut, or the bad code you write. The code without comments or just repeating the same few things to get from A to B when you could get there in a much more efficient or compact way.

While I could live with it, it severely limits what you can do and how you can reuse the code. So either I pay it now and clean up my mess, or I pay for it later when I need that code to output something more polished (say for a paper). In an effort to standardize some of the new plot styles I’ve made for this project I’m writing several functions to make it easier for me in the long run. I can just call the function and with one line of code create a very complex plot.

Which brings me to the second point, higher dimensions. I ask a lot of my functions. When I create one, it needs to be well documented (inputs, outputs, and the stuff in the middle), but it also needs to be very flexible. For example, one function I wrote for what I was calling “last paper” was a way to determine significance. I made sure to write the function to be flexible enough that I could reuse it and I have already reused it for several other different projects (and I plan on using for this project).

Putting that extra effort in means I can save time in the long run or I would have to do the same thing repeatedly for each project. So I’m trying to put that same effort into this function I’m trying to write. The catch is I’m going to be asking the function to do a lot of stuff, like a lot, a lot. So I’m anticipating some of the use cases, one of which is the five dimensional case (surprisingly that’s the highest I need to go for this one). And it’s giving me some headaches wrapping my brain around how to get my code to use lower dimensional and higher dimensional cases.

The problem is each dimension represents something. It could be amplitude, it could be time, it could be simply the sensor channel. But at the end of the day the order that those attributes are set by the person using the code… that would be me in this case. As the number of dimensions increases, you need to keep track of those attributes because maybe you want to average across time instead of amplitude, or maybe you want to average over sensors, or power spectrum, etc. so order matters. If I take the average of the first dimension when I really mean the second, I could (in some cases) get a solution, but it wouldn’t be the solution I wanted.

I’m hoping to get it all figured out over the next few days, but it’s a process and there’s a lot of stopping and checking. I’ve also realized that I need to adjust a few things in my code that I didn’t anticipate because I want to add all this extra functionality so it’s a lot of work that I’m trying to hammer out so I won’t have to do it again. One day I’ll look back and be thankful I did all this, but right now I’m just trying to get it done.