We're a little crazy, about science!

It came from the fifth dimension!

Programming, you either love it or you hate it. Okay, let’s be honest, you either hate it or you tolerate it. There’s just so much to keep track of and as you start tacking on more and more, it can quickly become overwhelming. What’s worse is when you don’t have a whole lot of time to make sense of it all, you just push ahead as quickly as you can. Something has to give, it’s just a matter of what.

It’s been an interesting couple of days. I’ve been chipping away at some data processing and once again I find myself having to work in higher dimensions. I hate having to do that, but here we are. While this is a story about working in higher dimensions (and the pain that comes with it), it’s also a story about something called technical debt and how we have to come up with ways of paying that debt when the time comes.

Recently I collected some data for an idea I had, my “big idea.” It’s been a very exciting time and there’s been some interesting developments, but a lot of the progress I’ve made came to a halt after hospital-PI left for vacation. That’s because I crammed several weeks/months worth of work into a few days. It was a few sleepless nights there, but I got a rough answer to one of the simpler questions we could answer with our data, and I don’t regret it because it was a very exciting result.

As a consequence my code and a lot of the stuff I was churning out wasn’t good. It was enough, it was correct, but it wasn’t polished. It wasn’t something we could publish to say the least, but it was enough to use internally so we could figure out what we had and the exciting possibilities for future work. That’s where technical debt comes in.

Because I was speed running my code I didn’t give myself a lot of time to think about the best way to do something, instead I opted for the first way I could come up with. Hospital-PI has been out for almost two weeks now and there’s been minimal forward progress because I’ve been paying back that technical debt. I don’t like to let it build up because it always needs to be paid one way or another. Technical debt is the corners you cut, or the bad code you write. The code without comments or just repeating the same few things to get from A to B when you could get there in a much more efficient or compact way.

While I could live with it, it severely limits what you can do and how you can reuse the code. So either I pay it now and clean up my mess, or I pay for it later when I need that code to output something more polished (say for a paper). In an effort to standardize some of the new plot styles I’ve made for this project I’m writing several functions to make it easier for me in the long run. I can just call the function and with one line of code create a very complex plot.

Which brings me to the second point, higher dimensions. I ask a lot of my functions. When I create one, it needs to be well documented (inputs, outputs, and the stuff in the middle), but it also needs to be very flexible. For example, one function I wrote for what I was calling “last paper” was a way to determine significance. I made sure to write the function to be flexible enough that I could reuse it and I have already reused it for several other different projects (and I plan on using for this project).

Putting that extra effort in means I can save time in the long run or I would have to do the same thing repeatedly for each project. So I’m trying to put that same effort into this function I’m trying to write. The catch is I’m going to be asking the function to do a lot of stuff, like a lot, a lot. So I’m anticipating some of the use cases, one of which is the five dimensional case (surprisingly that’s the highest I need to go for this one). And it’s giving me some headaches wrapping my brain around how to get my code to use lower dimensional and higher dimensional cases.

The problem is each dimension represents something. It could be amplitude, it could be time, it could be simply the sensor channel. But at the end of the day the order that those attributes are set by the person using the code… that would be me in this case. As the number of dimensions increases, you need to keep track of those attributes because maybe you want to average across time instead of amplitude, or maybe you want to average over sensors, or power spectrum, etc. so order matters. If I take the average of the first dimension when I really mean the second, I could (in some cases) get a solution, but it wouldn’t be the solution I wanted.

I’m hoping to get it all figured out over the next few days, but it’s a process and there’s a lot of stopping and checking. I’ve also realized that I need to adjust a few things in my code that I didn’t anticipate because I want to add all this extra functionality so it’s a lot of work that I’m trying to hammer out so I won’t have to do it again. One day I’ll look back and be thankful I did all this, but right now I’m just trying to get it done.

2 responses

  1. I’m trying to think how technical debt tends to manifest at my job. I’m sure we have an occasional issue with code being inefficient or incomprehensible, but we also get things like this:

    “We didn’t build enough prototype hardware, and now the engineers are taking forever to debug the design because they’re all fighting over one test system.”
    “Somebody designed an interface with very marginal timing, and now the Worst Case Analysis is taking weeks.”
    “We mandated use of a big code block in this FPGA because it’s one we’d used before and we didn’t want to prove out a smaller one – but now the FPGA is taking forever to build because it’s too full, and debugging is very slow.”

    It’s extra frustrating because the decisions that create the debt usually get made by somebody else, and then I’m one of the people paying it off. Haha.

    Outside work, sometimes I feel like the Acuitas code base is one big rolling ball of technical debt. That’s an interesting situation because I’m sort of feeling my way through the dark, so there’s almost no way to “do it right the first time.” I write a new module and only then do I see how an older module ought to be transformed to support it. So there’s a constant frantic balance between jury-rigging in new things and overhauling old things, and some part of the code is always semi-obsolete or a mess.

    That’s me managing myself, and who knows whether it’s the ideal way to do things or not. But at least there, I’m the only person I can blame.

    Liked by 1 person

    July 27, 2022 at 12:24 am

    • Yeah I don’t know how you survive in an environment like that. I get annoyed too easy to be useful in situations where you’re forced to fix others messes. I mean I guess I could do it, but it would be frustrating for sure.

      I don’t think Acuitas counts! As you get better writing the code I’m sure you’re going back and changing things, but it’s like (pardon the pun) a living thing, so it’s always changing, even if parts of the code are a work in progress or parts you just ignore. In any case, it’s still incredibly impressive.

      Liked by 1 person

      July 28, 2022 at 1:07 pm

But enough about us, what about you?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.