We're a little crazy, about science!

When the assumptions fail

We make simplifying assumptions all the time. Assume the cow is a sphere. Assume a frictionless surface. Assume no air resistance. If you’ve ever taken a physics course, you’ve run into these at least once. Simplifying assumptions work because they get us an answer that is close enough to the “true” answer that we don’t need to do all the extra work of finding a more complex high accuracy solution. We like assumptions because they make our lives easier! But what happens when your simplifying assumptions turn out to be incorrect?

The task was simple, fabricate four new pieces of equipment. Well, simple enough. Mostly it was a giant soldering project, not unlike the dozens of other soldering projects I’ve done in the past. This was just extra tedious and time consuming, so I got to work and about halfway through, I realized I made a horrible mistake. I made a simplifying assumption that I didn’t know was true.

But I quickly connected some things, checked the outputs on a few of the connectors, and everything looked good. It appeared that my simplifying assumption turned out to be correct. Before we get into what that assumption was, I want to make it clear. I checked three of the parts I had made and in each case my random output tests (I did not check each individually that would take a long time) all proved that I was right, tested every input lead to the expected output.

The assumption was simple. The wiring across parts was the same. The parts were all the same and they were all manufactured from the same company, so the assumption was one I felt comfortable making because why wouldn’t they all be wired the same? You would want to standardize across parts so it felt like a no-brainer to me. After testing a few of the connections I was confident that this was true. In each case, the input lead to the expected output, so the continuity held no matter which part I tested. Thankfully I’m an idiot.

Normally I don’t like being an idiot, but today it actually saved me. Okay, it wasn’t the only thing that saved me, but it helped. See when we solder stuff we typically put heat shrink on the cables. Since I was soldering dozens of wires, I like to hide all that mess by adding one large piece of heat shrink over the top. It makes everything look nicer, sort of like how we store dishes and what not in cupboards because having them out in the open just looks messy.

So the story starts off with me being an idiot. I put the wrong size heat shrink on my cable before I started. See, the heat shrink fit just fine on the cable itself, but with all those extra wires soldered together, the diameter of the area I soldered was slightly larger than the diameter of the cable, so much so that the heat shrink that I put on the cable at the start of the project didn’t slide over the mess of soldered cables. This was the final part I needed to make and the first and only time I made this mistake, it’s always the last one, always. So, I had two choices at this point.

The first choice was just cut to it off and leave it. Afterall, who’s going to notice? Who’s even going to care? It’s research using custom made stuff, no one will judge. The second choice was to pay the idiot tax. I wasn’t about to cut the cables and resolder them together. That would’ve been too much work and the solder job was too fine, I wouldn’t have enough cable space for it all. So instead the second option was to desolder the connector at the end and slide the larger heat shrink tube over the end (I tried to do it without desoldering, it was millimeters too big to work. Just barely… but again I’m an idiot).

Well I may be an idiot, but I’m also someone who likes to do things right and leaving it without a heat shrink cover just wasn’t an option for me so I took out a piece of paper, jotted down the color code for each of the connectors so I could put it back together afterwards (not THAT big of an idiot, thank you) and got to work. After putting the heat shrink on without too much trouble, I started soldering the wires back to the connector, but I had a nagging feeling I was missing something so I took apart the connector to one of the cables I had soldered together to check it against my pinout diagram (which was really just a list of colors, nothing fancy).

Everything matched, except for four of the wires, they were flipped so instead of red, green, blue, black, it went black, blue, green, red. Odd… okay maybe I made a mistake writing out my pinout diagram, let me check a third. So I grabbed yet another part that I’ve made at random, there were only four, so it wasn’t too random I guess. Once I got it open I saw that it had the wiring matching the list I made.

Oh no.

So I powered through, soldered the connector the way I had originally written it out and set out to determine just how big of a problem this would be. There were about a dozen things I soldered together each with more than a dozen wires, you do the math, and it turns out they were all basically the same layout… basically, but not exactly.

I’m not sure why, if it’s an assembly error or intentional, but the same flipped color scheme for those four wires was found randomly across my parts. Internally they were consistent, so if one connects to two, meaning 1 —> 2 the wires were the same on 1 as they were on 2, so internally they were connected “properly,” but when I had piece 3, 4, 5, 6, … X and I required they be all wired the same way, well the little thing that wasn’t a problem for the original design suddenly turned into a huge problem. A problem I wouldn’t have caught if I wasn’t an idiot, so who’s the real idiot here?!… Oh right, that’s still me.

The thing is, I didn’t catch it in any of my random checks because they were just four of the dozens of connections I was verifying worked. It just so happens that, by luck or by laziness, I didn’t check those four particular input/output connections. I would’ve gone into long experiment thinking I was perfectly ready and had everything I needed and I would;ve been horribly wrong.

That’s not the scary part though. The scary part is I may not have caught it, even after the fact. We would’ve looked at the data and found something very interesting in some of our readings that didn’t make sense, but would’ve been found on multiple pieces of equipment because we had four of these things I just built and they each had at least one instance of this happening. It’s things like that, that keep me up at night.

Because, unlike our physics textbooks, where some of the answers are in the back of the book and the rest the teacher can give us, in research we’re the ones making the back of the book. We don’t know the answers, no one does. We’re trying to figure that out and it can be a bit terrifying at times having this realization.

And that’s the story of how I almost unknowingly ruined a whole experiment (well part of an experiment). Luckily I caught it! Then again, that’s the catch isn’t it? I only know about the error because I caught it, so what if I missed something else?

Cue the suspenseful music.

2 responses

  1. This reminds me of a time (embarrassingly recent, I think either this year or last) when I got out some stepper motor controllers I had made years ago, connected one to a motor, and it didn’t work. The motor chittered back and forth instead of turning. So I started swapping parts: different motor, same thing, but if I used a different controller board then it turned. I was sad because I thought the first controller had gone bad sitting on the shelf somehow, and that didn’t say good things about the reliability of my boards.

    And then I managed to notice that I had connected a pair of wires from the second controller – the working one – to the motor cable backwards.

    I tried hooking up the first controller “backwards” and it worked too.

    Then the memory finally started to surface: when I designed the boards, I hadn’t done the pin order quite right and those wires needed to be crossed over each other in order to get the right connections to the motor coils. I went and looked at some other motor/controller pairs I had in a project and sure enough, all those wires were crossed.

    I only managed to remember this and not think everything was broken because I accidentally hooked one of the controllers up “wrong.”

    All that to say, we can be idiots together. I guess sometimes you just need a mistake to cancel out your other mistake. Or, to quote a co-worker, “I outsmarted myself. But actually there was nothing smart about it; I outdumbed myself.” I’m glad you caught the problem; that sounds bad. I suppose that’s one reason replication is needed across studies and labs. The chance of two places making the same wiring mistake would have to be pretty small.

    The nightmare I get to have is that we didn’t test enough corner cases and something of mine goes to the customer with a bug in it. Or worse, goes to space with a bug in it. For the reprogrammable FPGAs, we might be able to uplink new images, so all hope is not lost … but it’d still be pretty embarrassing.

    Liked by 1 person

    September 29, 2022 at 1:03 am

    • Ha so it’s not just me! At least the controller wasn’t bad. Also good you didn’t fry something accidentally. Not saying I’ve ever done that of course…

      I’m going to use outdumbed myself often I think. It’s a perfect way to describe what happened. You know people who say stuff like, “that dog is so ugly it’s cute?” I imagine it’s like that, I was being so dumb I was smart!

      Ouch, space bugs. I would probably be that person, which is why I don’t work on stuff like that. Not that my mistakes are any less embarrassing!

      Liked by 1 person

      September 29, 2022 at 8:33 pm

But enough about us, what about you?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.