How to Fix Things

There’s a clear philosophy to fix anything successfully. These are near-universal approaches to just about any technical/interpersonal matter. This idea goes beyond break-fix and repair and can apply to any domain where change is necessary.

There are implementation- and component-specific details (such as those for computers), but that’s not the point of this essay.

The steps may sometimes only take a few seconds, but every fix employs a relatively straightforward, mundane procedure:

  1. Analyze and identify the purpose that makes the thing “broken”.
    • Consider the chain of events that creates the thing.
    • Observe what exists carefully, and what is precisely happening.
  2. Investigate the obvious things that could be wrong.
    • Poke at it to find out more information.
    • Reposition perspectives to find even more information.
  3. Work the edge case if Step 2 failed.
    • Investigate complexities to find anything weird.
    • Presume harmonizing issues to find really weird patterns that may line up.
  4. Repeat things to be sure you’re not wasting resources with the fix.
  5. Get the supplies and visualize the fix.
  6. Repair and observe what happens.
  7. Clearly document things that happened and what you did.

It sounds easy enough, but there’s more to it.

It’s all networked

Imagine a diagram of a network with nodes and connections between them:

This network can represent anything that accomplishes a defined purpose:

Reality is frequently atomic, where every object is the combination of smaller objects. Those smaller objects are the combination of even smaller ones, and so on.

Except for the specific neurodivergent state of autism, most intuitive thinking works in reverse to nature. We imagine the object as a collective unit first, then work downward into details.

Experience allows people to know which of those details affect results and which details are irrelevant:

  • A lightbulb will fail before a wire
  • Fuses fail before cables
  • Alternators fail before starters
  • Networks fail before software

In the above example, if node 3 fails, the whole system may go offline, which may make it “broken”. However, the entire thing except node 3 is perfectly fine. It only fails its purpose when someone expects it to do something that passes through node 3.

We can typically salvage the object if we mend or circumvent the broken component.

Therefore, troubleshooting is knowing where the broken node is.

This isn’t always easy. There are often networks or nodes beyond the current known network. This is a massive reason why great technicians constantly ask, “why?”

1A. Analyze the desired purpose

A network is the “state” of reality, but living beings ignore information dissociated from a predetermined purpose.

“Broken” is a misleading idea. Its best definition is “current events with the object haven’t satisfied the user’s prior experience of cause-and-effect expectations“.

At the beginning, the user doesn’t know what’s wrong. The thing created a result, but now it doesn’t.

Previously known experience is highly useful because it focuses the scope of what could be wrong:

  1. It specifies exactly what someone was doing when the thing broke, which creates a “start” point.
  2. The desired result becomes clear, which creates an “end” point.
  3. Those “start” and “end” ideas together mean all possibilities are “chains” between “start” and “end”. Extra possible avenues branch into separate chains.

To avoid any risks of misunderstanding, every single person who steps in to fix anything should be keenly aware, to the point of feeling it, of what the problem precisely is.

1B. Consider the chain

With the above diagram in mind, imagine someone saying, “I tried to make 8 happen, but it’s not doing it”. Without knowing which node they’re observing, it could represent any node on that network. The story is entirely different when the statement is, “I tried to make 8 happen through starting 15, but it’s not doing it”.

The first thing, more than anything else, is to find out the chain’s limits.

  • If the action is going to 8, and starts at 16, there are only a few possibilities (16, 8, or the 16-8 junction).
  • If there are many possible connections, the cause-and-effect relationship will be harder to deduce.
  • Each node exponentially adds more possibilities (as a matrix of known issues):
    • A = [A] = 1 possibility
    • A-B = [A,B,A+B] = 3 possibilities
    • A-B-C = [A,B,C,A+B,A+C,B+C,A+B+C] = 7 possibilities

The nodes will behave differently between things and people.

  • Things give instant feedback or, when there’s a time delay, a predictable response when timed.
  • People are complex networks in their own right. Every system test or query involving people carries the risk of changing the outcome or the network altogether.

Most issues are simple enough for this approach. More complex issues require careful perception and a shifting perspective to see whether things work.

1C. Observe everything

If you must deduce smaller links in the chain, draw from several approaches:

  1. Use your intuition and experience to check what is most likely to fail.
    • Bias easily sways this with experience. There’s no reason to investigate an obscure cause simply because you read about it in school. While this mostly requires hands-on experience, it is the quickest way to diagnose things.
  2. Test something in the middle of the chain.
    • If your test can slice the chain of events in half, you’ve exponentially reduced the number of possible problem areas (X/2 => √X).
    • To be sure, test both sides of that chain, since it might be two issues!
  3. Turn off every single feature or option and see what happens. If it doesn’t work, your chain has become much smaller. If it works, sequentially turn features back on until you see something stop working.

Pay close attention to every little detail. The issue could come from anything unusual (e.g., minor scuffs, a strange prompt, an unusual statement).

When a breakdown poses severe risks, it’s not uncommon for bias to blind us to patently obvious things.

  • We tend to imagine adverse consequences of a sustained failure, and it’s wisest to kill those thoughts.
  • We often ignore the dumb, obvious reason because we imagine the phenomenally unlikely experience we once heard about.

The answer to a far-reaching problem is often a mundane, common fix.

  • For that reason, it’s typically worth asking the intuition of the least educated or qualified person in the room.

2A. Investigate the obvious

Often, the end user or consumer has difficulty understanding the specifics of their problem. In their lack of understanding, they’ll lose patience with the problem’s persistence.

  • The user/member/patient often says something vague:
    • “My car won’t turn left.”
    • “The computer won’t turn on.”
    • “He is a bad person.”
  • If you can, ask for key information about precise elements as the event transpires.
    • Avoid jargon or diagnostic terms, since they might start misstating that jargon to feel important (e.g., “Yes, my brake caliper was making that noise.”).
  • This serves several functions:
    1. To the degree of their perceptiveness, you won’t have to revisit everything they’ve experienced in person.
    2. They may realize there are more details than they originally thought, which may give them more patience.
    3. They become informally trained to accurately perceive future changes, which can profoundly impact an entire group’s corporate culture over time.

Keep an eye out for the XY problem, which scales dramatically as systems become more complex:

  1. The user’s inexperience leads them to the details of X when they’re trying to solve Y. X and Y are entirely unrelated.
  2. Repeated support tickets and requests open up for the user’s X problem.
  3. Those tickets promptly close as highly qualified people consistently fix X.
  4. The user becomes increasingly frustrated until one of several conditions happens:
    • They either give up (and solve their problem without that particular product).
    • Someone in the organization notices that everyone keeps fixing X.
    • Someone else notices Y in an unrelated situation, leaving the user confused but satisfied.

2B. Poke at it

While completely focused, test it a few more times (if possible) by changing the origin and destination locations:

  • Remove the farthest-outputting part and watch what happens.
  • Feed in a different input, such as hand-cranking, or vary the quantities.
  • Insert a new input or output that gives more or different information.
  • Plug in another known-good screen or try a different mouse.
  • Access a different website or talk to a different person in that group.
  • Use a different computer on the network or try a different operating system.
  • Try a different fluid, or remove the extraction pump.
  • Use a different thought experiment.

2C. Reposition perspectives

To get more information, it helps tremendously to shift around perspectives.

Smart people tend to be better at finding problems.

  • One definition of intelligence is “a person’s ability to maintain multiple perspectives at once”.
  • This capacity magnifies if they have a favorable neurodivergence like ADD, autism, or schizophrenia.
  • However, smart people are also slower to act. Sometimes, just throwing things at a problem gives enough information.

The range of perspectives scales exponentially with a system’s complexity and poor design decisions.

Ironically, while redundant systems can be very useful in preventing failures, they can make failures more difficult to diagnose.

  • Assume A leads to B. Assume there are two more backup systems to lead A to B. This makes three systems to verify instead of one.
  • To discover the problem, create explicit distinctions between the systems (i.e., severing connections, using alternate inputs).
  • This mindset is mostly why hackers are the best technicians.

By this point, you’ll have figured out 95% of the issues you’ll ever encounter.

3A. Work the edge case

By this point, you’ll typically see the issue and know what needs to be replaced. However, there’s a comparatively small chance the cause of the issue won’t become apparent in a blaze of clarity.

The next step is to whittle away things it can’t be. Your purpose should be to make the likely chain of issues as small as reasonably possible:

  1. If it doesn’t cause any adverse consequences, try reproducing the issue again with different inputs.
  2. If it does create adverse consequences, ask for volunteers to experiment (who may be end users in some scenarios).
  3. Examine other ways to break the system the same way.

Generally, understanding likely failures is the best preventative measure. It requires gleaned experience, though (either yours or others working with the items in question).

  • This is recursive unlikelihood: something broken is already unlikely, and the likely fix isn’t working, making an unlikely unlikelihood.
  • Constant industry changes can quickly sabotage gains in this domain.

3B. Investigate complexities

Finding a cause becomes more difficult in proportion to the system’s complexity. Elaborately designed things (e.g., computers) almost guarantee you can never be 100% sure about any solution.

  • Error-tracking systems significantly reduce the need to investigate. It’s another system, though, which could also fail (e.g., engine codes are electrical signals).

For this reason, most advanced troubleshooting uses more in-the-weeds technical controls to keep the chains as small as possible:

  • Command-line prompts remove the uncertainty of a bad GUI and allow the user’s input to be more precise.
  • Before computerized throttle controls, auto mechanics operated the throttle on top of the engine instead of the accelerator.
  • Philosophers tend to imagine idealized scenarios rather than practical considerations that complicate matters.
  • Expert conflict managers tend to use low-context and simpler language.

3C. Presume harmonizing issues

Some edge-case issues can cause technicians to lose sleep trying to figure out why:

  • Two relatively innocuous edge cases can create unique and difficult-to-reproduce circumstances. One trivial failure will frequently interact with another trivial failure and cause a system to fall apart with zero intuitive predictability.
  • Occasionally, a connection between two elements can be defective while the elements themselves are known good.
  • Another hidden risk in diagnosing problems is when the chain has two or more points of failure.
  • Rolling out an update can break things. When something updates, it’s no longer known-good. Always keep backup versions ready to roll things back until you’ve implemented every applicable case.

Synergistic issues come more frequently in the presence of human failings:

  • Inherent design or engineering flaws
  • Poor maintenance
  • Previous people who “fixed” it incorrectly

If you don’t know, expect neglect and more issues to happen soon.

4. Repeat things

Since you’d prefer not to revisit the issue, you need to be 100% sure you’ve replaced all non-working components.

This part here is the most tedious portion of diagnosis, but it is also often the most neglected.

  • People with great problem-finding personalities also typically don’t like checking their work.
  • People often skip this one, especially if they’re impulsive or not particularly intelligent.

In human interactions, replicating the scenario is typically impossible (since people respond differently each time). It requires a combination of finely-tuned intuition and imagination.

It always requires at least 2-3 perspectives of that one part to get an accurate picture. You won’t know until you’ve tried it a few ways.

Every single part has at least two aspects to it:

  1. The part’s inner workings that do things (e.g., the database, the compressor’s components).
  2. That part’s connection to other things (e.g., the database GUI, the plumbing to the rest of the unit).

It’s typically difficult to tell, with any degree of authority, what’s actually wrong at first glance. When there were bad design decisions, this gets much worse.

  • The car may not receive power from a bad battery, a faulty alternator, or a blown fuse.
  • The forgotten message may be from a bad messenger, poor interpretation, or a vague message.
  • A failed command may be a bad API, a bad command to the API, or a bad network configuration.
  • Failed recipes could be bad instructions, bad ingredients, or bad cooking skills.

5A. Get the supplies

To fix things, you need parts or supplies. When you have what you need, swap it out and close the ticket.

  • Since your time is important, you can add the old part to a “fix later” pile. It gets things back to running immediately, and you can close the ticket.
  • However, you must actually order the replacement parts before they fail for the planning to matter at all.

The easiest solution, if it’s attainable, is to get an identical copy of the thing:

  • For software, this is trivial on an internet-connected computer (or an OS with a reliable backup schedule). It may simply require copying drivers from another computer.
  • For hardware, you must have a good inventory management policy beforehand to have parts ready to go.
  • Arrange for logistics for new components beforehand. This is a significant aspect of good management.
  • However, it can become horrifyingly complex if the components are extremely rare or an organization’s security policy becomes Orwellian.

5B. Visualize the fix

It doesn’t have to be complete, but that person must be able to imagine where all the resources are located.

There are 3 distinct phases in any repair job:

  1. Disassembly – remove everything that may obstruct convenient access to the problem.
    • Keep everything neatly arranged and labeled to ensure the final phase runs smoothly.
    • In social contexts, this means finding a quiet, safe time to talk with someone.
  2. Repair – directly access the problem area to swap out a component.
    • Any new issues are a recursion of this 3-phase system, but smaller.
    • If the component is a cable, securely attach the new cable to the old one, then yank the old cable out and detach its end, leaving the new cable in place.
    • In social issues, this is extremely difficult because the individual may not want the fix. At that point, consult ideal conflict-management approaches.
  3. Assembly – combine everything back together.
    • For a physical object, securely tighten everything in place to prevent it from moving later.
    • In social contexts, this means carrying on as if nothing had happened.

6. Repair and observe

This requires very close monitoring to ensure it works correctly, which is often easier with a second observer.

It’s always a good idea to keep tools available and in working order. While your needs will vary by industry, it’s almost always worth keeping a few tools around:

  • A high-quality multi-tool
  • Hammer
  • Variously sized screwdrivers
  • Needle-nose and snub-nose pliers
  • Crescent wrench
  • Hex wrench set
  • Utility knife
  • Protective gloves
  • Headlamp and flashlight

Both the decay of any individual part and the discovery of failures usually follow a logarithmic curve. Unfortunately, in gargantuan systems, a lot of time can pass between events. With enough time, everyone forgets what happened last time, or all the familiarized technicians no longer work there.

When updating, make sure the thing is as immovable as possible. Particularly savvy people update/replace on the fly, and sometimes multitasking:

  • Always secure and ground the objects you’re working with, even if it’s with tape.
  • Keep everything organized to prevent something from interacting inappropriately.
  • Never permit distractions to an important conversation.
  • Keep everything labeled as you go.
  • During critical stages, never make more changes than necessary.

Many managers imagine breakdowns are a great time to upgrade. This is only a good idea if it’s replacing an entire system. That upgrade will likely cause more things to fail later at the junction between the new thing and the old one. It’s only cost-effective if it worked elsewhere.

7. Clearly document things

It’s critically important to thoroughly and precisely document what you actually did:

  • If something failed, it may fail again and present a different issue further down the chain.
  • Clarify any slight deviation from what it was before you touched it.
  • Even if it’s your solo project, memory is fickle, even 2 days later.

If you don’t write it down, it didn’t happen.

A note on technical debt

“Technical debt” is creating future work by badly fixing something right now.

  1. A “duck tape and baling wire” solution is always available.
  2. It “fixes” the problem but isn’t durable enough for heavy use.
  3. That “fix” will mean a failure in the near future.
  4. An actual fix will be more expensive later, to undo the earlier “fix” that made the work environment workable.

Technical debt is only worth the cost in two circumstances:

  1. You have an emergency.
  2. You’ll soon be decommissioning the system.

Frequent emergencies often stem from accrued technical debt. At that point, consider investing in a better system.

Technical debt isn’t a big deal in an edge case, but it will destroy any efficiency gains in a common case.

Technical debt lingers with the object. This can be dramatically severe in a long-term project, or just about everything with software development (where the actual system is a bunch of constantly copied rules).

Documentation can help curb at least some technical debt, but the best cure is never to have it. Most people want to ship quickly, which runs counter to human nature.

From the beginning, think months and years ahead. An edge case can become the de facto common case over time, especially with frequent use. Things frequently scale over time, so it’s often not a question of if, but more of what.

One particular example is any fix that involves tape. Connecting two parts either needs a more robust securement (e.g., adhesive) or a better solution (e.g., plumbing or welding).

A note on long-term improvement

As much as possible, keep spare resources around before you need them.

  • This applies to spare parts, especially those most likely to break.
  • You’ll frequently need more time than you’d expect to diagnose problems.
  • You’ll also need plenty of space to work in.
  • As you scale, assign a designated area (e.g., a workbench) with all the necessary parts and space. It’s sometimes worth hiring someone to help with fixing as well.

Frequently, a network has a distinct pain point where things break down more often. There are several ways to work through it:

  1. Keep swapping out the components, which can be time-intensive and won’t scale well.
  2. Find a way around it, which may require clever hacks. Sometimes, if it’s better than the original solution, it might become the new standard.
  3. Invent something that creates the desired result from an entirely different angle.

Some perfectionists obsess about getting everything “just so”, where the object is as immaculate as possible. However, the tradeoff for this is typically not worth the effort, and they should settle for “good enough”.

Besides giving things more meaning, one major benefit of fixing things rather than replacing them is that it makes us smarter. “Getting inside” something makes us more qualified to know everything around it by association.

For this reason, the most skilled people at repairing are also often inventors who start businesses.