AAA-DBA.com

A blog site for database enthusiasts

Rip Off the Band-Aid When Quick Fixes Cost You More

This morning, I had a bit of an epiphany, and it’s a little embarrassing to admit. For the past month, I’ve been dealing with a slow bathroom drain. Like many people, my first thought was to pour some Drano down it. It worked a little, so I poured more. It improved slightly, so I added even more. This cycle went on for weeks.

Then, this morning, I accidentally dropped something down the sink. So, I decided to take the drain apart (just a small part to unscrew) and pulled out the plug. It was covered in disgusting hair and gunk. I cleaned it off, rinsed it, and put it back in the drain. Suddenly, the drain was completely cleared! The whole process took me about 5-10 minutes.

As I stood there, I thought, “What the heck? I could have fixed this ages ago.” Instead, I wasted money on Drano and spent time pouring it down the drain, waiting, and rinsing with hot water. All of that took time too.

Then it struck me that this tendency to choose quick fixes instead of addressing the underlying issues is common. For instance, businesses might “upsize the hardware” instead of optimizing problematic queries, which often results in escalating costs—a scenario DBAs frequently encounter worldwide.

Or companies that pay employees just enough to prevent them from leaving, but not enough to keep them truly satisfied. When these employees eventually depart, the cost of hiring replacements at higher market rates, often with less experience, can be significantly greater. The money thought to be saved ends up being spent anyway.

Some companies decide they don’t need a DBA, even when their entire product’s performance depends on the database. I was once approached by a company that had 30 SREs but no DBA, and they were struggling with daily issues because they had no one to consult on performance problems. They had reached a point of desperation, facing daily outages, and their only solution had been to continually upsize their hardware. However, this approach was becoming unsustainable as the costs skyrocketed. They had run their entire enterprise without a DBA for five years. 

A different company decided to minimize its reliance on the database to avoid the need for a DBA. However, this decision backfired when an application issue happened and none of the engineers were available or on call. As a result, the application was down all night, leading to significant downtime and potential losses.

Choosing manual interventions over automation is another common example of short-sighted decision-making. Often, tasks are done manually because setting up automation seems too time-consuming or complex. However, this approach leads to wasted hours and potential errors, whereas automating these tasks could save significant time and ensure consistency.

Other factors to consider include:

  • Team Dynamics: When one person is tied up with manual work, it adds pressure on the rest of the team, who must pick up the slack for other tasks.
  • Technical Debt: It only grows as more layers are added, and code becomes more complex.
  • Performance Issues: Untreated performance problems can become expensive.
  • Long-Term Costs: Short-term fixes can lead to accumulating costs over time.
  • Bottlenecks: Band-aid solutions can create bottlenecks, slowing down overall.

This experience got me thinking, “Why do we default to these quick fixes?” Is it because the time and effort needed for a proper solution are underestimated, or is it just an avoidance of the discomfort of dealing with the root cause?

What strategies do you or your company implement to address issues head-on, rather than applying temporary “band-aids”? And personally, what are some “band-aids” in your life that need to be ripped off and addressed? How are your bathroom drains?

2 responses to “Rip Off the Band-Aid When Quick Fixes Cost You More”

  1. Daniel Kimberlin Avatar
    Daniel Kimberlin

    Great article. I see this all the time. It starts out as a quick fix and devolves into tech debt that weighs on an org because those band-aids are usually steeped in tribal knowledge. Some I’ve seen are shrinking a DB daily, restarting databases or app pools daily to give them the best foot forward. Putting status updates in application code to run before the “problem query”, using hints to force plans to do things like forceseek or force specific joins and leaving them there and that gets disseminated to the entire org as the panacea for all performance issues, putting option recompile in stored procs to circumnavigate parameter sniffing issues, etc. It takes effort to clean up some of these things but the end result is always a cleaner, happier system. Thanks for the chat, Amy!

    1. Daniel Kimberlin Avatar
      Daniel Kimberlin

      *Stats updates

Leave a Reply

Your email address will not be published. Required fields are marked *