The Reality of Imaginary Problems - In the context of Software

Many factors lead to ineffective software development: from the tools used, to team communication, to the personal motivation of developers, the testing approach to the organisational leadership matrix. Among these, there’s a fundamental issue that often results in poorly conceived software: chasing perceived or, more often than not, non-existent problems, that take shape through the process to prove some kinda relevance by one or many involved directly or indirectly in the process.

Let me double click on the last part - non-existent problems. Most complicated or broken software is not designed to be overly complex or dysfunctional. It’s just designed to do something other than its intended purpose.

Consider a situation where you are a <fancy title here> developer working in a media company that runs a music streaming platform. They need a custom backend system that can handle the delivery of music albums, videos, and blogs, generate advertising revenue, and sell promotional products.

The system’s requirements might include:

Quick server response times in <a region>, along with real-time streaming and downloading capabilities for podcasts.
System stability ensuring that it doesn’t crash or freeze for 99.99 percent of users, ideally never.
Compatibility with Google AdWords and potential integration with other third-party ad providers.
Real-time linking and updating of the product catalog from <music erp store>, possibly with personalized recommendations based on user consumption history.
Integration with Facebook Live or even an in-house live streaming solution if feasible.

These requirements distill through many layers of management before they reach your team - the team of developers. There will be course of development with ‘regular checkin’ including these layers in your org and all seems well. Yet, when your team delivers the MVP a few months later, it’s far from what the original requirements craved for💩. The application hangs on startup, the advertising selection interface is counter-intuitive, twitter product links are broken, and the Facebook Live stream is sluggish🔥.

You as a developer are astonished becuase, the prioritized items that were due for delivery - according to the person <fancy title here> who distilled the requirements to your team were:

A sophisticated recommendation system.
Real-time transcript generation of all streams.
Ultra-fast loading times across the globe.
A proprietary streaming protocol to replace Facebook Live.
A feature to easily integrate with more than 20 ad exchanges.

So, you’re effectively part of a team who has ‘delivered’, but considered a failure 👿. Go figure 😨. Understandably, you & your team is confused & angry, rightfully so, becuase you’ve gone to hell and back to deliver this on time!

The problem lies in the misinterpretation of the original objectives. The development team prioritized non-essential, exciting technical challenges over the core, functional requirements. This misalignment may be further amplified when communication between the team and stakeholders gets filtered through multiple layers (coughs product managers, project managers, delivery managers, and <fancy title> managers), each adding their own interpretation of what’s needed 💩.

Imaginary problems are often more fun to solve than real ones. Extremely intelligent people play competitive games, construct and solve problems, and write books that aim to answer abstract questions about the human condition, all of them for free. A mediocre programmer, however, will probably charge you a fair amount to build a simple Android app. That’s not because mediocre programmers are harder to find than geniuses, but because the former activities are all fun, while the latter can be quite boring.

How many imaginary problems a software developer can create depends on their imagination and the complexity of the real problems they are tasked with. This is not unique to developers. Other departments like sales, HR, legal, and even accounting have their own ways of creating imaginary problems. These arise when individuals overemphasize minor issues related to their roles or take on more responsibilities than necessary, just to prove their relevance.

One dark aspect of this phenomenon is that the existence of imaginary problems can be used as a means of justifying team or organizational growth. An example is the banking industry. Despite the technological advancements, online banking software is often complex and cumbersome, a reflection of the industry’s hierarchy and its resistance to change. Not because the task of transferring and storing numbers is hard, but because addressing the real issues might threaten job security and established order.

Imaginary problems often stem from boredom or from long communication chains. When requirements pass through multiple individuals, even with the best of intentions, misinterpretations and alterations are inevitable. Sometimes, these changes happen due to misunderstandings or a desire to make tasks more exciting.

When the real problems get lost and replaced by imaginary ones, complex software solutions arise. This often happens when boring tasks are ignored in favor of more challenging and ‘interesting’ problems. But this results in software that does not meet the needs of the end users and wastes time and resources.

The dark force

There can often be an even darker reason for the existence of imaginary problems: problems can help a team or a company grow, and can even become an integral part of its function.

Ever heard about the trio of IT experts who realized that maintaining online banking security wasn’t as herculean as perceived? They engineered a foolproof banking application from scratch, harnessing functional design principles and memory safe languages, and began transitioning major banks to their impeccable framework.

Probably not, because such a trio doesn’t exist. Yet, there are numerous developer teams, struggling to understand basic principles such as “rollbacks,” consistently creating banking software.

“It’s a challenge to make a person understand something when their livelihood depends on their ignorance of it!” — Upton Sinclair

Seeding imaginary problems and accounting for complex software that solves for these imaginary problems - sure, that’s one way to claim ‘busy’ and ‘percieved’ progress or ‘useless’ celebration, but how does something like go unnoticed? Isn’t this basic common sense 😵?

Well, isnt this is kinda similar to -

The boardroom often turns a blind eye to the management indulging in “conference trips” involving exotic locations and generous allowances for “miscellaneous” expenses. In return, the management overlooks the board’s less than transparent practices.
Encouraged by department heads to maintain an image of perfection, management turns a blind eye to those spending more on their office decor and personal assistants than necessary.
Because supervisors don’t question their grandiose power displays, department heads ignore those supervisors who, instead of reducing overheads, focus on creating impressive presentations.
Because team leaders don’t complain about their superiors’ lack of understanding of basic software, supervisors ignore the team leaders discussing complex system integration instead of addressing slow response times.
Team leaders, not recognizing that their bosses barely understand coding, don’t question their developers who, instead of diagnosing the slow server response times, keep overhauling the user interface using the latest JavaScript frameworks.

It’s a vicious cycle of tackling non-existent problems—from the CEO who doesn’t understand that securing another big investor won’t compensate for customer dissatisfaction, to the software intern who doesn’t realize that redesigning the booking interface won’t solve the issue of overbooking.

But everyone keeps trying to solve these imaginary problems, because if they stop and focus on the real problems, they might realize the whole system is fundamentally flawed 😇.