Over the past few years I’ve found a need for a new term to describe my approach to software development, which challenges the age-old dichotomy of Waterfall vs Agile - both of which fail to produce software systems and organizations that are easy to maintain in the long term.
Eventually, I came up with the term Holistic Software Development - a mindset and a methodology for developing software that emphasizes interconnectedness and flexibility. It retains the iterative nature of Agile, but It moves away from rituals and ceremonies, and focuses on nested iterations that encompass product discovery and management, sustainable software engineering and customer obsession to make sure that we’re always in touch with the changing reality.
The Failures of Traditionals Approaches
While most people view Waterfall as somewhat of a joke, with rigid stages of specification, design, development, testing and delivery, many would-be Agile practitioners in fact engage in mini-waterfall episodes scoped to a sprint or several sprints - sometimes referred to, jokingly, as Scrumfall. This creates unsustainable development practices that can be characterized using the following facets:
We spend too much time on advance, low-level planning, with little regard to changes in the market, the product or the organization. Often these planning efforts are concentrated in a quarterly effort, sending the entire team into a resource-consuming vortex lasting several days.
Since we have already spent a lot of time laying plans (and making commitments), we’re now driven to spend even more time estimating the effort required to achieve these plans. But as most developers know, these estimates are rarely met - leading to bloated estimates, and actually reducing the total output of an individual developer.
Given a strict plan and a false assumption that the future can be predicted, we go on to write detailed design documents to help us prepare for all expected features that we have planned, but these design documents are often stale as soon as the first line of code has been written.
Last but not least, the plans themselves often reflect a misinformed process of feature ideation, since not enough effort is given to understanding what the users really need and to continuously collect feedback from them.
In attempting to predict the future, we err twice. In addition to wasted effort in writing design documents, we also try to defend ourselves from uncertainty by creating elaborate designs that could solve any future surprise we might come upon. This results in over-engineering and architectures that are too complex and costly - both of which often not only fail to make changes easier, but actually make it harder, since assumptions during upfront design are often based on limited information.
Fear of change also prevents us from cleaning up our codebase and architecture, as often we’re too busy trying to meet impossible deadlines and do not invest enough in automated tests.
In attempting to achieve the plans we laid, and based on unfounded estimates, we set deadlines that are often unrealistic. We expedite execution, often compromising on quality, and when these deadlines are eventually not met, the whole organization often goes into blame games.
This creates rivalries between discipline-based teams such as Product, Engineering, QA and Customer Success, preventing us from creating a shared vision for the entire organization.
Often reality differs from our plans, requiring us to adapt or change our behavior. But the sunk costs, having spent so much time on planning and designing, often cause us to oppose the needed changes to schedules, features, or teams.
When fear, rivalry and despair spread in our organization, people burn out, and it is often the best employees who leave first, since it’s easiest for them to find better jobs. As time goes by, we are left with those employees who couldn’t find a better job, causing the culture to deteriorate and the quality of our deliverables to drop.
Having skipped automated tests and refactoring efforts, we are left with software that’s hard to change, and as time goes we incur more and more technical debt, increasing the cost of change to the codebase. Similarly, having spent a lot of effort on an elaborate design, and having little to no automated test coverage, we often end up forcing changes on misaligned architectures, creating a chimera of what was supposed to be and what we actually need.
Since every change is costly and difficult, we find it increasingly difficult to react to changes in the market or reality, to deliver features to users in a timely manner, and to incorporate any feedback from our users into our plans, resulting in a ship that is very slow to turn.
Making Agile Agile Again
In contrast with Scrumfall, Holistic Software Development focuses on information flow, simplicity, high confidence, very short iterations and frequent reality checks. These help create feedback loops between our users and our stakeholders, between our stakeholders and themselves, and between our engineers and the codebase. Healthy information flow helps prevent inner-organization rivalries and make sure everyone works towards the same shared vision.
We form loose teams (often dubbed Pods or Squads) according to current needs, composed of individuals of complementary disciplines, creating cohesion and reducing friction between disciplinary groups such as Product Managers or Engineers. We change the structure and composition of our teams as often as needed, helping knowledge propagate throughout the organization, and helping employees form productive and lasting relationships. Continuous mentorship and pair-programming inside teams and between teams also make it easier to move people between teams without fearing loss of knowledge. Since these teams are better aligned to the needs or the product or our users, each team can change course on its own. Instead of one big ship to turn, we have a flotilla of smaller, more agile boats.
Instead of elaborate, long-term plans, we limit ourselves to high-level plans and continuously refine, adapt and change them according to how reality is changing. This in turn reduced the need for effort estimation and low-level technical design documents, freeing us to spend more time executing, and minimizing sunk costs. Since our plans are less detailed, it’s easier for us to make room for changes originating from user feedback. In addition, any new information gained during development is fed back into the continuous planning process, allowing us to better adapt to unforeseen complications. Any change in plans can easily be reflected in our delivery forecasts, removing the need for setting strict deadlines - and the burnout failing to meet them.
We develop our software using Acceptance-Test-Driven-Development (ATDD), which focuses on writing automated tests from the perspective of a user, yielding a test suite that’s unlikely to break as we make changes to the implementation of our software (by refactoring, changing dependencies, replacing architectural components, etc). ATDD facilitates Emergent Design, guaranteeing that in any given time our software design is the simplest possible one, easy to refactor and ready to accommodate any unforeseen requirement. This is often done in pairs, with a more senior engineer mentoring a junior one, helping us to onboard new engineers, further increase knowledge sharing, and strengthen professional relationships.
The high level of confidence gained by ATDD combined with the low cost of change gained by using Continuous Delivery techniques allow us to expedite the time-to-market for any given feature, since it’s not coupled to anything else in a periodic Release Train, further reducing risks in deployment, and improving the ability of any single team to achieve their current goals. Reliance on automated regression tests helps us minimize or eliminate manual QA altogether, expediting the release process. Any bug that is detected is first reproduced in a failing test and only then fixed, so it can never recur. This is easy, since the software is always testable, having been written using ATDD.
We constantly observe reality, and accept it as it is. We change course as soon as needed, rather than blindly following our original plans by inertia or fear of accepting sunk costs. Any delay is communicated to stakeholders and users rather than trying to circumvent it by overwork or cutting corners. Our agile flotilla helps us make sure we can quickly adapt to changing landscapes - whether from our competitors, our users, or global events such as pandemics and wars.
The following table summarizes the key differences between Scrumfall and Holistic Software Development:
Holistic Software Development
Delayed time-to-market; Coupling; Release Train; Feature bloat; Gold Plating
Continuous Delivery of individual features; Micro-goals achievable by small teams; Flexibility as a moat.
Long backlogs often ignored; wasted planning efforts create strict schedules
Minimal, high-level planning; long-term planning in bullet points, Just-In-Time distillation and expansion of feature requirement
Design and Architecture
Premature optimization; fear-driven; over-engineered
Emergent Design via ATDD yields simple, changeable software
Quality and Stability
Unrealistic deadlines lead to compromises in quality; Automated tests are harder to write and often skipped altogether
Test-first approach yields testable software and an ever-growing suite of regression tests; Bugs reproduced by writing tests can never recur. little to no manual QA given extensive and dependable automation suite
Micro-management; miscommunication between employees of different disciplines; Burnout and attrition; Bus Factor is high risk; Disciplinary-based teams such as Product, QA, Frontend, Backend, etc
Pods or Squads are formed and changed as needed; Synergy thanks to healthy information flow; happy and satisfied employees; knowledge propagation thanks to pair-programming and frequent mobility;
Fear-driven behavior; rigid structures; accidental complexity; fear of change; uninformed decision making; restricted information flow; fragmentation in priorities and vision
Fearless execution due to reduced cost of change, simplicity and improved confidence; Informed decisions and shared vision thanks to continuous information flow