Refactoring is an Investment

December 16, 2018

Inspired by J.B. Rainsberger asking, “What are the values of software?” in his talk, “The Economics of Software Design” at , starting around 4:30 where he talks about how to answer this manager’s question:

“My manager gets angry when I refactor.
What should I do?”

And JBR says that you should say…

“I’m just trying to reduce volatility in the marginal cost of features.”

I don’t think this works for me.  My problem is that it’s been my experience that it’s typical for the team or line managers I’m working with to insist that this is my problem, as a professional software developer — that as a professional, I should eat the cost or use professional techniques to make the volatility problem go away, so that he doesn’t have to deal with it.  “That’s what I hired you for — to make good estimates.  ‘volatility’ issues are your problem.”

So I find it more helpful to say, quite honestly, that…

The reason I’m doing refactoring is to reduce costs.

Now this is simple, direct, honest, and easy to understand.  And reducing costs is something that they want.  But they don’t believe me, because refactoring clearly requires time (and hence costs money) to accomplish, so it “can’t possibly reduce costs.”

Actually, it can.  Because refactoring improves the structure of the code without changing its functionality.  And this means that refactoring reduces the maintenance costs of the code, in the future.

“But I don’t want you making ‘investments’!”, says the boss, “I want you pounding out code!”

That may sound like a rational and reasonable objection.  But if my boss makes it, then I have to conclude that therefore, we should not be developing software.  We may as well cancel the project and walk away.  Because software development is always inherently a process of making an investment now, by writing the software, in hopes of a return on investment in the future.  If that’s not what we want, then clearly, writing software is the wrong thing for us to be doing.

What we need to be talking about is the rate of investment that we should be doing at each stage of the project.  Near the start, we should be investing more heavily, so that we can reap the rewards of that investment through the largest portion of the project.  Near each major deliverable, we should reduce risk and investment, for the benefit of the short-term goal.

Every working day, software developers like me make hundreds of decisions, to invest in improving the quality of the code, or to mortgage the future, for a short quick burst of productivity.

We often do ourselves no favor by saying that “refactoring reduces costs long-term, by making an investment, short-term,” because it’s easy for our boss to assume that “long term” means years, when, in fact, many refactorings provide immediate benefits that repay the costs within a very short time.  I’m often looking at refactoring changes, or improvements, that have pay-off times best measured in hours, days, or weeks.  Only the more ambitious process and tooling changes may require some number of months to recover the investment.

And nearly every investment you make that improves your productivity on the project, continues to provide benefits for the lifetime of the system.

I find this incremental decision making process more useful than the “do it well or do it poorly” choice that JBR offers the project manager, starting just after 12:00 in the video:  For the same reason you can’t shift an organization directly from yearly releases to weekly releases, you also can’t shift an organization directly from the “bad curve” to the “good curve” — you have to make incremental change, to be successful.

I proposed more realistic formulas, from the COCOMO estimation model that a realistic range of costs to develop the software on a project would be from 3.2 * KLoC ^ 1.05 on the low end, and 2.8 * KLoC ^ 1.20 on the high end.  As JBR explained in his video, the “good curve” starts “higher” (more costly), but do to the “gentler slope of the 1.05 exponent, we’ll find that the “bad curve” of the 1.20 exponent, while it starts lower, it quickly goes much higher, and the overall cost quickly becomes overwhelmingly more expensive.

My objective is to move the exponent from 1.20 down towards 1.05, in incremental steps, so that we will have sufficient resources to be able to produce half way decent software.  This will also increase up-front costs, raising the constant factor from 2.8 up to something closer to 3.2.  But this has little effect, relative to the compounding benefit of the process improvements.

What is the Value of Software?

December 16, 2018

I was inspired by J.B. Rainsberger asking, “What are the values of software?” in his talk, “The Economics of Software Design” at , where he suggested that three of the values of software (out of “7 or 8 million”) are…

  1. Features
  2. Design
  3. Feedback

He goes into detail as to the value of each, but I kept thinking that…

The value of software is automation.

The reason we write software is to automate (business) processes.  That’s what computers can do.  That’s what software does.

Automating processes can be a good thing because it can…

  • increase speed / performance of the operations
  • improve consistency, including enforcing rules
  • reduce cost and waste

But we do have to recognize the trade-offs that exist:  Nothing in the world is “completely free.”  When you automate a process, you have to deal with…

  • loss of flexibility — ad-hoc manual processes are quite flexible
  • the required up-front investment to understand the process and automate it
  • the related ongoing operation and maintenance costs.
    • maintenance is often the cost to achieve or regain some of that flexibility that you need. 

So what about the Values of Software?

Features are good.

But features are not business value.  Features that are not useful to the business users have no business value.  Features that they don’t know about or don’t use have no business value.  Features that they’re afraid to use have no business value.

Only features that contribute to the successful achievement of useful business process have business value.  Software features that contribute to increased sales, reduced costs, and/or contributing factors like brand recognition, customer and employee satisfaction, product function, etc. have business value.

Is it true that 45% to 65% or more of features are unused?

Probably.  It depends.  Software built for multiple customers probably has many unused features.  Purpose-built software of a single application for a single customer may have fewer unused features.

But the more requirements are “fixed up front,” rather than added incrementally as needed, the stronger the motivation to “pile in” all possible requirements — a process that inevitably leads to “feature bloat,” with many unused features.  If you only get one chance to “state all the business requirements” up-front, and it will be very difficult to add or change anything later, then most people will quite sensibly try to “throw in” practically every possible thing they can think of that may have value.  What would be their motivation to limit themselves to only the items with the highest overall payoff — particularly when “someone else is paying the bill”?

Also, the normal inevitable change of any business, typically around 1% change in overall “business requirements” every month, according to some sources, will inevitably “leave behind” some functionality that had business value at some time, but is no longer of use.

So it’s probably best to develop and deliver features that have high business value frequently.  The feedback you get from delivering usable features that deliver business value can help prioritize subsequent work and focus it on delivering the most highly valued features first.

Design is good.  Good design is good.

But a great design of a useless thing does not save it from being a useless thing.

Good design makes it more likely that the software will successfully implement the intended requirements (both functional and non-functional).

Good design typically also reduces the cost of future changes.  In the best case, this might be “insightful” design — that anticipates likely future changes successfully.  It would be a waste of money to invest and build for requirements that won’t be implemented until some time in the future.  But allowing flexibility for future changes instead of inhibiting them can be cost effective.

There is always a risk that some entirely unexpected future requirement or opportunity will be entirely incompatible with the abstractions we used to build our current “good design.”  With experience and insight as to a very rough range of plausible future changes, we can mitigate this risk.  But it can never be completely eliminated.

Feedback is good.

I don’t really see software as providing feedback directly, itself.  But deploying working software for real use certainly enables receiving much more useful and realistic feedback than having even highly knowledgeable people speculate as to what might happen in various hypothetical situations.  Actual implementation in the “real world” often reveals unexpected details and difficulties that we could easily miss otherwise.  And it’s hard to ignore or deny a problem that is actually happening, as compared to ignoring or dismissing a hypothetical problem or fear that might arise in a conceptual planning session.

Regarding the other 7-8 Million Software Values…

I think I’ll leave those to another talk or article, too.    ;->