Of software "rot", fertility, cities and innovation

Let's please look wide, and even when we think we are looking wide – look wider still.

Last week, R. Hanson wrote about the coming "rot" of software and innovation in general. On impulse, I commented (whole thread here): Software isn't cheese. It doesn't rot. And real innovation is much more than what transpires from this post.

Here, I explain what I meant with that comment, namely that, even IF "rot" were the most appropriate term for software, it very often happen in ways, for reasons, and with consequences, quite different than those described or implied in that post. In general, I found that description of how software is or can be maintained, and that of innovation in general, lacking. Here's why.

Software competition?

Hanson starts from the famous declaration that "Software is eating the world", a great intuition by a guy who sadly totally lost the plot later, as you may also see here and here. Starting from there, Hanson describes the expanding economy of the last decades as fueled by an expanding software industry in a virtuous cycle in which, even if some software company fails (all text in italic is Hanson's quotes, with emphasis mine):

better solutions [will be] likely on their way, offered by more numerous suppliers who compete more strongly for your business... [and] If some suppliers die or raise their prices, other superior options will probably become available soon. So the usual practice of mostly ignoring future issues works out pretty well.

More numerous? "IT Software is quite a concentrated market, with the top 20 players representing approximately 75% of the market", say the analysts. Competition? After decades of Microsoft, then Google, Facebook, Apple and a few others... squashing competitors by "embrace, extend and extinguish", or farming a startup "culture" whose main "exit strategy" for startups is to "sell the company to a bigger one for a profit", and having your app bought by Google is "the ideal outcome for most entrepreneurs"? THIS would be an industry of more numerous suppliers who really want and care to compete more strongly for customers?

Rewriting from scratch?

Speaking of software maintenance, Hanson writes that [your software company] might be sloppy and let software rot faster than it would otherwise have done. But if your product is near state of the art in features, then expanded revenue from your expanded base of customers will usually let you rewrite it all from scratch later. So you worry mostly about staying state of the art, and less about how fast your software rots.

Of course, yes, eventually all software programs become completely useless or unusable, if nothing else because their initial requirements and functions have become completely unnecessary in the current world. And of course, as with any other rule, there are exceptions.

But in most cases, borrowing from J. Spolsky and a StackExchange thread on software "expiration dates" among actual software developers:

• "the single worst strategic mistake that any software company can make [is] to rewrite the code from scratch"

• "[if] software is a mess it is because [its developers]] made it a mess, and it’ll only get and STAY clean if they clean it."

• "continuing to keep the software producing business value.. isn’t accomplished by... opting for a rewrite [but] by a lot of discipline, belt tightening, and dogged, incremental improvement", that is the exact opposite of "worry later"

• more of the same here and here

No more hardware or software?

All this software innovation will be lost… like tears in rain (image source: Silicon valley memes)

Hanson continues noting that we are heading (more on this later) towards a world of shrinking innovation in all fields, software included, in which

computer hardware stops improving, and the accumulation of durable hardware may basically end the computer hardware industry. [In such a world] who will want to pay more for new hardware that isn't better than plentiful durable old hardware?

Using "old" products as long as they do the job instead of sending mountains of perfectly good hardware to the landfills every year just because some stockholders want more money seems pretty smart economics to me. This said, what that shrinking would surely end is only the growth in volume of hardware manufacturing, but so what?

Technically speaking, population decrease (see below) plus saner software regulations that impose open standards and adversarial interoperability exactly to foster real competition and innovation could reduce the depletion of raw materials for electronic devices enough that we may keep making and using, without any real loss in standards of living, the same hardware components of today for decades. It would be a political choice, of course, but not necessarily a tragedy or defeat.

In a shrinking world without software innovation, most software suppliers will not be able to count on increasing customer revenue to fund routinely rewriting their software from scratch when it rots. And there won't be new tools to let new versions be better than old ones.

Even ignoring that "new major versions every year = innovation" is true only in marketing... As said above, if you let software "rot" by "worry mostly about staying state of the art" instead of keeping it clean, you'd better change job.

Above all, all the many cases like Microsoft Office, where even the users perfectly happy with previous versions or with alternative suites see them "rot" and are forced to buy every new version (plus hardware that can bear it) only because everybody, starting from governments, is so clueless to stick to Microsoft's default choice of a proprietary format, are not "rot". That's just ignorance of the very basic nature of digital documents, that couldn't care less of politics and ideologies, and of the self-harming refusal to get it right (1).

As suppliers drop out, once competitive software areas become duopolies, then monopolies, then non-maintained products.

The world would abandon entire alternative computing ecosystems... so that they could better share the costs of supporting the few remaining ecosystems. The bitten losers would be those who came to depend on complex systems with too many features that the world could no longer support.

See above about "more numerous suppliers". It would be great if monopolies" were a future concern, not the situation we are in now. No, the bitter losers will be the individuals, companies and states who will:

• keep falling into lock-in traps instead of demanding open standards for file formats and communication protocols (1) that would make it immensely easier to migrate from one software supplier to another, or

• completely ignore, as it seems the case here, the existence and potential, especially in "declining" scenarios, of the Free/Open Source Software management model

Open standards, plus making it mandatory to open source the software code of a company when it dies, and making the following maintenance finally sustainable, at much smaller costs, by Universities or everybody else, could solve most of the problems that Hanson fears.

Basically, what Hanson seems to fear is not a future in which software or society in general "rot" and everybody is real sad because what we call innovation today is dead. It's a just a world in which the software industry has finally reached maturity, developers don't run like lemmings and most software has a Free/Open Source license, because that has become the only model that's compatible with reality. But it doesn't mean it's automatically bad, or an age of general decline for society.

On a side note... The unglamorous mere maintenance of "old" software may be the only kind of software job really within reach of two quickly growing, but much overhyped class of "software developers":

• the millions of kids we are training to code without really worrying to teach them how to think and vote critically

• the millions of adult workers left jobless by automation and AI, exactly because they were born without the skills to get a 6-figures Silicon Valley job of making "really innovative, ever expanding software", and yet are told by Industry 4.0 pipers that "upskilling", usually by learning to code, will get them better jobs

Innovation my foot. Indeed, hiring exclusively for the absolutely critical but dull, low-paying routine maintenance of existing software the whole overwhelming majority of those "coders" that could never do any other software job would be the only way to recover the costs of their "upskilling". No, wait, AI could do that maintenance too, right?

The really important issues, that is loss of innovation in... fertility and cities

Much of Hanson's post deals with fertility and innovation, and may be summarized by these quotes, edited for brevity:

Roughly thirty years from now [both] world population [and the world economy] will peak, [and then] fall by a factor of two every one or two generations. The world's innovation rate will shrink a bit faster [and] seem to grind to a halt within a few generations.

In a shrinking world economy, falling populations would raise the cost of supplying city services to partially populated cities. Which would induce people to abandon some cities in order to fill up a few populated cities, to keep city service prices low.

[In] declining scale economies... it would probably take over a millennia for this to induce sufficient poverty to raise fertility. And, alas, learning about this dark decline scenario will not be enough to scare nations into implementing the feasible fix: borrowing to pay enough to parents to induce over replacement fertility.

Instead, rapidly growing insular fertile subcultures like the Amish and Haredi Jews will eventually restore growth [and] dominate the world population. Eventually... there’ll room for new innovation, at least of the sort approved by these once not very tech friendly fundamentalist religious communities.

Hanson IS right to be concerned about the aging, decrease in fertility and sustainability of cities tha are coming (as a matter of fact, that's why I started following him). I'd seriously like to read his opinion on my thoughts about certain approaches to fertility and on my vision of the least unfesible way to "supply city services to partially populated cities", that is here (why do it) and here (how).

However, regardless of which political side one is or isn't, the current world economy is so... stupid, no matter from which side of politics one looks at it, that I struggle to see its upcoming peak and shrinking as a surely bad thing. But that's not my main issue.

Innovation, or REAL progress?

As "fertile subcultures" go... as a minimum, it's unfair to call "not very tech friendly" any culture, just because it doesn't blindly swallow whatever PASSES for "innovation" today, or the general principle that "more" and "new" are always surely better, by definition. The Amish, for example, more than being anti-technology tout court, are just "more thoughtful", seriously "intentional about it" and perfectly able to sell million dollars’ worth of products online. Catholic Social Teaching, as I wrote in 2013, had "already conceived, decades or centuries ago, solutions so advanced that technology has been able to build the necessary tools only in recent years".

Finally, exactly because I share certain long term concerns, it worries me to see them mixed with pretty weak arguments about a technology as vital as software, and with what is becoming, apparently blinded by a need for a growth so perpetual and so narrow to be the very "definition of metastasis", a really small, really unimaginative, dangerously narrow vision of innovation.

Any moron can look and do great with ever expanding money and other resources at hand. It's when constraints increase that real (as in "really meaningful and game-changing") innovation emerges.

We need to be, exactly to prepare in time for an age of smaller population and resource-limited cities, less "very hype friendly" and more "very appropriate- tech friendly", especially when software is concerned.

We need less fake innovation and more real progress, from whatever group, instead of chasing as innovation anything, as long as it did not exist last year, or it's from the very same thinking and guys who created problems in the first place. And we already have all the software we need to innovate (yes, even using AI) where it most matters and it's more urgent. As things stand now, it’s sticking to certain ways of thinking that may create the most rot.

Usual final reminder: if you like what you read here, please remember that, besides paid subscription, you can support my work in these other ways. Thanks!

1. If you have no idea what I'm talking about, please read this