Category: Software Engineering

Two issues facing software in the 2020s

I have been thinking about the problems in software a bit lately. As I see it we have 2 core issues, the first is the diseconomies of scale in software. The more and bigger the software, the more it costs to develop. The second issue is software security and maintenance. 

Diseconomies of scale are an issue because we keep writing more software. And the expectations of that software continue to rise. Smartphones started out with phone calls, texting, and surfing the web. Now your smartphone tracks your exercise, interacts with your smart home devices, does payments and more. As the smartphone becomes your main connection to the digital its complexity increases, and at the same time the expectations for everything else increase. 

I have four potted plants in my apartment. I could have smart water meters in each pot that track when I need to water those plants. In the past no one would have expected your plants to be smart. Today it is an obvious next step for people who’s plants always die. 

Since your smartphone is in many ways you, long term I expect things like door locks to transition from keys to phones. Cars have already made this transition. My car unlocks via a button press. Using the actual key requires me to first remove it from the dongle. 

Now a serious question is, “Why do you need a dongle to start your car?” Your phone has all the capabilities of the dongle. You don’t use the key anyway, what is the point of the dongle? Extend that question to everything in your life. 

Why not have a way to check on your phone if you left the oven on? 

Why not have your water heater hooked up to your phone so you know if your kids left you any hot water?

Why not have a GPS dog collar hooked up to your phone so you always know where your dog is?

Gradually everything moves to your phone, and as more things move to your phone you will begin to ask “Why isn’t this on my phone yet?” 

The phone ends up as an incredibly complex, absolutely essential part of your life. And as it does so it pushes everything else to connect to it. 

We are on a trajectory that yields an incredible amount of computerization. Who is going to write all that code? It is going to be incredibly expensive with today’s linear coders. 

Software maintenance is the second big issue. We keep building more and more software, but we aren’t getting any better at maintaining it. Software is a weird animal because it is never ‘done’. You can reach a point where software is feature complete and seems perfect. But not really, hackers could discover a vulnerability you must patch at any time going forward. If that happens you need to have people who can update your software quickly. But if you reached feature complete 10 years ago and laid off all the programmers, your project is now poisonous. No one can use it until you hire and train people to update that software. 

Software seems like it can be completed, but that is a trap. Security is not the only risk. Operating System updates are another problem. Microsoft and Apple constantly update their systems and API requirements. Customer expectations change as the software marketplace changes. 

The final trap is what happens when you get off the treadmill. Say your software is complete, it works perfectly and has no vulnerabilities. So you let it sit with minimal updates, a skeleton crew to keep it running. Well, ten years from now no one will be willing or able to modify that software. Maybe it’s Cobol, or Java 8 or Typescript, but the tech is now old. Any programmer who works on it will have to spend time learning technology that is worthless long term. 

Blockchains are the first mega scale software daemons

A software daemon is some bit of software that runs independently of human control. We have been creating them for most of computing history. But the majority are not that important. Nobody knows their names. Nobody really cares that much if they have bugs. If one stops working you restart it and move on with your life. 

Cryptocurrency blockchains are also daemons. Each blockchain is just software that performs whatever tasks it is programmed to do. The interesting differences are the scale and the distributed nature of the blockchain. Millions of people contribute hardware to run blockchain software as opposed to a cronjob running only on my laptop. 

The Ethereum blockchain is a billion dollar distributed computing platform powering things like DNS names (ENS) and art NFTs. Anyone can submit a ‘contract’ to Ethereum as it is essentially a daemon that runs other daemons. Compared to the log daemon that collects logs on your computer, Ethereum is immensely larger. Restarting your log daemon is essentially a free operation. Restarting Ethereum from zero would cause billions of dollars of losses. 

Due to their decentralized nature it is also impossible to ‘stop’ a blockchain software daemon. Ethereum exists on millions of computers and as long as even one of those copies continues to operate Ethereum will continue to exist. 

People also care enough about these blockchains that they have names. You have the ‘Ethereum Classic’ and ‘Eth2’ daemons which run very similar software, yet their identity is actually important to humans. Millions of copies of Linux and Windows are running right now, but none of them are important enough to have names. 

Why don’t people use functional languages more.

“There are only two kinds of languages: the ones people complain about and the ones nobody uses” — common quote

This is a fun statement if you think about it. Because the languages everyone complains about are obviously better than the ones nobody uses. It seems controversial but a simple thought example shows why it has to be this way.

Imagine a world where the mainstream languages were worse than the niche languages like Haskell or Crystal. In this world Haskell has a clear 2x productivity advantage over Java. Since Haskell is clearly better than Java most projects that use Haskell will execute roughly twice as fast. Double productivity is enough that companies using Haskell will be able to achieve the same results with less money spent on developers and thus achieve higher profits. We should see companies that don’t use Haskell going out of business. Eventually we would expect all greenfield development to happen in Haskell. 

You might say well, “Haskell isn’t really twice as good as Java, it is only 20% more productive overall”. Well, 20% averaged out over hundreds of thousands of software engineers is a huge amount of money. My employer would very happily take a 20% increase in productivity if it were on the table. 

However, in the real world we don’t see all greenfield development happening in Haskell. Instead lots of greenfield development happens in Java, Javascript or the .Net ecosystem. 

There has never been a consensus moment in the industry around niche functional languages. Instead people say they wish they could use them while making actual money writing Java code. 

Personally, I would really like to use a better language than Java. Unfortunately, we don’t seem to live in that world. 

Expecting people to become ‘Senior’ Engineers in 3 years is ridiculous.

Becoming a ‘Senior’ engineer in three years is a ridiculous concept. Programming is a complex enough endeavor that you can’t learn the basics of most areas in 3 years of work experience. By the 3 year mark great programmers will have achieved a high level of expertise in one programming domain. For example 3 years might be enough time to master web development with Django. But web development with Django is just one of many domains within computer programming. Mastering Django does not require a software engineer to learn anything about operating systems, low latency computing or graphics computing. In some environments web developers might not have any involvement with DNS or how their software is made available to users as deployment is handled by a infrastructure as code team. 

 In my case I was promoted to ‘Technical Architect’ and team lead in under three years at my first job. I was really good at my job and worked well in the consulting business. At that point in my career I knew a lot of stuff application development, frontend development, automated testing, devops, etc. But I felt under qualified and wanted more money so I ended up taking a demotion to move to FAANG. 

Why can people get promoted to ‘Senior’ Engineer so quickly in the software industry? The main reason is the expansion of the workforce. With double digit employment growth each year, there are a lot of inexperienced programmers in the workforce. Having three years of experience is a big deal in an environment where a large portion of working programmers have under three years experience. 

The issue is that three years is not really enough time to master software development. Honestly, in three years you are more of a ‘journeyman’ programmer than a senior one. You wouldn’t expect a three year senior to be able to bootstrap a new engineering team or to set the engineering culture at a new company or division. There is simply too much to learn about how software works. 

In my particular case, I felt that my ability to deliver projects was great. But when it came to designing software abstractions or building software architecture, I felt like I didn’t know enough to do a good job. I had read books like ‘Domain Driven Design’ on software design patterns, but knowing patterns and actually designing software are not the same thing. In my opinion software agency work is not the best place to learn software architecture design. You simply don’t stay embedded on projects long enough to see the longterm costs and evolution of your designs.