What makes a low tier software engineering culture? 

In my conceptualization a low tier software engineering culture is one where engineers have low wages, low accountability, little responsibility and little control over the software development lifecycle. In my experience this is typically due to economic reasons more so than ineptitude. 

One example is specification work for hardware integration projects. This article talks about how a culture of spec work killed the Japanese software industry for decades https://www.disruptingjapan.com/the-forgotten-mistake-that-killed-japans-software-industry/. These kinds of projects have inflexible deadlines, clear feature requirements and no customer flexibility. Customer’s can’t shop around for toaster software. The result is that engineers have little freedom or control over the software product. And since the software is merely a cost factor, little compensation is available for the developers. 

Another example is the internet service provider industry. Companies like Comcast, Cox, Charter Communications hold regional monopolies over consumers. The software they build on isn’t the product and customers typically have no alternative source for internet. As a result around the 1990s most of these big cable companies outsourced their information technology departments. 

These types of companies typically have little competitive pressures around the software they produce. The software just needs to work on average. You can’t shop around for a different operating system for your sedan. As long as the car works on average the software is good enough. 

The problem is that these kinds of software culture provide very bad worst case scenarios. Take a look at this article on security vulnerabilities in automobiles https://samcurry.net/web-hackers-vs-the-auto-industry. Automobiles manufacturers as an industry seem to produce very bad outcomes in software. 

The kinds of software engineering cultures that produce software that ‘merely works’ seem to produce software that is insecure, buggy and doesn’t ‘delight’ customers. But this is driven by business dynamics more so than teams or methodologies. Kia has no competitors in the market of software for Kia cars. You have no alternative to the software that operates your smart fridge. 

Funnily enough there is a solution to this problem. Toasters, fridges and cars are all powered by general purpose computers. The same kinds of computers that we all use everyday. Just change the automobile into a platform that accepts an operating system. Just like any personal computer can be converted to any operating system you chose. Is that going to be easy? Probably not, but structurally there is little difference between your PC and the PC that runs your automobile. 

Low tier engineering cultures are hard to fix. Due to the way the business works there is little money or freedom available to give to the engineering team. As a result skilled members of the team can easily move elsewhere and get paid significantly more. It is very hard to avoid a dead sea effect across the entire company and you end up with massive security issues like the automotive industry. 

Please don’t learn software architecture off the internet 

The internet isn’t the best place to learn software architecture. While you can quickly read about dozens of novel ways to structure software applications the internet tends to pull people towards poor solutions. 

Software architectures are ways of structuring software to support various demands. 

– development speed

– hardware platform (web, mobile, point of sale, drone, etc)

– team structure 

– ability to scale aggressively (If we get a big customer can we scale 100x quickly)

– correctness       (How often do we produce an incorrect result)

– recoverability    (how quickly can we recover from failures)

– supporting a particular load range (100-300 request/second)

– software engineering team competency

The problem is that the internet talks about interesting architectures preferentially over appropriate architectures. If you learn about architectures on the internet you will hear primarily about solutions developed for global scale corporations. These corporations typically employ the highest paid, most experienced engineers on the market and have large operating margins to work with. 

These ‘cool’ architectures typically support billions of operations per day and involve the efforts of 100s to 10,000s of engineers. Then the architect for a team of 20 engineers reads a blog post and decides we need ‘microservices’ or we need ‘event driven’. This sounds reasonable in a meeting. We ‘need to be able to scale if we get a big client’ sounds true. But it’s not. 

People do not understand software scaling. To humans ‘one million records a day’ sounds like a lot. To computers that is only 11.5 records per second, something a single virtual machine running python can handle easily. 

One million 10KB records per day is a problem you buy one hard drive per year to solve. 

If you primarily learn about software architecture online your focus is driven towards optimal hyper scale solutions for hyper competent teams. Your team is not likely to be hyper scale or hyper competent. The ‘software best practice’ for normal teams is not the same as for google. Finding the appropriate software architecture for your company probably requires logging off and talking to experienced ‘enterprise’ engineers. 

Links Post

Real time analytics article

https://adamsinger.substack.com/p/demystifying-real-time-analytics?utm_source=url&s=r

Danluu on the issues with purchasing solutions.

https://danluu.com/nothing-works/

Nuclear Stuff 

https://austinvernon.site/blog/nuclear.html

https://whatisnuclear.com/economics.html

Are we really engineers?

https://www.hillelwayne.com/post/are-we-really-engineers/

SledgeConf July 15th 2022

4:30PM Pacific Time

https://sledgeconf.dev

Hey, I’m bringing back Sledgeconf on July 15th. We have two talks this time ‘Planning your Cloud Migration’ and ‘Practical Software Estimation’. 

“Planning your Cloud Migration”

Jay Manning joins us again to talk about planning and executing Cloud Migrations. Jay has extensive experience with AWS partners working through their Cloud journey. 

“Practical Software Estimation” 

I will be speaking on my research and interviews on the subject of practical software estimation. How do we estimate software today and what are the best practices?

Make sure to join the mailing list to get the invite link and blog updates! See you all in July!

https://sledgeconf.dev