Trying a new language won’t solve your development problems. 

I’ve had a conversation where someone proposes trying a new language like Go or Scala or Typescript for new development several times. Typically, the argument is that we have problems x, y and z with our current language and they would all be better under $new_language. The arguments are always things like x has better performance, y has better typing, z is more functional. 

There is no free lunch within modern programming languages. There are no opportunities to switch to a new language without paying new costs. Java requires a lot of boiler plate and is very heavy weight. Typescript is async by default and has non-reliable typing. Go has a litany of problems (tool chain, exceptions, generics, etc) downstream of Google making all the serious decisions for the language. Rust is very strict and gives you excess performance you can’t use. 

Switching between any of the modern languages means trading some benefits for some negatives. Unless you have a serious problem with your current language you are unlikely to gain anything. 

An exception is switching from Ruby, or Python to something like Java or Go. Then you would gain a lot of performance. But first ask yourself whether any of your services max out a single server. Are you even using instances larger than your laptop? Can the performance problem be solved by a refactor and re-architecture in the same language? Re-writing is time consuming and if your team already knows Ruby a re-architect in the same language will usually be faster and produce better code. 

Programming languages have trade offs. Switching is an expensive process. It always takes longer and costs more than you expect. And after it all by switching languages you are just switching old problems for new problems. 

What makes a low tier software engineering culture? 

In my conceptualization a low tier software engineering culture is one where engineers have low wages, low accountability, little responsibility and little control over the software development lifecycle. In my experience this is typically due to economic reasons more so than ineptitude. 

One example is specification work for hardware integration projects. This article talks about how a culture of spec work killed the Japanese software industry for decades https://www.disruptingjapan.com/the-forgotten-mistake-that-killed-japans-software-industry/. These kinds of projects have inflexible deadlines, clear feature requirements and no customer flexibility. Customer’s can’t shop around for toaster software. The result is that engineers have little freedom or control over the software product. And since the software is merely a cost factor, little compensation is available for the developers. 

Another example is the internet service provider industry. Companies like Comcast, Cox, Charter Communications hold regional monopolies over consumers. The software they build on isn’t the product and customers typically have no alternative source for internet. As a result around the 1990s most of these big cable companies outsourced their information technology departments. 

These types of companies typically have little competitive pressures around the software they produce. The software just needs to work on average. You can’t shop around for a different operating system for your sedan. As long as the car works on average the software is good enough. 

The problem is that these kinds of software culture provide very bad worst case scenarios. Take a look at this article on security vulnerabilities in automobiles https://samcurry.net/web-hackers-vs-the-auto-industry. Automobiles manufacturers as an industry seem to produce very bad outcomes in software. 

The kinds of software engineering cultures that produce software that ‘merely works’ seem to produce software that is insecure, buggy and doesn’t ‘delight’ customers. But this is driven by business dynamics more so than teams or methodologies. Kia has no competitors in the market of software for Kia cars. You have no alternative to the software that operates your smart fridge. 

Funnily enough there is a solution to this problem. Toasters, fridges and cars are all powered by general purpose computers. The same kinds of computers that we all use everyday. Just change the automobile into a platform that accepts an operating system. Just like any personal computer can be converted to any operating system you chose. Is that going to be easy? Probably not, but structurally there is little difference between your PC and the PC that runs your automobile. 

Low tier engineering cultures are hard to fix. Due to the way the business works there is little money or freedom available to give to the engineering team. As a result skilled members of the team can easily move elsewhere and get paid significantly more. It is very hard to avoid a dead sea effect across the entire company and you end up with massive security issues like the automotive industry. 

Please don’t learn software architecture off the internet 

The internet isn’t the best place to learn software architecture. While you can quickly read about dozens of novel ways to structure software applications the internet tends to pull people towards poor solutions. 

Software architectures are ways of structuring software to support various demands. 

– development speed

– hardware platform (web, mobile, point of sale, drone, etc)

– team structure 

– ability to scale aggressively (If we get a big customer can we scale 100x quickly)

– correctness       (How often do we produce an incorrect result)

– recoverability    (how quickly can we recover from failures)

– supporting a particular load range (100-300 request/second)

– software engineering team competency

The problem is that the internet talks about interesting architectures preferentially over appropriate architectures. If you learn about architectures on the internet you will hear primarily about solutions developed for global scale corporations. These corporations typically employ the highest paid, most experienced engineers on the market and have large operating margins to work with. 

These ‘cool’ architectures typically support billions of operations per day and involve the efforts of 100s to 10,000s of engineers. Then the architect for a team of 20 engineers reads a blog post and decides we need ‘microservices’ or we need ‘event driven’. This sounds reasonable in a meeting. We ‘need to be able to scale if we get a big client’ sounds true. But it’s not. 

People do not understand software scaling. To humans ‘one million records a day’ sounds like a lot. To computers that is only 11.5 records per second, something a single virtual machine running python can handle easily. 

One million 10KB records per day is a problem you buy one hard drive per year to solve. 

If you primarily learn about software architecture online your focus is driven towards optimal hyper scale solutions for hyper competent teams. Your team is not likely to be hyper scale or hyper competent. The ‘software best practice’ for normal teams is not the same as for google. Finding the appropriate software architecture for your company probably requires logging off and talking to experienced ‘enterprise’ engineers.