Burnout or health problem?

I experienced a lot of burnout last year. Usually, I can just outlast burnout and it goes away when I start a new project. This time it lasted most of the year and I didn’t really get back to normal until 2021. In my case the reason my ‘burnout’ lasted so long and was hard to get rid of is because it was actually a health problem. I got into this industry because I love coding but I didn’t really feel that way last year and even considered leaving the industry. I had a lot of troubling focusing with the single minded obsession you need to beat down tricky bugs. At the time I thought I had burnout, I was depressed, it had to be something like that right? 

Well in this case it wasn’t burnout or depression, but my diet. I’ve had issues with various foods making me sick in the past, but never really isolated the cause beyond ‘don’t buy bread with preservatives in it’. I eat out a lot and while I’d thought about doing an exclusion diet several years ago, and have built apps in the past to help people isolate food ingredients they have issues with, I’d never actually done an exclusion diet myself. 

In the end I started an exclusion diet in January after moving into my new apartment here in Phoenix. The effects in my case have been totally worth it, as I now know which foods I can and cannot eat if I want to think clearly. I can’t really say I understand the medical causes behind things, but a number of foods give me a combination of brain fog, headaches and stomach discomfort. It isn’t life threatening beyond making me suicidal on occasion. But I really can’t afford to eat anything with Sage in it without ruining an entire day. 

So far I’ve had to exclude Sage, preserved meats, and wheat or gluten products. I can eat most other foods without a problem.  

Software advances are slower than you expect.

Most people think of technological advances using the eureka metaphor. But software doesn’t work like that. Take a clear technological advance like a self-driving tractor. They are on the market now but there was no eureka moment, no sudden breakthrough. Self-driving technology just advanced to the base level required for tractors on clearly mapped fields, then a team of software engineers built a working system over several years.

There’s no invention or breakthrough moment, just a slow build in none software capabilities than an investment in building out software to leverage those capabilities. 

There were no software engineering advances that enabled the self-driving tractor. Instead machine vision improved until it was good enough to unblock the software solution.

What advances in software are in sight?

Reproducible builds are one of my favorite advances in software recently. Strictly speaking reproducible builds have been available for quite some time, but that doesn’t mean it isn’t a real advance. Having a trustworthy build environment where you can debug the inputs vs the outputs for different machines is a big benefit. Without reproducible builds software engineers end up spending extra time fixing build failures and figuring out dependency resolution overrides. 

Golang is in my opinion an advancement in software engineering. Instead of focusing on productivity for a single or small number of engineers, Golang is focused on maximizing productivity for large software projects with thousands of engineers. It does this by simplifying the language as much as possible to ease readability and eliminate magical effects. Everything in Golang is very procedural and easy to follow. 

Easy to use gradual typing is a more recent advancement in the Python and Ruby worlds. You can now build your MVP in a dynamic quick evaluating language and then after you reach product market fit you can add types to the code base. Typing is no longer and either/or thing but a sliding scale where you can chose the most opportune time to transition. Overall I think gradual typing has huge advantages compared to the old standard of rewriting the codebase in Java. 

January Links

Scientists Say You Can Cancel the Noise but Keep Your Window Open

They will integrate these speakers into windows/walls and make it smaller

Concept of ‘feature store’ for typed ML model inputs (tensors, vectors, etc)

https://www.logicalclocks.com/blog/feature-store-vs-data-warehouse

VM performance tests, very good blog series.

https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_1.html
https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_2.html

Compressing for pub/sub results in great savings.

https://blog.lawrencejones.dev/compress-everything/

The meta programming problem with functional programming in software leviathans.

Few of the software leviathans are built in functional languages. Facebook uses PHP/Hack, Google Java, C++, Amazon Java, Netflix Java. The common consensus about functional languages is that they provide large benefits over object oriented and procedural languages like Java. One particular claim is that functional languages like Haskell can do the same work in 1/10th the lines of code. If functional languages really are better we would expect to see the big tech companies investing heavily in adopting functional languages. We might even expect them to create a functional programming language just for their use case, but instead Google created Go possibly the least functional programming language created in the 21st century. What is going on here? Why aren’t functional programming languages being adopted in the biggest software systems on the planet? 

People have argued that inertia is the explanation for the low adoption of functional programming languages in massive software projects, but I think the evidence is in the opposite direction. Google created an entire new language that was intentionally less functional than Java. Facebook started on PHP and then extended that language into Hack. They could have used that energy to completely adopt Haskell. 

My suspicion is that the real reason functional languages are not used in massive software leviathans is meta-programming. Meta-programming enables software developers to create custom domain specific languages, literally adding new programming syntax and expressions to the code base. This is an incredible power and can make a lot of problems much easier. But meta-programming does not scale.  

In a software project with 10,000 software engineers. At this scale the limiting factor is not our ability to write clean and concise code. The main issue is understanding the effects of changes to the code base. A change might take a month to research before changing 500 lines of code. Not doing your research upfront more likely then not will result in you starting the project than realizing 2 weeks in that your approach will never work. Then having to start over. 

Meta-programming falls under the set of programming constructs that are easier to write than they are to read. This is true for all code of course, but in large code bases reading Golang code is reliably easier than reading Lisp code. 

In a algorithmic metaphor, Golang code complexity scales at O(n^2) vs Lisp code scaling at O(n^3). 

Software Leviathans

Dis-economies of scale, why FAANG pays high salaries, the dominance of Java

The top end of software engineering jobs are dominated by what I’ve started thinking of as ‘Software Leviathans’, large software systems that are staffed by thousands of engineers. A few that come to mind are Amazon Alexa, Amazon.com, Google Search, Salesforce, Facebook.com. These are not “monoliths’ or large services that do everything. Instead they are the result of combining 100s of smaller ‘micro-services’ into one massive software product. 

These leviathans do many many things, few people on the planet can claim to know all of the features of facebook.com. It is quite possible that there exists no single list that enumerates every feature in that product. 

Similarly, development on these systems happens in parallel across many teams. It it is essentially impossible for any one person to keep track of everything that is being added to the system. 

Leviathans are too big for anyone to understand. It doesn’t matter what architecture or runtime choices are made. It could be one massive JVM, a million lambda functions, a hundred thousand docker containers or thousands of micro-services. Even if you work on the leviathan, you won’t have any real understanding of the total state of the system. Each engineer will be aware of and communicate with a tiny fraction of the total number of people working inside the leviathan. 

Leviathans are heterogeneous systems. The do not do ‘one thing well’. Leviathans do everything you can think of. Google.com is a search engine, but it’s also a calculator, an advertising system, a web scraper, a hotel booking tool, a flight booking tool, and many more. Leviathans grow in parallel, across myriad tentacles of functionality. New features emerge all the time usually to the surprise of other engineers on the project. 

Leviathans are difficult to work in. Despite appearing to be a sea of constant change from the outside. Any change made inside the Leviathan is extremely expensive in engineering hours. There are thousands of potential interactions each engineering team has to consider when evaluating changes to their system. The architecture must be constrained heavily to support parallel development in environments where coordination between different teams is impossible due to scale. Engineers working on a software leviathan spend a relatively small fraction of their time actually writing code as compared to debugging issues, research, coordinating changes, and documenting. 

Leviathans are interesting because they are the ‘core’ services powering the digital world these days. Their scale is at top of the chart in the software engineering world and as a result they expose the limitations of software engineering. 

Software diseconomies of scale are at their most evident in these software leviathans. They are massive projects with huge numbers of the best engineers working on them. But development is slow per engineer and code quality is not clearly superior to industry best practices.