I started my software engineering career in consulting. We did application development for large companies that did not have enough engineers on staff to get their projects done. We had a Statement of Work and a list of stories to do. Then we handed over the code and went back to our nice benches. Consulting was a big thing, we had executives, million dollar contracts, etc. I often ran in situations where I was spinning my wheels waiting for people to make decisions or to sell another contract. We ran some internal projects, but they were always an after thought.
Now I work in a product company somewhat similar to the companies we used to consult with. We have what is essentially an unlimited amount of work. There isn’t any time for anyone to be ‘on the bench’. We have enough work for double our headcount. I have 5 real, valuable projects that I want to work on. It is just an issue of prioritizing and getting things done. But the thing is, a lot of stuff is falling through the cracks.
We don’t have time to do everything, but we do need a lot of things improved.
We could use an expert on CI/CD pipelines to come in for a month and upgrade our pipeline. We could use a java performance expert to come in and help us cut end user latency by 50%. We could use a consultant to help us break up the monolith. We have opportunities for a dozen consultants, but from the outside you don’t see the internal workings of the product company. You can’t see that we have hundreds of ways to make the product better and no where near enough work to do it.
Following on my earlier post “How much is turnover costing your company?” this post is about how much using non-standard technology stacks costs your company. Lets say company A is high performance company with a custom functional programming stack that uses Haskell and Miso for development. Company A doesn’t ship a lot of bugs to production, in fact they run CI/CD with deployments to prod everyday. Their SAAS product is one of the best in the industry and is growing at a good rate.
Now Company A has a competitor in the form of Company B. In this case B does stand for Boring. Company B offers a SAAS product that is almost identical to the one Company A sells but Company B’s has more bugs because they use Java. In fact Company B uses Java with Spring Boot and ReactJS. Both Company A and Company B started in 2018 when React was already mainstream. Company B also uses CI/CD deployments but they can only deploy to production once a week because there are so many bugs. Company B also hired a QA Engineer to help support the pipeline because of an embarrassing outage that CompanyA used in a humiliating Ad.
In January 2019 both companies raise a round of founding for 10 million dollars. The mandate from the VCs is to 10x the company in size and destroy the competition. Both companies need to start hiring fast. Throughout 2019 CompanyA has an advantage, there are a bunch of passionate Haskell engineers that want to work on their product. Both Company A and B reach their growth targets and hit 100 engineers that year.
Fortunately, things go well for both companies in 2020. The market is growing and revenues are increasing. The VCs call for more growth. Now we want to grow engineering to 500 engineers.
Now Company A has a problem. They already hired all the passionate Haskell+Miso engineers they could find. Any new engineers will take 12 months to train on their technology stack. CompanyB doesn’t have this problem everyone still remembers how ReactJS and Spring Boot work even though they are no longer the top technologies. Company B only needs 6 months to train a new engineer on their technology stack.
Lets run the numbers. How much is it going to cost to onboard 400 new engineers for each company?
We will keep our 2 year average tenure estimate and use 12 months of onboarding for company A. Each engineer has a total cost of 200k including salary, equity, benefits, taxes and insurance.
Each engineer is going to cost around 100k to onboard at Company A.
Each engineer is going to cost about 50k to onboard at Company B.
Now the next question, how long will it take each company to get all 400 new engineers fully operational?
My assumption is that it takes 1 fully onboarded engineer to train a new engineer. Each company has 100 engineers to start so at the beginning the first 100 engineers will train the next 100. Then each company will have 200 engineers to train the next 200. Then things calm down a little and we have 400 engineers to train the last 100.
After 12 months Company B has double the trained engineers Company A has. At 18 months Company B is finished onboarding and can have all 500 engineers focus on the product. Company A won’t catch up for another 18 months.
Most people will agree that Company A’s functional programming stack is not going to make up for the extra engineer years that Company B has at this point.
The software industry has a turnover problem. The general consensus is that it takes around 6-12 months to fully onboard a new hire. Essentially, you hire someone for 100k a year. Then it takes roughly 6-12 months before that person starts producing work at the 100k level. This would be fine if Software Engineers did not change jobs frequently, but we tend to switch jobs every 1-3 years for higher pay.
Let us consider the case of a company with a complex project and slightly high turnover. They need a full 12 months to onboard a new engineer and they can expect that engineer to stay at the company for 2 years in total. If onboarding proceeds linearly that company ends up paying 2 years pay for 1.5 years worth of results. The company is spending 25% of salary on training for the lifetime of that employee.
If the company managed to get the average lifetime per employee up to 3 years they would spend 3 years pay on 2.5 years of work or roughly 17% of salary on training.
A company in this situation could afford to give an 8% raise if it kept an employee around for an additional year.
I had the opportunity to take the [CKA](https://www.cncf.io/announcement/2016/11/08/cloud-native-computing-foundation-launches-certification-training-managed-service-provider-program-kubernetes),
‘Certified Kubernetes Administrator’ exam because my employer is trying to get ‘Certified Kubernetes Service Provider’ status, which requires 3 certified administrators. Now, why would Certified Kubernetes Administrators or service providers be valuable for your company? The CKA is good because it certifies a base level of knowledge and ability in kubernetes administrators. Things you would expect from CKAs are the ability to debug clusters, perform upgrades, bootstrap clusters and application deployment tasks.
What and how does the exam test administrators? The exam tests your ability to perform operations against the kubernetes api. The entire exam period is spent in the command line with kubectl on a standard linux shell. The test validates your ability to ‘get things done’ in the kubernetes environment.
Overall, I have a very positive outlook on the exam. I spent about a week preparing. I read a lot of the kubernetes.io documentation, ran through ‘kubernetes the hard way’ three times and had worked with kubernetes on the application side previously. I passed on the first try, but I did need all of the exam time and had to skip a few of the hard problems.
I don’t think that the CKA is essential for devops or kubernetes admins, but it is a good exam and great for filling some of the gaps you might have in your knowledge.
