Pros and cons of working at a consulting company

It seems like we get lots of questions on the subreddit r/cscareerquestions about working at a consulting company. I worked at a small company that did a mix of staff augmentation, application development, devops, and agile process consulting. Here are some thoughts on the pros and cons.

Pros

  • Work on lots of projects and technologies 
  • Willingness to train you on the job
  • Teams rotate frequently – get to know lots of people 
  • Don’t have to worry about legacy code as much
  • No Oncall rotation 

Cons

  • Don’t run code in production 
  • Travel a lot
  • Don’t own your code or service 
  • Experience is shallow and short term

I worked at a software consulting company for about four years right after college. I had the choice between the consulting company that did work in the cloud or a company that made a medical software system, I went for the cloud. The company had around 100 employees with some shifts up and down. As far as work, we did the whole gamut from staff augmentation, agile training, ‘digital transformation’, migrated people to the cloud, built platforms as a service, and more. 

During my consulting days, I worked on a ton of projects for different companies, wrote code in different languages, and traveled a bit. For the first two years I worked on a variety of java projects for a cable company. We did around 90% of our business with that company at the time. Working with cable company employees is kind of a drag, they were focused more on ‘resting and vesting’ and frequently missed deadlines. 

My next project was with a new client in Boston and we started flying in. I was tapped for staff augmentation to help the client create a devops team. They ran their own datacenter and mainly had system administrators. I worked with this team on typical infrastructure automation as we set up a Platform as a service for the companies projects on AWS. I learned a lot at this client, managed to insult one of their directors and almost got fired, and that I didn’t particularly enjoy devops. 

Next I worked on a few Machine Learning proof of concept projects for a client and became a certified kubernetes administrator. My former employer invested heavily into Kubernetes and it paid off with press and connections. My last year I spent as a team lead running development of an internal project that we hoped to resell to clients. That project is what finally motivated me to find a new job.

I worked around 40 hours a week except when I was onsite with a client when it was more like 60 hours. Getting onsite with clients is actually pretty exciting, and I didn’t have to do it too often. I traveled around once a month for one of the years at this job. 

Pros

Work on lots of projects and technologies 

I worked at a consulting firm early in my career and getting experience with lots of technologies and experiencing different company cultures really helped. I have a much better idea of the industry after working with 10 companies in 4 years than if I had worked at one company for that time. 

Willingness to train you on the job

My employer was extremely willing to train people on the job. Part of our business model was billing out cheap junior developers with just enough supervision to get things done. The company paid for me to get certifications in Cassandra and Kubernetes. 

Teams rotate frequently – get to know lots of people 

My longest project lasted 9 months. In consulting once a project ends, the team disbands and you start up on a new project with a new team. Over time you get to work with almost everyone in the company directly. You will also work with lots of different clients and need to build rapport and a good working relationship quickly.

Don’t have to worry about legacy code as much

I didn’t work with a lot of legacy code as a consultant. We were mainly brought in to help launch new projects. A six month project really doesn’t generate that much legacy code. Then once times up, you move on to another project. 

No Oncall rotation 

As a consultant I never had an oncall rotation. We didn’t have any services of our own. 

Cons 

Don’t run code in production 

One of my greatest frustrations as a consultant was that I didn’t get any experience running code in production during that job. We wrote the code then handed everything off to our clients operations teams. I think this made it harder for me to find a new job. Since I started running my own code in production I have learned a ton.

Travel a lot

I didn’t have to travel that much as a consultant. My best year I did around 10k miles. Of course as you move up the ladder, you will end up traveling more. 

Don’t own your code or service 

As a consultant you are working for another company. Their teams will own the code. Their Product department makes the calls. You will often just get the requirements and be shaking your head at how wrong they are. Decisions will be made by the client that you don’t like. But the client will get what they want 99% of the time. 

Experience is shallow and short term

The flip side of working on such short projects an so many different technologies is that you never get to master any of them. One project is in react the next project is in Angular, the next project is devops, with totally different tooling to learn for each of them. Consulting is not great for building deep knowledge as a software engineer. 

Everyone uses (failing) software all the time.

Because you use it all the time at least one piece of software is broken for you at all times.

I stopped using Facebook after my freshman year of college, but recently got pulled back in by a Facebook group. As a result I now have the pleasure of enjoying a 10+ second loading phase every time I open the homepage. 

Recently, I tried to buy a CODE mechanical keyboard on the wasdkeyboards.com website. But every time I submitted my order it failed. I tried different browsers. I had to look into the console to find out that a http request was failing to find a paypal advertising domain that my PiHole blocks on the network. To buy my keyboard I had to tether wifi from my smartphone. A non-technical user wouldn’t have been able to find out why the order failed because there was no error message. There was a spinning symbol that just disappeared after a while without a message to the user. 

Everyone uses software all the time now. We have smartphones, smart TVs, smart refrigerators and smart homes. If you use 100 programs a day, 99% uptime means one program is down for every person. If every application manages 99.9% uptime, one out of a hundred people is experiencing software brokenness everyday. 

Then realize that billions of people have smartphones now. 

99.99% * 1,000,000,000 = 100,000. 

If your software has a billion users and works 99.99% of the time, its down for 100,000 people all the time. 

Software updates are a liability for customers

Software Updates are a liability for customers. Imagine a business, say a restaurant, which uses some POS (point of sale) software to run its business. Every time the software is updated the business’s employees have to relearn some of the software. That costs money. They would rather buy software that never needs to be updated.

You might say “Well, what if there are security vulnerabilities”. Fixing those vulnerabilities still costs money. Customers will have to update their software which costs money even if there is no functional change. Its essentially saying, “Hey, Customers, we shipped you a defective product, please spend money to get the update which fixes this vulnerability”. 

Many companies have moved to SAAS (software as a service) which solves the problem of getting customers to install updates by forcing them to. Now, instead of customers constantly digging in their heels and refusing to update anything, we can ship them updates constantly. 

Customers hate software updates. They are slow, buttons move around, and people lose the fluid mastery that they painstakingly learned. Software should empower customers, every time you ship an update that breaks fluid mastery, customers are kicked off of Olympus. 

Links Post — Q4 2019

Here are some interesting links I’ve come across over the last few months. 

htop explained 

https://peteris.rocks/blog/htop/#htop-on-ubuntu-server-16-04-x64

4 Types of Documentation 

https://www.divio.com/blog/documentation/

Making a Language Server 

https://nickmqb.github.io/2019/11/24/building-a-language-server-for-muon.html

Don’t move to the Cloud to increase CPU utilization

I have worked with a bunch of companies that launched major initiatives to move their hardware onto the public cloud. None of those companies managed to get their CPU utilization over 10%. At my current job we run Java with 10-20% cpu utilization and 90% memory reserved for the JVM. The standard software development approach does not result in amazing hardware utilization rates. 

The consulting clients we worked with at my previous job expressed a lot of interest in increasing efficiency. We theorized complex tagging and did proof of concepts with Cloudability. But I do not recall actual savings coming out of it. Although there was a lot of complaining about the AWS bill being high. 

One Fortune 500 company I worked with had a dev environment with a huge number of hosts (1000+) that were basically never utilized at all. That company only used continuous delivery in development, not for production deployments. The other obvious issue at that company was their reluctance to rely on AWS Autoscaling groups to handle load spikes. They allocated for peak load despite running in the cloud. 

One concern that came up a couple times was that Autoscaling groups have a scaling response rate in the order of minutes. In the event of a traffic spike, it might be 5-10 minutes before extra hosts come online. 

If you are worried about unexpected instantaneous peaks write a fallback. Serve a landing page our of cache and sit tight. There is no magic solution to instantly increasing your traffic by 100x without scaling preemptively. 

The largest websites in the world scale ahead of time. We know when we will get lots of traffic historically. You know when your Super Bowl add is going live. Scale up a week before hand. Run load tests to make sure you can handle the traffic. 

Run your servers at 30-60% utilization. Build a fallback page for big instantaneous peaks. Most importantly know ahead of time what your traffic is going to look like so you can prepare.