Please don’t learn software architecture off the internet 

The internet isn’t the best place to learn software architecture. While you can quickly read about dozens of novel ways to structure software applications the internet tends to pull people towards poor solutions. 

Software architectures are ways of structuring software to support various demands. 

– development speed

– hardware platform (web, mobile, point of sale, drone, etc)

– team structure 

– ability to scale aggressively (If we get a big customer can we scale 100x quickly)

– correctness       (How often do we produce an incorrect result)

– recoverability    (how quickly can we recover from failures)

– supporting a particular load range (100-300 request/second)

– software engineering team competency

The problem is that the internet talks about interesting architectures preferentially over appropriate architectures. If you learn about architectures on the internet you will hear primarily about solutions developed for global scale corporations. These corporations typically employ the highest paid, most experienced engineers on the market and have large operating margins to work with. 

These ‘cool’ architectures typically support billions of operations per day and involve the efforts of 100s to 10,000s of engineers. Then the architect for a team of 20 engineers reads a blog post and decides we need ‘microservices’ or we need ‘event driven’. This sounds reasonable in a meeting. We ‘need to be able to scale if we get a big client’ sounds true. But it’s not. 

People do not understand software scaling. To humans ‘one million records a day’ sounds like a lot. To computers that is only 11.5 records per second, something a single virtual machine running python can handle easily. 

One million 10KB records per day is a problem you buy one hard drive per year to solve. 

If you primarily learn about software architecture online your focus is driven towards optimal hyper scale solutions for hyper competent teams. Your team is not likely to be hyper scale or hyper competent. The ‘software best practice’ for normal teams is not the same as for google. Finding the appropriate software architecture for your company probably requires logging off and talking to experienced ‘enterprise’ engineers. 

The next 10 Years of Remote Work

This post was developed in conjunction with my talk of the same name for Spring into SledgeConf 2021

My belief is that most tech startups going forward will be 100% remote. Setting up an office for a company that might not last isn’t a great idea. And after 2020 spending your runway on an office is basically wasting money. 

The complication is that remote work requires more discipline from the team than office based approaches. If you have unmotivated employees, managers, management skills, or outdated processes you will not be as productive as a remote company. Going forward there will be a dichotomy between startups that try to succeed with the colocation approach and startups that commit to remote work. My belief is that overtime collocated startups will consistently fail against remote startups. The reasons remote startups pull ahead are lower fixed office costs,  easier access to talent at lower prices and that remote work requires more disciplined processes. 

Fixed office costs

Offices being expensive is an obvious fact for most people. Offices are lowest common denominator environments with constant pressure to cut costs. Workers consistently complain about noise and distractions in ‘open plan’ offices but they have stuck around because they are cheap. 

Remote work allows everyone to setup their own ‘dream’ office space. You can get whichever chair you want. Any combination of monitors and other peripherals. You can play loud music or install noise canceling tiles to ensure silence. Workers also save hours each day compared to commuting, not to mention the environmental benefits. 

Easier access to talent

Access to talent is a major problem for startups. Before achieving profitability startups have limited time to find software engineers and limited money to pay them. Hiring remotely allows startups to access a large number of employees who live in low cost of living areas. This can either be lower cost of living cities, cheap small towns or even other countries. 

For example entry level software engineers typically make over 100k in ‘major’ software areas like Seattle, San Fransisco, San Jose and New York City. But those same engineers will typically make 60-70k in second tier cities despite having comparable amounts of skill and experience. 

Higher Process Bar

A remote first company requires better processes and more motivated employees than the typical office micromanagement scenario. Failing to set up good processes in a remote environment will result in unmotivated employees that don’t do any work. You can’t just assume communication will happen in a remote environment. In the office employees gossiping can make up for a lot of bad communication processes. Whereas in a remote environment employees need to go out of their way to setup a zoom call. 

 Companies with good remote work processes will experience higher average productivity than was ever achieved in classical ‘open office’ environments. 

In a remote work environment you are forced to invest more in onboarding, training and cross team coordination. In the office you can simply walk over to the frontend team’s section and introduce yourself to the new team member. But working remotely without a process to include the new guy, he could just sit in the figurative corner for weeks without interacting with anyone. 

There isn’t any way for new hires to ‘overhear’ conversations about services or context that they don’t know about when working remotely. So those thing have to be documented and included in the formal onboarding process. Which means you actually have to document your software for real.

Meetings need to be orchestrated better when done remotely. You need to have an agenda and focus on solving the problem. 

One meeting that works very well remotely is the architectural document review. A 60 minute meeting would go as follows. 

  1. 20 minutes to read and comment on the document in google docs
  2. Short discussion of each text comment made in google docs starting from the top of the doc
  3. Discussion of action items and any other final comments

What happens next?  

The pandemic caused a massive shift in software engineering working conditions. We went from maybe 5% remote to 95% remote work during the crisis. Now that things are normalizing I expect a gradual shift back towards 90-10 or 80-20 split of hours worked in office vs remote. The fact of that matter is that a lot of people like working in an office. And many bosses think micromanaging is necessary. 

Going forward there will be a lot of demand for remote software engineering jobs and many companies will jump on the chance to acquire programming talent at competitive prices. 

The in-office premium.

Going forward I expect to be paid more to work in an office than to work remotely. Working in an office costs me more in time and transportation costs. It also limits my freedom during the day. To make up for the inconvenience of needing living within commuting distance of jobs I expect to be paid an additional 20-30% premium over what I would charge for remote work. 

Legal hurdles 

The legal burden for international remote work is way too high. Each country has different requirements and the company is essentially required to establish a corporate entity in every country they have employees in. Even if they do not actually do business in that country. 

Even in the United States having a remote worker in New York city could require a company to start paying additional taxes to New York state due to establishing a ‘nexus’ in that state. 

Over time I expect these legal hurdles to be simplified or simply ignored. Functionally, it doesn’t matter where code is written. Remote workers will push for legal accommodations. Nations will be forced to compete by providing supportive legal conditions for remote workers and employers. 

https://news.ycombinator.com/item?id=17022563

Other Perks 

Less workplace drama

One thing that I didn’t expect to come out of remote work is that annoying coworkers are a lot less annoying when they aren’t in your face. You can basically avoid all non-work related communication with people you don’t like. For example we have one guy on my team who goes on rants and never gives people a chance to cut in and steer the conversation back on topic. In the office I would end up listening to this guy talk for minutes at a time without getting a chance to talk. Now since I’m not sitting across from the guy I can avoid his negatives and appreciate his dedication and work more. 

Corporate Culture

For example processes like 6 month RSU vesting are not tenable in a remote first world. You cannot afford for disgruntled employees to wait 6 months for a vest date before they quit. 

Spring into SledgeConf!

https://sledgeconf.dev

Our next SledgeConf is coming on Friday May 21st at 4:30pm PST!

Join us for discussion of the next 10 years of remote work. 

My presentation for this SledgeConf is titled. 

“The next decade in remote work.” 

I’m joined by Mason Traylor presenting!

“How to waste Everybody’s Time with Remote Meetings”

Agenda:

4:30 Introductions 

4:45 Presentation “How to waste Everybody’s Time with Remote Meetings”

5:00 Questions + Discussion

5:15 Presentation “The next decade in remote work.” 

5:45 Questions 

6:00 Promotional Round table 

6:15 Unstructured Discussion time

The zoom link will be sent to the mailing list the day of SledgeConf. Please go to SledgeConf.dev and sign up for the mailing list. If you are one of my connections on linkedin the zoom link will also be available there.

https://sledgeconf.dev

Pros and cons of working at a consulting company

It seems like we get lots of questions on the subreddit r/cscareerquestions about working at a consulting company. I worked at a small company that did a mix of staff augmentation, application development, devops, and agile process consulting. Here are some thoughts on the pros and cons.

Pros

  • Work on lots of projects and technologies 
  • Willingness to train you on the job
  • Teams rotate frequently – get to know lots of people 
  • Don’t have to worry about legacy code as much
  • No Oncall rotation 

Cons

  • Don’t run code in production 
  • Travel a lot
  • Don’t own your code or service 
  • Experience is shallow and short term

I worked at a software consulting company for about four years right after college. I had the choice between the consulting company that did work in the cloud or a company that made a medical software system, I went for the cloud. The company had around 100 employees with some shifts up and down. As far as work, we did the whole gamut from staff augmentation, agile training, ‘digital transformation’, migrated people to the cloud, built platforms as a service, and more. 

During my consulting days, I worked on a ton of projects for different companies, wrote code in different languages, and traveled a bit. For the first two years I worked on a variety of java projects for a cable company. We did around 90% of our business with that company at the time. Working with cable company employees is kind of a drag, they were focused more on ‘resting and vesting’ and frequently missed deadlines. 

My next project was with a new client in Boston and we started flying in. I was tapped for staff augmentation to help the client create a devops team. They ran their own datacenter and mainly had system administrators. I worked with this team on typical infrastructure automation as we set up a Platform as a service for the companies projects on AWS. I learned a lot at this client, managed to insult one of their directors and almost got fired, and that I didn’t particularly enjoy devops. 

Next I worked on a few Machine Learning proof of concept projects for a client and became a certified kubernetes administrator. My former employer invested heavily into Kubernetes and it paid off with press and connections. My last year I spent as a team lead running development of an internal project that we hoped to resell to clients. That project is what finally motivated me to find a new job.

I worked around 40 hours a week except when I was onsite with a client when it was more like 60 hours. Getting onsite with clients is actually pretty exciting, and I didn’t have to do it too often. I traveled around once a month for one of the years at this job. 

Pros

Work on lots of projects and technologies 

I worked at a consulting firm early in my career and getting experience with lots of technologies and experiencing different company cultures really helped. I have a much better idea of the industry after working with 10 companies in 4 years than if I had worked at one company for that time. 

Willingness to train you on the job

My employer was extremely willing to train people on the job. Part of our business model was billing out cheap junior developers with just enough supervision to get things done. The company paid for me to get certifications in Cassandra and Kubernetes. 

Teams rotate frequently – get to know lots of people 

My longest project lasted 9 months. In consulting once a project ends, the team disbands and you start up on a new project with a new team. Over time you get to work with almost everyone in the company directly. You will also work with lots of different clients and need to build rapport and a good working relationship quickly.

Don’t have to worry about legacy code as much

I didn’t work with a lot of legacy code as a consultant. We were mainly brought in to help launch new projects. A six month project really doesn’t generate that much legacy code. Then once times up, you move on to another project. 

No Oncall rotation 

As a consultant I never had an oncall rotation. We didn’t have any services of our own. 

Cons 

Don’t run code in production 

One of my greatest frustrations as a consultant was that I didn’t get any experience running code in production during that job. We wrote the code then handed everything off to our clients operations teams. I think this made it harder for me to find a new job. Since I started running my own code in production I have learned a ton.

Travel a lot

I didn’t have to travel that much as a consultant. My best year I did around 10k miles. Of course as you move up the ladder, you will end up traveling more. 

Don’t own your code or service 

As a consultant you are working for another company. Their teams will own the code. Their Product department makes the calls. You will often just get the requirements and be shaking your head at how wrong they are. Decisions will be made by the client that you don’t like. But the client will get what they want 99% of the time. 

Experience is shallow and short term

The flip side of working on such short projects an so many different technologies is that you never get to master any of them. One project is in react the next project is in Angular, the next project is devops, with totally different tooling to learn for each of them. Consulting is not great for building deep knowledge as a software engineer.