Away team work

Away team work is a critical component of high performance software organizations. It is a way for high priority teams to work around other teams’ manpower constraints to deliver software. Without a well established culture of away team project work your organization will default to a standard of ‘shut up and wait while your item is in the backlog’. 

What is away team work exactly? Away team work is when your team implements a feature or integration in another team’s codebase. Strictly speaking in away team work the host team does not review or approve pull requests. Your team’s senior engineers will get approval at an architectural level then handle pull request review themselves. 

The Away team process allows your team to unblock itself when dependencies don’t have manpower to spare. The key component is first that your team not the dependencies team does the work. Both the implementation, testing and code review. Some effort obviously is required of the host team, but it should be minimized. 

Away team work acts as a release valve on the conflict between the host team’s priorities and potential client’s goals of delivering features to customers. If the host doesn’t have manpower available your team can provide the engineers to make it happen. 

Without away team work your organization will have to make more tradeoffs on the scheduling side. More often than not you’ll have to pick a migration over the new feature. But Away team work would have enabled your organization as a whole to deliver both. 

What do you need to make away team work happen?

The key thing is that you have to make a formal process that specifies the maximum standards. You can’t allow teams to be squeamish about it. If the requirement is that two senior engineers approve that’s fine. Or maybe only senior engineers can do away team work in your company. Thats fine but you need a formal standard to avoid negotiations happening on a per project basis. 

The core problem of micro services – how many feature can you fit through the pipeline before it breaks down?

At Amazon I had the chance to watch a monolithic service reach the point where it had to be split up into microservices. When I started we had about thirty software engineers contributing to our service. We had the great idea of developing a framework to speed up feature delivery for everyone in Alexa Shopping. The framework ended up working and the number of people contributing code to our service shot up over the next two years. As of July 2021 we had around 200-300 people in our partner support slack channel.

What happened is that we gradually spent more and more time supporting tests in our CD pipeline. First we had an oncall who did operational support and pipeline release support. Then once that person got swamped and complaints about release frequency got louder we added a second person to operational support. Then we had one person doing releases and outages with a second doing partner pull requests and office hours. 

Growth continued during this time and we attempted a number of changes to federate features and split out responsibility. We split end to end tests into separate suites so it would be easier to find out who knew what a feature was supposed to be doing. This helped a lot, prior to federating the test suites our oncalls would spend a lot of time deep diving end to end tests to figure out what was going on. Afterwards it was a lot easier to find out who we could ask for help. 

One thing that was a huge failure was expecting other teams to debug and fix their end to end tests in the pipeline. Typically, team A has a launch deadline and it asking us to deploy now. We get their code to the staging environment and we see test failure from teams B, C, and D. Teams B, C and D do not have a launch deadline so they are not prioritizing fixing their end to end tests. 

Another big failure was splitting each teams rules into a different repository. We ended up with ten repositories with one relevant file in each. It could have just been a folder in the original project. Plus it was a lot harder to figure out where to put things with ten different projects. The nail in the coffin from my perspective was that the rules were still deployed together. 

One final issue was integration tests. We wanted people to write integration tests, but everybody (including myself) avoided it as much as possible. The reality was the DSL for our end to end tests was significantly better than for integration tests. It was hard to reason how you were testing your specific feature at the API level. But in the end all our feature changes were directly testable at the end to end test level. It was just a lot quicker and easier to write an end to end test than API level test.

Finally, we reached the point where we had to split up the monolith purely because the pipeline was blocking too many teams. It was a risk because in the short term we knew it would increase operational overhead and delay needed upgrades. Unfortunately, the team lost about 50% of its people especially the most experienced ones. And I was one of them so I don’t know how things ended up. 

Self-hosted is not cheaper or better. Your options are pay Amazon insane markups or deal with a Platform as a Service built by the lowest bidder.

We often see posts on hacker news about how Amazon charges an insane markup for hosting or data transfer, or anything. Then someone will post a comparison where they price out an equivalent amount of hardware and show that it would cost 1/5th or 1/10th the cost. They then assume, obviously, hiring an IT department that can run this hardware is an achievable goal. 

Sorry, no it’s not achievable. Most companies cannot build a decent IT team that could run their own servers. They don’t have the ability to organize or hire the IT people needed. Not to mention they would never be willing to pay the salary costs. Quite simply, most businesses that try to run their own hardware will have a lower quality internal platform than if they just paid Microsoft. Whats the point of having faster hardware if your software stack is stuck in 2005 and managed by college drop outs?

Most companies are not overflowing with smart programmers who want to run bare metal hardware. Those people are moved to higher paying jobs programming the actual products of the company. 

Collected Sledgeworx

The collection is out now!

I’ve finally released my collection Collected Sledgeworx on amazon.

I started blogging to share my thoughts and vent frustrations. My first blog was about futurism and politics, my second on fitness, my third was Sledgeworx.io. I ended up being much better at a writing about software than anything else.

In this book I have collected many of the posts on the blog from its inception until the second half of 2021. These writings can be found on the blog in similar form. But you will find that the works in this book have been edited and supplemented in small ways.

Thanks for being a reader as we continue along to 2030!

What happened to Seviipay

Seviipay is a SAAS startup I launched in Summer 2021. The idea was to bring best in class UI/UX to native cryptocurrency payments. Basically the stripe of cryptocurrency payments. The MVP worked on mobile and desktop just for Ethereum. I did a few product discovery calls without getting any buyers.. Eventually the project kind of stalled and I moved on to other projects. 

The big thing here was motivation. My motivation for leaving my job was not Seviipay. The idea for Seviipay just came up at a time when I was really motivated to get out. Handling burnout and starting a business at the same time is not a great idea. 

Another issue with Seviipay and solo-bootstrapping in general is that you need a lot of skills to get a functional business working. Sales, Accounting, product, UI/UX, marketing, etc. I did not have enough of those skills at the beginning of the project to make it work. There is a reason startups usually launch with a product founder and a sales founder. 

One other weakness of Seviipay or me I guess is that I do not have web design skills. And I have struggled to make Seviipay visually appealing. I am a written word guy so if I thought of a product that was cli only I would build it. 

My skill stack now has grown to range from backend development, javascript, content marketing, copywriting, and a bit of sales skills. But that still isn’t enough to make Seviipay work, I need strong web development skills and more sales power. 

What is next for Seviipay

Seviipay is on hiatus for now. I’ve gotten a new software job and am getting up to speed for that. Also working with some friends on helping start up an agency. The next thing I need to do is hire a web developer to help with the UI and team up with a sales person to get a real sales process going. 

I’m also going to look into how I can make Seviipay a real Web3 business. Right now its a cryptocurrency SAAS that relies on Web 2.0 to join the blockchain with the old school internet.