Continuous Delivery of Bugs


Agile Rapid Delivery means you deliver bugs to your customers everyday. 

Every once in a while I read an article about how software has become horrible over the last decade. Software has gotten slower, it uses too much memory, it has too many confusing features, why are developers producing worse software? 

Part of this is coincidence. We remember software being fast, but that was during a time when CPU speeds doubled every year. By the time you had used a program for a year, there was a processor out that could do it in half the time. Now Moore’s law ambles around at low double digit performance improvements. 

Another issue is feature complexity. “The old software just worked, why did they have to change it?” With the move to Software as a Service we can constantly add new features. 

Frequency matters if you deliver buggy code to your customers once a year on CD-ROMs. They are going to complain about buggy code once a year. If you deliver buggy code to your customers everyday, they will probably give up and accept their fate. 

Please don’t build yet another internal ‘tool’

We all know the feeling. You have just started a new contract and its time to find out the truth about their software stack. Do they still use Ant despite the fact that Maven and Gradle have been options for over 10 years? Are you about to find out that their website is ‘really’ built on a 20 year old Pearl web framework that hasn’t been used for anything anywhere else? 

Once you get past the initial shock of finding out what they doubled down on after it was wildly known to be a horrible idea. You get to find out about the internal tooling. Maybe you hear someone say, “we use the tools that are right for the problem”. Or maybe someone says “We don’t want to force people to use tools that are not a fit for their use case.” That is a sign that there might be more than a few internal tools lurking in the company wiki. 

In your first week you hear about how we have our own web framework and try to figure out how dependency management works here. You find it somewhat nifty that you can submit multi-repository Pull Requests to their custom version of bitbucket, except with a few less bells and whistles. Alright, we have our own RPC framework that uses XML kind of like how SOAP used to work. 

Then over the next few months you realize all the tools are custom. The company hasn’t taken advantage of any of the advances the rest of the industry has over the last 20 years. They have continuous integration and continuous delivery but it is all custom. The end to end test framework your project relies on was built by another team that disbanded quietly last year and is now entirely unsupported. You have the options of taking over the project (fuck no) or migrating to another promising internal end to end test framework. That team promises they will fully support it once they hit 1.0. 

Name a tool used in the industry. Go ahead, name one. “Alright, Terraform”, oh, Terraform is nice, but we have an internal tool Declarative Template Manager which allows you to write templates in Pearl. DTM templates are faster and supported as a first class citizen in our companies closed source deployment system. 

-you- “Cool, I need to track down this bug, how can I search the logs?”

-old time dev- “We use grep.”

-you- “Alright, how do I get to the logs?”

-old time dev- “WoodShed, were you keep logs, here is the link”

Its a link to a wiki page with the standard internal documentation you know and love. 

-you- “So how do I get the logs out of WoodShed?”

-old time dev- “Well, people who know what they are doing use Wood Chipper. WoodChipper is the best tool to use to grep logs from WoodShed. “

-you- “I just want to do a quick search, is there a web portal that can do that?”

-old time dev- “Well, we have SawMill, it can pool all the logs for your teams servers onto one host so you can ssh into one place and use grep!”

Now you might be thinking no serious software company would allow their engineers to ssh into production hosts just to grep logs, but you would be wrong. After all Devops means everyone has ssh access in production. You might think that the top software companies would have a better solution than STORING ALL THE LOGS IN S3 AND THEN GREP’ING THEM.

If you thought any of those things you were wrong. Every company has some private tooling, over time that ‘some’ goes from a couple tools to dozens and then hundreds. The reason software engineers love startups is because they get to dump all the legacy code (older than 2 years) and build something new. Thereby escaping the Chuthulu-esque horror building up in the company codebase. 

It’s amazing how much time not having standup adds to my day.


Now that I am working as an Individual Contributor again I actually get to do focused work. My team has standup at 10:45, I start thinking about standup at about 10:00 or so. What did I do yesterday? I put together some notes and then go back to working for a bit. After standup I have a few minutes of work and then go to lunch around 11:30 to avoid the rush. 

Contrast this with our ‘no meeting’ day. I get into the office and have an additional 90 minutes of uninterrupted work before lunch. The real difference is that I get two 4 hour blocks of uninterrupted work in a no-meeting day as opposed to one or zero on a day with meetings. 

Back to being a full time dev 6 month review.

Time has gone by faster than I expected. At this point I have been back as a full time developer individual contributor for about six months. It had been about 2 years since I did any Java development and the getting back into it was harder than I expected. It doesn’t help that the Java projects here are large and the system is massive. We have dozens of Java services with 100,000s of lines of code and we integrate into a lot of core business systems outside of our organization. This is the most complicated project I have worked on personally, with problems spanning from prices, voice UI, internationalization of voice UI, international law and massive java projects. 

The Software Development Lifecycle here is very mature and the best way to work I have come across in the business world so far. We practice what could be described as ‘informal agile’ We have sprints and stories and quarterly goals like many places. Compared to the Scaled Agile Framework we have a laughable amount of process. It took me a few months to realize that my open calendar was here to stay. 

Java is a mature language with lots of support for anything you could imagine, but it is still a painful language to use. On the flip side, I have finally figured out how annotations work. And I have been experimenting with Lombok. I have returned to the world of XML configuration and spring (not boot). 

All the less fun part of development is back too. Really long builds, massive numbers of dependencies. Pull request churn so you have to rebase your changes constantly. But I get to write code again and my technical expertise is increasing again instead of declining.

Overall, I am pretty happy with my transition back into development. Would I have done it for a pay cut, no. But since it came with an increase I am in a pretty good place. 

Software Engineer Resume Handbook

resume_handbook

Software Engineer Resume Handbook

Introduction

Many Software Engineers would rather ignore their resume. It reminds you of how few companies get back to you when you apply. But when you are looking for a new job, you end up scrambling to update your resume. Hopefully, this writing will help you create a resume that gets the results you want. 

Important Sections

Your resume has five important sections contact information, your education, work experience and skills. If space allows a personal projects section and goal statement can be added. 

Contact information

For contact information you want to include your name, email address, phone number and the city you live near. Recruiters will contact you by phone or email. Including your actual address is unnecessary, simply state the city you are in ex. “Miami, FL”. Recruiters are never going to physically mail you anything. They just want to know whether you will need to relocate or not.

Education

In the education section you want to list which college you went to, the year you graduated and your GPA. If you have a relevant masters degree list that too. I took my GPA off after a few years of industry experience because it was under 3.0. After five years in industry it is probably best to remove your GPA even if it is a 4.0. By that point in your career your GPA is irrelevant compared to your experience in the workforce.

My education line looks like what you see above. It includes the university I went to, what I majored in and the year I graduated. Since, I have a Minor in Computer Science it would be worthwhile to list it above.

Do not list the classes you took unless you have nothing else to fill space. In general companies do not care what classes you took as long as your major was computer science, software engineering or something closely related. 

If you went to community college and then transferred to a university, you do not have to mention community college.

Your education section should be very short. Five lines at the longest. It should be easy to skim past without thinking about. When the recruiter is skims your resume she should think “ah, the candidate has an appropriate college degree”. 

Experience

The experience section is the most important part of your resume. It should be the largest and clearly visible in the top half of your resume. This section is where you convince the hiring manager to interview you. Many companies will post a general job posting for a ‘Software Engineer’, then depending on your experience and interview performance they will decide to make an offer for a particular level like Junior or Senior software engineer. 

I like to split the experience section up by job. Then have a small paragraph for each project at that job that I want to highlight.

You want to fill the section with the highlights of your career. When did you demonstrate the most leadership, technical acumen and success? That is what you put in your experience section. 

Some of your experiences will have been bad. Projects fail, people get laid off, etc. A simple rule of thumb is “never volunteer negative information”. This goes double for your resume. Never include negative information in your resume. 

If your GPA is bad and you barely graduated college with a 1.5 GPA, do not put it in your resume. If you failed Calculus 2 three times do not put that in your resume. If you are a felon and do not have a legal obligation to disclose your criminal history, do not put it on your resume. If the company wants to know they will ask you.

Were you fired from your last job? If so, do not put that on your resume. 

Employment Dates

All you need to include is a range of months during which you worked at the job. Do not put January 13, 2017 through March 5, 2018 on your resume. The recruiter does not care how many days you were employed at a job. Put January 2017 – March 2018 as your employment date range. If the recruiter cares they will ask you about how many days you were employed during a phone screen. 

Do not say anything negative in your resume about any of your jobs. You might think a job was a horrible failure that you were glad to escape after a year. But your resume should describe that job as a successful stepping stone where you shipped desperately needed features to hungry customers. 

This blurb implies a number of things. I say that I led a team, mentored developers guided the architecture of a project and that I did so with Go, React.js and AWS. I do not say that I have soft or hard skills. I say that I used soft and hard skills to do useful things. 

My friend had an internship which he hated, where his job consisted of writing scripts for his gaming mouse which scraped insurance company websites to check customer eligibility. He wanted to gloss over that internship because he hated it. I had to challenge him on it because even though he felt the internship was a failure it could be phrased as a massive success. How many people help hospitals verify patients’ insurance during an internship? In my internships we built internal tooling that maybe 3 people ever used. My friend built a system that verified insurance coverage for hundreds of people. But he didn’t want to list that on his resume because he remembered it as a horrible experience. 

In this example I included two projects that I worked on while I had the title “Java Developer” at a company. 

You should include non-technical jobs

Include as many jobs as you can fit into the Experience section. If you have one programming internship and three jobs where you worked as a dishwasher, list all of them. Add a short paragraph describing what you did and learned at your non-technical position. Wait until you have had four technical jobs before you remove non-technical roles from your resume. 

Skills

The skills section is often highly positioned in engineers resumes. We value the technologies, languages and frameworks we have mastered and want to put them on top. Resist that urge. The tools you have developed skills with probably only overlap a little with the stack used at your next job. Put the skills section near the bottom of your resume. Do everything you can to make it as small as possible to free up space for the experience section. 

Hiring managers do not get a lot of value out a simple list of technologies you are familiar with. Unless you are a perfect match they will have to spend months training you on their technology stack. 

Unless you are a serious expert on a technology do not list a skill level. You are not ‘Advanced’ at Java or Object Oriented Design or anything else in your ‘Skills’ section. 

Companies are looking to fill positions for generalist software engineers and hire experts to solve particular problems. If you are an expert on a subject your resume should be focused only around your expertise in that one thing. A Cassandra database expert should not be looking for general purpose software engineering jobs, he should be focusing on jobs that want a Cassandra expert. 

As a Generalist you are expected to be able to muddle through any problem that is thrown at you. Companies will throw you at whatever they want no matter the ‘level’ of expertise you put on your resume. 

The hiring manager has better visibility into what the average skill level is than you do. You could list yourself as a ‘Kubernetes – Beginner’ and be the most knowledgable person that company ever interviewed. Only people who do dozens of interviews a year have the visibility necessary to compare you to the rest of the job market.

Some people worry about being ‘grilled’ about anything they list in the skills section. So they try to protect themselves by putting ‘beginner’ or ‘intermediate’ next to the skill under the belief they will be grilled less. If you are a junior engineer, people know you are not an expert at anything. Becoming an expert requires a lot more effort and time than most junior engineers have invested into software engineering. Even if you put ‘Expert at distributed systems’, recruiters will see your two internships and they will not believe you. If you get asked about something admit what you do and do not know. If you are backed into a corner, just say “I know a little about that technology, and I am looking for jobs where I use it in my work.”

If you are an expert in something you do not want to put it in your ‘Skills’ section, put it in the ‘Experience’ section!  If you gave a talk on Kubernetes at Kubecon, that gives you enough expertise to interest people. Do not relegate an interesting experience like that to the skills section. If you were the Java performance expert at your last job then relate an experience where you saved the company hundreds of thousands of dollars in hardware costs. Don’t list actual expertise, tell a story about it. 

Aim to list twenty skills at most. You want to be able to talk about them all intelligently. More skills is not necessarily better, the recruiters is looking for a few keywords that match the job posting. Your goal for the skills section is to give the recruiter the keywords they are looking for and to hint at the breadth of your skills.  

Goal Statement (Optional)

The goal statement is a short paragraph that you can use to help the hiring manager understand what you are looking for. It can be helpful when applying to companies that make general job posts for ‘Software Engineers’ and then rank candidates based on experience. Use the goal statement to let them know what role and grade you are looking for. 

“Senior Software Engineer building today’s software products and platforms, training developers to maximize their abilities, and supporting business success.“

The line above is one I have used. It gives an idea of what role I am looking for and how I view my role in a company. The goal statement is optional, you do not need one. If you are short on space cut it and devote more to the experience section. 

What about Personal Projects?

Personal projects are not one of the major sections on your resume. I include links to my github and blog where people can find my personal projects. I would prioritize describing personal projects before hobbies, but describing job experience is just better. If you have a bunch of personal projects and only one job or internship, make sure to use more space describing that job than you do on your personal projects. 

Style

Styling a resume is a worthwhile, but risky process. Unless you are looking or jobs as a designer or creative artist, showcasing your artistic talent is not going to contribute to getting a job. Hiring managers and recruiters are skimming through dozens of resumes at a time. Your first priority is to make sure that the important sections are easy to find on your resume. Use bold headings that make it obvious that this section is about your education and that section is about your work experience. 

You can add flair to your resume to help standout. Try to avoid anything fancy, a resume is not art. Any flair should be evaluated on whether it makes your resume more difficult to read. Unless you are 100% sure your flair does not make your resume harder to read cut it. 

Sentences with Trailing words

Watch out for sentences with trailing words. Those trailing words are taking up an entire line that could be describing your experience. Try to modify that sentence to use less space or to completely fill two lines in your resume.

Trim Extra Space

Trim as much extra space as you can. Careful formatting can save you dozens of lines in a one page resume. 

Cover Letters

Cover letters often come up when people discuss job applications. A cover letter is a one page letter you write to persuade a company to interview you. You want to write about why you are a good fit for the position and how you are motivated especially to work at that company. 

Two major strategies you can use are to write a custom cover letter for each job you apply for or to maintain a template cover letter which you modify a little each time you apply to a job. 

I use a cover letter template for the majority of the jobs that I apply for.  I maintain a ‘Application Developer’ cover letter and a ‘Devops Engineer’ cover letter then do a little customization for each position I apply for. The jobs I am qualified for are generalist Software Engineer and Devops so my cover letter do not need to change a lot. 

Custom cover letters are good for highly competitive companies and positions that you really want. The problem is that you will end up spending several extra hours for each application and still might not get a response. 

Why I am having a good time at Amazon

This post is about some of the good parts of working at Amazon as a Developer. Believe it or not, I have been working at Amazon for almost 6 months. I work with massive Java micro-services and XML configuration everyday. There are so many systems that it takes years to figure out what is going on. 

Mentorship

The top thing I am enjoying is mentorship. My coworkers, managers and lead engineers are experienced and very competent. They have worked on massive systems at Amazon for 5-10 years and have a host of experience.  I like being one of the dummer people in the room even if its inevitably temporary. 

Best Practices

At my last job we studied and taught best practices, but we had a hard time actually, you know, practicing them. At Amazon we actually do CI/CD. We have integration and E2E tests. Our team practices supports our own services in production. This is the experience I wanted to have before I got pulled into teaching courses and best practices. 

Bigger impact

My team builds the core platform for all sales on Alexa. We have customers all over the world in a number of languages. It is not like the 9 months I spent working on a product that was canceled after a merger. 

More interesting problems

Just today I was looking into some latency issues after a Product Owner did some introductory research. We pulled up the metrics and started analyzing p50 vs p99 vs p99.9, etc. Do the latency spikes correspond with deployments? Do they correspond with GC? What does the profiler say is happening during this API call? 

What is the same? 

The great things that I have enjoyed previously in the tech industry are pretty much the same. My hours have not changed. My flexibility is the same. The general culture is pretty much the same. 

What is worse?

Amazon is immense and it is impossible to know everything that is going on. There are often 4 or more slightly different solutions to a problem. How do you know which one is useful? Well, you have to rely on context. The wiki is comparable to a certain cable company I used to work for. Each team makes their own docs and their own documentation standards. 

Links Post

https://presumably.de/monorepos-and-the-fallacy-of-scale.html

Monorepositories are an interesting topic. The more I use similar systems the more I think monorepos are the way to go.

The Product Model vs Project Model

https://nickmchardy.com/2018/12/doing-products-instead-of-projects.html

Nick McHardy writes about how his organization moved from doing ‘projects’ to organizing around ‘products’. Especially interesting for me since I just moved from a project organization to a product organization.

Sensible Software Engineering

https://www.scriptcrafty.com/2019/02/sensible-software-engineering/

How do we get out of the local optimum of software development theory we are in now? We can be somewhat confident that there is a better programming paradigm out there than the modern Java ecosystem. Technically, there is an ideal development paradigm, even if finding it is near impossible. It takes us around 20 years as an industry to explore one paradigm. We don’t know what we can do to find a better paradigm within the constraint of how much it costs to really explore even one.

This reminds me of the multi-armed bandit problem. You want to find the optimal slot machine to play. But you still have to create things as an industry every year. 

Pipelines should automatically reject change-sets that fail the integration tests.

CI/CD is magical when it works. Write your code, open a pull request, get approvals and merge. Then the pipeline takes care of everything from then on. Integration tests are run end to end and UI tests pass. Finally, your change is deployed via a canary release into production. 

My team runs a full CI/CD pipeline, after a Pull Request is approved and merged it is deployed through several stages of integration tests and deployed to production. This pipeline gets a lot of use with around 10 engineers working on it at any given time. Unfortunately, because of how our infrastructure works, integration test failures often break the pipeline. 

Changes do not proceed through the pipeline independently, instead commits build up in front of slower parts of the pipeline, particularly our integration and end-to-end tests. During busy times several pull requests will be working their way through the pipeline together.  

Our integration tests can fail for many reasons other than code changes. A partner team could have a failed deployment that is upstream of us, a configuration file could have been changed, a test account could have expired, and more. When the pipeline does fail, we have to track down what caused the failure. If it was a code change we will revert the commit, otherwise we typically have to work with another team to get their systems working. 

When a pull request breaks the integration or end-to-end test the pipeline will block at that stage. Then we will revert the changes and redeploy the master branch. In the worst cases several commits will have to be reverted and redeployed into the pipeline before it will start working again. 

Fixing the pipeline will often take an entire day as the fix has to move through each stage in the pipeline until it reaches the one that failed. Then there will be a flood of changes as all the commits that built up at the failing stage move onward. 

I would like to fix this, but doing so will likely make the pipeline slower on average or require changes to how our Pull Request workflow functions. 

Prepare notes for Standups

Daily standups are a standard part of the software engineering life. Over the last four years I have had a standup of some kind nearly every workday.

Standup is a ritual, you circle up with your coworkers and take turns trying to remember what exactly you did to justify your salary yesterday. Every third update devolves into a 5+ minute derailment of the meeting that would really prefer not to follow. Then when it is finally your turn you spill out your update as quickly as possible. Only to slowly realize 3 minutes later that you forgot to mention half of the things you did the previous day. 

The solution to standup is pretty simple.  Just take notes.

I keep a checklist of tasks in the Mac notes app. I check off the item when its done. Then in standup all I do is read through my list. 

Just keeping a little list of tasks makes it easy to remember what you did for standup. It is easy and will save you the embarrassment of not having anything to say to justify your salary during the meeting.

A Certification is not the same as having a skill 

People often ask on r/cscareerquestions, “Can I just get this Java Certification instead of doing a Bootcamp or getting a CS degree”. And the answer is that the Certification is not going to help you as much as you think it will. Computer Programming is still a young field, gate keepers are just not that important yet. Becoming a Real Estate Agent requires you to get a License or Certification from the State certifying you to buy and sell houses for people. There is nothing like that for Computer Programming. Anyone with any amount of skill or experience can declare themselves a ‘Software Engineer’. There is no governing body which can ‘certify’ your skills. The current standard is Whiteboarding problems and timed hackerrank.com questions. Your return on investment from investing time into hackerrank.com will be much higher than getting a Certification. 

How Software Engineers demonstrate their skill to the market is through whiteboarding questions. Get good at that first then worry about Certifications. 

Does this make Certifications worthless in Software? I don’t think so, but it is not going to be worth it for junior developers trying to break into the market. One of my employers paid for me to get the ‘Certified Kubernetes Administrator’ certification from the CNCFF. Why did they do that? So that they could qualify as a CNCF partner organization and get free marketing and a better aura of expertise. 

I put that certificate on my resume, but it has not gotten me any Kubernetes jobs. If your company is going to pay for a certification go for it. Otherwise, I do not think it is worthwhile for you to pay for it personally.