Here are some interesting links I’ve come across over the last few months.
4 Types of Documentation
Making a Language Server
Here are some interesting links I’ve come across over the last few months.
4 Types of Documentation
Making a Language Server
I am working on some materials around onboarding new engineers.
When was the last time you onboarded a new engineer?
Do different team members use different development environments to run integration tests locally?
Do you have architecture diagrams that are up to date as of last month?
Do you have an up to date list of the APIs you consume and what their SLAs are?
Do you have a list of all the internal tools your team uses to complete their work?
Do you have a list of all the 3rd party tools your team uses to complete their work?
Do you have a list of all the configuration files that your services use?
Do you have a list of all the resources your services consume, object storage, disks, caches, VMs, queues, etc?
The typical agile process involves 5-10 people in a room. They go through one sentence descriptions of tasks, and each secretly comes up with their own estimate before every turns over their cards like in the final round of poker.
Then there will be a couple minutes spent discussing the estimates which will end with the average of the “votes”.
These meetings are usually confused by someone misunderstanding the breakout of one particular feature into one sentence descriptions, causing the group to backtrack and re-estimate several of the stories.
Most of the team will think about each task for a total of 2 minutes total.
The intention is that the project manager will take all this data and gradually calibrate the estimates.
I have spent a great deal of time participating in these agile “estimation” sessions and I can assure you that the estimates I produce in this setting have almost always been far off the mark.
Projects commonly run over what they were estimated, and sometimes a project estimated to take a month actually takes a day if you get the right person to work on it.
After 3.5 years working in this environment the whole process seems pretty pointless. Everyone knows the estimates are wrong and plans for it. Why do we bother doing this process that is known to be unreliable at best?
In construction estimates are done off of detailed blueprints using well understood heuristics, equations and the price of materials.
In software estimates are done off of one sentence descriptions of heterogeneous tasks and the gut feel of developers about how long something will take.
Agile Rapid Delivery means you deliver bugs to your customers everyday.
Every once in a while I read an article about how software has become horrible over the last decade. Software has gotten slower, it uses too much memory, it has too many confusing features, why are developers producing worse software?
Part of this is coincidence. We remember software being fast, but that was during a time when CPU speeds doubled every year. By the time you had used a program for a year, there was a processor out that could do it in half the time. Now Moore’s law ambles around at low double digit performance improvements.
Another issue is feature complexity. “The old software just worked, why did they have to change it?” With the move to Software as a Service we can constantly add new features.
Frequency matters if you deliver buggy code to your customers once a year on CD-ROMs. They are going to complain about buggy code once a year. If you deliver buggy code to your customers everyday, they will probably give up and accept their fate.
We all know the feeling. You have just started a new contract and its time to find out the truth about their software stack. Do they still use Ant despite the fact that Maven and Gradle have been options for over 10 years? Are you about to find out that their website is ‘really’ built on a 20 year old Pearl web framework that hasn’t been used for anything anywhere else?
Once you get past the initial shock of finding out what they doubled down on after it was wildly known to be a horrible idea. You get to find out about the internal tooling. Maybe you hear someone say, “we use the tools that are right for the problem”. Or maybe someone says “We don’t want to force people to use tools that are not a fit for their use case.” That is a sign that there might be more than a few internal tools lurking in the company wiki.
In your first week you hear about how we have our own web framework and try to figure out how dependency management works here. You find it somewhat nifty that you can submit multi-repository Pull Requests to their custom version of bitbucket, except with a few less bells and whistles. Alright, we have our own RPC framework that uses XML kind of like how SOAP used to work.
Then over the next few months you realize all the tools are custom. The company hasn’t taken advantage of any of the advances the rest of the industry has over the last 20 years. They have continuous integration and continuous delivery but it is all custom. The end to end test framework your project relies on was built by another team that disbanded quietly last year and is now entirely unsupported. You have the options of taking over the project (fuck no) or migrating to another promising internal end to end test framework. That team promises they will fully support it once they hit 1.0.
Name a tool used in the industry. Go ahead, name one. “Alright, Terraform”, oh, Terraform is nice, but we have an internal tool Declarative Template Manager which allows you to write templates in Pearl. DTM templates are faster and supported as a first class citizen in our companies closed source deployment system.
-you- “Cool, I need to track down this bug, how can I search the logs?”
-old time dev- “We use grep.”
-you- “Alright, how do I get to the logs?”
-old time dev- “WoodShed, were you keep logs, here is the link”
Its a link to a wiki page with the standard internal documentation you know and love.
-you- “So how do I get the logs out of WoodShed?”
-old time dev- “Well, people who know what they are doing use Wood Chipper. WoodChipper is the best tool to use to grep logs from WoodShed. “
-you- “I just want to do a quick search, is there a web portal that can do that?”
-old time dev- “Well, we have SawMill, it can pool all the logs for your teams servers onto one host so you can ssh into one place and use grep!”
Now you might be thinking no serious software company would allow their engineers to ssh into production hosts just to grep logs, but you would be wrong. After all Devops means everyone has ssh access in production. You might think that the top software companies would have a better solution than STORING ALL THE LOGS IN S3 AND THEN GREP’ING THEM.
If you thought any of those things you were wrong. Every company has some private tooling, over time that ‘some’ goes from a couple tools to dozens and then hundreds. The reason software engineers love startups is because they get to dump all the legacy code (older than 2 years) and build something new. Thereby escaping the Chuthulu-esque horror building up in the company codebase.
Now that I am working as an Individual Contributor again I actually get to do focused work. My team has standup at 10:45, I start thinking about standup at about 10:00 or so. What did I do yesterday? I put together some notes and then go back to working for a bit. After standup I have a few minutes of work and then go to lunch around 11:30 to avoid the rush.
Contrast this with our ‘no meeting’ day. I get into the office and have an additional 90 minutes of uninterrupted work before lunch. The real difference is that I get two 4 hour blocks of uninterrupted work in a no-meeting day as opposed to one or zero on a day with meetings.
Time has gone by faster than I expected. At this point I have been back as a full time developer individual contributor for about six months. It had been about 2 years since I did any Java development and the getting back into it was harder than I expected. It doesn’t help that the Java projects here are large and the system is massive. We have dozens of Java services with 100,000s of lines of code and we integrate into a lot of core business systems outside of our organization. This is the most complicated project I have worked on personally, with problems spanning from prices, voice UI, internationalization of voice UI, international law and massive java projects.
The Software Development Lifecycle here is very mature and the best way to work I have come across in the business world so far. We practice what could be described as ‘informal agile’ We have sprints and stories and quarterly goals like many places. Compared to the Scaled Agile Framework we have a laughable amount of process. It took me a few months to realize that my open calendar was here to stay.
Java is a mature language with lots of support for anything you could imagine, but it is still a painful language to use. On the flip side, I have finally figured out how annotations work. And I have been experimenting with Lombok. I have returned to the world of XML configuration and spring (not boot).
All the less fun part of development is back too. Really long builds, massive numbers of dependencies. Pull request churn so you have to rebase your changes constantly. But I get to write code again and my technical expertise is increasing again instead of declining.
Overall, I am pretty happy with my transition back into development. Would I have done it for a pay cut, no. But since it came with an increase I am in a pretty good place.
Software Engineer Resume Handbook
Many Software Engineers would rather ignore their resume. It reminds you of how few companies get back to you when you apply. But when you are looking for a new job, you end up scrambling to update your resume. Hopefully, this writing will help you create a resume that gets the results you want.
Your resume has five important sections contact information, your education, work experience and skills. If space allows a personal projects section and goal statement can be added.
For contact information you want to include your name, email address, phone number and the city you live near. Recruiters will contact you by phone or email. Including your actual address is unnecessary, simply state the city you are in ex. “Miami, FL”. Recruiters are never going to physically mail you anything. They just want to know whether you will need to relocate or not.
In the education section you want to list which college you went to, the year you graduated and your GPA. If you have a relevant masters degree list that too. I took my GPA off after a few years of industry experience because it was under 3.0. After five years in industry it is probably best to remove your GPA even if it is a 4.0. By that point in your career your GPA is irrelevant compared to your experience in the workforce.
My education line looks like what you see above. It includes the university I went to, what I majored in and the year I graduated. Since, I have a Minor in Computer Science it would be worthwhile to list it above.
Do not list the classes you took unless you have nothing else to fill space. In general companies do not care what classes you took as long as your major was computer science, software engineering or something closely related.
If you went to community college and then transferred to a university, you do not have to mention community college.
Your education section should be very short. Five lines at the longest. It should be easy to skim past without thinking about. When the recruiter is skims your resume she should think “ah, the candidate has an appropriate college degree”.
The experience section is the most important part of your resume. It should be the largest and clearly visible in the top half of your resume. This section is where you convince the hiring manager to interview you. Many companies will post a general job posting for a ‘Software Engineer’, then depending on your experience and interview performance they will decide to make an offer for a particular level like Junior or Senior software engineer.
I like to split the experience section up by job. Then have a small paragraph for each project at that job that I want to highlight.
You want to fill the section with the highlights of your career. When did you demonstrate the most leadership, technical acumen and success? That is what you put in your experience section.
Some of your experiences will have been bad. Projects fail, people get laid off, etc. A simple rule of thumb is “never volunteer negative information”. This goes double for your resume. Never include negative information in your resume.
If your GPA is bad and you barely graduated college with a 1.5 GPA, do not put it in your resume. If you failed Calculus 2 three times do not put that in your resume. If you are a felon and do not have a legal obligation to disclose your criminal history, do not put it on your resume. If the company wants to know they will ask you.
Were you fired from your last job? If so, do not put that on your resume.
All you need to include is a range of months during which you worked at the job. Do not put January 13, 2017 through March 5, 2018 on your resume. The recruiter does not care how many days you were employed at a job. Put January 2017 – March 2018 as your employment date range. If the recruiter cares they will ask you about how many days you were employed during a phone screen.
Do not say anything negative in your resume about any of your jobs. You might think a job was a horrible failure that you were glad to escape after a year. But your resume should describe that job as a successful stepping stone where you shipped desperately needed features to hungry customers.
This blurb implies a number of things. I say that I led a team, mentored developers guided the architecture of a project and that I did so with Go, React.js and AWS. I do not say that I have soft or hard skills. I say that I used soft and hard skills to do useful things.
My friend had an internship which he hated, where his job consisted of writing scripts for his gaming mouse which scraped insurance company websites to check customer eligibility. He wanted to gloss over that internship because he hated it. I had to challenge him on it because even though he felt the internship was a failure it could be phrased as a massive success. How many people help hospitals verify patients’ insurance during an internship? In my internships we built internal tooling that maybe 3 people ever used. My friend built a system that verified insurance coverage for hundreds of people. But he didn’t want to list that on his resume because he remembered it as a horrible experience.
In this example I included two projects that I worked on while I had the title “Java Developer” at a company.
You should include non-technical jobs
Include as many jobs as you can fit into the Experience section. If you have one programming internship and three jobs where you worked as a dishwasher, list all of them. Add a short paragraph describing what you did and learned at your non-technical position. Wait until you have had four technical jobs before you remove non-technical roles from your resume.
The skills section is often highly positioned in engineers resumes. We value the technologies, languages and frameworks we have mastered and want to put them on top. Resist that urge. The tools you have developed skills with probably only overlap a little with the stack used at your next job. Put the skills section near the bottom of your resume. Do everything you can to make it as small as possible to free up space for the experience section.
Hiring managers do not get a lot of value out a simple list of technologies you are familiar with. Unless you are a perfect match they will have to spend months training you on their technology stack.
Unless you are a serious expert on a technology do not list a skill level. You are not ‘Advanced’ at Java or Object Oriented Design or anything else in your ‘Skills’ section.
Companies are looking to fill positions for generalist software engineers and hire experts to solve particular problems. If you are an expert on a subject your resume should be focused only around your expertise in that one thing. A Cassandra database expert should not be looking for general purpose software engineering jobs, he should be focusing on jobs that want a Cassandra expert.
As a Generalist you are expected to be able to muddle through any problem that is thrown at you. Companies will throw you at whatever they want no matter the ‘level’ of expertise you put on your resume.
The hiring manager has better visibility into what the average skill level is than you do. You could list yourself as a ‘Kubernetes – Beginner’ and be the most knowledgable person that company ever interviewed. Only people who do dozens of interviews a year have the visibility necessary to compare you to the rest of the job market.
Some people worry about being ‘grilled’ about anything they list in the skills section. So they try to protect themselves by putting ‘beginner’ or ‘intermediate’ next to the skill under the belief they will be grilled less. If you are a junior engineer, people know you are not an expert at anything. Becoming an expert requires a lot more effort and time than most junior engineers have invested into software engineering. Even if you put ‘Expert at distributed systems’, recruiters will see your two internships and they will not believe you. If you get asked about something admit what you do and do not know. If you are backed into a corner, just say “I know a little about that technology, and I am looking for jobs where I use it in my work.”
If you are an expert in something you do not want to put it in your ‘Skills’ section, put it in the ‘Experience’ section! If you gave a talk on Kubernetes at Kubecon, that gives you enough expertise to interest people. Do not relegate an interesting experience like that to the skills section. If you were the Java performance expert at your last job then relate an experience where you saved the company hundreds of thousands of dollars in hardware costs. Don’t list actual expertise, tell a story about it.
Aim to list twenty skills at most. You want to be able to talk about them all intelligently. More skills is not necessarily better, the recruiters is looking for a few keywords that match the job posting. Your goal for the skills section is to give the recruiter the keywords they are looking for and to hint at the breadth of your skills.
Goal Statement (Optional)
The goal statement is a short paragraph that you can use to help the hiring manager understand what you are looking for. It can be helpful when applying to companies that make general job posts for ‘Software Engineers’ and then rank candidates based on experience. Use the goal statement to let them know what role and grade you are looking for.
“Senior Software Engineer building today’s software products and platforms, training developers to maximize their abilities, and supporting business success.“
The line above is one I have used. It gives an idea of what role I am looking for and how I view my role in a company. The goal statement is optional, you do not need one. If you are short on space cut it and devote more to the experience section.
What about Personal Projects?
Personal projects are not one of the major sections on your resume. I include links to my github and blog where people can find my personal projects. I would prioritize describing personal projects before hobbies, but describing job experience is just better. If you have a bunch of personal projects and only one job or internship, make sure to use more space describing that job than you do on your personal projects.
Styling a resume is a worthwhile, but risky process. Unless you are looking or jobs as a designer or creative artist, showcasing your artistic talent is not going to contribute to getting a job. Hiring managers and recruiters are skimming through dozens of resumes at a time. Your first priority is to make sure that the important sections are easy to find on your resume. Use bold headings that make it obvious that this section is about your education and that section is about your work experience.
You can add flair to your resume to help standout. Try to avoid anything fancy, a resume is not art. Any flair should be evaluated on whether it makes your resume more difficult to read. Unless you are 100% sure your flair does not make your resume harder to read cut it.
Sentences with Trailing words
Watch out for sentences with trailing words. Those trailing words are taking up an entire line that could be describing your experience. Try to modify that sentence to use less space or to completely fill two lines in your resume.
Trim Extra Space
Trim as much extra space as you can. Careful formatting can save you dozens of lines in a one page resume.
Cover letters often come up when people discuss job applications. A cover letter is a one page letter you write to persuade a company to interview you. You want to write about why you are a good fit for the position and how you are motivated especially to work at that company.
Two major strategies you can use are to write a custom cover letter for each job you apply for or to maintain a template cover letter which you modify a little each time you apply to a job.
I use a cover letter template for the majority of the jobs that I apply for. I maintain a ‘Application Developer’ cover letter and a ‘Devops Engineer’ cover letter then do a little customization for each position I apply for. The jobs I am qualified for are generalist Software Engineer and Devops so my cover letter do not need to change a lot.
Custom cover letters are good for highly competitive companies and positions that you really want. The problem is that you will end up spending several extra hours for each application and still might not get a response.
This post is about some of the good parts of working at Amazon as a Developer. Believe it or not, I have been working at Amazon for almost 6 months. I work with massive Java micro-services and XML configuration everyday. There are so many systems that it takes years to figure out what is going on.
The top thing I am enjoying is mentorship. My coworkers, managers and lead engineers are experienced and very competent. They have worked on massive systems at Amazon for 5-10 years and have a host of experience. I like being one of the dummer people in the room even if its inevitably temporary.
At my last job we studied and taught best practices, but we had a hard time actually, you know, practicing them. At Amazon we actually do CI/CD. We have integration and E2E tests. Our team practices supports our own services in production. This is the experience I wanted to have before I got pulled into teaching courses and best practices.
My team builds the core platform for all sales on Alexa. We have customers all over the world in a number of languages. It is not like the 9 months I spent working on a product that was canceled after a merger.
More interesting problems
Just today I was looking into some latency issues after a Product Owner did some introductory research. We pulled up the metrics and started analyzing p50 vs p99 vs p99.9, etc. Do the latency spikes correspond with deployments? Do they correspond with GC? What does the profiler say is happening during this API call?
What is the same?
The great things that I have enjoyed previously in the tech industry are pretty much the same. My hours have not changed. My flexibility is the same. The general culture is pretty much the same.
What is worse?
Amazon is immense and it is impossible to know everything that is going on. There are often 4 or more slightly different solutions to a problem. How do you know which one is useful? Well, you have to rely on context. The wiki is comparable to a certain cable company I used to work for. Each team makes their own docs and their own documentation standards.
Monorepositories are an interesting topic. The more I use similar systems the more I think monorepos are the way to go.
The Product Model vs Project Model
Nick McHardy writes about how his organization moved from doing ‘projects’ to organizing around ‘products’. Especially interesting for me since I just moved from a project organization to a product organization.
Sensible Software Engineering
How do we get out of the local optimum of software development theory we are in now? We can be somewhat confident that there is a better programming paradigm out there than the modern Java ecosystem. Technically, there is an ideal development paradigm, even if finding it is near impossible. It takes us around 20 years as an industry to explore one paradigm. We don’t know what we can do to find a better paradigm within the constraint of how much it costs to really explore even one.
This reminds me of the multi-armed bandit problem. You want to find the optimal slot machine to play. But you still have to create things as an industry every year.