Reducing toil with AI

LLMs are at the point where they can reduce toil for software engineers across a host of use cases. Here we will explore a few I’ve thought of.

Reduces time spent on toil 

  • Resize this button -> AI
  • Refactor this function -> AI 
  • Connect these endpoints -> AI 
  • Write more tests -> AI 
  • Add more comments -> AI 
  • Create PlantUML diagrams from a sketch -> AI 
  • First pass code reviews -> AI

Reduces time spent researching small things

  • AI is a better stack overflow
  • Examine stack traces 
  • Easier to write architecture documents 
  • Faster development of small utilities

What AI still cannot do

  • Test if a library will work for your use case 
  • Respond to outages 
  • Decide the product direction
  • Argue with stakeholders 
  • Yell at people who want to do stupid things

A few opportunities to reduce operational toil

  • AI can review graphs and notice changes
  • AI can check if a website is down
  • File bug reports 

For me the most valuable use of Claude as a coding assistant has been how it makes getting started much easier. Usually, to program a swift game I’d have to spend a couple hours breaking into swift development and building up my program off of examples. Claude was able to create a basic version of the game I wanted off of a detailed prompt. It didn’t manage to create the game in one shot, I had to edit the code a bit to get it to run. But it turned a project that would have taken me a few days into one that took a few hours.

An Excellent AI use case: Generating PlantUML and Mermaid schemas.

If you’ve ever worked with PlantUML or Mermaid before it’s easy to forget the domain specific language used to build the diagrams. You sketch out your whiteboard diagram, discuss it with coworkers and are ready to start on your architecture document only to sit there stumped trying to remember how to convert your drawing into PlantUML.

Well the good news is that Claude can do it for you. Just take a picture of your whiteboard and Claude can convert it into Mermaid or PlantUML for you. I tried both and Claude is better at Mermaid but can do PlantUML.

Unfortunately, Claude’s first attempt had an error. I fiddled with it a bit in the editor, but Claude was able to fix it faster than I was.

Claude fixed the issue and I was able to generate the following PlantUML visualization.

Here is the original image compared against the PlantUML Claude generated. The relationships are correctly mapped even if the physical placement is a bit different.

Lastly, I asked Claude to do the same thing with the Mermaid diagram visualizer. Mermaid does the same things as PlantUML, it’s just newer and easier to use.

To my surprise Claude not only managed to do it in one attempt. But Mermaid is built into the Anthropic UI.

If you’re ever stuck trying to remember the PlantUML DSL just ask Claude to do it.

Links October 2024

Great article on mac setup

Lots of tools are outdated by default. Got to take advantage of this blog post since I got a new laptop this summer.

https://matt.sh/setup-2021-late

Operating Systems use out of date assumptions

Interesting talk on how modern systems on a chip differ from how we imagine computers to actually work. Over the last two decades hardware has gradually abstracted away many features from the actual operating system. The computer board is now a collection of sub-computers which cooperate to mimic the operation of an actual traditional computer. 

The system describe here is reminiscent of a micro service environment where you need to communicate with many different protocols to get the job done. I wonder if we can explore migrating ‘cloud’ microservice techniques down to the CPU level. 

Passive Radar for finding meteors

You can detect meteors using a set of clock synchronized radios. The way it works is you monitor a reference frequency at perhaps 180MHz and can detect changes in it as meteors burn up in the atmosphere. 

https://en.wikipedia.org/wiki/Passive_radar
https://britastro.org/wp-content/uploads/2019/11/Detection_of_meteors_by_RADAR.pdf

A clear sign you are overdoing microservices 

Fine grained services

Microservices have been the thing for over 15 years. They are great in large companies with CI/CD environments. But as your situation drifts farther away from the ideal microservice use case traps abound. 

Building a new service for one endpoint 

If you find yourself having a conversation where you need to create a new endpoint somewhere, but adding it to any of your existing services would break the concept of that microservice. Turning it instantly into a ball of mud with no clear purpose. You have fallen into this trap. microservice does not mean each service has only one HTTP endpoint. That use case is better served with Cloud functions like AWS lambda. 

The problem here is that we have gone too far in splitting up the monolith. Splitting a monolith with 100 HTTP endpoints into a dozen or so services with eight endpoints each is great. Splitting up a monolith with 100 endpoints into 100 services is counter productive. Instead of having an actual purpose the single endpoint microservice becomes the xyz endpoint microservice. 

Endpoints are things that microservices empower. An endpoint in of itself should never justify the creation of a microservice. 

KPIs for Software Engineers

Key Performance Indicators are a common business practice. They are a quantifiable measure of performance for a specific objective. Occasionally, I am asked to create my own KPIs as an individual contributor. I think it is a bit strange, after all I work for the company, you’d think they would tell me what the KPIs are!

Ideally we want to avoid metrics which are created arbitrarily by team members. Hours worked is a great example of this especially in remote work environments. In hourly workplaces employees check in and check out to validate hours worked. I have never seen a software company do anything comparable. Typically, the software company employee is asked to fill in timesheets based on their personal memory with zero accountability. That scenario does not make for a good KPI. 

For our KPI metrics we have the following criteria.

Countable 

KPIs need to be quantifiable. It should be easy to number how many of X someone completed

Verifiable

There should be impartial systems tracking KPI completion. We want to use systems like PagerDuty, JIRA, Github, etc to source our KPI data. 

Valuable

KPIs should relate to valuable activities for the company. We want our staff focused on things that are important. 

Individually attributable

KPIs should be based on individually attributable work. We want to avoid subjective judgements of how much of task X engineer 1 did vs engineer 2. We also do not want to encourage infighting over who gets credit for which tasks. 

Here are a few measurable things we could use for KPIs. 

  • lines of code 
  • commits 
  • bugs fixed
  • tests written 
  • Pull requests opened | accepted / rejected
  • Pull request comments
  • Pull requests reviewed | approved / rejected
  • story tickets completed
  • stories estimated 
  • Projects lead 
  • Projects delivered / Epics delivered 
  • architecture documents published
  • architecture documents reviewed / comments / approved / rejected
  • Documentation pages published
  • oncall response time when paged
  • pages recieved daytime/off hours
  • Prod deployments 
  • meetings attended
  • Pair programming sessions attended
  • Junior dev questions answered (answers documented in wiki / private stack overflow)

KPIs are a useful concept for businesses to track their performance. But they are often really ideal for business groups to examine their performance. Individual contributors rarely can claim responsibility for things like increasing the subscription renewal rate from 1% to 10%.

While this list is not exclusive it should include most of the trackable numerical things that software engineers do in their job. Then if you need to come up with some KPIs for yourself you can just pick from this list.

If you can think of some more good metrics for software engineers let us know!