Figure out how to debug everything, kill blackboxes

 Avoid blackboxes in your tech stack. It is tempting to say “oh, this piece of software is acting up, but its not our team’s responsibility its xyz team’s problem”. Or this vendor’s software is acting up, “Guess we have to take 30 minutes off while we wait for it to come back up”. Sometimes this pattern shows up in the use of 3rd party libraries. 

It is tempting to allow the resistance to figuring out another piece of software block you from making progress on your project. But letting yourself take the easy route will slow down your progress as a software engineer. A blackbox is just a piece of software you don’t understand yet. You have worked with and figured out hundreds of programs in your career. This one isn’t anything special. 

Don’t let a bug in the end to end framework prevent you from making progress. Figure out the problem, open a ticket against the team that owns it. If you can make a fix yourself. The last thing you want is to get nothing done because you were waiting for someone else to fix a problem that you could have fixed, but “didn’t have to”. 

If your team has a docker compose stack for local testing, figure out how to get it running. Fix the documentation, test your software e2e locally. 

If there is an old integration test suite that intermittently fails, don’t let those failures block your work. Fix the failures, mock the flakey dependencies, don’t just sit there and let some software waste your time because you didn’t want to put the effort into deciphering how it works. 

Working from home

I’ve been lucky enough not to be laid off yet, and have been plunged headfirst in working from home. Before this pandemic I avoided working from home because I got distracted pretty easily and found it easier to get work done in the office. I’ve typically been a high performer and like to maintain work life balance so falling behind because I worked from home is not an option. 

Now I’ve been required to work from home since early march. The first couple weeks were rough. Working in a new setting, big changes going on in the world, combined to make focus difficult. I fell behind where I wanted to be on several projects. It was all dragging me down.

Late last week, at the time of writing, I made two big changes which seem to be helping a lot. They are getting better sleep and using the Pomodoro technique to focus on work. 

To get better sleep, all I did was start closing my blinds in the evening. I live downtown, so at night there are lots of light shining into my nice large windows. The view is good, but the light degrades sleep quality. I noticed a big difference the first night and have been feeling consistently good this last week.

My version of the Pomodoro technique is to do 40minute ‘sprints’ of focused coding time. This gets me past the initial hump and doing solid work. Then I take a short break and circle back for another round of focused work. This is similar to what I would do in the office, work for a while, then chat with coworkers or go to the lunch room. Except now I just stay in my apartment and stretch or workout. 

https://en.wikipedia.org/wiki/Pomodoro_Technique

Working from home is still a bit rough. But I think I’m past the hump of adapting to it. Now I just want to optimize a few things like my monitors and getting a motorized standing desk. 

Expecting end users to customize the experience is madness

 Don’t do it to yourselves

Don’t do it to customers 

Do the work to make a good product

Enterprise software sucks. Its not bought by the people using it, but by a guy wearing a suite on the 37th floor the day after eating a fabulous steak dinner paid for by Oracle sales guys.  By the time you start using it, it is bought and paid for. Suck it up and learn how this pile of code works. 

Internal enterprise software is another beast. Constantly underfunded, built by interns that just learned object oriented programming, and designed by the CEOs cousin, it is not the greatest. 

Know what will ensure that your internal software is never improved in a meaningful way? Make customizing it the default workflow. Just have every engineer at the company load up a GreaseMonkey script that adds in the features that PAAS should have by now. 

The problem is fixed for the graybeards. Sure, every new employee will spend six months realizing that all the people who are getting anything done have customized the UI so extensively its not recognizable as the same product. 

When they said go use ‘deployment ladder’, they meant use ‘deployment ladder’ with 12 GreaseMonkey scripts installed. Where are those scripts? You might ask, the answer is always ‘in the wiki’. Searching for the name of the thing in the wiki does not result in finding the thing, like it would in google for an open source project.

Having everyone customize the software does not result in a good product. It papers over a shitty UI by fragmenting it even more. After a while no one with any power in your organization realizes there is a problem because they have 50 GreaseMonkey scripts installed, and haven’t looked at the actual ‘base’ UI in 5 years. 

Save yourself millions in on-boarding. Invest in good tools. Put the work into offering a great default workflow. Don’t end up in a situation where the graybeards can’t even understand the workflows the new hires are dealing with. 

Business Intelligence what is it?

Business Intelligence is the process of exposing the results of customer workflows in an easy to consume fashion for Business Analysts, Accountants and Executives. 

When your product is software that your customers’ use, it can be hard to know how customers are using it in the large. For BI you log whatever the customer did in a format that is easy to query via SQL. For purchasing flow you log.

{
    "customerId": "ABC",
    "itemId": 1234,
    "time": "12:34 AM",
    "price": "$13.87",
    ……
    "sessionId": 987
}

Then analysts can figure out usage patterns, discover and fix bad customer experiences and generally figure out what customers are doing. 

How is this different from normal logging? 

Programmers care about exceptions, lines of code and values of variables. The business cares where the customer was in the workflow, what did the customer click on, what happened after that. If there was a silent retry that the customer didn’t notice the business analysts don’t care. 

They care about what customers are doing and how the software responds more so than what the software is doing. 

BI logging is typically preserved durably over at least months and needs to adhere to a schema that makes aggregating data together easy later on. This can be achieved with normal structured logging or by adding a separate code flow just for ‘Business’ logs. 

If you like my writing, please buy my book on Amazon.
The Sledgeworx Guide to Getting into Software


Multiple promotions for solving the same problem

One failure mode I’ve seen in software organizations is multiple promotions for building the same solution to a problem. Suppose, in your software organization you have 10 teams working in a similar problem space. In one year you see two promotions for building an asset management pipeline. When I saw the second promotion announcement, I was thinking to myself “Wait, didn’t Tyler get promoted for building an asset management pipeline?”

Any Vice-President who’s organization has this issue should be thinking hard about where it went wrong. That engineering culture is completely broken and can only be fixed by rooting out the leadership and then replacing senior engineers. 

Why is this such a big deal? Because its a symptom of several serious problems in your organization.

No information sharing between teams

One team built a solution to this problem in Q1, 6 months later another team built another solution to that problem. Why couldn’t they have shared asset management pipelines? If its a valuable thing to have why did one team go 6 months without an asset management pipeline?

Promotions are being gamed

Two people being promoted for building similar solutions to the same problem is a sign that your promotion process is being gamed. Redundant projects being lauded as keystone accomplishments is ridiculous. The manager of the 2nd team should have at least caught that this project was already a keystone. 

Important shared infrastructure is being ignored

If two teams are building the same infrastructure to solve similar problems it should be a shared service. Otherwise, you are paying to build twice and to support the system twice. This is software we should be able to shard this or multi-purpose the pipeline.

In this particular organization, the root issue is that the asset management system is horrible. Because the quality of that system is bad you have dozens of teams working around pain points with hacks. The solution is not to build asset management pipelines, the solution is to recognize how important this asset management system is and invest in it appropriately. 

Every asset management pipeline this organization built was a waste of effort that should have been invested at a higher level. 

If you like my writing, please buy my book on Amazon.
The Sledgeworx Guide to Getting into Software