A software daemon is some bit of software that runs independently of human control. We have been creating them for most of computing history. But the majority are not that important. Nobody knows their names. Nobody really cares that much if they have bugs. If one stops working you restart it and move on with your life.
Cryptocurrency blockchains are also daemons. Each blockchain is just software that performs whatever tasks it is programmed to do. The interesting differences are the scale and the distributed nature of the blockchain. Millions of people contribute hardware to run blockchain software as opposed to a cronjob running only on my laptop.
The Ethereum blockchain is a billion dollar distributed computing platform powering things like DNS names (ENS) and art NFTs. Anyone can submit a ‘contract’ to Ethereum as it is essentially a daemon that runs other daemons. Compared to the log daemon that collects logs on your computer, Ethereum is immensely larger. Restarting your log daemon is essentially a free operation. Restarting Ethereum from zero would cause billions of dollars of losses.
Due to their decentralized nature it is also impossible to ‘stop’ a blockchain software daemon. Ethereum exists on millions of computers and as long as even one of those copies continues to operate Ethereum will continue to exist.
People also care enough about these blockchains that they have names. You have the ‘Ethereum Classic’ and ‘Eth2’ daemons which run very similar software, yet their identity is actually important to humans. Millions of copies of Linux and Windows are running right now, but none of them are important enough to have names.
Voice assistants are wide software. In the industry we call it the ‘long tail’ of functionality. There are hundreds of ‘tasks’ that your Alexa or Google assistant can perform for you. You probably don’t know that most of them even exist. But not knowing that these tasks exist costs you nothing. The fact that you can buy pizzas via Alexa has no impact on your ability to get news briefs. You can do either without ever engaging with the other.
This ‘long tail’ attribute makes voice assistants extremely wide software. Alexa can do hundreds of things which could be stand alone applications. But the trade off is that voice assistants are extremely vast and don’t do anything particularly well. Over the time the ‘main’ functionalities will be refined and optimized. But Voice assistants will always suffer from the ‘long tail’ problem in that they have extremely wide feature sets.
Wide software spreads across multiple domains. More domains means leaky abstractions and mapping software.
Deep software focuses on a single domain. Perhaps it is an order book or a workflow execution engine. Deep software has a clear purpose and domain. Wide software does e everything.
Over time deep software converges on clean abstractions and easy to understand code. Wide software on the other hand is never finished. Wide software is naturally expansive. There is always a reason to add a new functionality to a voice assistant. And in fact there is no real barrier to entry. Adding a new functionality to a voice assistant is a net positive to the system as a whole. The negatives of adding a new domain are already baked in and many customer will enjoy the new functionality.
Deep software can be finished. It can solve a problem in one domain and be done. Hadoop is an example. No one has heard about miraculous developments in HDFS this decade. Hadoop is essentially feature complete and in maintenance mode. In reality development continues, but is it really new stuff or refinements?
Wide software cannot be ‘finished’. Wide software is an infinite sinkhole. Adding more code makes the sinkhole more valuable so more code keeps getting added. There is no real way to ‘solve’ the problems of wide software. You can partition wide software such that each domain exists separately. But if you allow one domain to reference another, now you are back into the pit.
Software Leviathans which I’ve discussed in another post (https://www.sledgeworx.io/software-leviathans/) are wide software. Supporting more domains typically increases the value of the leviathan as a whole. A voice assistant which can order dry cleaning is better than an assistant that can’t order dry cleaning. Overall there isn’t a trade off between the two. One has an additional ‘ability’ with no downside to adding that ability. You would have to make a voice assistant that only handled one domain to escape this constraint.
Wide Software isn’t magical. Wide software does too many things to be incredible at any of them. Since there are countless features the team has to spend a lot of time making sure they don’t break anything. In software leviathans not breaking things is particularly difficult because nobody actually knows what all the features are.
Since wide software is always being pushed to add something new. Energy and design focus are constantly shifted towards new features and problem domains. New domains expect old domains to support new features.
Wide software suffers another problem which is that even if some domains in the project are invested in continually say music for example. Even if music functionality is iterated on again and again. That particular domain being awesome doesn’t change the flavor of the beast. It is still a ball of mud, dirt, rocks, etc.
My belief is that most tech startups going forward will be 100% remote. Setting up an office for a company that might not last isn’t a great idea. And after 2020 spending your runway on an office is basically wasting money.
The complication is that remote work requires more discipline from the team than office based approaches. If you have unmotivated employees, managers, management skills, or outdated processes you will not be as productive as a remote company. Going forward there will be a dichotomy between startups that try to succeed with the colocation approach and startups that commit to remote work. My belief is that overtime collocated startups will consistently fail against remote startups. The reasons remote startups pull ahead are lower fixed office costs, easier access to talent at lower prices and that remote work requires more disciplined processes.
Fixed office costs
Offices being expensive is an obvious fact for most people. Offices are lowest common denominator environments with constant pressure to cut costs. Workers consistently complain about noise and distractions in ‘open plan’ offices but they have stuck around because they are cheap.
Remote work allows everyone to setup their own ‘dream’ office space. You can get whichever chair you want. Any combination of monitors and other peripherals. You can play loud music or install noise canceling tiles to ensure silence. Workers also save hours each day compared to commuting, not to mention the environmental benefits.
Easier access to talent
Access to talent is a major problem for startups. Before achieving profitability startups have limited time to find software engineers and limited money to pay them. Hiring remotely allows startups to access a large number of employees who live in low cost of living areas. This can either be lower cost of living cities, cheap small towns or even other countries.
For example entry level software engineers typically make over 100k in ‘major’ software areas like Seattle, San Fransisco, San Jose and New York City. But those same engineers will typically make 60-70k in second tier cities despite having comparable amounts of skill and experience.
Higher Process Bar
A remote first company requires better processes and more motivated employees than the typical office micromanagement scenario. Failing to set up good processes in a remote environment will result in unmotivated employees that don’t do any work. You can’t just assume communication will happen in a remote environment. In the office employees gossiping can make up for a lot of bad communication processes. Whereas in a remote environment employees need to go out of their way to setup a zoom call.
Companies with good remote work processes will experience higher average productivity than was ever achieved in classical ‘open office’ environments.
In a remote work environment you are forced to invest more in onboarding, training and cross team coordination. In the office you can simply walk over to the frontend team’s section and introduce yourself to the new team member. But working remotely without a process to include the new guy, he could just sit in the figurative corner for weeks without interacting with anyone.
There isn’t any way for new hires to ‘overhear’ conversations about services or context that they don’t know about when working remotely. So those thing have to be documented and included in the formal onboarding process. Which means you actually have to document your software for real.
Meetings need to be orchestrated better when done remotely. You need to have an agenda and focus on solving the problem.
One meeting that works very well remotely is the architectural document review. A 60 minute meeting would go as follows.
20 minutes to read and comment on the document in google docs
Short discussion of each text comment made in google docs starting from the top of the doc
Discussion of action items and any other final comments
What happens next?
The pandemic caused a massive shift in software engineering working conditions. We went from maybe 5% remote to 95% remote work during the crisis. Now that things are normalizing I expect a gradual shift back towards 90-10 or 80-20 split of hours worked in office vs remote. The fact of that matter is that a lot of people like working in an office. And many bosses think micromanaging is necessary.
Going forward there will be a lot of demand for remote software engineering jobs and many companies will jump on the chance to acquire programming talent at competitive prices.
The in-office premium.
Going forward I expect to be paid more to work in an office than to work remotely. Working in an office costs me more in time and transportation costs. It also limits my freedom during the day. To make up for the inconvenience of needing living within commuting distance of jobs I expect to be paid an additional 20-30% premium over what I would charge for remote work.
The legal burden for international remote work is way too high. Each country has different requirements and the company is essentially required to establish a corporate entity in every country they have employees in. Even if they do not actually do business in that country.
Even in the United States having a remote worker in New York city could require a company to start paying additional taxes to New York state due to establishing a ‘nexus’ in that state.
Over time I expect these legal hurdles to be simplified or simply ignored. Functionally, it doesn’t matter where code is written. Remote workers will push for legal accommodations. Nations will be forced to compete by providing supportive legal conditions for remote workers and employers.
One thing that I didn’t expect to come out of remote work is that annoying coworkers are a lot less annoying when they aren’t in your face. You can basically avoid all non-work related communication with people you don’t like. For example we have one guy on my team who goes on rants and never gives people a chance to cut in and steer the conversation back on topic. In the office I would end up listening to this guy talk for minutes at a time without getting a chance to talk. Now since I’m not sitting across from the guy I can avoid his negatives and appreciate his dedication and work more.
For example processes like 6 month RSU vesting are not tenable in a remote first world. You cannot afford for disgruntled employees to wait 6 months for a vest date before they quit.
Our next SledgeConf is coming on Friday May 21st at 4:30pm PST!
Join us for discussion of the next 10 years of remote work.
My presentation for this SledgeConf is titled.
“The next decade in remote work.”
I’m joined by Mason Traylor presenting!
“How to waste Everybody’s Time with Remote Meetings”
4:45 Presentation “How to waste Everybody’s Time with Remote Meetings”
5:00 Questions + Discussion
5:15 Presentation “The next decade in remote work.”
6:00 Promotional Round table
6:15 Unstructured Discussion time
The zoom link will be sent to the mailing list the day of SledgeConf. Please go to SledgeConf.dev and sign up for the mailing list. If you are one of my connections on linkedin the zoom link will also be available there.
My belief is that most tech startups going forward will be 100% remote. Setting up an office for a company that might not last isn’t a great idea. And after this last year spending your runway on an office is basically wasting money.
What would a ‘best practice’ startup look like if it launched today?
The tech stack really depends on what you are doing. There are a bunch of right answers just do your do diligence and commit to one.
For an MVP I would go with Ruby on Rails with Heroku and postgresql. Don’t waste your startup’s time on infrastructure as code, use a complete platform as a service. Infrastructure as code provides next to no value to customers and eats insane amounts of time. Use the simplest cloud architecture you can whether that is Heroku or copying code straight into EC2 hosts. Kubernetes is great but your startup should achieve product market fit long before you consider that beast.
If you are doing machine learning Pytorch or tensor flow are good options. For low level or high performance computing Rust and modern C++ are great.
The only things I would be careful about here is in picking the framework. You want a ‘batteries included’ framework. If you are in a rush you don’t want to have to dig through a bunch of github pages to figure out which authentication library is best.
There are a lot of options here. Just make sure you have a good videoconferencing application, email and a messaging app.
Slack, Email, Zoom
Github/Gitlab, source hut, etc
Teams is another great option with everything included.
Regular one on one meetings with teammates is a huge benefit for remote workers. I recommend scheduling a variety of optional meetings where you can get face time in with teammates.
Donut is a slack app that will automatically schedule 1 on 1s with different team members.
Document based decision making is one of the best ‘mechanisms’ I’ve encountered at Amazon. Technical proposals are written up into an architecture document and then reviewed in a team meeting. The first half of the meeting is spent reading the document and making comments in google docs. Then the second half is spent discussing the proposal and comments.
Agile Kanban style workflow
Kanban is the most realistic way to organize work on a software team. It supports coordination, helps avoid duplicating work and eliminates process.
Do a monthly and quarterly planning process. Leadership should identify the business areas that are the priority for the next 1-3 months on a regular basis.
At Amazon we have a bottom up planning process where teams and groups put together a document of their goals and priorities which is then reviewed and approved by the executive team. A bottom up process shouldn’t be possible in a startup because there is no ‘executive’ team just a few founders.