Turtle Generator Project Idea

New programmers sometimes ask me “what project should I work on next?”. This project is one I drafted up for myself because I wanted to build a more complex application with Django and pytorch.

The Turtle Generator Project is an attempt to create a website on which people can submit pictures of turtles and vote on whether machine generated turtles are “turtle” or “not turtle”. The user submissions and votes form a GAN or generative adversarial network, both classifying and producing pictures of turtles, although we may use user submissions as part of our dataset of turtles.

Pages / Components

Draw or Submit turtle component
Drawing component where a visitor can use their mouse or touchpad to to draw a turtle and submit it to the turtles dataset.

Vote component
A component consisting of a picture of a turtle generated by our backend algorithms and a button which says “turtle” or “not turtle”.

Architecture

Django frontend + PostgresDB
serves web pages and handles user interaction
votes and turtle drawing submissions are submitted to the MachineLearner system via Celery
submits “turtle image” request to celery queue — gets turtle back

Possibly Reactjs or just django templates 

Machine Learner
online machine learning system based on pytorch
takes celery tasks and either
generates a picture of a turtle
adds a vote submission to the training data ( classification )
adds a turtle picture submission to the training data

Celery + Redis
Message queue used to handle queuing training tasks

Software as a Service benefits both companies and consumers

I was surfing reddit this week and I saw a great comment from u/Swordbow. 

“Thus as the obligation stretches across time, the cash flow must stretch across time.” 

–Quote from u/Swordbow on reddit

Computer software isn’t like classical machinery in that bugs and vulnerabilities can be discovered after release that destroy the security of the application. A year after release you might have to perform an urgent patch to protect your customers from hacking. If your company goes bankrupt during that year, customers are in a very bad situation. Software needs to be maintained and unlike typical machinery it has to be maintained by the creators not the end users. 

Selling new releases of software is hard. Programs can be copied at zero marginal cost and don’t really wear out. Getting customers to buy a new version every year is a lot harder than getting customers to pay a monthly subscription. With Software as a Service instead of getting paid when you sell a new version of your software, you get paid as long as customers use your software. And you don’t have to worry about dealing with upgrades. 

Companies can reduce consumer surplus by switching to a SAAS model that requires customers to pay as long as they use the software. With SAAS all customers pay for every update. 

Creating a good Project README

Project README flies are typically an after thought in the software development process. If a question comes up repeatedly it gets added in an unstructured fashion. This is unfortunate, because the people who need READMEs the most are new engineers who joining the team. They don’t know any of the teams jargon. They probably don’t have a good understanding of what the project does. And they probably don’t understand the internal architecture of the project. 

You want the first part of the README to be an introduction to your project. Answer the question “Why do we have this service?”. 

To help new engineers use as little jargon as possible, and define terms in the README. 

Include a summary of the architecture of the project in the README. It should cover what abstractions you are using and why you picked the ones that you did. If you use any patterns that are not included in every project at your company make sure to mention them in the README. The last thing you want is for people to take over the project from you, not be able to figure out why you chose these abstractions and then removing them from the codebase. 

You README should also include the  steps to get the project running. What permissions and credentials do new engineers need to run builds and integration tests? Who should they contact to get those permissions? Make sure to include the common failure cases that new engineers ask questions about. 

Include a summary of the typical build process for the project. If you use make, write explanations for every make command you support and when they should be used. If you use a standard build tool like Maven, mention the extensions and plugins you use. “We use the Jacoco Plugin to ensure 80% code coverage, if you add a Spring configuration class you can add it to the ignored list for Jacoco.”

If you have integration or end to end tests in a different package reference in your README. Include an example of typical usage of the external package and expect people to read the README for that package if they run into trouble. Make sure to include common failure cases in the test suite. If external dependencies commonly cause your integration tests to fail, call out how a new engineer can determine that is the case and what they should do in response. 

Example Table of Contents for a README 

Introduction
    Why does this project exist? 
    Where can I find additional documentation?
    Where can I find our CI/CD infrastructure?

Architecture
    What is the basic architecture of the system? 
    MVC, SPA, messaging, RPC
    Do we have any managed thread pools?
    What are our asynchronous tasks?

    What patterns do we use in our codebase? 
    Explain any unusual patterns you use and why you need them.


How to get builds running
    What tools are needed to run builds?
   What build commands and flags should a new engineer be using?

How to get Tests running
    What tools to use to run unit/functional/integration/end-to-end tests
    Are any external packages needed
   How to retrieve the external packages
   Basic commands for any external packages 

   How to know if the tests passed or failed. 

Rework book review

I read REWORK by Jason Fried and David Heinemeier Hansson the founders of basecamp. The book is a series of short 200-500 word ‘sections’ that elaborate on a point. No wasted space or pages full of empty words where the point has already been made. As a result the book flows incredibly well. It is a quick and light read. The ideas in the book are commonsense lessons learned from running a successful small business. A lot of the ideas are shared with agile and the ‘lean startup’ schools of thought. But REWORK is a superior book to the ‘The Lean Startup’. Comparing the two books its clear Hansson and Fried understand the space better. 

A few points from the book stuck with me so I will go over them. 

Don’t write it down

The top customer complaints will come up so often you will never be able to forget them. You shouldn’t need a long list of customer issues, if you are listening to your customers regularly you won’t be able to ignore the top issues. If you get ten customer complaints each day and five of them are the same issue, you know what to work on. 

The myth of the overnight sensation

“And on the rare occasion that instant success does come along, it usually doesn’t last —there’s no foundation there to support it.” — page 196.

I liked this phrasing of the overnight sensation. These days social media constantly spams us with success stories and lavish lifestyles we could be living. But if you are relying on luck to succeed it might not come a second time, and then you don’t have anything left. 

Don’t scar on the first cut

Policies are only meant for situations that come up over and over again. You create a policy to make a common problem easier to solve. Without a policy you have to rely on judgement and escalating up the chain of command. That is expensive, but having a policy takes all the flexibility out of the situation. Don’t create policies unless its obvious that the issue is common and thinking about it is wasting people’s limited time. 

Four letter words

Don’t use the words “Easy”,  “Fast”, etc. Things are rarely done fast or easily. If they could be we would have done it already. Using those words implies things that we probably don’t know. 

Inspiration is perishable

If you want to do something, you have got to do it now. You can’t do it later because you won’t be inspired to do it later. 

Looking from the inside out.

I started my software engineering career in consulting. We did application development for large companies that did not have enough engineers on staff to get their projects done. We had a Statement of Work and a list of stories to do. Then we handed over the code and went back to our nice benches. Consulting was a big thing, we had executives, million dollar contracts, etc. I often ran in situations where I was spinning my wheels waiting for people to make decisions or to sell another contract. We ran some internal projects, but they were always an after thought. 

Now I work in a product company somewhat similar to the companies we used to consult with. We have what is essentially an unlimited amount of work. There isn’t any time for anyone to be ‘on the bench’. We have enough work for double our headcount. I have 5 real, valuable projects that I want to work on. It is just an issue of prioritizing and getting things done. But the thing is, a lot of stuff is falling through the cracks. 

We don’t have time to do everything, but we do need a lot of things improved. 

We could use an expert on CI/CD pipelines to come in for a month and upgrade our pipeline. We could use a java performance expert to come in and help us cut end user latency by 50%. We could use a consultant to help us break up the monolith. We have opportunities for a dozen consultants, but from the outside you don’t see the internal workings of the product company. You can’t see that we have hundreds of ways to make the product better and no where near enough work to do it. 

Solutions Pyramid

In College Computer Science Majors worry a lot about the optimal solution to algorithmic problems. What is the optimal solution? Is it in logarithmic or polynomial time? Did you pass or fail the test? My Algorithms class had a group class competition to see which team could find the best solution to a particular algorithmic problem. 

Then when they start looking for jobs they run into whiteboarding interviews where they are supposed to find the optimal solution in 50 minutes so we have time for questions. 

But once you enter the workforce, no one cares about the optimal solution anymore. If you look at the Solutions pyramid above there only one optimal solution. But right underneath there are 10 excellent solutions and a 100 good solutions. In the corporate world a good solution is an A grade, and an acceptable solution is passing. Typically as long as a enterprise software project avoids clearly ‘bad’ algorithms it will probably meet its throughput and latency goals. 

When your team has found multiple approaches to solving a problem and can’t seem to agree on the ‘right’ one. Ask yourself whether any of the solutions are in a higher level of the pyramid than the others. Don’t try to find the best solution, try to get a solution from the highest level of the pyramid implemented. And if all the proposed solutions are from the same level of the pyramid, sit back and relax, it doesn’t matter who wins.

You should not “build it here” even if your core competency IS Software.

Some companies try to avoid developing software that doesn’t directly contribute to making money. Other companies do the opposite and try to only use internal software. Falling into the “not invented here” trap.

Large software companies fall into a different failure case where they build large massively integrated internal development platforms. 

At the beginning these companies were solving a problem that had no existing solution and they correctly chose to create internal solutions. 

But it has been a long time. You should not be maintaining your own containerization system anymore. Use Docker. 

But for a big tech company moving to docker could be a five year effort costing millions. It is easier to keep the old system.

That large tech company gradually becomes trapped by internal software, that the rest of the industry has never heard of. As the company’s internal tech stack drifts away from the rest of the industry employees pay less attention to the mainstream.

Eventually the company loses perspective on what they are missing. The people who have worked there the longest who have the most power to improve things think the internal software is better. 

These companies are big enough that they have internal developer conference where employees can learn about the latest in internal tooling. 

Today most of the components of these stacks are worse than the industry mainstream alternative. But there is no upgrade path on the horizon.

A better way to do Agile Estimates

The standard agile estimation process falls into typical estimation traps that make creating accurate estimates difficult. Major traps are  relying solely on expert judgement,  rushing the team to estimate in a high pressure setting, like a sprint planning meeting, and we often estimate work without clearly defining the scope.

When estimating, we want to prioritize Counting, Computing and Judgement, in that order. The best way to estimate something is to have a record of how long things took historically, then create a complete set of stories for the work. To create the estimate you simply count up the stories. In construction you might have a certain number of feet of drywall you need put up in your house. To estimate the cost you would multiply the number of feet by the going rate per foot of drywall and end up with a pretty good estimate. 

In software we work with myriad heterogeneous tasks that we don’t have industry standard data for. So we have to rely on Computing and Judgement to create our estimates. To compute an estimate you could do something like reasoning that since your historical rate of completing stories is 2 days for the company, we will create an estimate based on (2 days * #stories). That is computation will create a rough estimate at best so it would be worthwhile to round it to one significant digit. What you should not do is use your judgement to tweak the estimate after you compute it. Don’t say to yourself, “Our team is the best in the company we will only take 1.5 days per story” if you don’t actually have data to back up that rate of 1.5 days per story.

Our last resort in estimation is to use the judgement of experts to tell us what they think is a reasonable estimate. The problem is that in Agile we rely on Software Engineers who have no training in estimation and our estimating process encourages them to take shortcuts when creating estimates. 

The standard agile process today is to put a bunch of engineers in a room and have them play ‘planning poker’ where they are expected to use expert judgement on a short timeframe, without access to any historical data. 

We want to avoid pressuring people to ‘rush’ when estimating. If data is available we should have time to look it up and compare with previous projects. 

I propose a tweak to the standard sprint planning meeting called ‘Serious Agile Estimating’. Before sprint planning most teams will have a backlog grooming session. After that meeting prepare a spreadsheet of every story that your team needs estimates for. Next fill out the estimation form with the stories for the next sprint and send it out to each team member before the sprint planning meeting. 

The sprint planning meeting will be split into 2 phases, estimation and discussion. The estimation phase will take up the first 30 minutes of sprint planning and should be carried out in silence. Each team member should have a laptop and the estimation form. Team members should take this time to read through the stories creating best, most likely and worst case estimates. After 30 minutes, ask everyone to submit their forms online. 

The discussion phase of the meeting is focused on discussing the differences between people’s estimates. Was there work that we forgot to break up into stories? Do members of the team have differing opinions on how much work different tasks are? What data did team members use to support their estimates? It is important to remember that we do not change estimates during the discussion phase. 

Never make estimates off the cuff or during a meeting while people are talking. Having a hard rule of ‘never make estimates during a meeting’ would be the best. But in my experience it is more reliable to block off focused time in a meeting than to have everyone do it beforehand. 

Serious Estimating Steps

  • Create a list of groomed stories to estimate ahead of time
  • Give each team member a checklist and estimation form (See the example form included at the bottom of this post.)
  • Have each team member estimate stories individually then meet to compare your estimates. 
  • Do not make any estimates during the discussion
  • Require team members to submit a complete set of estimates before the meeting. 
  • Don’t just average the estimates and call it good enough. You need to discuss the differences among individual results, don’t just take the calculated average automatically. 

The big difference is that team members create estimates on their own with ample time to think. No count downs, no poker, estimates are written down and then discussed. 

Finally, I strongly recommend reading Software Estimation Demystifying the Black Art by Steve McConnell which is the source of many of the ideas in this post. 

The book can be found here: https://www.construx.com/books/software-estimation-demystifying-the-black-art/

Google Docs Link:

https://docs.google.com/document/d/1H0CkBlGcRCXxPFMQ3QlXuPzjEUJJRNw/edit?usp=sharing9y9dIw6HnRGm

Bookshelf Post

I have gotten back into reading Software Engineering books. It has been a good change of pace to study a bit in the mornings. My current focus is on Estimation. I want to come up with a better more accurate way for us to estimate software projects in agile environments. 

These two books are great references. Designing Data-Intensive Applications is a good read and believe it or not my workplace still uses Java 8 so Java Performance is not out of date.

I got these two management books back when I was a manager for a few months. But I haven’t dug into them yet. 

Agile Software Development was a textbook back in University. It is a funny book since it has about as much space devoted to Agile as it does to Object Oriented programming. The Mythical Man-Month is fun because of all the references to technology in the old days before the modern web. 

Building Products for the Enterprise is a product book, but I really like how it really explains what those Product Owners are up to all day. 

How much does using Haskell instead of Java cost your company?


Following on my earlier post “How much is turnover costing your company?” this post is about how much using non-standard technology stacks costs your company. Lets say company A is high performance company with a custom functional programming stack that uses Haskell and Miso for development. Company A doesn’t ship a lot of bugs to production, in fact they run CI/CD with deployments to prod everyday. Their SAAS product is one of the best in the industry and is growing at a good rate. 

Now Company A has a competitor in the form of Company B. In this case B does stand for Boring. Company B offers a SAAS product that is almost identical to the one Company A sells but Company B’s has more bugs because they use Java. In fact Company B uses Java with Spring Boot and ReactJS. Both Company A and Company B started in 2018 when React was already mainstream. Company B also uses CI/CD deployments but they can only deploy to production once a week because there are so many bugs. Company B also hired a QA Engineer to help support the pipeline because of an embarrassing outage that CompanyA used in a humiliating Ad. 

In January 2019 both companies raise a round of founding for 10 million dollars. The mandate from the VCs is to 10x the company in size and destroy the competition. Both companies need to start hiring fast. Throughout 2019 CompanyA has an advantage, there are a bunch of passionate Haskell engineers that want to work on their product. Both Company A and B reach their growth targets and hit 100 engineers that year. 

Fortunately, things go well for both companies in 2020. The market is growing and revenues are increasing. The VCs call for more growth. Now we want to grow engineering to 500 engineers. 

Now Company A has a problem. They already hired all the passionate Haskell+Miso engineers they could find. Any new engineers will take 12 months to train on their technology stack. CompanyB doesn’t have this problem everyone still remembers how ReactJS and Spring Boot work even though they are no longer the top technologies. Company B only needs 6 months to train a new engineer on their technology stack. 

Lets run the numbers. How much is it going to cost to onboard 400 new engineers for each company?

We will keep our 2 year average tenure estimate and use 12 months of onboarding for company A. Each engineer has a total cost of 200k including salary, equity, benefits, taxes and insurance. 

Each engineer is going to cost around 100k to onboard at Company A. 

Each engineer is going to cost about 50k to onboard at Company B.

Now the next question, how long will it take each company to get all 400 new engineers fully operational? 

My assumption is that it takes 1 fully onboarded engineer to train a new engineer. Each company has 100 engineers to start so at the beginning the first 100 engineers will train the next 100. Then each company will have 200 engineers to train the next 200. Then things calm down a little and we have 400 engineers to train the last 100. 

After 12 months Company B has double the trained engineers Company A has. At 18 months Company B is finished onboarding and can have all 500 engineers focus on the product. Company A won’t catch up for another 18 months.

Most people will agree that Company A’s functional programming stack is not going to make up for the extra engineer years that Company B has at this point.