git format-patch and git am

# format-patch creates a patch file
# the -1 $COMMIT arguments take the last commit and put it into the file
git format-patch -1 14ab6d…..

#Applies the commits in the patch file onto the current branch
git am ../folder/2019-10-17/0001-SLEDGE-32

#This git log command pulls the history of a file into one patch
git log --pretty=email --patch-with-stat --reverse -- path/file_or_dir 
git am <  path/to/file_or_dir 

Turtle Generator Project Idea

New programmers sometimes ask me “what project should I work on next?”. This project is one I drafted up for myself because I wanted to build a more complex application with Django and pytorch.

The Turtle Generator Project is an attempt to create a website on which people can submit pictures of turtles and vote on whether machine generated turtles are “turtle” or “not turtle”. The user submissions and votes form a GAN or generative adversarial network, both classifying and producing pictures of turtles, although we may use user submissions as part of our dataset of turtles.

Pages / Components

Draw or Submit turtle component

Drawing component where a visitor can use their mouse or touchpad to to draw a turtle and submit it to the turtles dataset.

Vote component

A component consisting of a picture of a turtle generated by our backend algorithms and a button which says “turtle” or “not turtle”.

Architecture

Django frontend + PostgresDB
serves web pages and handles user interaction
votes and turtle drawing submissions are submitted to the MachineLearner system via Celery
submits “turtle image” request to celery queue — gets turtle back

Possibly Reactjs or just django templates

Machine Learner
online machine learning system based on pytorch
takes celery tasks and either
generates a picture of a turtle
adds a vote submission to the training data ( classification )
adds a turtle picture submission to the training data

Celery + Redis
Message queue used to handle queuing training tasks

What I learned from a year of Devops

In 2017 I had the opportunity to spend a year working as a devops or platform engineer. I have mainly worked as a software engineer before so moving in to an automate and operate role was a bit of a leap. This was a fully remote engagement where I was embedded with and helped bootstrap the client’s first platform team.

The project was building out continuous integration and delivery for a client of ours that had no AWS experience. Before they brought us in they ran all their systems in their own datacenters in a windows and .Net environment. We came in to assist with the move into the cloud and to help transition the company from .NET to Java, javascript and microservice development.

The first few months we focused on building out the CI/CD with Jenkins pipelines and a great deal of AWS cli scripting. Once we got the basics working teams started to come out of microservices training and began developing against it. This was the start of operational support for us and started a bit of a scramble while we tried to balance new features and the stability of the platform with hiring and onboarding.

We used jenkins pipelines, docker and cloudformation to provide our users with a solid customizable pipeline solution. Using our default templates development teams could easily bootstrap their pipeline with CI/CD from dev to canary deploys in production. If they needed more than a stateless microservice we enabled them to provide cloudformation templates in their github repository that would be run with each deploy to ensure the AWS environment was bootstrapped for their needs.

We started out with the intention of using Jenkins pipelines with ansible to automate things, but the client’s team was more experienced with CloudFormation and as a result I ended up writing most of our initial CI/CD code in a combination of groovy and AWS cli calls. This proved unwieldy and eventually led us to using Groovy + Cloudformation for nearly everything. Cloudformation works but it is locked into AWS and its programming model is a somewhat awkward. Cloudformation’s saving grace is the first class integration and editor. Next time I would recommend starting with a commitment to Terraform or Ansible.

In the 3rd quarter we started work on implementing Canary deployments. Our solution ended up being a combination of a customized client side load balancing http client and jenkins pipelines. I started us off with a proof of concept that proved easier to write than we expected which put us on good footing for the rest of the project. One of the client’s employees took advantage of the space we had to rewrite the shared jenkins pipeline library in more idiomatic language which turned out to be a great improvement.

We went live in Q4 and I moved on to another project. I am moving back into application development, I ended up doing 100% automation scripting instead of the 50-50 split I was expecting. So it will be good to get back to writing applications.

Starting a new Go project is delightful.

I started a couple projects recently, a mock crypto exchange and my latest project a unicode manipulation library. But what struck me is that its really simple to get started. You need the go runtime and a GOPATH setup.

Then you specify the package and the main method and that is a valid program.

package main

func main(){}

Above is my program so far. It is a short program with a bit of exploratory code that converts strings into their unicode rune ids. Back when I mainly used Java, I would have had to setup an IDE, integrated maven and think about package structure. In Go I just have a main.go file, if I need dependencies I create a /vendor directory and usego get github.com/gin-gonic/gin to pull the dependency.

Overall the lightweightness of the Go tooling makes it very easy to build small programs.