don't worry, it's probably fine

Notes from the Week #2

28 Sep 2018


It’s been an absolute blinder of a week with some big wins and big learnings - thinking about marginal gains, evangelising approaches to colleagues, trying to keep myself accountable, and my team’s novel approach to code improvement.

Marginal gains

I’ve been working with one of the core product development teams during my 20% time and spare moments that I can grab, making small changes to incrementally improve their build times. They were running into issues where we were delivering too quickly (yes!) and were being bottle-necked by the deploy.

The team and I learned some important lessons about how powerful the accrual of marginal gains can be:

  • We started using .parallel() in some of the integration tests, once the stream operations were determined to be safe to run concurrently. Gain: ~30 seconds
  • A pre-deploy gate checks the existence of assets on AWS S3 - we moved from call-per-asset to a single metadata fetch for an S3 directory. Gain: ~1 minute
  • The java app in question makes calls to S3 during operation - importing aws-java-sdk-s3 rather than aws-java-sdk removed over 50% of the build artifact size. Gain: ~1.5 minutes

In total we managed to remove almost 3 minutes from our deploy times - to put this in perspective we deploy the application between 5-10 times a day. By the end of the week we will have saved ourselves over an hour of time that can now be spent getting faster feedback.

Our approach of trunk-based development combined with deploying straight to production means our testing and deploy times are absolutely crucial, and we’ll be using the learnings from this to inform future improvements.

Tests and deployment processes need absolutely as much care and attention given to their efficiency as the production code itself.

(This article on the British Olympic Cycling Team is a great example of applying the concept of marginal gains to gain incremental advantage)

Evangelising to colleagues

We have fortnightly Tech Talks on Thursday afternoons - these run the spectrum from “here’s a cool thing I built”, through conference reports, to “getting a bit ranty” about technologies or processes.

I gave a 20 minute talk on developing software in the open, a technique I’ve gained a lot of enthusiasm for over the last year (and since I became Team Lead). Shift has been actively developing projects in the open recently (see our java-config-fallback, hiera-secrets-manager, and unruly-puppet) and I took this talk as an opportunity to explain the virtues and philosophy of the idea.

Below is a cribbed version of my talk containing some of what I feel are compelling reasons to start developing software in the open.

  • Eyes on the code
  • Having lots more eyes on our code than just the writers increases the chance of finding defects that we’ve missed
  • Leveraging the Hawthorne Effect that people behave differently when they know they’re being observed.
  • We build code as if we were a consumer, not the consumer and so think harder about design decisions
  • It’s good for the ‘free market’
  • Reducing barrier to entry in an industry is good
  • When ‘anyone’ can start up an ad-exchange or an RTB bidder, we open the door to more possible innovation.
  • Free stuff is great
  • Lots of services offer free plans for open-source software
  • Not just CI (e.g. TravisCI/CircleCI) but dependency checking, code coverage, and security patching.
  • Thinking harder about APIs and contracts
  • It’s easy to be ‘sloppy’ with inter-app or inter-library contracts if you control both of them.
  • Developing libraries in the open can compel a stricter compliance to APIs and versioning.
  • Security
  • You can’t put credentials in code any more
  • Well, you can, but you can’t get annoyed when someone uses your credentials.

A group of developers at Unruly are advocating a move towards this approach, so watch this space for more!

Keeping myself accountable

I get excited about some of the work we do and often take on more than I can handle, so I’m working on ways to visualise what I’ve agreed to do (and in some cases, give other team members the opportunity to take these things off my hands).

We had a new member join our team last week, and we’ll be celebrating by redrawing our team’s wipeboard - I’ve requested a section for myself on the board where I will put my own queue of tasks and ‘todos’.

I am aiming to keep myself more accountable to my team, visualise my tasks so it becomes obvious when I’m doing too much, and give them opportunities to step up and take responsibility for themselves.

Experiments and Gardening

Qaiser, our Agile Coach, has introduced a way of formalising process experiments that we run. These are little cards with fields like:

  • Experiment Champion - who’s proposing the experiment?
  • Problem - what are we trying to address?
  • Success criteria - how will we know if it works?

We’re running an experiment to capture time spent on making our codebases more habitable - we’ve found it difficult to capture this as technical improvement work or business-value-driven stories.

We call this ‘gardening’ - a pair can opt to do a bit of gardening when we’re deciding on work for the day. The odd name comes from the notion that gardens need constant attention to flourish: pruning old branches, trimming overgrown shrubbery, and more.

Our codebases, when not given ample care, can turn into something that feels like a gnarly and overgrown garden and a small amount of tidying and care (like refactoring tests, cleaning up documentation) can prevent us from having to hack through a jungle at a later date