Monday
Monday was pretty meeting-focused.
We had a huddle on deriving a set of SLOs from our initial graphite SLIs. The outcome of the session was that our metrics needed further refinement - what we actually want is to have a response time bound for well-formed requests and a different threshold for number of queries that time out or are invalid rather than overall request latency.
Our second session was a retrospective on how we handle ‘walk-ups’ - Shift is pretty lucky that we surrounded by our customers which keeps feedback loops tight but we become overloaded by questions and distractions. We used index cards to keep track of the number of walkups broken down by subject, and decided to productise it into a Google Form for longer-term storage and analytics.
Tuesday
I paired with Narayan on Tuesday to make some efficiency improvements to our generated firewall configurations. We’ve been less than judicious with some of the templated rule-sets and this was an opportunity to smooth out our global Puppet runtimes. We did this by putting feature flags on our security-related puppet classes and started to turn off parts that weren’t being used.
I also went along to my first weeknotes meetup! :D
The first venue that Steve suggested turned out to be mostly booked by a speed-dating event so we ended up decamping to a cocktail bar nearby. I always like meeting new people (big shout-out to Dan and Giuseppe) and it’s a weird sensation to hang out for the first time with people who you only know from the Twitter-sphere, but good times were had by all even if I did forget to actually eat and had a delicious two-pint dinner instead.
Wednesday
As tradition dictates, it was 20% time day - I finally got around to releasing my ProtonMail DNS terraform module and pushing it to the terraform registry.
I set aside a bit of time for attempting an upgrade of part of our Puppet systems and it turns out that it’s going to be a fair bit more work than I thought - we’re using the open-source version and we’ve architected it in a way that worked when we first brought it up but makes it harder to incrementally scale.
Thursday and Friday
I did a fair bit of pairing this week, in total!
I paired with Petrut on AWS optimisations and with Seng on improving the state of our SSL certificates, both of which required doing a fair bit of Terraform-ing (I swear this is 90% of my development time now, the rest is Python). I miss TDD’d app development. :(
Speaking of app development, Stephen made an initial release of a small puppet-token Slack app that builds on the data that we started piping into DynamoDB from our Puppet runs.
We have a monolithic shared puppet codebase and because we practice trunk-based development we also use a physical mutex to make sure only one team is committing/deploying at a time. This token, an android plush, often requires developers to go and “search” for it to acquire the lock (a practice that worked when we were small but Shift is working to make more scale-appropriate).
We can type /puppet-token into our Slack and it will tell you where it thinks the token is!
An improvement to our process, but the next step will be using these events (START, FINISH) to determine whether a run is in progress and use that as a mutex instead of our token. I’m excited for more of these small human-centric improvements.
This weekend, I ordered a copy of Shoshana Zuboff’s The Age of Surveillance Capitalism - probably required reading for working in ad-tech? - and I’m looking forward to eating through it, I’m going to regenerate my GPG keys, and then quite probably bake a Lemon Drizzle cake (which will require me taking at least some of it into work).