Lorin Hochstein on complex systems, resilience engineering, and human adaptability in software.
Last week, GitHub Actions experienced another incident. As is typical of a GitHub public writeup, it was published very soon after the incident, but there also aren’t isn’t a lot of detail. But let’s see what we info we can glean from it. Database saturation T…
I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-a-service (do people even say that anymore? in the cloud. This doesn’t just apply…
Recently, two AI-related pieces of content caught my attention. The first was the blog post On-Call is Now Theatre by Boris Tane. He argues that AI agents are now capable of doing the majority of on-call work that is currently being done by humans, and that we…
In yesterday’s post about the recent GitHub outage, there was a detail in the writeup that I didn’t say anything about: the autoscaling policy on the service with the saturated Istio sidecar. Originally this was caused by an Istio sidecar pod reaching its conc…
It wasn’t even a week ago when I wrote about a major GitHub incident. Yesterday, they had another big incident, which lasted almost eight hours. There’s a public write-up already posted, which is surprisingly quick. While I’m personally very impatient to read…
The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at t…
On August 6, 2026, GitHub had a pretty rough incident: GitHub Actions was degraded for about nine hours. GitHub posted a public incident write-up. It’s only a few paragraphs long, but there are some interesting details in here. This was yet another incident th…
As a fan of resilience engineering, I often differ with people on where we should focus our scarce engineering cycles in order to improve reliability. I thought it would be a useful exercise to brainstorm some of the differences in focus between what I’ll call…
There’s a new software reliability conference that just spun up called Software Should Work, and I had a chance to give a talk there. I took the opportunity to speak about one of my favorite topics: saturation. Here’s a recording of it. Links from the talk
Over the years, mathematicians, logicians and computer scientists have developed various calculi. If you have a background in computer science, you’ve likely heard of the lambda calculus, a model of computation that was developed by Alonzo Church. If databases…
The other day, Reginald Braithwaite posted the following toot. For posterity, I’ve also included my own response to it: Braithwaite’s post is dripping with sarcasm, but make no mistake, incident reports written entirely by LLMs is coming. And I am not looking…
The Journal of System and Software publishes a regular column called Dear Researchers: The perspective of software practitioners. Each column is an open letter to the software engineering research community from someone who works in tech. It’s edited by Austin…
I can’t bring myself to read text if I believe it to be AI-generated. Now, I ask LLMs questions all of the time, and I do read those answers. I frequently use tools like ChatGPT and Claude as replacements for Google for answering specific questions; that’s not…
Modern software systems contain within them a mind-boggling level of complexity. As software engineers, we make this complexity manageable through techniques like decomposition, information hiding, and abstraction. We endeavor to break our systems up into comp…
At this point, you have no doubt heard about GitHub’s availability woes over the past several months. In April, Mitchell Hashimoto (of Hashicorp fame) wrote a post about how he is moving his Ghostty project off of GitHub: …for the past month I’ve kept a journa…
We cut middle managers across the organization because AI allows us to have more direct reports per manager while still measuring and mentoring our teams effectively. – Matthew Prince, How I Choose Which Cloudflare Employees to Replace With AI My PhD research…
I’m a betting man; I just enjoy making bets, even when there are no stakes at all. And when you talk about bets, you end up talking about odds. It turns out that reliability is also about odds, even though we don’t use the language of odds in our domain. Consi…
“Flipping the bozo bit” is an expression from the software world. Think about a time when you reached a point where you simply stopped respecting the opinion of a particular person, most likely a co-worker. From that point on, you disregarded what they said. T…
Here’s a famous optical illusion, which was developed by the American neuroscientist Edward H. Adelson. Even though square A appears darker than square B, the two are, in fact, the exact same shade of gray. It’s such a powerful illusion that, even knowing the…