Mike's Notes
An excellent article by Martin Sustrick on why accountability is lacking. With lots of examples. A long read and lots of quotes to follow up on.
Resources
References
Repository
- Home > Ajabbi Research > Library >
-
Home > Handbook > Teams > Culture
Last Updated
13/05/2025
Accountability Sinks
By: Martin Sustrik
250bpm: 22/04/2025
Accountability engineering in institution design
Back in the 1990s, ground squirrels were briefly fashionable pets, but
their popularity came to an abrupt end after an incident at Schiphol Airport
on the outskirts of Amsterdam. In April 1999, a cargo of 440 of the rodents
arrived on a KLM flight from Beijing, without the necessary import papers.
Because of this, they could not be forwarded on to the customer in Athens.
But nobody was able to correct the error and send them back either. What
could be done with them? It’s hard to think there wasn’t a better solution
than the one that was carried out; faced with the paperwork issue, airport
staff threw all 440 squirrels into an industrial shredder.
[...]
It turned out that the order to destroy the squirrels had come from the
Dutch government’s Department of Agriculture, Environment Management and
Fishing. However, KLM’s management, with the benefit of hindsight, said that
‘this order, in this form and without feasible alternatives,* was
unethical’. The employees had acted ‘formally correctly’ by obeying the
order, but KLM acknowledged that they had made an ‘assessment mistake’ in
doing so. The company’s board expressed ‘sincere regret’ for the way things
had turned out, and there’s no reason to doubt their sincerity.
[...]
In so far as it is possible to reconstruct the reasoning, it was presumed
that the destruction of living creatures would be rare, more used as a
threat to encourage people to take care over their paperwork rather than
something that would happen to hundreds of significantly larger mammals than
the newborn chicks for which the shredder had been designed.
The characterisation of the employees’ decision as an ‘assessment mistake’
is revealing; in retrospect, the only safeguard in this system was the
nebulous expectation that the people tasked with disposing of the animals
might decide to disobey direct instructions if the consequences of following
them looked sufficiently grotesque. It’s doubtful whether it had ever been
communicated to them that they were meant to be second-guessing their
instructions on ethical grounds; most of the time, people who work in sheds
aren’t given the authority to overrule the government. In any case, it is
neither psychologically plausible nor managerially realistic to expect
someone to follow orders 99 per cent of the time and then suddenly act
independently on the hundredth instance.
— Dan Davies: The Unaccountability Machine
***
Someone – an airline gate attendant, for example – tells you some bad news;
perhaps you’ve been bumped from the flight in favour of someone with more
frequent flyer points. You start to complain and point out how much you paid
for your ticket, but you’re brought up short by the undeniable fact that the
gate attendant can’t do anything about it. You ask to speak to someone who
can do something about it, but you’re told that’s not company policy.
The unsettling thing about this conversation is that you progressively
realise that the human being you are speaking to is only allowed to follow a
set of processes and rules that pass on decisions made at a higher level of
the corporate hierarchy. It’s often a frustrating experience; you want to
get angry, but you can’t really blame the person you’re talking to. Somehow,
the airline has constructed a state of affairs where it can speak to you
with the anonymous voice of an amorphous corporation, but you have to talk
back to it as if it were a person like yourself.
Bad people react to this by getting angry at the gate attendant; good
people walk away stewing with thwarted rage.
— ibid.
***
A credit company used to issue plastic cards to its clients, allowing them
to make purchases. Each card had the client’s name printed on it.
Eventually, employees noticed a problem: The card design only allowed for
24 characters, but some applicants had names longer than that. They raised
the issue with the business team.
The answer they've got was that since only a tiny percentage of people have
names that long, rather than redesigning the card, those applications would
simply be rejected.
You may be in a perfectly good standing, but you'll never get a credit. And
you are not even told why. There's nobody accountable and there's nobody to
complain to. A technical dysfunction got papered over with process.
***
Holocaust researchers keep stressing one point: The large-scale genocide
was possible only by turning the popular hatred, that would otherwise
discharge in few pogroms, into a formalized administrative process.
For example, separating the Jews from the rest of the population and
concentrating them at one place was a crucial step on the way to the
extermination.
In Bulgaria, Jews weren't gathered in ghettos or local "labor camps", but
rather sent out to rural areas to help at farms. Once they were dispersed
throughout the country there was no way to proceed with the subsequent
steps, such as loading them on trains and sending them to the concentration
camps.
Concentrating the Jews was thus crucial to the success of the genocide.
Yet, bureaucrats working on the task haven't felt like they were personally
killing anybody. They were just doing their everyday, boring, administrative
work.
The point is made more salient when reading about Nuremberg trials.
Apparently, nobody was responsible for anything. Everybody was just
following the process orders.
To be fair, the accused often acted on their own rather than following the
orders. And it turns out that the German soldiers faced surprisingly mild
consequences for disobeying unlawful orders. So it's not like the high-ups
would be severely hurt if they just walked away or even tried to mitigate
the damage.
Yet, the vague feeling of arbitrariness about Nuremberg trials persists.
Why blame these guys and not the others? There were literally hundreds of
thousands involved in implementing the final solution. The feeling gets even
worse when contemplating German denazification trials in 1960's:
Today they bring even 96-year-olds or even centenarians to justice just
because they worked somewhere in an office or sat in a watchtower. But if it
had been done like that in 1965, more than 300,000 German men and women
would have had to be imprisoned for life for aiding and abetting murder.
[...] It had to be kept under control and suppressed, because otherwise it
would have been impossible to start anew. [...] There was a secretary at the
Wannsee Conference and no one considered putting her in jail. And yet all
these terrible orders were not typed by those who issued them. They were
typed by women.
— Interview with historian Götz Aly (in German)
***
At the first glance, this might seem like a problem unique to large
organizations. But it doesn't take a massive bureaucracy to end up in the
same kind of dysfunction. You can get there with just two people in an
informal setting.
Say a couple perpetually quarrels about who's going to do the dishes. To
prevent further squabbles they decide to split the chores on weekly,
alternating basis.
Everything works well, until one of the spouses falls ill. The dishes pile
up in the kitchen sink, but the other spouse does not feel responsible for
the mess. It’s not their turn. And yes, there's nobody to blame.
***
This is what Dan Davies, in his book The Unaccountability Machine: Why Big
Systems Make Terrible Decisions — and How the World Lost Its Mind, calls
"accountability sinks".
Something that used to be a matter of human judgement gets replaced by a
formal process. Suddenly, nobody makes any deliberate decisions. Instead, a
formal process is followed. In fact, the process may be executed on a
computer, with no human involvement at all. There's nobody to blame for
anything. There’s nobody to complain to when the process goes awry.
Everybody’s ass is safely covered.
In any organization, incentives to replace human judgement by process are
strong. Decisions may be controversial. More often than not someone's
interests are at play and any decision is going to cause at least some
bitterness, resentment and pushback. Introducing a non-personal process
allows the decision to be made in automated manner, without anyone being
responsible, without anyone having to feel guilty for hurting other
people.
The one most important contribution of Davies' book is simply giving this
phenomenon, that we all know and lovehate, a name.
And a name, at that, which highlights the crux of the matter, the crux that
all too often goes unsaid: Formal processes may be great for improving
efficiency. The decision is made once and then applied over and over again,
getting the economies of scale.
Formal processes may improve safety, as when a pilot goes over a check list
before the take-off. Are flight controls free and correct? Are all the doors
and windows locked? And so further and so on.
Processes also serve as repositories of institutional memory. They carry
lessons from countless past cases, often unknown to the person following the
process. Problems once encountered, solutions once devised, all of that is
encoded into a set of steps.
But in the end, deep in the heart of any bureaucracy, the process is about
responsibility and the ways to avoid it. It's not an efficiency measure,
it’s an accountability management technique.
***
Once you grasp the idea, it’s hard to unsee it. There's a strange mix of
enlightenment and dread. Of course it's about accountability sinks!
All of the popular discontent in the West today is fueled by exactly this:
A growing rage at being trapped in systems that treat people not as humans,
but as cogs in a machine. Processes, not people, make the decisions. And if
the process fails you, there's no one to turn to, no one to explain and no
one to take responsibility.
This is why even the well-off feel anxious and restless. We may have
democracy by name, but if the systems we interact with, be it the state or
private companies, surrender accountability to the desiccated, inhuman
processes and give us no recourse, then the democracy is just a hollow
concept with no inner meaning.
You can't steer your own life anymore. The pursuit of happiness is dead.
Even your past achievements can be taken away from you by some faceless
process. And when that happens, there’s no recourse. The future, in this
light, begins to feel less hopeful and more ominous.
It’s eerie how much of today’s political unrest begins to make sense
through this lens.
The backlash against experts? Understandable. After all, they’re seen as
the architects of these inhuman systems. The skepticism toward judges? It
fits. They often seem more devoted to procedure than to justice. Even the
growing tolerance for corruption starts to look different. Yes, it’s bad,
but at least it’s human. A decision made by someone with a name and a face.
Someone you might persuade, or pressure, or hold to account. A real person,
not an algorithm.
And every time you’re stuck on the phone, trapped in an automated loop,
listening to The Entertainer for the hundredth time or navigating endless
menus while desperately trying to reach a human being who could actually
help, the sense of doom grows a bit stronger.
***
But let's not get carried away.
First, as already mentioned, formal processes are, more often than not,
good and useful. They increase efficiency and safety. They act as a store of
tacit organizational knowledge. Getting rid of them would make the society
collapse.
Second, limiting the accountability if often exactly the thing you
want.
Take the institution of academic tenure. By making a scientist essentially
unfireable, it grants them the freedom to pursue any line of research, no
matter how risky or unconventional. They don’t need to justify their work to
college administrators, deliver tangible results on a schedule, or apologize
for failure.
The same pattern emerges when looking at successful research institutions
such as Xerox PARC, Bell Labs, or DARPA. Time and again, you find a crucial
figure in the background: A manager who deliberately shielded researchers
from demands for immediate utility, from bureaucratic oversight, and from
the constant need to justify their work to higher-ups.
Yet again, venture capital model of funding new companies relies on relaxed
accountability for the startup founders. The founders are expected to try to
succeed, but nobody holds them accountable if they do not. The risk is
already priced in by the VC. Similarly to the role of the Bell Labs manager,
the VC firm acts like a accountability sink between the owners of the
capital and startup founders.
***
On October 1st, 2017, a hospital emergency department in Las Vegas faced a
crisis: A mass shooting at a concert sent hundreds of people with gunshot
wounds flooding into the ER at once. The staff managed to handle the
emergency in a great way, violating all the established rules and processes
along the way:
At that point, one of the nurses came running out into the ambulance bay
and just yelled, “Menes! You need to get inside! They’re getting behind!” I
turned to Deb Bowerman, the RN who had been with me triaging and said, “You
saw what I’ve been doing. Put these people in the right places.” She said,
“I got it.” And so I turned triage over to a nurse. The textbook says that
triage should be run by the most experienced doctor, but at that point what
else could we do?
Up until then, the nurses would go over to the Pyxis, put their finger on
the scanner, and we would wait. Right then, I realized a flow issue. I
needed these medications now. I turned to our ED pharmacist and asked for
every vial of etomidate and succinylcholine in the hospital. I told one of
the trauma nurses that we need every unit of 0 negative up here now. The
blood bank gave us every unit they had. In order to increase the flow
through the resuscitation process, nurses had Etomidate, Succinylcholine,
and units of 0 negative in their pockets or nearby.
Around that time the respiratory therapist, said, “Menes, we don’t have any
more ventilators.” I said, “It’s fine,” and requested some Y tubing. Dr.
Greg Neyman, a resident a year ahead of me in residency, had done a study on
the use of ventilators in a mass casualty situation. What he came up with
was that if you have two people who are roughly the same size and tidal
volume, you can just double the tidal volume and stick them on Y tubing on
one ventilator.
— How One Las Vegas ED Saved Hundreds of Lives After the Worst Mass
Shooting in U.S. History
As one of the commenters noted: "Amazing! The guy broke every possible
rule. If he wasn't a fucking hero, he would be fired on the spot."
***
Once, I used to work as an SRE for Gmail. SREs are people responsible for
the site being up and running. If there's a problem, you get alerted and
it's up to you to fix it, whatever it takes.
What surprised me the most when I joined the team was the lack of enforced
processes. The SREs were accountable to the higher ups for the service being
up. But other than that they are not expected to follow any prescribed
process while dealing with the outages.
Yes, we used a lot of processes. But these were processes we chose for
ourselves, more like guidelines or recommendations than hard-and-fast rules.
And in an aftermath of an outage, adjusting the process became as much a
part of the response as fixing the software itself.
There is even an explicit rule limiting the accountability of individual
SREs. The postmortem, i.e. the report about an outage, should be written in
a specific way:
Blameless postmortems are a tenet of SRE culture. For a postmortem to be
truly blameless, it must focus on identifying the contributing causes of the
incident without indicting any individual or team for bad or inappropriate
behavior. A blamelessly written postmortem assumes that everyone involved in
an incident had good intentions and did the right thing with the information
they had. If a culture of finger pointing and shaming individuals or teams
for doing the "wrong" thing prevails, people will not bring issues to light
for fear of punishment.
— Site Reliability Engineering, How Google runs production systems
***
On February 1, 1991, a jetliner crashed into a smaller, commuter propeller
plane on the runway at Los Angeles International Airport. The impact killed
twelve people instantly. The small plane, which was pushing the larger one
ahead at a speed of one hundred and fifty kilometers, exploded in the fuel
tank and caught fire. Passengers tried to get out of the burning plane, but
not everyone succeeded. The death toll eventually rose to thirty-five.
Here's a great article from Asterisk about the event:
At the LAX control tower, local controller Robin Lee Wascher was taken off
duty — as is standard practice after a crash. After hearing about the
propeller, she knew she must have cleared USAir flight 1493 to land on an
occupied runway. As tower supervisors searched for any sign of a missing
commuter flight, Wascher left the room. Replaying the events in her mind,
she realized that the missing plane was SkyWest flight 5569, a 19-seat
Fairchild Metroliner twin turboprop bound for Palmdale. Several minutes
before clearing the USAir jet to land, she had told flight 5569 to “taxi
into position and hold” on runway 24L. But she could not recall having
cleared it for takeoff. The plane was probably still sitting “in position”
on the runway waiting for her instructions when the USAir 737 plowed into it
from behind. It was a devastating realization, but an important one, so in
an act of great bravery, she returned to the tower, pointed to flight 5569,
and told her supervisor, “This is what I believe USAir hit.”
[...]
The fact that Wascher made a mistake was self-evident, as was the fact that
that mistake led, more or less directly, to the deaths of 35 people. The
media and the public began to question the fate of Ms. Wascher. Should she
be punished? Should she lose her job? Did she commit an offense?
[...]
Cutting straight to the case, Wascher was not punished in any way. At
first, after being escorted, inconsolable, from the tower premises, her
colleagues took her to a hotel and stood guard outside her room to keep the
media at bay. Months later, Wascher testified before the NTSB hearings,
providing a faithful and earnest recounting of the events as she recalled
them. She was even given the opportunity to return to the control tower, but
she declined. No one was ever charged with a crime.
[...]
If you listen to the tower tapes, you can easily identify the moment
Wascher cleared two planes to use the same runway. But if you remove her
from the equation, you haven’t made anything safer. That’s because there was
nothing special about Wascher — she was simply an average controller with an
average record, who came into work that day thinking she would safely
control planes for a few hours and then go home. That’s why in interviews
with national media her colleagues hammered home a fundamental truth: that
what happened to her could have happened to any of them. And if that was the
case, then the true cause of the disaster lay somewhere higher, with the way
air traffic control was handled at LAX on a systemic level.
When you read the report of the investigation, your perspective suddenly
changes. Suddenly, there is no evil controller Wascher who needs to be
publicly punished. Instead, there is a team of controllers who, despite all
the broken radars, poor visibility and distracting duties, heroically ensure
that planes do not collide day in, day out.
And if the result of Washer giving a honest report about the incident is
that a second ground radar is purchased, that the interfering lights are
relocated, or that various less important auxiliary tasks are not performed
by the flight controller in charge, the effect on air traffic safety is a
much bigger than what could be achieved by firing Wascher. Quite the
opposite: Punishing her would have a chilling effect on other experienced
controllers. At least some of them would be unwilling to take responsibility
for things beyond their control and would eventually leave just to be
replaced by less experienced ones.
***
Some of my former colleagues at Google were part of the effort to save
Obamacare after the website that people were supposed to use to subscribe to
the program turned out not be working. Here's Wikipedia:
The HealthCare.gov website was launched on the scheduled date of October 1,
2013. Although the government shutdown began on the same day, HealthCare.gov
was one of the federal government websites that remained open through the
events. Although it appeared to be up and running normally, visitors quickly
encountered numerous types of technical problems, and, by some estimates,
only 1% of interested people were able to enroll to the site in the first
week of its operations.
Suddenly, a website was in the center of attention of both the media and
the administration, even the president himself. As Jennifer Pahlka writes in
her excellent book Recoding America: Why Government Is Failing in the
Digital Age and How We Can Do Better:
If the site failed, Obama’s signature policy would likely go down with it.
With this threat looming, suddenly the most important policy of the
administration would live or die by its implementation. It was the first
time the highest priority of the White House was the performance of a
website.
Having the support from the highest places, the people in what was to
become US Digital Service, outlined a plan:
He had a straightforward two-step plan. Step one was to recruit a small
team of technologists with relevant experiences and skills [to fix the
website]. Through a trusted network, he reached out to a set of remarkable
individuals who signed up to jump into the fire with him. Step two was to
win the trust of CMS—an agency [responsible for implementation of the
website] that, like all other agencies, really, was highly skeptical of
people from outside and resistant to their interference. The situation was
tense, but Todd made it clear that he and everyone who came with him were
not there to assign blame. They were there to help. The result was the
opposite of the usual response to failure. Instead of the hardening that
tends to come with increased oversight—of the kind Weaver would later
experience working on satellite software, further limiting what he could
do—the CMS team suddenly found themselves with something they hadn’t
realized they needed: a group of smart nerds they could trust. Like most
other agencies, they knew how to acquire technology and technology services;
after all, they’d issued those sixty separate contracts for healthcare.gov
alone. But now they had people on their team who could look at the code, not
the contract terms. That, it turned out, made all the difference.
— ibid.
Again, sidestepping accountability had a beneficial effect. It managed to
cut through the seemingly unsolvable internal hurdles and unstuck a stuck
system.
***
Dominic Cummings, chief advisor to Boris Johnson during the COVID crisis,
recounts:
At the peak of COVID craziness in March 2020, on the day itself that the PM
tested positive for CoVID, a bunch of people come into Number 10 sit around
the table and we have a meeting and it’s about supplies of PPE to the
NHS.
They say, “None of this PPE that we’ve ordered is going to be here until
the summer.”
“But the peak demand is over the next three to four weeks.”
“Sorry, Dominic, but it’s not going to be here.”
“Why not?”
“Well, because that’s how long it takes to ship from China.”
“Why are you shipping from China?”
“Well, because that’s what we always do. We ship it from China.”
But A, we need it now and B, all of the airlines are grounded. No one’s
flying anything.
“So call up the airlines, tell them that we’re taking their planes, we’re
flying all the planes to China, we’re picking up all our shit, we’re
bringing it back here. Do that now. Do that today. Send the planes
today.”
We did that. But only the Prime Minister could actually cut through all the
bureaucracy and say, Ignore these EU rules on Blah. Ignore treasury guidance
on Blah. Ignore this. Ignore that. “I am personally saying do this and I
will accept full legal responsibility for everything.”
By taking over responsibility, Johnson loosened the accountability of the
civil servants and allowed them to actually solve the problem instead of
being stuck following the rigid formal process.
***
Finally, consider the mother of all accountability sinks: The free
market.
We want government to enforce the rule of law, to enforce the contracts and
generally make sure that the market operates freely. But we also explicitly
don’t want government interference in the day-to-day workings of the market
beyond ensuring compliance with the law and taxes.
Much has been written about how markets act as information-processing
machines, how they gather dispersed data from across society and use it to
optimize the allocation of scarce resources.
Much less is known about how the lack of accountability gives entrepreneurs
the ability to take huge risks. If your company fails, the blame is yours
and yours only. No one will come after you. There’s no need to play it
safe.
While ignoring the law of supply and demand may have been the primary cause
for the failure of communist economies, the fact that the management of
every company was accountable to the higher-ups and eventually to the
communist party must have meant that they tried to avoid any risks at any
cost, which eventually led to terrible performance in implementation of new
technologies — even if they were discovered by the scientists — and business
practices and to the overall economic stagnation.
***
By now, you probably have a few ideas of your own about how accountability
sinks can be creatively incorporated into institutional design.
But before I wrap up, let me make a few basic observations:
- Formal processes are mostly beneficial and they’re not going anywhere. Any
complex modern society would collapse without them.
- Not every formal process is an accountability sink. A process you design
and impose upon yourself doesn’t absolve you of responsibility when things
go wrong. You remain accountable. On the other hand, a process imposed upon
you from above often incentivizes blind adherence, even when it’s hurting
the stated goals.
- Not every accountability sink leads to rigidity and cover-ups. A process
can be designed in such a way as to shield those affected from the
accountability, but at the same time not to impose any pre-canned solutions
upon them. (E.g. blameless postmortems.)
