M&Ms: Appeal to Interest

With people forget about reason on the 103rd edition.

Hi Team,

I want to tell you a story that has two endings: one ending is true, and the other is fake.

I have a software engineering friend that once made a big mistake. He deployed some code that was quickly exploited by the internet. It cost the company tens of thousands of dollars in a few hours.

But this bug was particularly bad because it involved two different teams. 

So the engineer could not fix this bug himself if he wanted to. So instead of fessing up that he made a mistake and raising the alarm with the other team to get involved and fix it right away, he figured when the other team found it, he could blame it on them. Blame it on a communication problem and call it a day. As the old saying goes, when it comes to bugs, "founders keepers." 

So he closed his laptop and went home from work that Friday, and over that weekend, the mistake cost the company millions of dollars.

But before you assign all blame to my friend, you also need to know that his company had a particularly bad culture around reporting bugs and problems. They did not want to hear one bit that someone made a huge, costly mistake. No such thing as a "blameless culture" there.

This company believed in a policy called "Name Them, and Shame Them."

They believed that blaming people for the problems they introduced ought to teach them not to introduce more problems. And the managerial class at the company was rewarded and even promoted based on low defects in their teams.

So because my friend stayed quiet, by the time he came into work on Monday morning, the problem had cost the firm millions of dollars in damages. It had made the news, which caused even more people to exploit it.

The CTO waited for him at the front doors and said, "What have you done? We are ruined." The engineer blamed it on the other team, as planned, but it didn't matter; soon, no one would have a job at the firm. Not him, not the other team, no one. As the CTO exclaimed, “They were ruined.”

They would soon all be on the news for their failure as a company. Folks at other firms wouldn't even want to bring them in for an interview. 

But can you blame those other companies? Think about it, who wants the people that were around for the Lehman Collapse to work at their Bank? 

Silicone Valley Bank apparently was good with it.

Of course, I am being facetious here because SVB and Leham Brothers collapsed for entirely different reasons. I am still trying to prove a point about how that all gets spun up.

But as I said in the beginning, this story has two endings. This was the fake ending.

My friend’s story didn't end that way; other stories have. Knighted Capital was in the news in 2012 for losing $460 million:

"On August 1st, 2012, Knight Capital deployed a new software update to their production servers. At around 08:01 AM, staff in the firm received 97 email notifications stating that Power Peg, a defunct internal system that was last used in 2003, was misconfigured.

This was the first warning sign.

At 09:00 AM, the New York Stock Exchange opened for trading, and Knight Capital's first retail investor of the day placed an instruction to buy or sell their investment holdings.

Just 45 minutes later, Knight Capital's servers had executed 4 million trades, losing the company $460 million and placing it on the verge of bankruptcy. Some shares on the NYSE shot up by over 300% as High-Frequency Trading algorithms from other firms exploited the bug.

Ultimately, Knight Capital was fined an additional $12 million by the Securities Exchange Commission due to various violations of financial risk management regulations."

But back to my friend.

He actually worked at a company with a blameless culture regarding software engineering bugs. At this firm, it was in everyone's best interests to alert everybody else and fix a bug when it was discovered. Do a Root Cause Analysis (known as an RCA in the Software Engineering world) of the problem, discuss it without ever blaming anyone or penalizing them, learn from it, then move on.

And my friend had a great manager who understood that they have to move fast and break things, but they also move fast to fix things. And they won't blame each other when they break things. They fix it, learn, get better, and move on. Companies that operate like this make their systems antifragile; their systems benefit from mistakes because the same mistakes never happen twice. And instead of being penalized, people that harden systems this way are rewarded, even promoted.

So, in the real ending, as soon as he noticed the bug, my friend fessed up to the communication issue. 

He immediately told his Boss what had happened. He engaged the other team to help too, and they came up with a fast fix. He even offered to work that weekend to ensure the damage could be reverted; this was still possible since it was caught so fast.

His Boss offered to help, and together they offered that option to their business partners.

But the business partners looked at the sales and attention this brought the company on the internet. And they decided that although it was not ideal, the few thousand dollars that had been burned had now led to a huge spike in sales. There was no need to revert anything and risk losing the "goodwill."

But then also, that feature my friend was working on after it was fixed, along with a bunch of new attention, led to the highest growth in sales the company had seen so far. 

And sure enough, my friend was promptly promoted within a few months.

But if my friend handled this differently, this could've been much worse, and in many other places, it is.

There are a few lessons from this story:

• Be honest because the opposite compounds problems.

• Don’t deploy on Fridays.

• Be careful what you punish people for.

  • Create a blameless culture because blaming people will usually not yield the desired results.

• Be careful what you reward people for.

  • Even something as logical as rewarding people for fewer defects can have the exact opposite effect you intended.

And one of the final lessons, as Ben Franklin wisely said, “If you would persuade, appeal to interest and not to reason.”

My friend was told that in his company, bad things would happen if they weren't allowed to learn from their mistakes. And great things would happen when they did. He didn't need more reasons than that; it was in his best interest to always do the right thing.

---

While you are here, if you are interested in inspiration to make your first few dollars online, I highly recommend our other Newsletter, of which I am a co-author and regular contributor of:

Three Things: Productivity gains, American Geography, Children of Poor Immigrants

Here is a fun fact about productivity gains as expressed by Charlie Munger:

“Two hundred years ago, aided by the growth or technology and the growth of other developments in the civilization, the real output per capita of the civilized world started going up at about two percent per annum, compounded.

And before that, for the previous thousands of years, it had gone up at a rate that hovered just a hairs-breadth above zero.”

Two percent might not sound like much but compounded annually for a few hundred years; it will lead to an incredibly wealthy world.

And I think productivity might soon explode even higher.

---

You might have seen this thread by Michael, it made a huge splash, but as an immigrant to the U.S. and a huge believer in America, I feel compelled to share it with you.

It has a few surprising things in it, such as aspects of U.S. geography (besides the oceans) that have contributed to its dominance.

But besides the geography, as the Matthew Principal proves, winners keep on winning. They usually have more resources to bring to bear on problems.

And America has other massive advantages; the fact that it is so tolerant of immigrants is a massive benefit. Many nations have stagnated after they became wealthy.

But America gets a fresh injection of ambition and hunger with wave after wave of immigrants.

And I had a fun and respectful debate with my friends online on this back-and-fourth if you are interested in that.

---

The contributions that immigrants, and their children, have made and continue to make to America are tough to deny.

---

Sometimes You Have to Run

Write of Passage cohort 10 is running at the moment, and one of the students wrote one of the most deeply touching articles I have read this year.

It serves as a reminder to live now.

---

One Meme: As a large language model…

---

As always, thank you for reading.

-Louie

P.S. You can reply to this email; it will get to me, and I will read it even if I can’t always reply in a timely manner.