M&Ms: Make Noise

Sometimes its not enough to add value in the 88th edition.

Jan 09, 2023

Hey Team,

Years ago, our tech company had another major software outage.

The cause of the outage was something the core API team pushed out that Thursday before the long weekend. They had so many things to deliver and were so short-staffed that no one at the production review meeting was surprised about the outage. At this point, it was all so routine.

But that Monday, at the production review meeting, felt like the last straw.

I could see leadership was visibly upset, and things had to change.

My tech leads and I watched quietly from the corner as engineering leadership discussed what went wrong this time.

Then our firm spent the next month or two shoring up the core API team and our production environment.

Tech leadership began negotiations immediately with the business that feature development had to stop until we could get a handle on things. Our firm needed to hire more great API developers for the core API team and fix all the bugs before we could add new features. An amazing new process was agreed upon with the business and implemented because of this, "Error Budgets." If we in engineering violated our SLAs and broke customer promises, with bugs, by a certain agreed-upon amount, then all engineering teams went into paying down tech debt mode. On top of that, internally, this incident was a "code purple," the highest severity level, invented for this particular case. It was a "code purple" because the color purple was the core color of our firm.

During the "code purple," all attention went to the core API team and what they needed.

Unfortunately, this also meant my teams, and I wouldn't get any help. Even if we desperately needed it. My teams built and ran automated Ads, emails, and personalization, among other things, for our company. We ran a deeply important set of systems; we brought all the traffic to the site. But we tried extra hard to avoid bringing down our team's systems. We created a situation where everything looked like it was running smoothly and normally, even though we were burning ourselves like a candle to achieve it.

Put yourself in leadership shoes; why should my teams and I be prioritized if everything is smooth?

But, truthfully, my teams needed help. My leads were chewing my ear off; I had to do something. That's when it hit me; the squeaky wheel gets all the grease. If there are no outages caused by us, then who the hell will pay attention to us? Who in their right mind will prioritize us and give us what we need?

That's when I knew it was time to turn the other cheek, get some rest, and allow some engineering sins to occur. Basically, it was time to let some bugs slip through the cracks.

Then, sure enough, shortly after the code purple, my tech leads and I were up on the podium at the production review. We were at the production review, explaining why our team had an outage. "We are short-staffed, we are burnt out, we messed up," I exclaimed on the microphone as I stood next to my tech leads. Even though I had prepared for that podium ahead of time, I wasn't lying; we were short-staffed.

Now, my teams and I were the lost sheep.

"Suppose one of you has a hundred sheep and loses one of them.
Won't you leave the other ninety-nine in the open country and go after the lost sheep until you find it? And when you find it, you joyfully put it on your shoulders and go home.
Then you call your friends and neighbors and say, 'Rejoice with me; I have found my lost sheep.'
I tell you that in the same way, there will be more rejoicing in heaven over one sinner who repents than over ninety-nine righteous persons who do not need to repent."
-The Holy Bible, Luke 15

It may seem wrong that heaven would rejoice so much over the repenting sinner until you realize we are all sinners.

Shortly after that production outage, my teams and I became a huge priority to the firm. And our marketing business partners were entirely on board; after all, they wanted more features and fewer bugs. They wanted us to have more resources of all sorts.

Then everyone rejoiced as my teams hired people and cranked out feature after feature, and kept our production incidents really low.

The lesson here is that sometimes you've got to make noise; it's not good enough to be "perfect" and add value quietly in a corner. Especially if that means you are burning yourself out to keep up that perfection.

Sometimes, the right thing to do is rest. Let a damn sheep run.

That's the only way to have a party and rejoice when it's found.

Three Tweets: Definition of success, Survive, Internet Game

The co-founder of Netflix has an incredible definition of success:

“I resolved a long time ago to not be one of those entrepreneurs on their 7th startup and their 7th wife. In fact, the thing I’m most proud of in my life is not the companies I started, it’s the fact that I was able to start them while staying married to the same woman; having my kids grow up knowing me…”

I am striving for the exact same thing in my entrepreneurial journey.

I don’t always agree so strongly with David Sacks, but as I try hard not to return back to a big company (unless I started it), this advice from him for 2023 is rock solid.

Daniel Vassallo, the founder of small bets, had an incredible thread this week (this is just the first tweet of many) detailing some really important lessons about the internet game and the creator economy.

One Podcast: Risky Business

A few weeks ago, I was on a podcast where we discussed the tech industry, how it’s changing, and how it might evolve. It was released yesterday.

A few other things we covered:

– How to go from IC to engineering manager.

– How to transition from Big Corp to entrepreneurship.

– How to find great people to learn from.

– Tips for my younger self.

Two Memes: No cold showers, It depends

I have been reading a lot of Charlie Munger lately. Charlie and Buffet have been incredibly successful, basically bucking all of the rules we hear are required to be a successful entrepreneur.

And while this tweet is meant to be satire, it highlights something deeply important:

A lot of what we have been convinced success requires is a lie.