Admittedly, valuing things that break is very hard. Engineers hate things that break. And if a thing is broken, we consider it has no value at all. Computers and software are tools, and a broken tool is useless. Organizations are made up of people. And (lets be honest) most of them are broken. It takes a lot of effort to keep it un-broken. And there are many people whose job it is to keep it that way. And its a very hard job.
This post was inspired by a Rands article titled “Everything Breaks”.
It is up to you to make failure a competitive advantage. Failure is an opportunity to learn…When things break, you must learn completely from the failure. — Rands
It is important to value things that break, software, architecture, teams, organization, and culture, because that is where you learn the most information about what you’re doing. Not only do you learn how to fix things by paying close attention to how things break adopting this mindset will prevent a number of other problems.
If you’re making decisions based on looking closely at why something, anything, broke, you’re not going to make decisions too quickly based on emotion, politics, or hubris. When something actually breaks everyone is paying attention. There is nothing to hide, no whispers to silently work around things. Everyone is watching. If someone spoke up about thing X breaking before it broke, you better acknowledge it. Because they are watching too.
Again, this is hard. Making improvements by staring brokenness in the face is not for the faint of heart. It is a sign of maturity. Because Everything Breaks. And if you don’t believe that you’re not ready for it.
The things that break are not the mistake. The mistakes happen after they break. The mistakes happen in your response to the brokenness. Again, everything breaks — you cannot stop that. You can only control your response, you can choose to learn from what breaks, or not.
Does your competition believe that everything breaks? I doubt it. The baseline is that you’re both going to make the same mistakes at least twice. Maybe more. How ahead of the game would you be if you only ever made them twice. And then in two years, only ever made the same mistakes once? How much time, and therefore money, would you save?
Read more: Valuing Things That Break
Our daily interactions have an inherent limit. A network of people that we can manage can only be so large. There are 24 hours in the day, and we need to get things done so we can’t talk to everyone we’d like to, let alone reach outside of our bubble to find something new.
There was a time in my career where I never left my small bubble. I was effective. But I didn’t grow very much. I didn’t learn very much. Only when I began to value outside perspectives did things change.
Not only did I begin learning and growing as an engineer I also gained confidence. Somehow, while learning there is a ton I don’t know, I gained confidence — it still doesn’t make sense to me, but thats how it happened.
There are lots of places I looked to for outside input from people I didn’t work with, or didn’t go to school with:
As individual engineers there is a lot of wisdom we can find in the wider world. We can find good advice for technical problems and solutions, as well as personal and career questions we have. I think getting outside our immediate problems broadens our horizons and can help us avoid burnout. Strangely enough, it has offered me a glimpse of hope at times in my career that, “Yes, things can be different.”
As a small team or small organization, looking outside is something I’ve rarely experienced. I’m not sure why, but I have experienced a real aversion on teams I belong to in seeing what other outside groups are doing — unless its your direct competition, then they care immensely and its always political. I think a team that can value outside perspective, rather than dismiss it, is a mature team.
We can’t solve problems by using the same kind of thinking we used when we created them — Albert Einstein
Valuing outside perspective is going to show you things you don’t know yet. And I guarantee you’re going to find answers to problems you have you never would have thought of. A good leadership team is actively doing this for a mature business, and they’re doing it at a high level. But it rarely travels down the ladder to the rest of the organization, at the level where the rubber meets the road.
Read more: Valuing Outside Perspective
When describing what Software Engineers are like to “non-computer-y folks” I like to use the word chef. There is obviously a science to cooking, but we recognize that it is artistic as well. The job of software shares both those aspects.
I don’t want to describe Engineers as just “programmers” or “creators”. Because what we create does not solely come out of our own heads. We are always embedded in an organization that serves something other than the software. The software is never an end in itself, to shameless steal a line: “The computer is the thing that gets you to the thing.”
The artistic part of this job is in gaining understanding on what you’re actually being asked to do, and marrying it with the technology at your disposal. I seems to me that many Engineers only care about the technology end. They do not value a holistic understanding.
Engineers need to value understanding the actual requirements. More often than not you find out that there were a number of real requirements you never knew about. No one knew about them. Because the data presented to you was never properly interrogated. That is an art. Finding the real requirements underneath the supposed requirements that presented to you.
Engineers need to value understanding the strategy of their organization. Sometimes people never think to tell the engineers about The Strategy(TM). Without knowing the strategy you want to execute on you will absolutely make the wrong technology decisions. Your success will be random, rather than the result of planning and execution. Aligning your technological choices with the strategy is an art. We are seeing so many new tech stacks coming out of Facebook, Google, LinkedIn, etc. because their strategies necessitated creating new technology. Necessity is the mother of invention. Technological invention is all about fighting against constraints. Without knowing your strategy you won’t know what constraints you can live with — and which you cannot.
Engineers need to value communication, to value being understood. I know when I was younger I wondered “What they hell do we need with managers? And that entire marketing department?” Yea, you need it. We need to value communication, both internally and externally. Engineers need to write well. They need to write documentation. When they move up the ladder, they’ll need to write proposals. They’ll need to email the CTO. Communicating well is an art. If no one can understand you, you won’t grow, you won’t be effective, and you will be frustrated. For a very long time.
We need to value deeper understanding, because we are not just programmers punching cards and feeding them into a machine. I would bet money that engineers who are unsatisfied at their job are not valuing one or more of these understandings.
Read more: Valuing Understanding
I have a hard time with meetings. It would not surprise me if most people have a problem with meetings. I have a “Broken Windows” theory about meetings. Once one person, or a few people, demonstrate that they do not value the meeting we are all in together; more people follow suit and stop valuing the meeting as well. (Note: I find the “Broken Windows” social theory about communities to be nonsense.)
Engineering teams have a unique problem regarding meetings. Engineers are makers, we are chefs, we like to create. This manifests in two ways; either they want to run the meeting their way, or they don’t value meetings because it is not creation. In each organization you have to find a way to have valuable meetings. The only thing worse than having bad meetings is not having any meetings — and all the repercussions and bad work that result.
In my opinion there are several ways to demonstrate that you value a meeting. These are not the only ways to demonstrate it, but these are my ways.
Bad meetings are like a cancer. Bad participants are like a cancer too. Eventually they ruin good meetings by poisoning people to stop valuing meetings. I have to try very hard in these situations to be productive, and it doesn’t always work.
You’ll notice I said nothing about laptops here. I think they are often a red-herring; a symptom not a cause. The cause is that people are not valuing meetings, so they’ll start fiddling on their laptop. By removing the laptop you haven’t solved the problem. They still don’t care about the meeting. Laptops are just a tool, sometimes you need one. I think you’ll notice that people who value meetings aren’t fiddling on their laptops.
Do you share these values about meetings? Do you have additional things you would add? Would you remove anything?
Read more: Valuing Meetings
Observability is an attribute of a system. A system either has it, or does not have it. Most systems don’t have it. Even if they have some amount of reporting and system metrics — it doesn’t mean your system is observable.
A system is observable when you can ask questions, new questions, about how its performing (both technically, and the business goals) without having to make any changes or deploys. As an engineer who has built a lot of systems, I do not actually think I’ve built one that is observable. It is a high bar — but worth achieving.
This is a very technical issue, but it is actually a solution to a business problem. I am very encouraged to see clear communication from engineers about it. You can follow the #o11y hashtag on twitter. The hashtag and conversation came out of the DevOps community, which has been focused on merging infrastructure with their larger product organizations, rather than a line-item on a budget somewhere.
To make a long story short to create an observable system you have to merge the monitoring of system health and business metrics. Many times, business metrics can be very vague and fuzzy. But if you’re an e-commerce site you want to know how much a 5% slowdown in response time affects order conversion — in order to know how much to spend on infrastructure & engineering to avoid the 5% slowdown! Otherwise you’re just guessing. Hitting a quarterly goal is actually very vague and has a large number of dependencies.
It wouldn’t be hard to write a single report that correlates % slowdown with order conversions. But then, if you wanted to know “What is our % of repeat customers when we shipped them something late?” do you have to write another report? If you do, your system isn’t yet observable. There is nothing wrong with that. And you have to decide how far you want to go here.
The larger and more complex your system is — the larger the engineering challenge to get all that data. But if you’re losing this information (because you’re not capturing it) you don’t know what you don’t know. That is scary. That is how someone comes along and eats your lunch.
Read more: Observability: a goal