Solving Problems & Saving Time through Software and Crushing Entropy
Powered by Cecil & Hyde.
I am sure the majority of managers in my industry would recognize Goodhart's law:
When a measure becomes a target, it ceases to be a good measure
What surprises me is how often this maxim gets forgotten! Whenever there are discussions about quantifying "How good are our estimates?", "Is our productivity increasing or decreasing?", or "Are our teams happy and healthy?" folks routinely believe that these questions can be targeted for improvement by (more or less) direct measurement.
Management routinely accepts that with humans "by observing something you change its behavior." They understand that the measurement will have some margin for error and believe "it is close enough." And they are probably correct. It is likely that it is close enough.
Once an engineer knows you're monitoring their productivity, and then you set productivity targets their ears start burning. Now they know that their next promotion relies on productivity improvement. They know their favorability within the org relies on productivity remaining the same. And, they know that if their productivity ever goes down they might get fired. And somehow management believes they will still have an honest measurement of productivity?
Managers have to realize that they cannot have their cake and eat it too. I believe this is a constraint when creating a well-functioning organization. If you're interested in quantifying these kinds of things, you can not also talk about "making estimates better" or "increasing productivity" . I truly believe once you do that the game is over, and slowly but surely you will lose the trust of your team.
If that isn't controversial enough for you try this one on for size. Common wisdom for increasing the productivity of your startup is to hire more people—rapid growth! I believe that is incorrect, it is only part of the story, at best.
Hiring people increases your potential capacity—not "real" capacity. Real capacity is how many things you can do at once. Potential capacity is about how many things you can do at once in the future. I think Will Larson illustrates the problems with rapid growth where you are always hiring more people:
Just how challenging this is depends on how quickly you can ramp engineers up to self-sufficient productivity, but if you're doubling every six months and it takes six to twelve months to ramp up, than you can quickly find a scenario in which untrained engineers increasingly outnumber trained engineers, and each trained engineer is devoting much of their time to training a couple of newer engineers.
In any scenario close to the one that Larson describes there is zero question that you've hired more people, and more people are able to do more things. But have you increased productivity, or have you decreased productivity? If you haven't put a considerable amount of effort into making onboarding effective, which is incredibly difficult, the chances are you will experience decreased productivity.
Odds are the executives told you: "Bigger, Better, Faster" and your response was: "Give me cash and I will hire people!" As a manager how are you going to manage this constraint? Eventually, people get trained, but what are you going to do until then, and how will you manage expectations with executive leadership?
Read more: Two Beliefs About Organizational Constraints
If you've read a handful of my posts I don't think you would be surprised if I told you I am an academically-minded person. There are a fair number of causes for this (which I am learning about more and more) and there are plenty of reasons why that would be a surprising development.
As a kid, I was "good at math". The truth is more that it came very easily. However, it never engaged me. It was easy and boring. I still have not done a lot of math in my day, though I wish I did more. I wish they taught math differently because when I see the kind of work that Grant Sanderson does at 3 Blue 1 Brown I am positively riveted. Even though I am entirely incapable of actually doing that math—it would take a lot of work now to even attempt getting there.
The link between high-level math that looks so interesting, my "academic-ness" nature, and my career in software revolves around the question of "How do we know what we know?" In my mind this boils down to two famous sayings: "All models are wrong, but some are useful" and "Don't confuse the map for the territory."
Grant Sanderson wields his understanding of high-level maths by fitting a solution to a real-world problem. He answers "How do we know how to solve this problem?" by reframing the problem in terms of a system of math that can generate an accurate and useful answer. For those who don't enjoy math, but have seen "Hidden Figures", this is precisely what Dorothy Vaughn did to solve their trajectory problems by picking a different mathematical system that would give them an answer they can use. In the view of people without this kind of mathematical mastery, it looks like they are wizards, or they are merely cheating—making up answers as they go along.
We know that computers only do what they are told. Decades ago you had the ability to see and understand everything the computer was doing. Not today, there are too many layers and too many authors of code in every machine running today. Computers are still doing exactly what they are told. There are just too many authors yelling at the computer.
This is the root of bugs. The computer is doing what it is told. The problem is us. We got an unexpected response. It is our understanding that is flawed. The "territory" is what it is. Our "map" is wrong.
We are wrong constantly. This means that we should always be learning. Many people are afraid of being wrong. I've never understood that perspective. Learning oftentimes means you were wrong. After all, it's rare that we go out of our way to truly learn something from zero knowledge. More often we are adding to the knowledge, and more importantly, changing the knowledge we have. Updating our maps, not treating them as dogmatic truth to rely on.
Read more: Epistemological Models
About the time I was heading off to college I was introduced to an interesting phase. I am not entirely sure where I heard it, but I believe it was from my Aunt. Throughout her career she worked at IBM, Dun & Bradstreet, and Hewlett Packard. The saying went something like this: "If you need something to get done, give it to someone who is very busy."
As a kid who had never had a white-collar job before I didn't understand it. It still flies in the face of conventional thinking; surely you should give a task that needs to get done to people who have free time to do it. Only as my career has advanced am I beginning to understand the inherent wisdom in the phrase.
The first layer of understanding this wisdom is Parkinson's Law: "work expands to fill the time allotted." In the context of delegating an important task that needs to be completed; if you give the task to someone with enough free time the task will take longer. Delegate accordingly.
The second layer of understanding this wisdom, you'll discover, is that some people aren't interested or motivated. It is incredibly rare to see a skilled and ambitious person twiddling their thumbs waiting for something to do between 9 and 5. That said, staring at the ceiling trying to figure out the right decision to make counts as working for many of us. If something is urgent, out of the ordinary, or outside someone's comfort zone (and they haven't agreed to a stretch goal) you may reconsider delegating to this person.
Make no mistake, this is not about "a lack of trust". You trust the person to do their job, if you don't they shouldn't work there. But oftentimes things that need to get done lie on blurred lines.
Read more: If You Want To Get Something Done
There are some engineers that excel at delivering small pieces of high quality work. Like a metronome they consistently deliver. There are other engineers that pull large and complicated projects across the finish line, without the sense of a death march or moving the goal posts. You need both these kinds of engineers in order to stay afloat.
Even better yet there are engineers who can do both of these things at the same time. In addition to shipping the big project there are some smaller pieces in there as well.
If you don't have an organization that can ship the small things, you'll die by suffocation. The communication and administration cost will be so high that you'll never be able to keep track of everything, no matter how hard you try. Why? Because there is a never-ending list of small things that will always grow. If you never ship against this list it will never get smaller.
If you don't have an organization that can ship the big things, you'll fall behind. New competitors will outflank you. You'll be left behind by technological changes and engineering techniques. Slowly your organization will lose its standing in the market.
Your folks will naturally gravitate towards one of these poles. Try and cultivate people at both ends. You'll need them.
Read more: Both Big & Small
There is a short post that I cherish called: Complexity Has To Live Somewhere. Sometimes engineers need to be reminded that if everything was simple, clean, and easy, we wouldn't be getting paid. It is our job to create the interfaces, the abstractions, and the patterns that place the complexity where it belongs.
The truly insidious part about complexity is accidental complexity. To use a metaphor it is: "the cost of doing business." Accidental complexity is all the other crap that you have to do before you can do what you need to do. There are lots of reasons for accidental complexity; leaky abstractions, tech debt, and technical decisions made before your time.
I've written before how writing software is more of an artisan process than a construction process. And it is because this is still a young immature industry, despite how large and monied it is. We're still learning how to do this. Humans have been constructing buildings for thousands of years, we've figured out a few things and standardized on nearly everything. That said, there are still leaky abstractions and technical debt in construction.
I live in a house constructed in 1905. This has taught me a lot. Nothing is square, level, or plumb. That said, if you've ever walked into new construction—nothing is square level or plumb there either. Over that time there have been decisions and bad work. As an unskilled construction worker I can get by fixing things in this house. But the true expert, the master of their craft in construction and carpentry, can walk in here and produce work that results in perfection despite what is underneath.
We aren't close in the tech to allowing unskilled workers to make real progress. Hell, we're still trying to help ourselves.
Most of the startups that cater to other developers are working to eliminate accidental complexity for others by taking it on themselves and asking folks to pay for it. Many open source projects, I'm looking at you Kubernetes, are designed explicitly for removing this accidental complexity. To band together and try to make managing all the servers in a data center look like managing a single server. You can only reason about things that can fit inside your head. K8s is an operational metaphor that tries to get everything into your head so you can reason about it.
I look at Lambdas, and messaging services (whether it's AMPQ or Kafka), and I see the attempt to make a distributed system made up dozens of CPUs and hundreds of gigs of memory operate as if it was on one system. By making servers "not matter" you can think about your running application as a logic puzzle that runs on one system.
This is not limited to system design. We do this when we architect our code as well. We create interfaces and patterns that put complexity in the one spot where it is necessary to deal with. And we attempt to wall off each different complexity from one another.
This is the trend we've been on in cloud computing—eliminating anything and everything we can that makes applications look distributed. We don't always succeed. Just like in construction nothing is level, square, or plumb. When the abstractions leak we need the experts, the master artisans.
We are trying desperately to do more with less. Fewer engineers, fewer meetings, because those are the most expensive pieces of this puzzle, the network effect born by communicating. I'm not sure we're succeeding yet, but we're trying like hell.
Read more: Layers, Complexity, and Trends