Failures are interesting. They tend to teach lasting personal lessons, but we are loath to share them. These lessons have profound impacts on our habits and viewpoints, but they’re difficult to recount. We need to get comfortable talking about failure.
Unfortunately, our brains are wired to screw this up. When confronted with a failure, we are conditioned to ignore it, make ourselves blameless, or create elaborate strategies to avoid the possibility of failure altogether. I’d like to present another alternative: embrace failure. It will happen. That’s ok.
How can we challenge our natural human bias? I’ve spent the last couple years digging into “pop” psychology and thinking about how books like “Thinking Fast and Slow,” “Antifragile,” “Drive,” “Blink,” and “Outliers” apply in a software development environment.
I’m most excited about the idea that if we can make failure “safe” then we can use it to learn things that we would otherwise be tempted to avoid and ignore because it might go badly.
A focus on failure, both deliberate and accidental, has been beneficial in moving my projects beyond the no defects, 100% test coverage type focus and towards finding innovative ways to detect and quickly reverse failure.
The video above was recorded at Agile and Beyond on May 5, 2017.
Transcript of the talk
[00:00:00] The title of this talk is looking for failure. My name is Steve Jackson. I'm a double agent at Test Double and Test Double's mission is to make software better. So that naturally means we have to get pretty comfortable with failure. If you have any trouble with failure and want to get in touch with us, you can email us at hello at test double.com.
[00:00:17] We might be able to help. This is my personal Twitter, Steve Jackson, and it's spelled weird because it's really common, right? And then my email Steve at test double. com. I'm interested in this topic of failure and seeing if there's any if it's resonating with people, I'd like to get a conversation started around with this.
[00:00:33] So if any of this makes sense to you and you want to share your story, I'd love to hear it. I love this quote from Thomas Edison, right? It really embodies that ideal of innovation to me, that if you just keep pushing, if you just keep trying, you'll get past all your failures and eventually be a success.
[00:00:51] That's not what this talk is about. This is not a talk about perseverance. This is a talk about identifying failure early, looking it straight in the face, and stepping into it anyway. What sort of things happen when you look to fail? So I'm a big believer in the law of two feet. I want everyone to get as much value as they can out of this conference.
[00:01:14] So what this talk is about is we're going to talk about failure as a learning aid. And how we can use it effectively or how it could be effective used to use effectively. I'm going to talk about some of the reasons why it's difficult to use failure effectively and all the different biases we have in place to make that really hard.
[00:01:31] I'm going to talk about some hypotheses that i'm actively testing to get myself better at dealing with failure. And then I want to close talking about how we can make it safe for other people to fail and some things maybe we should think about. We're raised from a young age to believe that if you're getting up early, you're doing all your homework, you're helping the old ladies across the street, eventually you'll be successful.
[00:01:53] But, if you're watching too much TV and doing drugs and hanging out with the wrong crowd, you're doomed, right? No chance. We see success as this linear thing that we assume that anyone that was successful obviously was successful their entire life and it's just a wonderful golden path, right? But it completely contradicts our own experiences.
[00:02:11] We know we have ups and downs, and good days and bad days, and that's how the world works. Objectively, success and failure are just the effects of an action or a non action that we take. If the effect is positive, we call that success. If the effect is negative, we call that failure. It's just feedback. I got this from Juergen Apella.
[00:02:37] This is his celebration grid. And what I really like about it is it takes these two binary outcomes of success and failure and point out that your mindset and your intent do matter. So up at the top here we have when you're successful by mistake. You're, wow, I managed to get lucky and pull that off anyway.
[00:02:53] And this is pointing out that there's really not much to celebrate there. It wasn't through any particular amazing skill that you did that. And then underneath we have your garden variety failures that we regret and obsess about for the rest of our lives, right? In the middle, if we're in an experimental mindset here, then either success or failure are fine.
[00:03:12] We're in a mindset where we can learn, so either way we can find something to celebrate in that. We didn't do exactly what we thought, but we learned from it and it was good. In this last column we have the idea of practices. That if you succeed by doing the right things, then you should celebrate that.
[00:03:28] That's great. But if you don't succeed while doing the right things, that's just the opposite of that other good luck. That's just bad luck and maybe we shouldn't be so down about that. Now where I would differ a little bit from the celebration grade is I think there is value in these other two types of failure.
[00:03:43] So I want to talk about that a little bit more. But if you're interested in this sort of thing, this comes from workout. This is a great book. Lots of nice visualizations about working with teams and getting your, improving the way that you work.
[00:03:58] Failures are interesting to me. They have a certain amount of stickiness that success doesn't seem to have. When I succeed at something, I immediately look to the next thing I have to do, right? It's a very temporary feeling, but a failure, man, I can sit there and think about that for a while. Think about that stupid thing I did back in high school, right?
[00:04:19] And I'll see someone that's successful, and we might try to copy their success. We want to emulate them. But we often miss something in the middle, because again, our brains just assume it's a linear path. So we miss the struggle and the growth that they had to go through to get there. But we're really good at pointing out other people's mistakes.
[00:04:38] Everyone's yes, obviously, Mr. Stormtrooper, you should check the wash for the red sock. We all know that. We've all learned that lesson before. Successes don't teach us a whole lot except that success is possible. But what happens when you fail? You reduce your overconfidence. You start rethinking your strategy.
[00:04:56] You question your assumptions. You want to redeem yourself. You engage. You start paying attention. Video games have an interesting relationship with failure. It's not until you fail that you understand how to play the game. I have to go and touch that turtle to find out he's not my friend and I shouldn't do that in this game.
[00:05:16] What's more interesting to me is that video games have different difficulty levels. Now, if the point of playing the game is to win, why would you play at anything but the easiest level? And I can imagine okay, maybe it's not very much fun, so we need different levels to challenge you a little bit more.
[00:05:31] Maybe that makes sense. But what I find interesting is a lot of players that play a lot of games start on the hardest level. Hardest difficulty level. Why is that? What I think it is that at easier stages, You can succeed by mistake more often. You just get by and you don't know why. But when you play on the harder levels, your mistakes are punished more harshly.
[00:05:54] And it's through these mistakes, it's through these failures that the right way to play the game is revealed to you. And you learn it much sooner than at the end of the game. Now you might say, that's interesting. I, maybe I want to look at treating my life like a video game. Maybe I want to try to fail fast and get that feedback and use that effectively.
[00:06:13] That sounds great. But unfortunately, we have this problem, right? Our psychology has evolved over the years, over the eons, to really dislike failure. Because failure used to mean that you were going to get eaten by a tiger. And so we want to stay very far away from that. We try to stay far away from failure.
[00:06:32] We don't want to deal with it. So there are basically four ways that we have, we treat failure. And the first one is that we ignore it. And this is considered a positive thing in basketball. We call this the shooter's mentality. Every shot I take is going in. I don't ever think about the ones that don't go in.
[00:06:51] We all want to be like Swaggy P here, of course. Money, every time, right? And it's common when someone asks you about something and you felt like a success and you recount it. If you ask me about that Angular project, I'll be like, we did these cool filterings, and our users were delighted about the way we could rearrange these things, and we wrote these cool custom directives, and I'll just forget to mention the days I spent debugging lifecycle problems, and camel casing attributes, and all these other things.
[00:07:19] Our brain just lets those slip from your mind. The reframe our failure so that it's actually a success. We'll lower the bar, right? So as an example, say I decided to go running this morning. Get my shoes on, I get out the door, get down the road, get about 10 minutes in, and I hurt. I'm tired. It's cold.
[00:07:44] I'm gonna turn around and go home, then I get to the end of the day, and I get to my calendar, and I've got this successive days of work done, like I'm 70, and I'm just gonna check today off, right? I got outside, I exercised, I tried. If I think, and I do this like subconsciously, right? Like I don't really reflect.
[00:08:00] If I stopped and thought about it, I'd be like my intent was to run five miles today. I barely made it one. The third thing we'll do if we fail is deflect, right? We'll find somebody else to blame it on. That's usually convenient. Or just as commonly, our ego will step in and protect us. And it'll say, yeah, other people might be able to do that thing, but I can't do that.
[00:08:22] I'm not an art person. You can't, I can't design that. Of course it didn't go well. I'm not a people person. Why am I talking to customers? That makes no sense. I'm not a morning person. Why am I up at seven o'clock in the morning? That's crazy. The fourth thing we'll do is we'll try to avoid the failure, right?
[00:08:41] This gets positive. This is where we actually start getting constructive with our failure. So we'll come up with strategies. So when my son was in kindergarten, He had to be at school at 7 30 every day and it was on my way to work and it's about 20 minutes away. So it makes sense that I'd get him ready and we'd go.
[00:08:58] I've got to get up, I got to get ready, got to get him ready. We've got to have breakfast. We've got to have lunches. I go shovel driveway, all that stuff to make sure that we can get to school on time. So naturally I start thinking about ways I don't want to be late, so I have to get up early.
[00:09:09] So I'll set an alarm here and then I'll put my phone across the room and I'll have another alarm there. Maybe I'll get all this stuff ready. I've got all these plans to make sure that I don't hit that failure. I'd like to suggest that maybe there's a fifth option. I will fail. At least some of the time.
[00:09:30] I don't like that, right? I don't want to say that. But if I start to accept that a little bit, an interesting thing starts to happen. I start thinking about contingencies. Okay I, if I fail to get up in the morning, what do I do? Am I going to call my kid off of school? Or maybe I'll decide, oh I drop everything, I get him to school, and then I'll come back and get myself ready.
[00:09:52] Or maybe, more interestingly, I'll decide to change the rules of the game. If I can't win, I don't want to play that game. So instead, maybe I'll find somebody to drive my kid to school. Or maybe the problem is getting a five year old ready in the morning. I'll hire a nanny to come help with that. Maybe I'll decide that we should move, or switch schools.
[00:10:11] 7, 30 that's not reasonable. Maybe I'll decide to switch jobs, so I don't have to work till noon, and now it doesn't matter anymore, it's no big deal. All these options and opportunities don't really occur to me until I accept that I can fail. And that superhuman effort won't just get me through every time.
[00:10:33] It's just this changing of perspective, like switching the lens a little bit that opens up new opportunities. So I learned this technique from Gary Klein, this is the pre mortem. How this works is, he suggested if you've had a hard time coming to a group decision, and you finally come to something and you have a plan in place, here's what he suggests.
[00:10:52] Imagine it's a year in the future, and we have executed the plan exactly as we intended, and it's gone terribly. We've pushed United Airlines out of the spotlight. We are the front page of the New York Times. What does that newspaper article look like? And that's when you start thinking about these things, like how can it actually go wrong?
[00:11:16] Because our default as humans is to always be optimistic. If I'm involved, it's gonna go well. And this changes that a little bit. It also gives you an opportunity, like there are probably some misgivings about this plan, and now they will emerge naturally. Ah, that was the thing I was worried about.
[00:11:32] But this can be hard for teams, right? People don't like to be negative, especially out of the box. So another useful tip is Six Thinking Hats, where you will come at the problem from a number of different perspectives and try out different things. But again, it's just really about changing your perspective a little bit to let in some more options and opportunities.
[00:11:49] Again, I don't like failure, but I think there's some value in getting better at it. So I'm working through some hypotheses, and I'm trying to experiment on these and see how they work and how they feel for me. And the first one is around this idea of regret avoidance. I learned about regret avoidance from Daniel Kahneman's book, Thinking Fast and Slow.
[00:12:09] This is an excellent book if you're interested in finding out all the different ways that your brain is trying to trick you into thinking you're a rational person. It's very excellent, and what I realized is that regret avoidance is probably the number one thing that holds me back, and here's how I think about regret avoidance.
[00:12:25] So imagine I have 10, 000 shares of Luigi stock. But I've decided, I want some Mario stock. And it happens to be trading at the same price. Has been for a while. But, I want something different. I want to find some Mario. And I find out, hey Todd. Todd has some Mario stock. And I say, hey Todd, would you like to trade with me?
[00:12:44] And Todd goes, nah, I've had this. I'm pretty happy with it. It's cool. It's fine. I go, okay. Justin, I hear you have some Mario stock. Would you be willing to trade with me? He says, sure. Great. Got what I wanted. Everything's great. Fast forward two weeks. Ouija stock has gone through the roof. It's Google plus Facebook.
[00:13:04] Greatest IPO ever. Justin is a billionaire. It's pretty obvious, how does Justin feel right now? Two thumbs up. He's a pretty happy guy. How do Todd and I feel? Objectively, we'd say we both should probably feel sad. We had an opportunity to be a billionaire, and we're not.
[00:13:23] But, almost everyone would agree that I should feel worse. Because I took an action that put me into this situation. Todd just stood pat. And that's what Regret Avoidance is about. Regret Avoidance is when my aunt calls, and I'm, I don't know if I want to answer it. And I love my aunt. I, we have great conversations.
[00:13:45] She's a wonderful person. Every once in a while, she wants me to fix her computer. And so it's maybe I'll let it go to voicemail. We'll see, right? So how do I get past this? And what I'm trying here is getting ready for failure, priming myself for this failure. Because I found that if I think something's gonna really suck, and then I do it anyway, it's usually not so bad.
[00:14:09] I can get used to it. Shortly after my son was born, my wife and I had one of those rare opportunities to get out of the house for a few hours. And so we start driving around, we come up to the movie theater, and G. I. Joe is playing. Okay, we'll go see G. I. Joe. And I was a little trepidatious about this, maybe I knew this was gonna be a bad movie.
[00:14:31] And I really loved G. I. Joe that's the highlight of my childhood, was going to my friends houses and playing G. I. Joe. So we went, saw the movie. And I was right. It was awful. But we had such a good time. We cheesed it up. We laughed at all the terrible things they were doing. And it was amazing.
[00:14:51] And if I think about this objectively, I'd say I should have a lot of regret around this. This is the one movie we saw all year. I wasted it on G. I. Joe, not Iron Man or any other good movie that came out that year. The other place I turn for this is to science. So we've all seen the scientific method before, right?
[00:15:10] You make some observations, you form a hypothesis to explain those observations, you experiment to test the hypothesis. Then from those results, you change your understanding, maybe do more experimentation. And Lean Startup has really jumped all over this idea of doing hypotheses and experiments.
[00:15:26] We're talking about it all the time. So we'll have a hypothesis, like if we add feedback to the application, it will help with user re engagement. Okay, so we're on an experiment. We'll add five star ratings, right? And we expect that 50 percent of the users will re engage with the app in the next couple weeks.
[00:15:44] And I've helped a number of startups work up and think through these experiments. And what I found almost universally is that we always craft the experiment to prove the hypothesis correct. Which is the exact opposite of how science runs its initial experiments. You always try to prove the hypothesis wrong because you don't want to spend the next several months or years Testing and retesting and refinding a hypothesis that has a flaw.
[00:16:11] You want to find those faster You want to fail sooner and the place I'm most comfortable with this is in my development habits. I like to do TDD I like to start with that failing test and I think about that as my control in the experiment The universe works the way I think it does Then, using my powers of software mastery, I change the universe and make my test pass.
[00:16:36] That feels really good. And I'm perfectly comfortable with this failure. I love these failures. In fact, if these tests pass, I get scared. The universe doesn't work the way I think it does, and it's a weird and strange place. I like these failures. I'm fine with them. Another place I think we could really embrace failure is around meetings.
[00:16:57] Okay, so how many people in the room have attended a meeting where the key decision maker didn't show up? They're very busy people. And in how many cases did you go ahead and have the meeting anyway? This is common and we make excuses for this, right? We're gonna help, we're gonna get through some of the discussion now.
[00:17:15] We might come to a consensus, we won't have to do all this. But it never works, we end up rehashing the same exact conversation once we get the person in the room. Or we rob them of the context that they need to make a good decision. She can't make a good decision because she doesn't have all the criteria in front of her.
[00:17:32] So I'd like to suggest that if you're in charge of setting up meeting agendas, maybe think about defining some failure criteria. And if you hit the failure criteria, meeting's over. Now you can create success criteria too. Nobody cares as long as the meeting gets sooner, right? This is all good. So my next hypothesis is around this problem of reframing.
[00:17:52] It happens so easily. It's subconscious. Like how do we keep this from happening? And so my thought here is that if I can get my goals out there, I might be able to do something about it. And so for this, I steal from Adam Savage. He had a great job with Mythbusters, essentially blowing stuff up for our education.
[00:18:10] And he says that the difference between screwing around and science is writing it down. So if I write down my goal, it's harder for me to subconsciously change what the goal is. Now I can still consciously do that's fine. I just don't want my brain doing weird stuff behind my back. I want to, let's get up front about this brain, what we're trying to do here.
[00:18:32] My third hypothesis was influenced by Nassim Taleb's book, Anti Fragile, and the way I think about that is like this. So for nine years I was a defense contractor. And it was very important to us that we had robust systems. They need to be able to handle all kinds of terrible conditions, and pressure, and everything else, and work fine.
[00:18:54] It was very important to us. So we wrote in Java, and what we would do is catch exceptions and try to recover from them, and try to handle them. So you'd be in a situation where it's like, Oh, I lost connection to the database. Tis but a scratch. Oh, my navigation system doesn't have a GPS feed. It's just a flesh wound.
[00:19:14] Have you had enough yet, eh? We spent a lot of time jumping through all kinds of hoops to keep this thing running no matter what. We didn't want to let our users down. Anti fragile has this idea that the opposite of a fragile thing is not actually a robust thing. It's an anti fragile thing. So a fragile thing is something like a wine glass.
[00:19:35] It doesn't take a lot of pressure. It doesn't take a lot of change. Just a little tiny chip and it's worthless as an object. A robust thing is something like a stone. You could throw a lot of things at it. It still stays a stone. It's still as useful as it ever was. It's resilient to disorder. But an anti fragile thing actually gets stronger when disorder is applied.
[00:19:58] And probably my favorite example from the book was the idea of building muscle. When you work out, you actually tear your muscle. Small micro tears. You cause it to fail. And then it's from that failure that it grows back stronger than it was before. And what's more is if you do not cause your muscles to fail, they will atrophy and die.
[00:20:17] This is a system that needs disorder. It needs failure to move forward. And when I think about software systems I think about that system where you fix this bug, and then this other one pops up, so you fix this bug, and now this one's over here, and you're holding this thing, and every time you touch it, it seems to crumble in your hands, and it's very fragile and terrible.
[00:20:37] So naturally, we want to keep us from changing it. So we create these things with change control boards, all this other stuff, and that'll make it robust because now we can't let any of the randomness and chaos and bad things into our system, and everything will be fine. But I feel like this way of thinking is doomed.
[00:20:53] Software has to change. Even if you manage to write bug free, perfect software the first time, and I'm sure most of you have done that, even if you pull that off, the world will change. The expectations for your software will change. You have to be able to change with it. Software maintenance does not mean that your software works the exact same as it did last month.
[00:21:19] If it did, it would be free. Software maintenance means that it's at least as useful as it was last month. And that requires change. So it seems like we need to get better at this sort of thing, but it's easy for me to stand up here and say, you know what, guys, we should fail all over the place and we should add more chaos and it's going to be great.
[00:21:39] Trust me. Who's behind me. Let's go. There are problems with this, right? There are a lot of situations where it's hard to take on more negative things. If we have debt, we're very, we feel very vulnerable. If I'm barely making ends meet, I don't want to talk to you about your startup idea. That's terrifying.
[00:21:58] If my system is one of those fragile systems and you want to introduce more failure to it, no, we're not doing that. That doesn't sound good to me at all. If I'm new to a team or if I have failed recently, I'm not going to want to add more failure to my resume or my reputation. It's not a good place to be.
[00:22:16] So my thinking starts to shift to like, how do we create safer failures that don't feel as bad when we're feeling vulnerable? The obvious one here for me seems to be that if you're going to fail, you would prefer to fail in the small, maybe get good at that and get to larger failures and the engineering organization that I feel has really taken this to heart is Netflix and they're, they they have these tools called the simian army and essentially what it does, there are various tools that will take out the things that you're dependent, your application or service depends on.
[00:22:50] So they have things like chaos monkey that will take away your dependent services and you're supposed to say running. And by the way, they run all this stuff in production. They have latency monkey, which will add time to the request. And you still have to give a good user experience. And they've built this into their products.
[00:23:05] This is going to happen. You expect this. So they've gotten good enough with this, that they have things like chaos Kong now, which will take down your entire AWS region. And Netflix is pretty reliable. It works in a lot of places with not very good network infrastructure behind it. For me personally, this is really about pushing out my comfort zone a little bit.
[00:23:26] If I can screw up in front of my friends, then maybe I can make a mistake in front of my coworkers. Then maybe I can mess up something for my clients and eventually maybe I can fall flat on my face in front of an audience. And yes, as we add and invite more disorder and bad things into our system, into our lives, bad things will happen.
[00:23:47] But there's also the flip side of this. There's good luck that comes with the bad. And so as we open ourselves up to these different opportunities and interactions, we get good things. I answer the phone and my aunt has playoff tickets. Great! I'm glad I took that risk. My next hypothesis is that if we can cause a failure to happen and then reverse it, like we expect this to happen, then maybe we can use those same techniques when the random failures that we don't expect happen.
[00:24:17] Really what I'm talking about here is building undo into everything, right? It's great to be able to just get back to a good state when you're in a bad place. Either do undo buttons or get reset all the things. Sounds pretty good. And I feel like the place where we've really been pushing this forward is in the areas of continuous deployment and continuous delivery where we're getting pretty good at putting small little experiments out there and trying them and seeing what happens.
[00:24:41] And if they fail, they're still small and easy to reason about rather than the huge mega releases where we never knew what's going on. And again, I'll pick on Netflix. I learned about their deployment process a couple of years ago, and essentially you push a commit that causes a cluster to spin up in production with the new code, then the load balancer starts pushing some amount of traffic to the new code.
[00:25:01] And if it goes well, that becomes the new production. That's great. But if it goes badly, it gets dropped out of the load balancer. So it feels pretty safe to fail in this environment, right? Like your opportunity for screwing something up is pretty small and your infrastructure will save you.
[00:25:16] The place where my brain really started to get twisted on this is when I learned about Erlang and OTP, which is the Open Telecom Platform. So the thing about Erlang is that it can run lots of little tiny processes, like millions of them on a tiny laptop, and they start up super fast. So Erlang has evolved a very different error handling strategy than C based languages.
[00:25:37] In C, we always try to deal something with our errors. In Erlang, you just let it go. You let it fail. You let it die. And OTP immediately starts the same exact thing again. So again, it's very safe to fail in this environment. And this twisted my brain because I've spent so much time thinking about recovering from these bad things.
[00:25:57] And how do we handle this? And what do we do now? Just let it fail. Okay. We're really trying to get to the point is where failure is a non event. It's not, we don't get all the catastrophic fail. We don't get all the psychological baggage and damage from screwing something up. It's no big deal. We just resort to where we were.
[00:26:15] Now, I wonder about this a little bit, right? Because I talked earlier about all those benefits from failure that we'll slow down and we'll reduce our overconfidence. And are we going to still do that if we don't have all the baggage that comes with it? And so far I think it works because even though I don't feel devastated, I still don't like being wrong and I get a little annoyed that I'm wrong, right?
[00:26:38] And we're finding that a little bit of anger is actually a beneficial thing, right? It's energizing. Like you want to change it. It's, I got to fix that. It's driving me crazy. And if you can just get a little care mad about stuff, you can get a lot of things done. 90 percent of my Twitter feed appears to be powered by this magical energy.
[00:26:58] So this kind of gives me a framework for exploring things. First I set a goal. And then I write it down so I can't reframe. And the next thing I do is I think about failing at it. I don't like that. Get that taste in your mouth. Okay, so now what I do at this point is I don't give up, this seemed achievable 10 seconds ago. I can do this. So I start thinking about the failure a little bit. Are there things I can do to make this safer? Can I make it smaller? Can I come up with a way to reset out of it? Maybe if I take some friends with me, it won't be so bad. And it may be that this failure is just too big.
[00:27:34] It will hurt my friends. I'll end up crying in the corner. Okay, this isn't a suicide pact, right? This is fine. You can avoid failure. But if it doesn't seem like that kind of failure, I'll resolve to try. But first, I will write down what I think might happen here. And I tend to record failure criteria only because I have problems with sunk cost fallacy type problems, where I'll get 80 percent into the quicksand.
[00:27:59] It's I better keep going. That's what I should do now. So I try to write down what I think might happen and how I should get out of it. And then I'll try. So I'll try and maybe it'll go badly. I'll hit my failure condition. And so I'll pull myself out of it. Okay. I got to go around this one and that's fine.
[00:28:16] I learned something about in this context, this isn't a good idea. I need to do something else. Maybe I'll try a different idea next time, but usually, I'm done. I tried. Let's keep going. The second thing that might happen is that against all odds, I will succeed, right? And that feels amazing, like when you know you're gonna screw something up and you don't.
[00:28:36] Man, I am awesome. I'm so good at this. This is great. The third thing that might happen is that I will try and I will fail, and then I'll realize this failure is really not that big a deal. And maybe I'll come up with a way to fix it. And a lot of ways, this is what I'm striving for, because I want to build up this tool set of things to do when things go wrong.
[00:28:58] And so it helps me feel more comfortable in those situations. For instance, if a deploy to production goes badly, like I don't freak out because I have broken very many deploys to production and I have a whole set of ideas and tools to get back, get things back to good without freaking out.
[00:29:17] And then the fourth thing that might happen is I will try and I'll end up someplace completely unexpected. And this is really interesting to me because I still want to go this way. I want to get to my goal, but I didn't even know this existed as an option, right? So now I'm over here. Maybe I should keep going this way.
[00:29:36] Let me see what's this way. I have different things. But again, if I hadn't, if I just saw the failure and tried to stay away from it, I would have never really explored this as an opportunity.
[00:29:47] But again, It's easy to talk about failing when it's just you, and nobody notices, it's no big deal. Because failure in modern society has nothing to do with tigers. It has everything to do with people looking at you and saying, You screwed up, you're the mistake, why'd you do that? That's the sort of thing we want to avoid, because when that happens we naturally want to shell up, and hide, and not try anything very risky at all.
[00:30:12] I gave this talk at a meetup, and a man came up to me afterwards, And he said he had one of those situations where he screwed up something in production and everybody freaked out but he and his boss got together and they figured out and they got it back to good and they realized that there's some things they could have done differently there's some ways they can improve this and make it safer and so he left for the day feeling okay not the best day in the world definitely not the way he wanted it to go but he's going to keep his job and they had a chance to improve things were looking good.
[00:30:44] Objectively, this is the exact kind of failure I want. It was fairly safe, it was a learning opportunity, no real psychological damage done. He says he comes in the next day and he's excited, like he's gonna make this better. This is a good opportunity. And a co worker comes on up to the cuticle, saddles up and is that was a boneheaded thing you did yesterday.
[00:31:05] He said, I just froze up. I was afraid to do anything for two weeks because I might screw it up. All that imposter syndrome comes roaring back. Like I can't do this thing. I can't believe they pay me to do this. So how do we get past that? How do we get into a place where it's okay to screw up and we can learn from these things?
[00:31:28] This is hard. Like we all naturally build up these personas, this certainty and this confidence that we have. And it's very useful for us in an uncertain world to appear certain doctors understand this, right? If you go to a doctor, he's I don't really know what's wrong with you, we're going to run some tests maybe we'll try this thing.
[00:31:46] You don't want that doctor, you already feel vulnerable, you don't want to talk to this guy. You want this guy who's it's going to be fine, we're going to run these tests, don't worry, we'll figure this out, we're in this together. And I feel this way sometimes as a consultant. A client will bring a weird, strange problem I've never seen before, and I'm like, I don't really know how to handle that.
[00:32:07] But that's not the way I say it, right? I'll be like, okay we'll do this to reduce risk and we'll try these things to get feedback and don't worry, we'll figure this out because they're feeling vulnerable and I feel like I should help reassure them that everything's going to be okay. And I realized that I am speaking from a place of great privilege, right?
[00:32:25] If I screw up, it might reflect badly on me. Maybe it would take my company down. That's probably worst case scenario, but certainly nobody would think that some random white guy in Iowa is a screw up because I messed up. But if you feel like you're representing an entire religion, an entire race, an entire gender, then you're going to have a much harder time being okay with failure.
[00:32:50] You're not going to want to let your shields down. You know how hard it is to build this respect and reputation, and you don't want to lose any of it. I feel like my existing framework of experimentation isn't going to work here. I'm okay with doing things to myself and seeing how it feels, and that's fine.
[00:33:05] But I don't want to experiment with other people's feelings. So instead, I'm considering some questions, and I'd like to invite you to consider them as well.
[00:33:16] So the first question is, what do you celebrate? It seems pretty obvious to me that the thing we're really good at celebrating are those random successes that we're not supposed to celebrate, right? Justin's a billionaire, you won the lottery, we're going to write newspaper articles, we're all going to talk about that's the thing we want to talk about.
[00:33:36] Imagine you had a coworker, just had a successful launch. How would you celebrate that? And you found out that she had been working 80, 90 hours a week for the last month to pull this off. And so naturally, you want to celebrate, you want to take her out, you want to celebrate all that hard work and effort, and wow, way to go!
[00:33:53] But imagine the same exact situation, the launch had been botched. Our natural tendency is to go to correction, right? If you hadn't been so sleep deprived, maybe you should have asked for help. We say that we're celebrating the effort and the action. But we have a hard time taking it out of the outcome.
[00:34:13] There's a great story from IBM about Thomas Watson senior. IBM was in a bad place. I need to make some sales. Watson was going to lose his job. And one of his salesmen blew a million dollar deal. So that salesman comes into the office. He's got resignation in hand and Thomas Watson's what? And he explains where the deal went wrong, what he should have maybe tried differently and turns to leave.
[00:34:38] And Watson stops him at the door and says, I can't accept this. I just spent a million dollars on your education.
[00:34:48] It is hard for me to imagine being Thomas Watson in that scenario. I needed that deal. My job is on the line. But I can imagine what it felt like to be that salesman. How empowered I would feel. How supported I would feel. How I would want to redeem myself. How determined I would be to prove Watson correct.
[00:35:08] Can we find real, genuine things to celebrate when things go wrong? Can we actually celebrate the effort independent of the outcome?
[00:35:19] What sort of failures do you acknowledge? It's very easy for us to shoot, to put stuff under the rug. We don't like to talk about failure. I've been part of a number of initiatives, right? Where we have the big kickoff and we get everybody together. And this is going to be great. We're going to change everything.
[00:35:34] We're going to do it this way. And then two months later, nobody's talking about it. What happened? Nobody knows. So my brain naturally starts filling in details and they're not good. The stories I'm telling aren't really great. And I feel like we're missing such an opportunity here to talk about how our company works and what we value and what we believe in.
[00:35:57] It may be that we weren't ready for this particular thing. Okay. It may be that 90 percent of the employees really hated the idea, so we're not going to try to shove it down anybody's throat. Maybe we've discovered there's going to be a lot of work, and we didn't want everybody working 60, 70 hours to pull this off.
[00:36:15] All of these things tell useful stories. And I think by having to acknowledge them in the same place that you did the big announcement, you have an opportunity to use it as a positive experience. And if nothing else, it helps inform our next initiative idea that maybe we should check on these things.
[00:36:35] Likewise, what sort of things do you discuss in the open? Which ones do you try to hide? I was really impressed with the way that GitLab handled their recent data outage. They were open and transparent about the triaging they were doing the entire way through, and it felt pretty blameless. So much so that the person who did the typing mistake raised their hand and said, yeah, it was me.
[00:36:55] That's pretty remarkable, right? Most of us don't want to be singled out. And I can contrast this to S3, which had an outage not long after, and nobody knew what's going on. They, is everything okay? And everything's fine. Are you sure? I'm having a lot of trouble here. Like it took a number of hours for things to get resolved.
[00:37:12] And we'd all like for our dependencies and the people we work with to be 100 percent awesome all the time, but that's not reality. And I'd really rather work with people that are open and transparent so I know that things are being done and maybe I can help.
[00:37:29] What kind of exceptions do you make and which ones do you never allow? If the product owner can be 10 minutes late to every show and tell but I get slammed if I'm 30 seconds late to a stand up, I'm gonna notice that and I'm gonna key on that. And I'm not saying that everything has to be equal and fair.
[00:37:47] Equitable is a nice goal but implicit arbitrary rules destroy trust. And they destroy any ability to get to make a place safe So if you're going to have exceptions be explicit about them be open about the sort of things that you allow
[00:38:05] What is your own personal tolerance for failure when I think back on early management and mentorship opportunities? I had I screwed this up pretty bad. I was trying to help right? I would see someone going down the wrong path of no don't do that. You want to try this instead Oh, no, that, that seems risky.
[00:38:24] Let's try this instead. Like I'm trying to help. I don't want you to feel the pain of failure, so I will help you. Stay away from that. And I robbed them of a learning opportunity, right? Now, to be clear, I think you can learn from other people's failures, but it requires a lot of context. It requires a lot of why you almost have to practically step into the problem to understand the lesson to learn there.
[00:38:49] Secondly, I inadvertently made them more dependent on me. This was not my intention, but if every time you start something, someone's no, don't do it that way. You're naturally going to want to clear it with them first. You don't waste your effort. Not my intention, right? I wanted them to be independent thinkers.
[00:39:06] I had no interest in commanding, controlling everything, but that was the natural response because of the way I was acting, but the third thing was perhaps worst of all, I potentially ruined an opportunity for innovation. The amazing thing about novices is they don't know what's impossible. I have seen high school students do things that I didn't think couldn't be done.
[00:39:30] All my experience, all my baggage, all my context says you can't get here from there. And they did it anyway. So not only did I ruin opportunities for their learning, I ruined opportunities for my learning. And I potentially cheated the world out of an innovative way to look at a problem. All because I was focused on keeping people safe in a nice straight path where it's okay to go here.
[00:39:55] All right, this is the last question. What do you say about people outside your team during challenging situations? I have said some very not nice things about external partners in bad places, and everything's on fire. And I'll get on the phone with them, and I'll be very supportive. I'm kind. I want to help.
[00:40:14] I just want to get past this thing. But when I go back with my team, I might tell a joke. I might vent a little bit. It's safe. It's cool. We're all together. Think about the kind of lesson I'm sending, though. Hey, it's okay that you screwed up. We're gonna learn from this, it's fine. What do you imagine I'm saying as soon as I turn my back?
[00:40:35] We've already established I have this kind of behavior. Maybe it's okay that S3 goes down and everyone's valuable and it's fine. We'll get through this.
[00:40:46] The point of all this is that Change and disorder and chaos are the laws of the universe. We can't get past them. So I feel like we ought to join them and get better at it. The only thing change has lasted even longer than taxes. It's been around much longer and it will win over all other things.
[00:41:04] And there are opportunities here. We can do really amazing things. It is viscerally satisfying to see your vision come out of this chaos into reality. It feels so great to do these things. If I look at something like Wikipedia conceptually, think of all the noise that goes into Wikipedia. We don't care what you are, who you are, what you know, what your experience is.
[00:41:27] Everybody's welcome to add their own particular two cents. And somehow, a tremendous amount of signal comes out of this thing. Wikipedia has changed the world. It's hard to imagine getting by without Wikipedia nowadays. And likewise, I look at a place like Valen sorry, Venice, and I'm just amazed. Venice has been around for millennia.
[00:41:48] And it's built on the world's worst possible imaginable building material. Water is chaotic and fickle and ever changing and they have made it work for centuries. And then I think about some of our master planned communities that we have out there that got just a little bit of disorder and completely fell apart.
[00:42:11] It is our nature to just want to put things into binary buckets when we think about them, right? Success. Failure. Fragile. Robust. Republican. Democrat. Man. Woman. White. Black. Waterfall. Agile. And we're missing all this interesting stuff that's in the middle, because we hang out on the extremes. So I'd like to suggest that if you've been a hardline successist your entire life, you Maybe try on a little failure and see what it's like, because let's face it, if you're not trying new things, you're not growing, you're not learning, and you're probably not doing anything interesting.
[00:42:49] So I'm really interested in hearing more about this. If you guys have any thoughts or feedback, please come talk to me or send me an email. I'd love to hear from you. Thank you so much for your time and attention. I really appreciate it. Thank you.
