ReSolve Riffs with Annie Duke on Biases and Optimal Decision Making in Markets and Life

This is “ReSolve Riffs” – live on YouTube every Friday afternoon to debate the most relevant investment topics of the day, hosted by Adam Butler, Mike Philbrick and Rodrigo Gordillo of ReSolve Global*.

“Life – and investing – are more like poker than chess”. This week we had the honor to speak to none other than Annie Duke, former World Series of Poker champion, author of “Thinking in Bets” and “How to Decide”, and thought leader in the field of decision-making. Co-hosted by our good friend Brian Portnoy (founder of Shaping Wealth), the conversation covered:

  • How planning for a life in academia led her into a successful poker career
  • Applying her background in cognitive science to advance her game – Annie’s “Aha! moment”
  • Not all poker is created equal – tournaments, cash games, iterations, and strategic differences
  • Nature vs nurture – cognitive flexibility and turning a hedgehog into a fox
  • Cognitive Reflection Tests and why they may indicate a larger appetite for conspiracy theories
  • The Dunning-Kruger effect, overconfidence and naïve realism
  • The pervasive influence of the sunk cost fallacy and how we might keep it in check
  • Group think, cognitive laziness and why most people tend to anchor to the first response they hear

We also explored the illusion of explanatory depth (and why often the answer lies within the question), the effects of tight feedback loops, and multiple applications for life and investment decisions. Towards the end we caught on to the fact that Annie had actually been incepting us with ideas from her new book all along. We had been looking forward to this conversation for some time and it exceeded our expectations.

Thank you for watching and listening. See you next week.

Listen on

Apple Podcasts

Listen on


Subscribe On


Annie Duke
Author, Speaker and Decision Strategist

Annie Duke is an author, corporate speaker, and consultant in the decision-making space. Annie’s latest book, How to Decide: Simple Tools for Making Better Choices, is available on September 15, 2020 from Portfolio, a Penguin Random House imprint. Her previous book, Thinking in Bets, is a national bestseller. As a former professional poker player, Annie won more than $4 million in tournament poker before retiring from the game in 2012. Prior to becoming a professional player, Annie was awarded a National Science Foundation Fellowship to study Cognitive Psychology at the University of Pennsylvania. 

Annie is the co-founder of The Alliance for Decision Education, a non-profit whose mission is to improve lives by empowering students through decision skills education. She is also a member of the National Board of After-School All-Stars and the Board of Directors of the Franklin Institute. In 2020, she joined the board of the Renew Democracy Initiative.


Adam: 00:00:59Speaking of Master Class ladies and gentlemen. Here we are with none other than Annie Duke, co-hosted by Brian Portnoy, and guys thank you so much for coming. We are really excited for this episode so you know, can’t wait to get started. Mike, why don’t you go through your spiel.

Mike:00:01:18You got it. Yeah, just to make sure everyone is aware that nothing on this podcast or video is anything to do with investment advice, and please get that from qualified professionals not like the four scallywags on this particular show at the moment. And please feel free and encourage you to hit the like button, share, make comments and actively participate as we go along. And with that, let’s roll.

Adam: 00:01:48 Yeah, to get started maybe Annie I’m sure you need no introduction but I think it’d be great just to frame the conversation to give us a little bit about your background, maybe your academic and your other colorful background in the poker sphere, and tell us some of the common themes of the books that you’ve written, and then maybe tease us with a new manuscript that is on the table.


Annie: 00:02:16Okay. Well, I think the last part will come up organically so I’ll table that because the other stuff takes a little bit because I’m not someone who’s only done one thing in my life so it takes a second. I’ll start with after college. I won’t start with elementary school. I went to elementary school in Concord, New Hampshire. So after college  my plan was to become a professor, I went to graduate school at the University of Pennsylvania. I did five years there studying under Lila Gleitman and Henry Gleitman. Generally, I was basically studying learning, in particular first language acquisition but it was broadly a cognitive science program, so you were studying across all anything that would be encompassed by cognitive science. It would include judgment decision making, learning under uncertainty which is more the part that I was in perception that kind of thing, so it was a budding science at that point, this this sort of inter disciplinary approach to behavior. Did five years there, was on my way to be a professor. I had no plans to do anything else, but right at the end I actually just got sick. I had an illness where I had to delay all of my job talks for a year in order to recuperate and just physically wasn’t feeling great, so what during my time off, I was sort of constrained in my choices because like I didn’t want to reboot a new career, like another career or something because I was already had one. I was going to be  professor. I needed something that was really flexible in the hours because physically I just didn’t feel well, so it was like okay, I’m going to do something that’s kind of not permanent that’s going to give me some flexibility and so I played poker. And I think that given kind of what’s happened over the last 20 or so years with poker being all over television and obviously internet poker was a big thing for a while.

I don’t think that sounds quite as weird and unusual as it actually was because when I made this decision it was in the 90s, and there was no poker on television and it wasn’t on the internet. Poker was generally sort of grouped with Craps I guess. So essentially like when I told people well, I’m playing poker they’d be like, are you going to Gamblers Anonymous? So this idea that you could actually you know, generate a positive expected value by playing poker was kind of an unknown thing at the time, and the only reason why I actually happened to end up doing it was because my brother had done it before me. So he had gone off to New York and was studying with a grand master in chess and had deferred college for a year. I think he sort of ended up going for maybe six months or something, but it took him a long time to get back to it and he never did finish because he became very successful at poker. He found poker much more interesting than chess. So he had been playing poker for about 10 years before this happened and he was the one who actually suggested, why don’t you do this in the meantime?

And so I did it in the meantime, and when I sat down it was a little like that aha. Like oh everything I was studying, here it is at this table in this way that’s decision making under uncertainty and learning and how do you close these feedback loops when there’s luck and hidden information, and all the things that I’ve been studying about, like judgment decision making and bias and how do you actually solve for that in in this situation where it’s so incredibly high stakes and high speed? So I just, you know I sort of didn’t go back I just kind of didn’t go back is the way that I would describe it. And I ended up playing poker exclusively for about eight years. And then right in the early 2000s I got asked by a group of, a hedge fund actually to come speak to their options traders, and they wanted me to just come talk to them about how poker might inform the way that they thought about risk. And I actually ended up talking to them about how decision making on their uncertainty can distort your risk attitudes in a very negative way. That was what I ended up talking to them about. And I just remember like in that moment – by the way I was nine months pregnant when I gave that talk. I was barefoot because my feet were so swollen. But it was kind of in that moment where i realized like oh, there were a lot of things that I really loved about academics, right? Like I really love this idea of like synthesizing information and trying to figure out what does it mean and how would you use it, and I love teaching. And I kind of remembered that at that moment, and just kind of continued doing that and developed a consulting business and a speaking business that I overlapped with the poker for about 10 years.

And then in 2012 I retired from poker completely and had really had in mind that I wanted to write this book, Thinking In Books, that’s I had already written some books about how to play poker specifically from a decision-making framework, so you can think about that as kind of the mirror image of what I was doing in my business life at that point. But I really wanted to write Thinking In Bats, which it wasn’t called at that time. My editor, Nikki Papadopoulos shout out, came up with that title which was so great. And so I started working on the proposal and you know published it in 2018. I was very lucky for it to be a bestseller. Followed it with How To Decide. I now do lots and lots of consulting but if I’m going to do business consulting with you it needs to be a long-term relationship, because I discovered I don’t like things that are kind of surfacy so, I do two things. I’ll give a talk for like an hour, or I’ll actually like live in your organization for a while. I don’t like the two-day training thing. I don’t think it’s my sweet spot. I think it’s other people’s sweet spot. So I just don’t like to do things that aren’t in my sweet spot.

So now that’s kind of what I do, and then on the side I ended up back at the University of Pennsylvania and I’m a visiting scholar there. I work in Phil Tetlock’s lab, who wrote Superforecasting, for those people who, I’m sure everybody’s familiar with him. And I actually do research on forecasting, specifically counterfactual forecasting and now I’m buttoning up that PhD that I didn’t get because I went off for a life of vice and dereliction to go play poker.

Mike: 00:08:41That’s got to be a sweet spot on the resume.

Adam:00:08:44Okay. 10 years. Yeah, definitely vice poker player.

Annie:00:08:48 That’s right. Exactly, like derelict.

Mike:00:08:51 Derelict, exactly right. Ne’re do well.

Adam:00:08:56I was curious about whether you’re degenerate.

Annie: 00:08:58That’s what you say you’re such a degenerate.

Adam: 00:09:00No doubt. I might have been there. When you went to play poker and you said you sort of, you recognized that you were bringing to bear a lot of the formal topics that you had been working on academically. When you first started playing poker and as you as you climbed the learning curve, were you actually trying to apply those formal concepts to the game? Was that a conscious way you were thinking about how to approach the game, or did you find that there were a lot of natural sort of instinctive behaviors that you brought to bear? Like how do you reconcile those two sides of your game?

Formality or Instinct

Annie: 00:09:42That’s such a good question. I wouldn’t say that I started thinking about it like in this super explicit way until I started working on that as part of my career in 2002, right? When I got asked to speak to that group of options traders, that’s when like I really was thinking about it and what’s this kind of formal relationship between these two things that I’ve been doing. That being said, when I look back on what I was doing in poker, it’s really just applied, right? I always had very strict loss limits. That’s something that would be completely applied from what I was doing in cognitive science as an example. I was really actually, really obsessed with the resulting problem. I used to talk about it all the time and try to figure out with my peer group how are we supposed to avoid this. And obviously that’s outcome bias. Really just trying to deal with it and then you know I’ve done so much work thinking about these noisy systems and how are you supposed to actually extract signal from these noisy systems from a learning standpoint, and poker’s just such a noisy system.

So in retrospect like I wish that I had been thinking about it more explicitly from the start but when I look back on it it’s like, I was clearly thinking about it probably didn’t have the fidelity it would have, had I’ve been thinking about it more explicitly. And I will say that the quality of my game did take a jump when I started to be much more explicit about thinking about the way that those two things related to each other.

Brian: 00:11:15And did you find that the best of the best that you played with had sort of a natural instinct for those mental models already even if they weren’t explicitly doing decision journals at the poker table? They just sort of got it? I mean I know you’ve played with some of those the world’s best.

Annie: 00:11:35So Brian, it it’s interesting. It depends a little bit on what you mean by the best of the best. And I imagine that you can think about this in the—I think it’s very similar in the investment world as well. So there’s kind of a difference between someone who is really good at a very particular form of poker. So we could think about there’s kind of broad strokes, right? Are you in a tournament or are you in a cash game? And that division would create very different behaviors from your opponents, and that’s very different strategies and tactics that you might apply in order to be good in those two different environments, because the environments are clearly have a lot. You know in one case you have you will basically have the finiteness of your resources is really in your face in a tournament in the way that’s not in a cash game. And then even within those environments you have different games, right? You have stud games and draw games. You have flop games, you know there’s a different number of cards that you get in each of those.

So those also create a lot of different choices. So there’s basically sort of broadly two types of poker players. There the ones who just got really good at a thing, right? So maybe there’s somebody who’s just amazing at smaller buy-in, large field, no limit hold’em tournaments, and they’re just great at it, right? And then there are players who basically, you could invent a new game, tell them about it. They could sit down at the table with people who had played that game already before and they would beat them, because they just have this very deep conceptual understanding of, why are the reasons that I do things in different environments, right? How is the environment affecting the types of choices that I make so that if you tell me about a new game, I have those mental models to figure out like well, how easy is it to make a hand? How bluffable are my opponents? How much are they going to feel the pressure of going broke? Whatever those things might be, they just get it at that super duper conceptual level. So like in that category would be something like Phil Ivey, right? Like you guys could have been–you could invent a game that you’ve been playing for weeks, teach him the rules and he would eat your lunch, but never played it. So that’s what I mean by it’s a little bit different. Like it depends on what you mean by great, right.

But if you’re great in the second category, yes, you have all of those mental models. You’re thinking about that stuff all the time, and how do I sort of think about what the reference class for this particular game that I’m in is? And what does that mean for the way that people behave, and how can I see it from my opponent’s perspective and all that. And then in the first case you’re just like, betting here works.

Brian: 00:14:29Yeah you made me think of how Tetlock uses the foxes versus hedgehog analogy.

Annie:00:14:36That’s a very good way to think about it and so here here’s an example actually on the fox versus hedgehog. So the big idea in tournaments is that aggression is very good. And there’s a reason why aggression is very good in tournaments because when people feel their death is impending, meaning I’ve got a limited number of chips and I do not want to lose all of them because I will have to leave. Then you can imagine it’s much easier to push somebody around, right? So aggression is going to get rewarded in that environment where someone can feel their impending doom. But in a cash game—Okay, well, if I lose my chips, whatever I can buy more, right? It’s not always the case like some people are very risk-averse, but even the most risk-averse person is going to tend to be more risk-averse in a tournament than they are in a cash game, where you can renew your resources, right? So some people who are just great at tournaments, just have this big idea. I’m just gonna push people around. Aggression, that’s good.

What happens is when you stick them over in a cash game, they lose all their money, because they’re still trying to push everybody around and everybody’s like, hey, I can just rebuy. I call, right? And they don’t really know how to adjust. So that I think that framework of foxes versus hedgehogs is like a very good framework to think about the difference between those two things. And what we know out in real life is that hedgehogs can be very successful. If you have the right big idea at the right time, you can do really well.

Adam 00:16:12Can it be taught? Can that idea of cognitive flexibility or cognitive adaptivity be taught? I mean the way you speak of it seems like this is something you’re sort of genetically predisposed to, right? You just have this predisposition towards conceptual or cognitive flexibility or you don’t, right? But is there something that you can do to practice or to build or improve on that skill set?

Can it Be Taught?

Annie 00:16:40I think the answer is probably – so the answer is yes, you know I mean this is kind of what I’m doing in in the Phil Tetlock. Like can you train people to be better forecasters, and forecasts, obviously that requires kind of getting out of the moment into the future and trying to see things from different perspectives? And the answer is you can. But what we have to understand is that it’s incremental improvements, right? Like you can sort of teach people what the value of this stuff is and some people are going to respond to it more than others.

But the way the way that I try to think about it is, everybody has some sort of distribution of like their worst decision to their best decision, you know it’s like normally distributed. And are you ever gonna take the best decision maker in the world and the worst decision maker in the world and get the worst decision maker to look like the best decision maker? Probably not, no. Let’s just say no. But that’s okay because what really matters is for that person who’s the worst decision maker, can you make them a little bit better? Because if you make them a little bit better, sort of churning across all the decisions that they ever make in their life, that’s going to actually have a huge impact on their lives. So you know I think that I would sort of just reframe it, right, which is not can you turn a hedgehog into a fox, right? But can you make a hedgehog be a little bit more fox-like? And so I think the answer is yes, you can. But the other answer is, you can create decision making environments that kind of force that on them, and then you can start to get like, then you can start to get a lot higher quality decisions out of people that way.

Mike: 00:18:25So it’s a function of maybe two dimensions, one is a level of natural ability and then the next is trainability. And so for the optimal outcome you’d want very high natural ability with a person who has a high trainability which would lead to a likely very high outcome of efficacy in that?

Annie: 00:18:48That’s true. And also those things would be super correlated. If you have a lot of natural ability, you’re going to be much more trainable. I mean here’s an interesting thing, are you are you guys familiar with the cognitive reflection test? So you might have seen these questions like a bat and a ball cost ten dollars.

Mike: 00:19:05Oh, yeah, yeah, it’s a nickel.

Annie: 00:19:17All right. Well, here’s what happens. I’ll tell them now that you’ve given the answer away. Thanks.

Brian:00:19:20Talk about resulting .

Annie: 00:19:24So a bat–so here’s one of the questions. The bat and the ball cost $1.10. The ball is a dollar more than the bat. I mean, sorry, the bat is a dollar more than a ball, how much does the ball cost? And everybody says 10 cents. It doesn’t cost 10 cents because a dollar minus 10 cents is obviously 90 cents not a dollar. But what people do is just kind of this, you know it’s like a very thinking fast and slow kind of thing like system one says it’s a dollar more so what’s a dollar ten minus a dollar? And you come out with ten cents. So there’s a variety of these questions. There’s another one which is, there’s a pond with lily pads in it and the lily pads double every day. On the 49th day, the whole pond is filled. On what day is the pond half-filled? And people divided in half. But the answer is 48, right? Because it’s doubling every day, right? So I gave that one away too. And then there’s another one which is like–it’s something like a machine. One machine takes two minutes to create two widgets. If you have a hundred machines making 200 widgets, how many minutes will it take? And the answer of course is two. So anyway there’s all these things where you sort of end up at the wrong answer. So what they’ve shown it’s–so this is actually one of the most predictive tasks for your decision-making skill, it turns out. Because it has to do with sort of open-mindedness and how much are you stopping and thinking? How much is system one taking you down a path, or you know are you checking your work?

And just to give you a sense, you know these questions are quite hard, so I think if you take kind of like a state school people get sort of less than one right on average. And if you take MIT, they’re still only getting a little over two right. Okay, so even like people at MIT do not get three right on average. Here’s the interesting thing about it, why I think these things are correlated. Failing the CRT is also highly correlated with belief and conspiracy theories. And there’s an interesting twist on the CRT which is pretty fun. So let’s say that I ask you the bat and ball question and you say 10 cents. And I say to you actually that’s not right, do you want to try again? Now we can further divide people up into people who go back and try again, they may or may not get the right answer but they try again, and people who say no, I’m right. And the people who say no, it’s not wrong. You lie. It is ten cents, I figured it out. They are the most likely to believe in conspiracy theories. So I think this is what I mean by these things are correlated because what the CRT is getting at is this open-mindedness. It’s this ability to sort of check your work and think and am I wrong? If I were, wrong how would I be wrong? Those kinds of questions that I think are so incredibly important to great decision making, and that’s why I think it’s so predictive of decision making capability and particularly these things that have to, where decision making really goes south.

Adam: 00:22:42But they reflect two different pathways, right? I think so sort of the first the first level there in the, what is it called the CRT test?

Annie 00:22:51The Cognitive Reflection Test. Developed by a guy named Shane Frederick over at Princeton, who’s quite a character. He was a graduate student of Richard Thaler’s I believe.

Adam 00:23:00Oh, okay. Makes sense. So the first level is how much? Yeah, like you said sort of how much level one versus level two inclination you have in the way you make decisions, I guess. And then when you tell them the answer, then it’s more of a how to what degree or extent does what you say, how strongly you’ve internalized the value of what you said in terms of like what it means to your sense of self or your ego or whatever, right? It’s like writing something down. It’s like writing a goal down versus how invested are you in this answer, and how much does that mean to you?

Annie: 00:23:52I think it’s a little bit has to do with overconfidence. There’s definitely like some Dunning-Kruger in there, ability to see something from somebody else’s side, sort of like how trapped in the inside view are you? And then there’s you know naïve realism, the thing I believe is true and anybody else who believes otherwise is wrong. I don’t think that this particular thing has to do with like external validity like what will people think of me? I mean I think they really believe that they have the right answer. But they just can’t –it’s like very hedgehoggy, right? They just can’t say well, why would this person be telling me that I’m wrong, or is there another way that I could think about this problem or let me go back and do the math you know.

Adam: 00:24:43I  know. That makes sense.

Annie: 00:24:46Yeah. I mean, which obviously if you think about like conspiracy theories like, they’re not doing the math. No, it’s not… they’re  like, okay, so I’m saying that there are three million people who have kept a secret. And not one of them has ever let it slip, right?

Mike: 00:25:05There was no moon landing.

Annie: 00:25:10Think about how many people would have to be keeping that one a secret. There’s just stuff like that and then there’s just like illogical, right? Like vaccines as far as I can tell will both kill you but also we shouldn’t be giving them to any other countries because they should just be for us to cure us, and it’s like okay you just said they were going to kill you but they’re also going to cure you at the same time. Like that doesn’t jive. You know what I mean? Like there’s so many things like that where you’re going and from the outside looking in you can say this completely doesn’t make sense. But it’s a little bit like trying to tell someone that a dollar minus 10 cents is 90 cents right and they’re like, shut up.

Mike:00:25:55So how do you enlighten that person? How do you bring them along the trainability dimension?

Enlightening the Illogical

Annie: 00:26:04 It’s such a good question. There’s a few things. Remember I said some things about like create a really good decision making environment for people, that’s always really helpful. Part of that has to do with like discover information not in a group setting, so that you can actually see the perspectives of other people and get their rationales. There’s like a very famous question where you can see this happening, where it’s actually a little bit difficult. I’m sorry we’re getting down in the weeds on this stuff but I think this explains sort of why it’s really important.

Mike: 00:26:36We love the weeds. Just keep talking.

Annie: 00:26:39Okay. All right. So a woman comes into a jewelry store and buys a necklace, but she doesn’t have cash so she gives the store keeper, the jewelry store owner a hundred dollar check. He happens not to have any change lying around so he goes to the storefront next door and he gives the storefront next door person a hundred dollar check. And that person gives him a hundred dollars in change. Now he comes back and he gives the woman. The necklace was $78, he gives her $22 in change. Okay, the necklace cost –his cost for the necklace was $39 and now later on he finds out that the check was counterfeit and he needs to give the person next door a hundred dollars which he does. So the question is how much money did he lose? Do you want me to take a moment? I can stop and not just give the answer away.

Adam:00:27:42I think it’s $61 dollars.

Annie: 00:27:43Does anybody else have a different answer?

Mike: 00:27:52I think he lost $100.

Adam:00:27:54 Oh, sorry not $61. Was it $100?

Mike 00:28:00Well, he lost the value of the necklace and the $100. So, $139.

Adam: 00:28:04$139. Yep. Good call.

Annie: 00:28:07What do you think Brian?

Brian:00:28:10 I was gonna say $139.

Annie: 00:28:14Okay, so this is why we never discuss anything in a group. Okay, so the answer is 61.

Adam: 00:28:18Damn it.

Brian: 00:28:22You know Adam’s always the smartest guy in the room and I should just trust my instinct. All right. Yes, like I said 61.

Annie:00:28:30 I’ll tell you why. He got a hundred dollars from the person next door and gave him a hundred dollars back so that’s zero so we don’t care about that. The necklace cost was $39 and he gave her $22 and changed so it’s 39 plus 22 is 61. Okay, so here’s the thing. Imagine this is how we get these great decision-making environments. Instead of having that discussion like this, each of you had gone and done the math yourself. So you had written it out so when you said 61, you would have said all the stuff that I just said and you would have said your thing for 139 and we did it all separately, and then we came back and we just got to look at everybody’s answers, and then I could say well, hold on how come you think this? And we’re just much more likely to get a better answer number one and number two we get exposed to the fact that people do have actually a different point of view and we get to get their rationale on it.

So that’s one of actually the best ways that you can train this kind of thinking, is just allow people to get the maximum exposure to other people’s points of view and their rationale for those points of view so that you can understand there are different ways to comment something. And now for this there’s actually literally a right answer. But for a lot of things there’s you know a matter of subjective judgment, right? We could have a model of the market and both of us could recommend different things. And we just want to get those perspectives, so that we can see that our perspective isn’t the only one. And that’s actually both one of the best ways to improve decision making. Number one but also one of the best ways to train really good decision making where you start to learn to want to know how other, somebody else views the situation because you have enough times where you walk in the room and you’re like, I think it’s 139. And then you see the math from somebody else and you go hold on, it’s 61. And then you go, oh you know what I’m not always right. I ought to listen to other people.

Mike: 00:30:17It reminds me of the you know sort of the jelly bean experiment too, where guesses are kept separate from one another and distinct and don’t influence one another whereas if you do that in a group setting, all of a sudden whoever goes first anchors it, right.

Annie: 00:30:33You say it’s a hundred and everybody’s trying to figure out if it’s more or less than a hundred. Whereas if you said a thousand everybody would figure out if it was more or less than a thousand, and you would see that you get a spread. So yeah, so this is a way to actually use the wisdom of a crowd for complex decisions that don’t just have to do with like how many jelly beans are in a jar, but who should we hire? Where you can see that people look at things in a different way. There’s another interesting piece of that just by the way just on noise reduction, which is in Kahneman’s new book which is you can use an inner crowd. So one of the things you can do is write down your guess from a week ago and then guess again a week later, and you’ll see that those are often different. And it turns out if you take an average of those, you’re more accurate than either guess on its own. It’s called the inner crowd, you get to use the inner crowd. But that also teaches you that like, I did think different things on different days.

Adam: 00:31:29That is remarkable. Wow. So, I guess it’s–so your memory of the thought process that you brought to bear several days ago is weak enough that when you come back to it, you’ll use a different process or you’ll place different weight on the information that you have at your disposal or what have you and ensembling with yourself, is effective.

Annie: 00:31:55It’s actually really effective and it’s kind of to take what you said more broadly, our memories period are pretty bad . So I mean that’s kind of the whole thing with hindsight bias, right? It’s like you’re looking back on something and your ability to reconstruct your state of knowledge at the time is terrible. You know your ability to not mix in things that you found out afterwards into things that you knew beforehand, is terrible. When we reconstruct conversations we have had with someone, we reconstruct them in a way that fits a narrative that we care about and we don’t even know we’re doing it. Where our memories are just very bad because we don’t have a search function that’s like a computer where it’s sort of date and time stamped somewhere, and it just gets retrieved as the data artifact. I mean our memories are very contextually driven and so context matters and we create context all the time, and sometimes that context is just previous beliefs that we have that we’re trying to support.

Sometimes it’s like just what time of day is it or is there a mood or something like that that can cause you to have different guesses, or what were you just anchored to so when you’re looking at the jelly bean jar and you guessed last week and you guessed today, you may come up with different answers just because like you just saw something with a bunch of stuff in it. We’re cognitively just kind of like pushed around all the time. So once we know that, you know then the inner crowd becomes not so insane, but you know obviously you need to know that otherwise you would be like why would guessing at two different times help me? And it’s because you’re well your guesses are going to be different and they’re both inaccurate in different kinds of ways. So if you average across them you do and sort of in the same way that if the two of us guess, you know or the four of us I’ll guess, we would be inaccurate in different ways and if we took the average we would generally do better than any of the single guesses.

Mike: 00:33:46Where does that fall in the in the book How We Decide, which is just an exceptional set of tools that you propose for everyone to use and I’m trying to figure out, does that reside sort of in the knowledge tracker or is that more the– where is that covered in the sets of tools in that book? Because they’re really –

Annie: 00:34:06It’s not in How We Decide because that’s written by Jonah Lehrer. It’s in How To Decide.

Mike: 00:34:15I’m sorry.

Annie: 00:34:18It’s all good.

Adam: 00:34:19Dude, that’s your third yellow flag man.

Annie:      00:34:22Yeah, he gave away the answer.

Adam: 00:34:25Yeah, there was one–two with the before the pre-talk, nine was correct. That’s right.

Brian: 00:34:22Mike’s going to disappear through the floor.

Mike: 00:34:36Well, I also learned that if you say something with lots of confidence in that book that you can persuade other people of a wrong answer so I was just seeing if that is true.

Annie: 00:34:43That is so true.

Brian: 00:34:44:He got me.

Annie:00:34:47So this actually fits into a few places. I mean obviously the not– so just so that for people who aren’t familiar, the knowledge tracker is meant to be a way to look back on decisions and be better at reconstructing what your state of knowledge is. So essentially the biggest problem that happens with reconstructions is that we get memory creep. In other words like we learn a whole bunch of stuff after the fact and that sort of starts to creep into our memory of what we knew beforehand, you know. And like the simplest example of that would be like, you’re thinking about taking a job you ask my advice. I tell you all sorts of things you know which are like, it seems like a pretty good opportunity, it seems pretty close to this other opportunity that you’re already in or this other one that you’re thinking about and you know whatever, but nowhere in there do I say, you should never take this job, it’s gonna be a nightmare. The toxic, the culture is gonna be toxic and whatever. And you go and you take the job and it’s a nightmare and the culture is toxic and we have dinner later and I say I told you so. I knew it was going to be bad. And you’re like, I don’t remember that but okay.

So this is kind of what happens to us. So the knowledge tracker is basically saying look, you have to think, what did I find out after the fact, and I can you know, you’re trying to create that time stamp. What did I find out after the fact? What did I know before the fact? And given the knowledge that I actually had before the fact, what is the quality of that decision given that knowledge, right?

So, as an example, like you know the thing I open Thinking In Bets with Pete Carroll in the super bowl. He has no knowledge whatsoever that Bill Belichick has been practicing a play that would be a play where you might intercept the ball in that particular situation. Why do we know he didn’t know that? Because Bill Belichick had never ever run that play. There’s no tape. There’s nothing. He’s never seen the formation. There’s no reason that he would ever know that, and yet what people will tell me is well, you’re wrong that Pete Carroll made a good decision, because Bill Belichick had been practicing this play. It’s like, well, this is why you need a knowledge tracker, right? Did Pete Carrol know that before? He never said. If you fall for it a second time you know that’s like fool me once, fool me twice. Okay, so this is helping you do that but what I point out is that, but the thing is that you can do this knowledge tracking but if you actually record your knowledge before you actually enter into the decision, that’s actually first of all going to improve the quality of the decision.

But second of all it’s going to help with this retrospective look back. Because you’re not going to have to sort of reconstruct something. You’re actually going to have a record of it. That’s where this kind of thing comes in place. Because if we all take time to write down how much do we think that that jewelry store owner lost, right? And we all take time to write down what’s the math? You know what were we thinking? So on so forth, and now we have a record of what we all thought. Maybe we all decide from that. We’re more likely to actually figure out it’s $61.  But even if we don’t and we all come out of our group and we, poor you, we convince you that it’s $139 and we come out and then someone says you know $139 is wrong, it’s actually $61. We can now go look back and we can say well, what did we know? What was the math? What did we get wrong? Where was the error? Because we actually have it written down somewhere. So that’s where it sort of plays together. It’s a way to make it so that you don’t have to reconstruct it.

Mike:00:38:19And you don’t mis-reconstruct it too. Sorry.

Adam: 00:38:26No, it’s good. It’s just this sort of dovetails into something that we talk about a lot in terms of decision making which is, I mean Mike and I and all of the partners and many people we’re all big fans of your work but just like decision theory in general, and metacognition and we’re always trying to figure out how can we just make better decisions independently, as a team, etc? But you know we’re all familiar with the literature and we’ve read all the books and we’ve followed a lot of the protocols. And what we’ve discovered is that, and this is well documented too, right? Like you can know all of the rules, you can know and internalize the reality of all of these biases, but when it comes time to make decisions and reflect on them, knowing them does not make you invulnerable to them, right? So what are some of the metacognitive hacks? Because a lot of the like– you’ve written some great books, you know Kahneman has written some great books, Tetlock’s written some great books. And a lot of them sort of described the path too … and you’re in there. Forget it. You’re there.

So the, but the bigger question is or another question is, how do you make these things work, even though we know that when we know about them they’re still not guaranteed to work, right? And it seems like and you just described one of those which is document, right? But what are some of those? What are some of the other sort of tricks or hacks or tools that you can bring to bear to ensure that the knowledge that what we learn about how we think is, meaningful in how we operate going forward? Rather than just being something that we know, because becomes meaningful in the decisions we make and the lives we live?

Tricks of Metacognition

Annie: 00:40:27 Yeah. So let me just say a few things. One is, in all sorts of ways knowing about a bias doesn’t help. So I’ll give you an example. Most people are familiar with the sunk cost bias, sunk cost fallacy, so that’s just like–here’s an example. This is actually out of my new book. So, I tell you, hey, Brian what’s your favorite band?

Brian: 00:40:58Let’s just say U2.

Annie: 00:41:01Okay, there you go. Don’t ever drag me to a U2 concert please. I’m joking. I’m joking but also I don’t like U2.

Adam:00:41:12Sorry, not sorry.

Annie:00:41:12Sorry not sorry. But I’ll be over at the Jack White concert and you’ll be at the U2 concert. But anyway so U2. So U2 is doing an outdoor concert and I say to you hey, do you want to go to this outdoor concert that you U2 is doing? And you look up and the weather is gonna be freezing cold and rainy. So you know you’re gonna go to this outdoor concert, you’re gonna be standing around in the freezing cold and rain to watch them, and under those circumstances most people say, I’ll take a pass, right? Even though I’m offering it to you for free, I’m saying I got this extra ticket, do you want to go? And you’re like, no, it’s going to be like 32 degrees and ice cold rain in the middle of June for some freak weather experience . I have no interest in standing around. I’ll listen to the album.

But now imagine that you bought a ticket that’s non-refundable for a hundred dollars. So it’s non-refundable, right? Like you already bought the ticket and it’s the exact same situation and you have a choice about whether to go and stand in the cold and rain, and it’s going to be gross and freezing to go watch U2. And pretty much 100% of the people are like, yeah, I’m obviously gonna go. You’re like, okay, but you already have a state of preference that you don’t want to go and they say, but I don’t want to lose the money in my ticket, right? But okay, you already lost it. So that’s like very simple, sunk cost fallacy, right? So whatever your preference is if you wouldn’t go if it were free, you shouldn’t go just because you have a non-refundable ticket.

Brian:00:42:49For the record  I would have not gone and I would have eaten the hundred dollars.

Annie: 00:42:53Good. Absolute work.

Mike: 00:42:54You’re the guy offering the other guy the free ticket.

Annie: 00:42:59That’s right.

Brian: 00:43:00Mike, I’m actually emailing you and said you want my ticket, and you want to go.

Adam: 00:43:04Mike would totally go.

Annie: 00:43:12Most people obviously–most people are going but you know I can make it much less obvious and I could say well, it’s just going to be like 58 degrees and kind of windy and yucky and you know, whatever. But most people are gonna go in that situation and this obviously causes a whole lot of problems that have to do with projects that are going on too long, or like there’s big public works projects where this becomes a problem because money’s already been spent, so they continue even though it’s going poorly. So this is what people tell me. This is like if I had a penny for every time someone has said this to me, I would be rich. Oh I know about sunk cost and I actually don’t do it. Why? Well, because this is what I do, when I’m approaching the decision I say, if this were a new decision, would I make it? So you could imagine this with a stock right. You’re trying to decide whether to hold a position or not and you say well, if I had to buy this today, would I buy it?

So we think oh, okay, I found a great hack, it will work. So you know, that works not at all, like zero. It has no effect. But the problem is that because we think we’ve discovered something like, oh I understand saying something about sunk cost. The error is that people treat decisions in which they already have resources sunk into it differently than new decisions. So I know that and I’m a genius and I’ve come up with this fresh way to deal with it, which is to imagine am I if I were making the decision today would I do it? And so they think that they fixed it and they haven’t, and then that’s a really bad combination ,something that has remained unfixed but that you have high confidence that you did fix.

Okay, so this is just to say, when we’re in it, when we’re in the moment of decision, we’re really bad. So there’s two ways that we can actually help this because our intuition around this stuff is so bad that if we know it, we’ll fix it. One is to do it in advance, that’s one way to do it. So I could say, what would the weather have to look like for me not to go to this U2 concert that I’ve just bought this ticket for? And I could figure that out and then I could look at what is the day that the concert’s gonna occur? What is the probability that the weather will be below my benchmark, right? And  am I willing to take that risk for this non-refundable hundred dollar ticket? And then it having written down. If it’s below 52 degrees, I’m not standing out there, right? Then you are actually more likely not to go right?

Adam: 00:45:43Do you need to write that down before you buy the ticket?

Annie:00:45:46Yes you have to do it at the, before you enter into the decision, right? And that’s true of anything whether it’s a project or whatever. This is you know on Everest they have turnaround times, right? Like which just say if you’re not at point A at this time, you got to turn around and go the other way. So these are things where you’re just making the decision in advance. This is the way. And this doesn’t have to be just like, I’m not going to go. It could be, what are the circumstances under which I would continue? So like for example, if you have a position, you can say let me think what are the things that would cause me to press? What are the things that would cause me to hold? What are the things that would cause me to actually take the position off or partially take it off . These are all things that you can do in advance. So that’s one way that you can solve for this.

The other way you can solve for it is to get to the outside view, which is to say find out what somebody else thinks without telling them what you think first. Which is a little bit what we just did as an exercise, right? So this is one of the best. It does two things. It really helps to de-bias but it also helps to reduce noise. When we say let’s all go write down, what do we think the answer is and why? And we have some structure to the why, and this could be true whether it’s a hiring rubric right where we’re saying here are our values, this is what we’re trying to hire for, let’s all make a judgment about how good of a mentor is this person going to be? What is the quality of their coding? You know how good of a culture fit are they? Whatever, right? Just everybody makes their subjective judgments but it’s structured, in a way that we’ve decided in advance what the structure of that is so now we’re merging them. We decide in advance what the structure is and then we all do it independently.

And now all of a sudden you can get some of this really good de-biasing. Now what that means is that you have to get to that second order knowledge. I know about the biases. I know it’s bad. And one of the things that I know that’s sitting on top of that is that I can’t undo it by sheer force of will or some sort of Jedi mind trick. So therefore I must set some things up in advance that will help me to do this and I must put some structure to my choices. Now notice though what’s interesting, is that all of that causes you to write stuff down, right? Automatically you’re keeping a record, whether it’s the record of what we all thought about the necklace or the record of what my forecast was about when I would not go to the concert. All right, we’re done. I’ve solved everybody’s problems.

Adam: 00:48:21Yeah, that’s right.

Brian: 00:48:23Except for the fact that you don’t like U2 but we’ll take that offline.

Adam:00:48:29Brian is still reeling. I’m with Brian. I love you U2, man.

Annie: 00:48:31I know. You know I’m not a stadium rock person, that’s all. They’re great as far as stadium rock goes. Like if I were listening to stadium rock, they would be who I would listen to. But I’m just not a stadium rock person. When I was in high school, I just listened to the Grateful Dead all the time.

Adam:00:48:47I was going to say you’re a Dead Head for sure.

Annie: 00:48:52I don’t really listen to the Dead anymore except for nostalgia, but you can see how the Dead transforms into Jack White as a more edgy and you know adult version of that.

Adam: 00:49:02I’m tracking. I’m tracking that. So you just sort of described a set of tools that help to manage through the sunk cost fallacy.

Annie: 00:49:21Well, it will help through a lot of things. It helps with– it’s going to help you with over confidence. It’s going to help you with availability bias. It’s going to force you to go look up base rates. It’s kind of all in one.

Adam: 00:49:29That was my question. That’s a that’s a generalizable toolkit that you can bring to bear against a lot of different biases.

Managing Biases

Annie: 00:49:38Yes, exactly. And you know one of the best sort of principles being, you know I mean, this is this idea of like let’s get our feedback independently, is that groups stink. We think groups are great for de-biasing because we think oh, we’re getting other people’s opinions but mostly if you talk in groups you’re just making bias worse. So they’re horrible. They’re all sorts of like we talked before about you know the anchoring and things that come out of that, right? So it’s just like, get to the outside view. I mean that’s the thing that you need to do. You need to hear what other people have to say even if it’s the past version of you, or you on different days of the week right. Like we have to get out of the perspective of the moment as Kahneman said to me and I think it’s such a great statement. Find someone who loves you but doesn’t care about your feelings. What does that mean? Find someone who really wants you to have a good outcome, but isn’t worried that you’re going to be sad because they told you, I see it a different way or I don’t think it was $139. And I’m willing to tell you.

Mike:00:50:40 You as a recipient have to be willing to hear that though.

Annie:00:50:42Well, of course. That’s why he’s saying find someone who will do that for you.

Mike:00:50:46And yourself. you have to listen to that someone.

Annie:00:50:50Now I do have a trick that separates– now obviously if you can find someone to do that that’s great. But I have a trick for people who don’t want to hear it from you, and you actually would like to them to still be your friend, is one of the things that I’ll do is with people is I’ll just sort of shift it to the future. So the thing that people are defensive about are things that they’ve done in the past. You know particularly when they’ve acted on certain beliefs or whatever. So when somebody’s sitting like—I have a funny example in how to decide where someone’s talking about like, oh the last 10 people I went on a date with are such jerks, right? And of course every single person who’s ever been in that conversation is like, maybe you pick really badly or perhaps you’re a jerk. Really you hit 10 in a row? Like first of all, maybe you think that things are jerky that aren’t, maybe you’re too sensitive, maybe you’re without– like we all have that. Nobody actually thinks—

Adam: 00:51:42The point is it’s you, it’s not them, right?

Annie: 00:51:45 What do we all do? Oh, man. Because they’re like your friend. So the trick is instead of–now Kahneman’s friend would say, are you kidding me? They’re not. Let’s be honest here, Danny. They weren’t 10 jerks in a row, right. Okay, but that’s when you’ve invited that in but what happens when you haven’t? You just say, that’s horrible. I’m so sad that you dated all those jerks. Like when you go out on your next date and you’re going to pick somebody to go on a date with, what do you think you could do that would make it less likely that they’ll be a jerk? Now notice that in order for them to actually properly answer that question, they have to go back and think about, what was my behavior on the date? How did I pick? Was I just like swiping right on Tinder? Is it right or left? I don’t know. Whichever way you swipe on Tinder,

Brian: 00:52:40Not touching anyone.

Adam:00:52:43Stop virtue signaling, honestly.

Annie:00:52:44No, I’ve actually never used it. The fact that I haven’t used Tinder is. Actually– just tells people how old I am sadly. I’m sure if I were young-

Adam: 00:52:55I’m just kidding.

Annie:00:53:00But luckily I’m married and don’t need it. But anyway so whichever way you swipe that’s like this the good swiping thing or whatever. If it’s like oh, I was just swiping right on people who were showing pictures of themselves in the gym, right. Maybe I shouldn’t do that anymore. Nothing against people who spend all their time in gyms by the way and put pictures on Tinder of them doing gym things. I’m sure you’re lovely.

Adam: 00:53:22While listening to U2.

Brian:00:53:26Obviously listening to U2 blasting stadium rock.

Annie:00:53:28Stadium rock, exactly. Oiled up. But the point is they have to go think about those things. So instead of you just saying like maybe you’ve been really picking badly, you just say what are you going to do the next time to make it less likely to happen and that, then they circle back on their own and their hackles don’t get up in the same way and they can usually hear you a little bit better.

Adam: 00:53:55That makes a lot of sense actually. And Rick Haynes is on the comments and he’s asked a couple of questions that makes me think that he really wants to get to the bottom of this thing. So the first question—

Annie:00:54:06Bottom of what? Why I don’t like gym shots?

Mike:00:54:09No, it’s much worse than that.

Adam: 00:54:13So he says, I want to go to a concert Monday. I looked at the forecast, the probability of rain is 60%, what do I do? I paid a hundred dollars and it’s not refundable?

Annie: 00:54:26Well, wait, so can you make the decision on Monday?

Adam: 00:54:32Exactly, like it’s got to be made. You’ve got to have the criteria set before you buy the ticket, it is the point, right?

Annie: 00:54:38Well, first of all, you have to have the criteria set before you buy the ticket but in this particular case let’s say that you’re thinking about Monday, today’s Friday. If you have to travel to go do it right, it would have been good if you would decide that beforehand. But if you can decide on Monday, you can actually set the criteria today. That’s the good news. Is that you basically can do this. You can sort of get that advanced planning. Anytime you’re thinking into the future, you’re better. So you can say well, I’m looking and the probability of rain is 60%. I’m going to be prepared that there’s some chance that I might not go, particularly because I would go in some rain. Like if it’s sprinkling, I don’t care. If it’s driving rain, I may care. So let me actually sit down on Friday and say, what would the rain have to look like at that moment, right? And you have to set a time right? Because I have to leave at a certain time. So if at three o’clock this is what it looks like, then I will not go. So this is actually an interesting little trick because you can kind of think about, the one insight that’s true from the sunk cost problem is that we do want to think about the decision as new.

It’s just that if we’re actually deciding in the moment we can’t do it, but what we can do is say, well, now I’m going to sort of start like a new chapter and I’m going to make some– I’m going to set the benchmark for some later date. Which by the way secretly none of us ever think will actually happen. Because I’m like, you know, how is it June, 11th right? Like whoa! So we’re very surprised that the future actually occurs. So this is something actually that I was told by Ron Conway is the founder of SV Angel. He said when he’s working with them, you know founders, who are reluctant to shut something down, like you’re never going to convince somebody in that moment that they can’t turn it around. Because they’re all going to tell you no. But Angry Birds was Rovio’s 51st game. You know I know we can do this.

So he recognizes that and he says look, you know they didn’t set out the criteria when they started, fine. But now today, I’m not going to try to get them to shut it down today what I’m going to say is, okay, I agree with you. Like you’re totally going to kill it from here on out. Let’s see, what does your growth and error have to look like by, let’s pick a date. How many customers do you have to have acquired? Let’s pick a date. So what that does is it allows you to get the same–I mean obviously you’re going to delay it because if you’re at negative expectancy, you’d prefer to shut it down that day. But given that you can’t, what’s the worst mistake? Okay, I’m not going to get you to shut it down today. Let’s pick a date and say, what do you have to have achieved by this date? And if you haven’t achieved that, then let’s agree then that it’s time to do it.

So that’s basically what you’re doing in this case, right? You haven’t set out. Here are what my criteria are at before I buy the ticket for doing this. But now it’s Friday. I’m looking. I see on Monday it’s going to be raining. Let me actually set the criteria now because I know I’ll be more rational today than I will be on Monday.

Adam: 00:57:37Okay, that makes a lot of sense. So I want to make sure that we also touch on your experience working with people in the investment business, right? I know you’ve actually consulted for a variety of you know big name highly successful hedge funds and top investment firms, and so I’m just wondering you know what sort of patterns you’ve observed? Like do some types of organizations kind of load heavily on certain types of decision pitfalls or do the most successful types of organizations have certain decision making processes or structures or characteristics that really stack success in their favor? Like any broad strokes that you can kind of paint a picture for people who are trying to make decisions in these organizations?

The Broad Strokes

Annie: 00:58:34Well, it depends on kind of–obviously there’s a lot of different types of finance so let me just give you some stuff. There’s a lot of errors that people in high frequency trading, sort of options trading make, but the thing that they don’t make the mistake of, is trying to really model the data. And there’s obvious reasons why that is. The data is incredibly abundant and you know it’s like you’re getting the feedback right away. So you have a super tight and closed feedback loop and a lot of motivation particularly because the margins are so small that it’s just harder to fool yourself into thinking you know something that you don’t know.

That being said, those organizations do still tend to make a lot of errors. They still get stuck in their theses, and we’ve seen you know obviously we’ve seen recent examples of that. They don’t adjust their model according to changes in the world fast enough. They don’t think about their exits in the same way that they think about their entries, and that’s obviously a feedback issue. You naturally track all of your entries in a way that you don’t track your exits. So those are common problems in that area. In long/short, on a smaller scale but still on the same scale as in venture, what I find is really interesting is that I would say that the biggest error that people make there are saying, the feedback loops are long so therefore there’s no reason to forecast anything. Because I’m not going to find out for a really long time. And it’s like as if I’m not getting the feedback in two seconds that, what’s the point of doing the forecast and writing these things down? There’s a couple of problems with that.

The first thing is that a lot of times when I get brought in, it’s because they’re trying to answer questions where I say to them, well, show me the reasons that you made these decisions. And they’re like, well, we don’t have that, we didn’t record it. And I’m like why not? And it’s this well, why should I record something if I’m not going to know the answer for five years? And it’s like well, because five years later it’s sort of — oh my god, it’s June 11th. Because five years later you may actually want to know the answer to what was my thesis at the time, and so it would have been good to record that stuff. But separate and apart from that, it’s just an error in the way that you’re thinking about the world. Because nobody has ever made an investment, even if it’s not going to resolve for five years or even a year, where that investment didn’t rely on a prediction about intermediate states of the world in between the moment that you invest and the moment that you exit. It’s not like you invest and then the world stops for five years, right?

So simply put like, let’s say I’m making an investment in in a particular software company that’s leading edge. I’m clearly making projections about what the user appetite is going to be or their ability to execute or you know, how much the world is going to be looking for productivity software or whatever it is. There’s all sorts of stuff that I’m predicting in the simplest sense. If I invest in a company at series A, I am most certainly predicting that it will fund at series B, even though the ultimate exit doesn’t occur forever, right? That is clearly included in my forecast. And we just don’t take the time to figure out what are those things that would allow us to have those faster feedback loops so that our life looked more like an options trader or frankly a poker player. And it’s because we’re– you know first of all, I think the work is hard and people don’t really want to do it. And so you’re just coming up with these rationales for, well I don’t need to do this because the thing I’m predicting is too far away.

So you know and I include these interesting conversations so like on the, should you write down your forecast of series B? The initial reaction that I’ll get generally is something to be effective but I’m not investing them because I think they’re gonna fund at series B. So why should I forecast that? That’s not what I’m trying to achieve here. And so what I say to them is, oh, that’s interesting. So have you ever had a successful exit where the company didn’t fund it series B? And they say no. And I said well, then you’re forecasting that. So I think that the real hard work across any type of finance is to say, I have to have the discipline and not trust the world to discipline me in the way that it would in options trading, where the world is just disciplining me to make much more explicit the implicit assumptions in my choices, because I just have to.

But treat everything like that and say, there are things that are implicit in every single decision that I’ve ever made. I am better off making those explicit, writing them down, working with other people in independent settings and then coming together and discussing them as a group saying, here is my thesis. For my thesis to be true, here are the things that must develop in the world. These are the states of the world that would mean that I am right or right enough to make money because we’re never 100% right. And if these things changed in the world, that would mean that I need to change also. And we’re just not doing enough of that. And frankly I honestly and this is no aspersions, because I’m this way too, it’s because we’re just all cognitively lazy in the same way that evolution in general is just lazy. We’re all just sort of energy, you know, hogs. We don’t want to spend any energy, we want to conserve it all. And this takes energy. And even in organizations that I’ve worked a long time with where we have very, very structured decision processes, occasionally there’ll be some insane decision that’s made on Slack where I’ll just come in and be like, what happened? You just had a Slack conversation? Like there’s a thing over here we’re supposed to do. You’re like, oh, sorry.

Adam: 1:04:25Yeah, that’s hitting a little close to home, Annie.

Mike: 1:04:27Yeah, preach, preach girlfriend.

Annie: 1:04:33I apologize.

Adam & Mike:1:04:33No, no, no.

Annie:1:04:40You have an organization and you have a very structured decision process and then someone just says in a meeting hey, what are we going to do about remote work? And all of a sudden you have an in-person group discussion about something that’s incredibly important to your organization. Like it just starts and nobody says, hold on a second, let’s all be quiet. Let’s figure out what we need to know. What’s the information we need to have? What opinions do we need to elicit? What functions do we need to go talk to the leadership of? What are the subjective judgments that we’re looking for? People don’t like to– it’s just hard.

Brian:1:05:17Well, I think it also you know touches on the importance of culture. I mean not to add another level of meta but it’s like meta cognition like there needs to be kind of a cultural commitment to even engaging in these conversations. And we’ve talked about this for hours. It’s exhausting to go through these exercises like think about every writing down conditions for you know, how you’re going to approach decisions in a week or a year? Nobody’s going to end up doing that. So you need a culture that’s committed to that. There needs to be not just an individual, but you know sort of a way of doing things. And then you boil down to okay, we need to make decisions about these sorts of decisions. And then you need to make the decisions.

Annie: 1:06:59Never been like–it’s rare. I can count on one hand the number of times that an organization has hired me where everything is going just fine. I’ve had people talk to me when things are going fine and then be like oh, that’s really interesting, I totally think we should do that. And then I don’t hear from them and nine months later, they’re like our world fell apart. Come help us. Because you don’t really see what the cracks are until the world tells you that there are really big cracks. And then you go oh, no, things fell apart. Because these are things that take intention. And I think that unless we’re told that we need that intention in some way that’s really serious and you know very often an existential threat, then we don’t necessarily go and take the time to do it. I will say just kind of going back to what Brian said. I just want to be clear that there are lots of decisions that you can make super fast. And in fact, I would say one of the mistakes that I see is that people will spend a lot of time on decisions that they should be making quite fast, and essentially no time on decisions that they should be making quite slowly.

Adam 1:07:16So how do you identify in advance the ones that we should be making quickly versus the ones that should be more deliberate?

Making Decisions, Fast or Slow

Annie: 1:07:27All right. So, I’ll give you the framework. It’s just options and impact. And obviously options, they’re related to each other because optionality reduces the impact of realizing the downside. But if you take impact it’s like, okay what if this goes poorly how much do I care in the long run? So it’s interesting. Like people really worry about like should I buy this one stock? And it’s like if that stock goes down, you don’t care much in the long run assuming it’s in a well-constructed portfolio. So it’s like, once you kind of are like well, this is what I want the construction of my portfolio to be, and these are the sectors that I want to be in, and this is sort of how risky I want the stock to be. Once you’ve sort of done that, just like choose among a bunch of them. Like don’t spend a whole lot of time like worrying about A versus B. Once it’s already past the threshold because not that one stock isn’t going to affect you very much. The construction of your portfolio really matters, but the one stock probably doesn’t. What you order for dinner–like I mean we know those people who are like pick me last. What are you having? What are you having? What are you having? But this is the idea of like a minimum viable product right? It’s like release it to a handful of customers something that’s really stripped down. If it messes up, you don’t care because only five people got affected. So just do it fast. Don’t try to make it perfect. And put it out there and see what the world tells you, and that’s going to speed up the whole process. So that’s that impact side of things.

And then there’s optionality which we know like the more liquid the investment, the less work you have to do on it. If you’re going to tie your money up for 10 years, you ought to really think about it, particularly if it’s a lot of money which has to do with impact. But if you can move in and out of the investment, assuming that you’ll actually do so right, that you have your kill criteria set up in advance, then you don’t have to– you just have more room for error then and you can go a little bit faster. It doesn’t mean that you shouldn’t have a disciplined decision process. It just means that you don’t need to do as much in order to do it, if you can hedge, right? Like the more hedgeable something is, the faster that you can decide. Because you’re just protecting yourself against not getting it quite right. A silly example would be, if I’m climbing a mountain and I have ropes, I can go a lot faster and not worry as specifically or as much about the handhold that I’m about to grab to, because if I fall I’ve got ropes. But if I’m free climbing, I’ve got to map that route out and I’ve got to practice a lot more and I have to do a whole bunch of other stuff because if I slip, I’m dead.

So you know the ropes are just a hedge. So the more that you can hedge, you know you don’t have to go quite as fast. And we can actually take that into like a hiring thing right? Like you shouldn’t spend a whole lot of time trying to hire your interns, your entry-level people. You have a lot of them, so you have a portfolio, you just have to construct the portfolio. Well, it’s very, very quittable, right? I mean, those people– it’s very little organization or cultural damage for you to let somebody go who’s at that level in the organization. So you can just throw a bunch of spaghetti against the wall there. I mean, you want to have standards obviously and thresholds, but those thresholds can be a lot lower and you don’t need to take as much time in the recruiting process and so on so forth. But man if you hire a CFO, you better take some time. Because there is a lot of damage to hiring somebody and then letting them go from that position. If that doesn’t work out, it’s got a much deeper impact, much more long-term impact on your organization. And it’s harder to hedge against that. You can’t hire two CFOs in case one of them doesn’t work out. And it’s not nearly as liquid. You can fire them but it’s at a much greater cost.

So once we sort of understand that, we can get out of this 15 minutes of worrying about what we’re going to order off a menu. And we ought to stop saying–you know because I’ve seen this in organizations right. Like they meet someone and hire them without a process because they’re like no, I had lunch with them they were great. And it’s like really you’re hiring someone that’s a C level because you had lunch with them and you think they were great. But they’re like no, I can read people well. It’s like, okay. That’s fine for an intern. I met my friend’s kid, they seem fine, let’s bring them in.

Mike:1:11:49Yeah and then have the stop-loss which is that you know, they’re in their probationary period. They didn’t work out.

Annie: 1:11:55Right. Exactly. Which is why you stopped out your friend’s kid.

Mike: 1:12:01Yeah, exactly. In your new work and the work you’ve done, I’m fascinated by how you think about and construct the experiments that lead to the uncovering of the various biases and opportunities for improvement. And I know that the sort of the new work is quite– is more rare and I’m wondering if can you help enlighten us on how you think about those things? How do you construct a an experiment that’s going to explain the revelation?

Discovering Opportunities

Annie: 1:12:39I can give you a couple, an example. And thinking about the work that I do on forecasting where we’re asking people to forecast you know when probabilities, and you know certain outcomes and we’re using simulated environments so that we know what the answer is, because we can just run Monte Carlos. But it’s basically looking at, when you look at what good forecasting is, what are the skills that you need and what’s the outside view? Well, you need the base rate right? If you to be a good predictor, you have to actually be able to sort through patterns. So how can we train people on that? How do we help people understand when reversion to the mean is going to happen and when it’s not?

So like as an example, if there’s an accident on a highway at two in the morning that causes it to go down to one lane, it’s probably not going to affect things too much. But what I need to understand, is that going to be cleared by rush hour or not. Because if it’s going to be cleared by rush hour then I assume we’re going to just revert to the mean, and at rush hour traffic is going to look like it always does. But if it’s not going to get cleared, then I have to assume that actually I have to change my forecast away from the base rates right, for what traffic looks like at that time? So how do we think about how we would train people up on those kinds of questions? And so we create training around, we basically try stuff out and we say, can we create a module that’s on base rates that gets people to really understand what these are, and that you’re supposed to anchor there? Can we create a model that’s about you know a module that’s about pattern recognition?

And then basically you have people who get that training and people who don’t. And then you have them forecast the exact same environment. In this particular case like a simulated game and you say, what’s the effect? And it turns out we happen to come up with a training environment that has a very big effect. Now you know I’m basically, when I think about you know what are my instincts for creating the materials and the training that we’re trying to test, it’s coming from my work with organizations and my work in poker honestly, it’s like I have just had so many reps of trying to figure out like, what actually moves somebody to make a better decision? What are the things that you can tell them? What are the examples that they understand? What are the analogies that they can transport across different domains? And these things are hard because one of the things that we know is that transfer of training is quite hard to achieve.

So I’ve just been doing a lot of it, and then I just take those ideas and I test them in a laboratory environment and then sometimes it turns out that my instincts are right and sometimes it turns out that they’re not. And that’s the whole fun part is oh, this thing that I thought would work great didn’t work at all. One of the things that I just found out is that no matter how much you train people that you should care about the base rates. When you give them a weird outcome, they don’t care about the base rates at all. So if I tell you that when Brian and Annie play poker, Brian usually wins five dollars an hour from Annie. And I say that’s what– they’ve played a hundred times, over a thousand hours, right? 10 hour sessions and that’s what’s happened. And then I show you a game where Annie won $20 an hour from Brian and I say, what’s going to happen and the next time they play? Nobody says Brian’s going to win five dollars. And you can tell them about base rates and you can train them on it and it’s just very, very hard to get them away from that. So that’s actually a problem that I’m working on right now. I’ll let you know the results in a week. Which is, what can you do if it’s not just about pounding home the base rates that can get them to stop doing that? This is what I find out and I thought that would work. Well, I thought oh, well, they’ll be immune to that problem and it’s like no, they’re not.

Adam:1:16:35That is of enormous economic consequence in the investment industry, and I’m sure in every industry but I mean that’s we work in the investment industry and we see over and over again, and Brian you worked for many years in manager research. So I’m sure you can sort of speak to the recency effect and just how much attention people pay to recent performance versus sort of long term and we all know that like even long-term performance has very, very low signal.

Brian: 1:17:07And short-term might even be a counter signal.

Adam: 1:17:11For sure .

Annie: 1:17:13You know, if you just leave your money and you don’t touch it. You like, look at what happened all the people who panicked in 2008 they got really messed up, but the ones who didn’t they did really well. And then it’s like March of 2020 and you’re trying to tell people that and they’re like no, I’m selling. And you’re like no, but look, this is here historically. This is different.

Adam:1:17:35Have you done any experiments with financial data?

Annie: 1:17:39Not yet but maybe I will.

Adam:1:17:44It seems like a really interesting sandbox for decision making. It’s so clearly– there’s so much data available and it’s such a noisy process.

Annie: 1:17:58There are definitely people who have–Alex Imus is someone who I really recommend people go look at. He did a thing on dynamic trading and he’s found some interesting stuff but you know one of the very classic things that people find in training data is that retail investors. So this is just simple prospect theory, which is just people let their losers ride and they cash out of their winners like super fast to lock in the gains. And you can look across retail investors in large, large bodies of financial data. And what you will find is that retail investors blow through their stop losses and they never actually make it to the take game. They do that well before they ever hit that. So a lot of these things which are biases which you can see in these sort of laboratory environments do actually have actually been replicated across financial data.

Adam: 1:18:48I’m more sort of interested in whether there’s sort of a super forecasting framework that demonstrably works with financial data right or like with manager selection or you know some other decision making process of extreme economic importance?

Annie:1:19:12Well, you’ll have to have Phil on.

Adam: 1:19:13Happens all the time.

Getting Good Forecasts

Annie: 1:79:15You’ll have to have Phil on to talk about that. I will tell you this one thing that’s kind of interesting is, one of the best ways to actually get really good forecasts is to create betting markets in your organization. So just essentially you have a betting market and you know people can put money on either side so they’ve done this–Eli Lilly did this for a while with research where the research scientists got to bet on what projects they thought were gonna do great, and which ones you know obviously that also tells you which ones they’re shorting. So they did that and it was really predictive. It was much more predictive than like the project managers predictions, or anything like that super, super highly predictive.  And so they shut it down. So you might be wondering why.

Mike: 1:20:03Of course they did.

Annie:1:20:06Why? That’s the question? Why? But it’s not just them. I think it happened at Intel as well. Like basically any time that someone’s put this in, the answer is, it’s amazing. And then they shut it down and the reason is that even though a betting market is human beings offering their opinion, when I’m a manager and you do that I say, but what about my opinion? But my opinion is really good. I know stuff. Then why are you taking the decision making away from me? Even though they’re participating in that betting market. So there’s something about that that makes people feel unseen and unheard and not valued and not useful and all sorts of stuff and so they always end up getting shut down because they’re culture breakers.

Brian: 1:20:50That’s really interesting

Adam: 1:20:53Wow. Yeah, that’s an incredible insight.

Mike: 1:20:57–for people to start gaming that potentially in their….

Annie: 1:21:03It’s hard to do you know–it’s like trying- it’s like look, we all know what’s going to happen eventually to GameStop. It’s like you could game it but the market’s going to end up eventually. It can take a while but eventually someone’s going to get on the– you know the right money’s going to get on the right side of that bet. But no, they’re highly accurate. It’s just you can imagine it’s like ,well what am I here for? What about my opinion? Shouldn’t I be able to manage people? I want to make a decision about what projects keep going.

Mike:1:21:37What are they called? Ownership of your decisions, right?

Annie: 1:21:43It really,  really matters to people. I think it’s separate from ego actually. There’s a difference between feeling like ownership and agency that’s separate and apart, I think from ego. But at any rate they don’t work. So instead, what you want to do and want to do this for everything is like, just sort of start to transport some of the things that a betting market gets you right? Well, we’ve talked about those? What does a betting market have? Independence of point of view. So we’ve talked about how to get that in there right? Two, that people are expressing their opinion in a very precise way that’s what a price is. So we want to do that. Like if I say to you, what is the quality of this person’s mentorship abilities right? You’re not just free answering. It’s on a scale of one to seven. And that now I can see in a very precise way what that is and then create some stakes to it. And all the stakes do is create accountability. So the point is that when you’re recording all of this information, whether it’s a forecast or a subjective judgment or whatever it might be, that people look back at it. Not in a way to point fingers and say oh, you know you’re an idiot.

But in a way to say look, your forecasts are slightly off, let’s figure out how can we make those more accurate. So the fact that people know that this stuff is recorded, creates stakes. Because they understand there’s now a record of it that someone might go back and look at and then that also helps with the outside view separate apart from the independence of opinion because they’re like, well, let me imagine when someone’s looking at this later. How is this going to look for them? What do I think might be wrong in here? How am I going to get a good score on it? Which is all that a betting market is, is in the future how am I going to get a good score on this opinion that I’m logging right now?

Mike 1:23:24How do you balance the outside view with you know spectacular leaps in innovation? Nothing magic happens in the outside view. That’s always my challenge with the outside view. There’s no magic there. Like if you were to look at you know, Amazon’s the old example. The valuation is such that they would have to deliver every package on the planet and thus they are overvalued. But then they come up with AWS which changes the whole field and they continue to go higher. And that’s that sort of magic that’s not encountered. I suppose you have some sort of hypothesis about another view but you want to be aware of what the realities of the outside view are in order to help inform that. How do you capture the magic?

Annie:1:24:25Yeah, so that doesn’t actually go–again that’s that part of that training that I was saying which is you have to understand, do I care about the base rates or is this likely to run away from the base rates? So it’s that, is the accident going to get cleared before rush hour or not right? So one of the things–this is why you want to make explicit what’s implicit. Because if you’re going to say, for this valuation to be true, like and you can do a simple pre-trade or back cast right? Like let’s imagine it’s five years from now and Amazon looked cheap. Why was it cheap? And then when you have people do that independently, you start to see things like that right?

So as an example, you know we all know that there was a big shift when we went to you know, physical businesses that only have a certain amount to grow before competition crowds them out, you know versus something like Google which seems to you know this is where we get into power law. So the question is, was that foreseeable? Well, it was certainly foreseeable by some people because they invested in Google. You know at valuations that at the time would have seemed kind of crazy but they could see the future, because you have to say for this valuation to be right, what would have to be true? Well, this would have to be a space where you can just capture enough of the customers and have, create enough friction for someone to switch. That once they captured that value, they wouldn’t release it right? And that’s not true of physical businesses. In the same way but so you can actually end up seeing that and that’s, the kind of art of forecasting is understanding what are the questions that I’m supposed to be asking? How am I thinking about what would be implicit in that outcome occurring? Make it explicit. Explore those explicit things.

And then a lot of times you’re still not going to see it, but you’re going to be more likely to see it. So it’s not that everything reverts to the base rates, that’s not true. Sometimes there are changes like traffic in the road that isn’t going to get cleared, or an accident in the road where you just have to be able to see that. And you can imagine how that is. So let me imagine it’s rush hour and everything’s backed up. Why did that happen? Well, a truck must have you know– a semi must have turned over in the road and they couldn’t get it out. So you can start to kind of understand, what are the assumptions that would have to go into that? And you can do that with Amazon as well. Like for this to be right, what would have to happen? Well , could they do it off of just the packages? Well, no, so what else would have to happen? And then you can start to forecast the probability of those things happening. How realistic are those things? And then you decide, maybe you decide the market’s wrong and you short Amazon. Okay. I mean you know you’re just trying to be more accurate you’re not going to get perfectly accurate.

Mike:1:27:12The answers lie within the questions

Annie: 1:27:16That’s very good. I like that. By the way that sounds so wise. That was very good.

Brian: 1:27:26I just tweeted that out.

Annie: 1:27:30The answer lies within the questions. By the way there’s something called the illusion of explanatory depth. You should go look.

Brian: 1:27:40Okay. I think that’s a really fancy way of saying something very succinct.

Adam: 1:27:46Brian did you tweet that out on your GS elevator account or?

Brian: 1:27:50Yeah, yeah, exactly.

Adam: 1:27:55So Annie, just trying to summarize a little bit on the some sort of really fundamental takeaways. Like it seems like –okay, in terms of organizational decision making, there’s sort of a first step which is, is this a quick decision or does it require some reflection? The quick decision—

Annie: 1:28:21Wait. Let me just say–can I just have a step before that?

Adam: 1:28:25 Of course, please. Yeah.

The Decision Process

Annie: 1:28:26Know what a really good decision process looks like. Because even when you make quick decisions, knowing what a really good decision process looks like will help you make better quick decisions. It’s like a muscle that you do you and then, you also understand what can I get away with not doing? That always has to be the start. But then, yes. Now go to–so that would be like step zero. So now we’re at step one. Is it a quick one or—

Adam: 1:28:51Okay, yeah. So fair enough. So there’s this sort of this meta decision making process at the beginning that sort of formulates how good decisions are made right? And then step one is kind of, is this a quick or a fast or slow decision right? And in order to determine that, you’ve got a model which is impact and optionality right? So there’s sort of a rating system there and then maybe the organization sets thresholds of some interaction between impact and optionality above a certain threshold is something that we want to spend more time on right? And when we want to spend more time on it–

Annie: 1:29:35Let me just say that threshold will be different for different types, for different things . So like that threshold will be different depending on like is it an intern or yes. Okay, good.

Adam: 1:29:45Yes, right. Okay, so that’s it. That’s a good point. So who is involved in that decision right? It seems like even though it’s fast, it could still be–you could still ensemble several votes right? Like you can have– it might require several people’s votes or input on this right?

Annie: 1:30:09So you can literally just go, okay, everybody take out a piece of paper, write your number down. If that’s all you do, you’ll do better.

Adam: 1:30:19Exactly. Okay, cool. And then like there’s an average of those that above which now it moves into the more reflective decision making. And then it seems to me like there’s a Pareto Law there too. Where like 80% of decisions can be addressed coherently and systematically using a fairly limited number of different types of decision making schema right? So you can bring together different like grids or different like whatever– like there’s a limit–there’s a finite number of schema. You decide on what schema is most appropriate for this problem that we’ve decided requires more reflection. We bring that to bear. We bring the necessary resources in to fill out that schema. It all is done independently so that there’s no cross-contamination at the team level. Then the results are sort of collated. What comes next? Because I see a real such a good potential. I see it all the time in our organization where there’s–we form independent opinions. It’s not nearly as systematic or well-formed as what I’m describing. But let’s assume it is. We bring different opinions to the table but then there’s discussion and the discussion contaminates the ability for us to use the information that was derived independently.

Annie: 1:31:41Okay, yes. Okay, so first of all, you have to decide how is the decision going to be made. Because the people that you elicit feedback from are not necessarily the decision makers. So as an example, I just ran a process where everybody on the leadership team gave input but everybody on the leadership team was not making the decision. So the discussion was among the leadership team but then it went to a smaller group. Okay, so you have to decide, is this something that’s like–it has to be unanimous consent? Like in an investment partnership it might be we do this if it’s a majority saying yes, could be one decision maker. So you have to decide that in advance. So that’s separate and apart from the conversation. The reason why I say that is that people who are having the conversation need to understand that the goal of the conversation is not to make a decision in that conversation. It’s not for everybody to agree and to actually make the decision right then. That’s going to happen after what the goal of the conversation is which is to inform.

It’s to inform the whole group and inform whoever is eventually going to be weighing in on that decision. Even if it’s like we’re going to have the conversation then people are going to vote. The goal of that conversation is not to decide, it’s to inform the vote that’s about to happen.

Adam: 1:33:03They need to agree in advance on the decision policy right? So that everybody knows the objectives of each stage of the decision process.

Annie: 1:33:15Exactly. So now what do we do? All right. So let’s take a situation where I’ve said what is– let’s say we’re just doing a simple hiring rubric. Let’s make it really simple. And we have this question about mentorship which I’ve mentioned before, and it’s on a scale of one to seven, how strong of a mentor do you think that this person will make? All right. So everybody’s filled that out along with a whole–you can imagine a hiring rubric, a whole bunch of other opinions that you’ve elicited subjective judgments. And now we bring that together into a sheet– like it could be a Google sheet where I can see you know Brian, Mike, so on. We put this into a spreadsheet and now I can see everybody’s opinion and for each question I can see what number they gave. And now you can broadly do a couple things. One is you can say, here are the things that everybody has a lot of agreement on. This person is a very strong coder, everybody’s got five, sixes and sevens.

So everybody pretty much agrees on that. But here are places where we seem to have a lot of dispersion. Okay, so this would be like for example, I just ran like a remote work process and like we saw this a lot, like there was a question about being in person improves productivity. And it was a huge spread between one and seven across people. But it was it tended to be clumped, some people were sort of like seven, yes, I really agree with that. And some people were like, two, it doesn’t improve productivity. So here’s how it would go. Let’s take that question. So let’s say Brian was a two. Now you need very strong facilitation and I would say, okay, here’s all the places where everybody agrees and you acknowledge that.

And now I’d say Brian, can you just give your rationale for why you don’t think, you don’t agree that being in person improves productivity. And Brian would then give his rationale with no interruptions from anybody else. And Brian’s rationale would be, there’s too much tapping on the shoulder and people are trying to work and they can’t get any blocks of time, because they’re always being pulled into meetings, and whatever. So he gives that reason. And then I say, you know Mike, you were sort of in the same zone, why do you think what you know do you have anything to add to what Brian said? And you may or may not. Does anybody not understand anything from what Brian said? And you notice that framing, it’s not do you disagree. Is there something you didn’t understand in there? Normally, it’s silent at that point because normally people pretty much understand. Then you say Annie, you gave it a seven, you think it really improves productivity, why do you think that? And then identify the people who are more on Annie’s side and you explore that. And then you say, does anybody else have anything to add? And then you’re done. There’s no okay, what do we think? Because the whole—

Adam: 1:36:18But it seems to me that you’ve got a lot of dispersion in a category, it usually highlights that there’s a, different people have defined that in very different ways right?

Annie: 1:36:31It comes out in the rationale. That’s exactly what comes out in the rationale. And sometimes what you’ll find out is somebody will say, well, I was thinking about it not as if it’s left unchecked. But the reason why I thought that it improves productivity is because I was thinking about managers managing that situation really well and keeping people separate. So you actually find out that there’s agreement, right? Because people are allowed to give their rationale it’s not like they have a rating and then there’s nothing behind that you’re saying, why do you believe what you do? And then the a good facilitator will reflect that back and says, it seems to me that you’re defining this differently if I understand. Can I ask you? Brian, I think you’re defining it this way. Mike, I think you’re defining it this way. Do I have that right? So all of this is coming out in the conversation. The key is I have no intention of Brian and Mike ever agreeing with each other.

Because it’s not the point. Why would you have a team if everybody ended up agreeing with each other? Who cares? So now we find that out, everybody hears what everybody else has to say. And then, then how whatever the decision process is will happen. And you know maybe people vote now right? Maybe there’s one person who’s making the decision but now they’ve heard everybody’s input. And you’re exploring it in a way not meant to convince anybody of your point of view which makes for a very bad discussion. But it meant for me to be able to convey, why do I think the things that I do? And for me to hear you convey the same. And it’s just about me understanding. So exactly. If people are defining terms differently, that naturally comes out in that kind of conversation. Sorry, go ahead.

Adam: 1:38:22Is it worth at that point actually breaking that one thing into two, right. So it was productivity now we’re gonna define it’s –

Annie: 1:38:32Okay, you could do that. And what you would do in that situation is okay, I think we’re talking about two different things. Can everybody just write down their rating for thing one and write down their rating for thing two. So you can do that spontaneously as long as people write it down. You just don’t want people to discuss it before they’ve actually recorded how they think. So this happens all the time. Like I’ll be discussing like an investment opportunity and somebody will say, well, I think if we get this logo, it’s actually really going to increase the value of our funnel. And I’ll be like, hold on a second. That was a big sentence. Stop. could everybody just write down your estimate of how much this is going to increase the value of the funnel to have this logo, right? And then everybody will write it down independently right? So I do that all the time when you’re sort of in that information discovery and then I just stop the process right then and make people write it down.

Brian:1:39:29I mean this is fascinating that it touches on—I think an increasingly popular topic in organizational decision making which is cognitive diversity. Like how do you bring people who think differently into an organization? And obviously it’s a bit orthogonal to traditional definitions of diversity but it may be–and maybe because I’m obsessed with organizational culture. It kind of goes back to the, how do you handle bringing in people who think differently/can disagree, without being disagreeable? My experience over my career is that, that’s really hard to execute.

Annie: 1:40:09Yeah, so this creates a structure that allows that to happen. I mean I can tell you this was one of the best sentences that was ever said to me. I was brought into an organization that was unhealthy. I ran my first process that was similar to this. It was about a particular question and you know with the leadership team and we elicited all the feedback in advance. I was new to the organization so the way I figured out what feedback to elicit was to ask them to send me what feedback they thought I should elicit, and I did that all independently. And then combined that into themes and I elicited all of the opinions and then we had a big, two meetings back to back, on different days so that we could get through all of it. And I came in and I said, here are all the areas that everybody agrees and that I spent literally two minutes on that. And then we went through question by question, all of the dispersion and went through it all.

So now I do a debrief later with someone who was C-suite. And I said, I just would like feedback since that was the first time we were in the process, what do you think? And they said, I don’t think we’ve ever had a more helpful, like friendly, like you know not non-defensive, like open-minded conversation in this organization. So I said oh, that’s interesting. So you know that the only thing we talked about was people disagreeing with each other right? And they hadn’t realized it. And they were like what? Oh. I mean it was really like that reaction where I said– because I spent two minutes on here’s where you agree and they said oh, so open-minded and so cooperative and everybody seemed so friendly and happy and nobody was mad.

And all this stuff and I was like yeah, you know we only talked about places that were where you totally disagree with each other. But that’s what that kind of structure does because you know in some sense you know I think because of my games background, I just think about, how can you change what the rules of the game are? And here, the rules of the game are you know, to be really good at conveying why you believe what you do, and that’s it. But the rules of a game in a naturally occurring meeting are to get people to believe that you’re right and smart. And those are not the rules of this game.

Adam:1:42:40Yeah that’s a major shift in culture. One of the things it also avoids is the issue of a lot of situations where you’ve got this asymmetry between a person’s expertise or in you know insight and perspective that they bring to a problem, and their persuasiveness and or disagreeableness right? So often you’ve got a mix of people in a group and the person with the greatest amount of expertise or the greatest calibration for the problem, have the lowest weight in terms of the group weighting right? In terms of their opinion. Just because the qualities that they typically give weight to that weight are misaligned.

Annie:1:43:40I’m referring to something from this week. I know this unequivocally. What experts say is, it’s nuanced. They say it depends you know you can look at this from a variety of different ways. There’s different ways that you can model it. Even if you model it this particular way, you know depends on what your risk attitude is. Here’s what I think the lower bound is and the upper bound is. I can give you my point estimate but I’m telling you I’ve got a pretty wide band around that. And everybody goes zzzzzz…

Brian:1:44:09 What do you really think?

Adam: 1:44:13Sadly those people never make it on TV.

Annie: 1:44:16And then like Brian from sales says unequivocally.

Adam: 1:44:21Unequivocally, U2 is the best big stadium rock band.

Brian: 1:44:27Unequivocally.

The Facilitator in the Room

Annie: 1:44:28Unequivocally, right? Exactly. So this actually allows– this actually sort of tamps down that problem. I mean you know there’s this problem of just you know, there’s those three people who talk in every meeting. So you never hear anybody else. So this makes it so that everybody talks. And a good facilitator is specifically going to make sure that every single person has spoken by the end of that meeting. Because you’re looking through and you’re seeing where the dispersion is and you’re like okay, for this first question I’m going to call on so and so to start. Because there’s always a leader of the pack. You know like Brian’s going to represent the high numbers here and Mike’s going to represent the low numbers. And then you’re making sure that you’re calling on everybody to do that. So it doesn’t matter—

Adam: 1:45:13So implicit in this there’s a facilitator in every meeting.

Annie: 1:45:17You have to. You cannot have a good group discussion without a facilitator let me just say that… you can’t. By the way if it’s a small enough group I think if it’s like three people you probably can, but in a group of 10, it’s very hard without somebody facilitating. And it’s not that the facilitator couldn’t have also been an opinion giver, but like someone’s got to run that. You know one of the things that I’ve done just a spontaneous thing is like you know, you could have some sort of signal like if you have a placard you could turn it up and say, I have something to say and at least that creates some order to it. But in that particular case you wouldn’t have elicited the opinions in advance but once you’ve elicited the opinions in advance, someone has to be running it otherwise it. Otherwise it just erupts into people interrupting each other. But there’s something that’s really interesting when you have a good facilitator is that people stop interrupting because they know they’re going to get called on. They know they’re going to be heard. So there’s no reason to interrupt anymore because they understand like they’re going to get to say the thing they say they just have to wait a hot second.

Adam:1:46:36Yeah that makes a lot of sense. So I had something good. Don’t….

Annie: 1:46:33Don’t contact me for consulting right now. I’m at capacity and I will say no at the moment.

Adam: 1:46:39Well, you’ve done such a great job of demonstrating your value.

Annie:1:46:40Someone might call me for consulting. No really and no offense. I would love to help you but I’m just letting you know I’m at capacity at the moment. I can’t take on any new clients right now.

Mike: 1:46:52Congratulations.

Adam: 1:46:53So we don’t have like a an ending question on our riffs. No, no. But we are…

Mike: 1:47:03You can reach out to me here but I will not respond to you.

Annie: 1:47:07No, I will actually respond but the problem –By the way that doesn’t mean that you can’t contact me to give a talk. Like I can take a talk. I’m just saying like I can’t  do any– I just don’t have space in my schedule for. So contact me after December when I have turned in the manuscript of my next book. Then I will have some capacity but at the moment I have none. You know what I just did there? I just did it in advance because I know–because here’s the day when people call me I have trouble saying no. So see I’m saying in advance. Please don’t call me. I mean except I’m happy to talk. I’m happy to give a talk and I will respond to your email.

Mike: 1:47:52Or is it a classic takeaway close where it’s the second or third derivative in the behavioral economic continuum.

Brian: 1:48:02Guys, she’s just playing poker with us.

Mike: 1:48:08That’s right. We’re totally outmatched.

Adam: 1:48:09Tell us about your new book, Annie. Give us a teaser on your new book.

The New Book – Loss Cutting

Annie: 1:48:18Oh, okay. Yeah. So we’ve talked a lot about my new book that’s the secret. So my new book is actually on the topic of quitting, otherwise known as loss cutting. And it’s essentially just you know, the human beings just have a very strong tendency to over persist. So you know I think we talk about you know, stick to it. You know like just keep trying and you’ll succeed but you know the thing is that it’s more like people who succeeded stuck to it. It’s not so much that sticking to it makes you succeed, and I think that we get those, that causal relationship a little bit wrong. It’s a little bit more correlational. So the issue is like I mean it’s a lot what we talked about with like the concert ticket. It’s like once you enter into an endeavor, it is just very hard for us to extricate ourselves particularly when things aren’t going well and we feel like we’re going to lose to it. I’s a little bit different if you think you’re going to win. But when you think you’re going to lose so it’s very hard to get out of it, and we know this right? It’s hard to get out of bad relationships. Everybody says, I should have fired that person a lot earlier because firing is a form of quitting. It’s you quitting the employee. But you know everybody says, I did that too late. I hung up, on too long. I should have left that job earlier. This is just a very common problem and the science on this is deep and broad about what are the forces that cause us to stick around too long?

And I just don’t think that there’s a whole lot that’s been written about this. I mean Adam Grant has this wonderful new book Think Again, which is about changing your mind related. It’s certainly adjacent to what I’m talking about. Seth Godin has a book that just sort of lays out the problem called The Dip, which is you know like Seth’s book it’s about this thick but filled within. But you know it’s just like every word’s a gem. That’s what he does it’s all like a—protein

Adam: 1:50:11High density.

Annie: 1:50:13Yeah, it’s like a protein bar. I’d really recommend people read that but. So I’m just really thinking really deeply about loss cutting and how do you figure out like– how do we become better quitters? Because I think that you know for me the secret to success is quitting. Why? Well, because you have to decide under uncertainty. So at the moment that you make any decision, you have limited information and also just so you know, the world changes separate and apart from that. So the whole thing is what allows us to make great decisions under uncertainty, is that later on we get to change. The market goes down in a way we didn’t predict. We can do something about that. We can change course. We get in a relationship and it turns out the person actually was a jerk but we didn’t know it at the time, we can break up with them. We tend not to respond to those signals very well.

So that’s really what the book is about. It’s like how do we become amazing quitters? Because I think calling someone a quitter as a mean thing to say. It’s silly. It’s like people who are great at quitting, are completely awesome.

Mike: 1:51:22Isn’t that the new tech word of pivoting?

Annie: 1: 51:28This is why I’m writing this book because when we do talk about quitting and we try to say it in a nice way, we wrap it in euphemisms. Oh, I pivoted. No, you didn’t pivot. You quit. Why? Just say it. Somebody said that to me recently, they said what happened? Why did you pivot into poker from academics? I was like, what are you talking about? I didn’t pivot. I quit. I quit academics and I became a poker player. It’s okay. I wear it proudly. It was not a pivot. You know pivoting is like a little change of direction. It’s like a very slight– it’s not running off the court which is what people say pivoting in business is. And I think that the people who are you know–I think we should stop using euphemisms to describe it. It’s quitting. There’s nothing wrong with quitting. It’s not a bad thing. And part of the reason why we won’t quit is because we think it’s a bad thing. Oh, you have some kind of character flaw. Like have you ever heard– like if I said to you, Adam, you’re a quitter. Like whoever said that is a nice thing. What I’m saying is you are a flawed human being and it’s not true.

Brian: 1:52:43Versus you stuck to it like.

Annie: 1:52:43Perseverance.

Brian: 1:52:44Like resilience.

Annie: 1:52:46Resilience, all those things. It’s like, look, the opposite of that is you’re just stubborn and rigid. So the thing is on both sides of the equation you need context. Is it bad if you know your child gets an entry level position and then the first time that the boss says something mean to them they quit? Of course, it is. Because first of all, the boss probably didn’t even say anything mean to them. They were probably just like, why didn’t you get your project in on time? And they didn’t say it in a way that was like coddling, so what? Yes, that would be bad. So if you have that context in that particular case, quitting wouldn’t be a good thing. But there’s lots of circumstances under which it you know sticking with it is really, really bad. You know like you’re in a failing business, you ought to return the capital to the investors and you just grind it down until the last penny is gone. Why is that a good thing? I mean you persevered.

Mike: 1:53:45I think the relationship context for me makes it very clear. You’re in a bad relationship and you won’t quit it. Which by definition means you can’t get into a good or new relationship. And so by quitting you’re actually optimizing potential future successes by trimming what is not working.

Annie1:54:10Like last time I checked, there’s opportunity cost to everything. But the problem is our mental accounting is really strange. Like if I have an investment that I’m losing in, I won’t quit it because I don’t want to have failed, even though I could take those same dollars and put them into something else that might actually, actually make the money back. But we don’t. Our mental accounting doesn’t square that. We don’t think about the balance sheet as a whole. I can’t believe I lost all my money in that thing and now I got to get it all back as opposed to we’ll put your money over here and you will actually get it all back. But because it wasn’t in the same thing it doesn’t count.

Adam: 1:54:46Yeah, my experience is that the most experienced traders are the ones that have best internalized that view.

Annie: 1:54:55Right, exactly. Like you have to start thinking long term. I mean there’s just this time horizon problem. There’s a horizontal problem which is you’re not thinking across all opportunities that you have. So yeah, I mean exactly. If you stay in that relationship, then you’re not getting into a better one. Anyway it’s my new thing. I’m on a mission now. I’m on a mission.

Brian: 1:55:18To have everybody quit.

Annie: 1:55:22I want everybody to quit. I’m a quitter. I’m awesome.

Brian:1:55:25I feel a hashtag coming on.

Annie: 1:55:26Yes I’m a quitter.

Adam: 1:55:29Well that sounds like the perfect segue to quitting–to put a pin in this conversation which is sneaking up on two hours. But this has been just as good as I had hoped it would be. Annie just incredibly generous and insightful and engaging like just your energy is so contagious and all the comments have been amazing. And so thank you so much for your generosity and your time here. And can’t wait to catch up in real life sometime which I think might happen for all of us again.

Annie: 1:56:05Well hopefully we’ll all be walking 28 miles again.

Adam: 1:56:09Exactly.

Mike:1:56:11I also want to thank Brian for coming on and making the introduction. And thank you so much for sitting with us for a couple of hours and adding to the conversation and jumping in with the wrong answer on me so I wasn’t alone. I appreciate that.

Annie:1:56:23Just take with you forever that you had the right answer.

Brian: 1:56:28I knew it was 61. I just didn’t want to embarrass them again.

Annie: 1:56:33That’s good. That’s why you need to do knowledge tracking, Brian.

Mike:1:56:37Yeah. So last thing. Where can people find Annie? Where can people find you if they want to follow you in the social media areas and things like that?

Annie: 1:56:44Good question. If you want to follow actually me, go to Twitter @AnnieDuke. If you go anywhere else, it will not actually be me. I mean not that it isn’t like you know what I mean. I’m not on my LinkedIn. I think someone just posts some stuff on there occasionally. So don’t ask me on LinkedIn because I won’t talk to you because it’s not me. But so go over to go over to Twitter. Twitter is where it’s the actual me. And then you can go to my website You can subscribe to my newsletter. And then you can also go to the Alliance for Decision Education. And that is a non-profit that I co-founded. Brian has been deeply involved with it. Thank you, Brian. An incredible friend of the organization. And we are trying to bring decision education to every child in K through 12 in the world. World domination in the same way that people get social emotional learning now, or STEM has become a focus that you know, better decisions lead to better lives which lead to a better society. And I say #more decision education, less trigonometry.

Mike:1:57:58There you go. Preach. Something you can use in life.

Brian: 1:58:02Can I just like echo that and say I think it’s just the coolest organization and that what you know Joe who runs the day-to-day, is such a sharp guy and the content. I mean I’ve got three teenage kids. We all have kids of varying ages and the content that is available to teach them to think a little bit more sharply about day-to-day problems. Like you encounter it for three minutes and you’re like you wish that this was in their schools. so I think the vision…

Adam: 1:58:40Where can people find more about it?

Annie: 1:58:42Just look at the Alliance for Decision Education and we’ve got a lovely website. Dot-org, right. Yeah, it’ll come right up and we are very pleased to have just stellar people involved. Phil Tetlock, Barb Miller, Danny Kahneman, they’re all on our advisory board. We just have you know incredible people on the board itself and Gary Kasparov. Yes, he’s also on our advisory. Lots and lots of amazing researchers. Paul Slovik, he’s associated with us. Barbara Taversky, Michael Mobison is on our advisory board. Ted Seides. Brian has been incredible and amazing from just about the start from when we founded it. And just super smart passionate people who are helping us out here. So we’d love for people to get in touch.

Adam: 1:59:39What a stacked team and an amazing initiative.

Annie: 1:59:52Well, it’s a big– it’s a moonshot kind of thing so we need moonshot kind of people.

Adam: 1:59:47Yeah, well you’ve got it.

Mike: 1:59:49Brian, where can people find you?

Brian: 1:59:52People can find me also on Twitter @brianportnoy. I also unlike Annie, I think I do tend to my own LinkedIn account but I’m not there very much. So yeah, @brianpornoy on Twitter and then my company is Shaping Wealth. So just go to

Adam:2:00:12Love it. Just so everyone knows since this is a weekly broadcast. Next week we’ve got Chris Schindler on, who will be talking about the anti-factor systematic strategies particularly in commodities. And he has been, every time he’s come on he’s been one of our absolute most popular guests. It tends to be more technical and very investing focused. So tune in for that one and over the next coming weeks we’ve got Fred Pye and Matt Hogan and Dave Nadig coming on to talk about crypto and we’ve got David Fauchier on. There’s lots of great stuff coming up. So tune in next week and through the summer.

Mike: 2:00:55 And lastly make sure you like and share. This was an epic episode to share and comment on.

Adam: 2:01:01Absolutely.

Adam: 2:01:03So please do that. All right, key music. Thanks again guys. Bye.

Show more

*ReSolve Global refers to ReSolve Asset Management SEZC (Cayman) which is registered with the Commodity Futures Trading Commission as a commodity trading advisor and commodity pool operator. This registration is administered through the National Futures Association (“NFA”). Further, ReSolve Global is a registered person with the Cayman Islands Monetary Authority.