Disclaimer:00:42Mike Philbrick, Adam Butler, Rodrigo Gordillo, and Jason Russell are principals at ReSolve Asset Management. Due to industry regulations they will not discuss any of ReSolve’s funds while on this podcast. All opinions expressed by the principals are solely their own opinion and do not express the opinion of ReSolve Asset Management. This podcast is for information purposes only and should not be relied upon as a basis for investment decisions. For more information visit investresolve.com

Mike:01:10Welcome back. In the first eight days we talked about an awful lot of the intricacies of portfolio construction, the assembly of strategies and thoughtful ways to increase the likelihood of positive outcomes for clients. And we dug pretty deep and now we want to think about, you know, how do you develop ah, a quantitative strategy? What is a quantitative investor, how did they develop the rules, how much should I rely on a back test, and a myriad of other questions and we want to dig into that right now.

Rodrigo:01:40Yeah, I think one of the points of this is we get, and we spent eight episodes talking about systematic investing, assuming that everybody’s on board, but there is a group of people out there that are somewhat skeptical, about a back test and the quantitative methods and the like. So, uh, you know, people had the famous saying, “you’ve never seen a back test you didn’t like”, well there’s truth to that, but there are ways to get around that as well. So today we want to talk a little bit about our journey, how we went into back testing using quantitative methods to invest in clients’ portfolios. Why we think it’s the appropriate way to do things and uh, and talk about the trials and tribulations of doing so.

Maybe Adam, you can give us a little bit of the background, some of the Tetlock research that you read early on in our partnership.

Adam:02:28Yeah. Well I think it’s important for everybody to understand that we weren’t always purely systematic thinkers. Some of us came to realize the fundamental value of systematic thinking a little sooner than others. I think Rodrigo, you kind of came to it a little earlier in your career. I had more hubris coming into the investment career than some and, and spent a long time trying to create portfolios and run the investment process using discretionary thinking, trying to figure out how the machine works through narrative construction. And it wasn’t until after the 2008 financial crisis when I think Mike and I really began to understand the power of purely systematic thinking and part of that journey was getting the frying pan to the face of 2008, but then also stumbling along or across some of the top thinkers in this space.

And one of those thinkers is a guy named Philip Tetlock, who I think more people will be familiar now with Dr. Tetlock than might have been familiar with him back in 2009 when I came across him, but just for those who are not aware of his story, back in the mid 80s, Dr. Tetlock graduated from a post-grad degree in applied psychology and went to work in Washington at some think tanks and specifically he was taking notes at political intelligence committee meetings.

So he was documenting at each of these meetings what the senior generals and senior thinkers in the military and in politics were thinking about what was going to happen. For example in the Russian politburo, who was going to rise to prominence and what policies were they likely to enact and how did it impact U.S. foreign policy. And every quarter they get together and he would read out what these experts had said in the previous quarter and what he observed was that from quarter after quarter after quarter, these top experts, we’re getting it wrong.

And they would always have excuses for why they got it wrong, and so he decided that he was going to set out on a journey to determine whether or not anybody could make reasonable accurate or well calibrated forecasts in complex fields. And um, so over a span of about 20 years, he interviewed 284 experts about their level of confidence that a certain outcome would come to pass. He solicited forecasts and a bunch of domains, economics, politics, climate, military strategy, financial economics, etc.

So he accumulated about 28,000 forecasts and he was specifically interested in measuring forecast calibration. So for example, of an expert said he was 60% confident that an outcome would come to pass, will on average over many forecasts if, if you know, they were 60% confident, then those forecasts should come to pass about 60% of the time writes that if so, that expert was well calibrated.

And so just to get right to the meat of the results after 28,000  forecast were made, he determined that experts are less well calibrated than what someone might expect from random guessing. Uh, on average, experts delivered forecasts and confidence in their forecasts that were less well calibrated, that might be expected from, from random guessing  Not one expert, there were no outliers. Not one expert distinguished himself with better than random calibration.

Rodrigo:05:55So none of them did better than, than a coin toss. Right?

Adam:05:58That’s right.

Rodrigo:05:59And wasn’t one of the stats that the experts outside of their field of expertise, we’re better at predicting the future than the ones inside their field of expertise?

Adam:06:07Absolutely.

Rodrigo:06:08Absolutely crazy results.

Adam:06:10Yeah. And, and experts that were cited most frequently in by the media or in the news also exhibited worst calibration than those experts who toiled in obscurity. So, so, so there’s good news in here, right? The good news is that alongside these expert forecasts, Dr. Tetlock also ran some really simple systematic methods. For example, uh, over the short term, the current trend will continue in the same direction. And over the very long term, we should expect there to be a reversion to the mean. And, what we realized pretty, pretty quickly was that these effects overlapped with some well known and well documented phenomenon markets.

What will, what is the phenomenon when the trend that’s in that’s currently in place continues for the, for the next little while? Well, that’s momentum or trend following, and what is a strategy that takes advantage of ah, a stock that is very far away from its long-term mean valuation? Well, it’s value investing. And so that sort of crystallized or cemented our passion in, in systematic thinking and then you’ve got Daniel Kahneman and Tversky and just a whole slew of other research that has validated that approach.

Rodrigo:07:33You go all the way down the rabbit hole, and you find yourself ready to go. Right. And this is like you mentioned, I started doing quantitative methods straight out of university because I had been lucky enough to have a father who was a math professor and a statistician. So I went into the problem headfirst. So I, I knew I wanted to do something that was rules based and what I ended up doing for the first two months of my life is something similar than what you ended up doing Adam.

And you wrote in the book that we wrote, which was grab a spreadsheet, spend two weeks, have no sleep, have 38 columns, in your spreadsheet on different types of technical indicators that are going to create a fantastic robust method of investing. Found something that had a 65% annualized rate of return and, and Sharpe of four or five, put it to work only lose like 40% in a couple of weeks. Right? So there’s a big divide between understanding that systematic investing is valuable, to actually creating something that’s useful.

Mike:08:32Let’s just pause there just for a second and just. Okay. So what, how would we define a quant manager then? Can we, can we just do that for a second? So here’s our journey to quant. How might we just sort of put a little definition into that? Adam, you want to take a poke at that?

Adam:08:48In terms of just sort of defining systematic thinking?

Mike:08:50Yeah. Well just for the, for the broad group of listeners, right? So what is a quant manager?

Adam:08:55A quant manager is somebody who realized that creating rules when you are in a calm, thoughtful, rational state that are based on fundamental intuition around economics or how agents operate in market or some other fundamental basis and then examining how that rule would have worked if put to work in markets over many, many, many samples back through time, and then once you arrive at a methodology that you have reasonable confidence in, then you just systematically execute on those rules over and over and over again relentlessly without letting emotion or intuition get in the way. Right. I mean that really that I think is, is the way that we would think about quantitative thinking or systematic thinking. Did I leave something out there?

Mike:09:44No, I think that, that’s a great, a great definition and so now that we’ve, we’ve got a common place, a common premise in which to chat about that, you can see that the world is actually full of quant managers, right? If you think about an index, the Russell 1000, that is a quantitative strategy and one of the reasons that indexes are so hard to beat or keep up with is because they relentlessly execute based on those rules and so I would, I would urge you know, listeners to understand that quantitative investing surrounds us everywhere and that you should really consider that as you’re entertaining quantitative managers and understanding how they might fit in your portfolio is that they are pretty pervasive.

And you want that, you know as, as you mentioned Adam, you want that economic intuition. You want to be thinking back to first principles on why you’re able to harness this excess return through some means so that when the going gets tough, you can keep going. Right? Because there’s, there’s a saying that-

Adam:10:51Will be tough.

Mike:10:52You know and it will. It absolutely will. And you in and under duress, you do not rise to the occasion. You sink to the level of your training, and so when the bullets are flying, and you’re in the foxhole on any quantitative strategy, you’re going to have, to have really good discipline, really good understanding so that you can stay the course.

Adam:11:13Absolutely

Rodrigo:11:14You have the rules laid out, you have the infrastructure in place so that you just keep on pressing that button, executing. We’ll that list. Even Warren Buffet is a quantum investor.

Mike:11:23Yeah.

Rodrigo11:24His quant strategy is just in a notebook and a checklist. Right?

Mike:11:27Sure.

Rodrigo:11:28And then he’s got a nice pool of money where he can relentlessly execute without getting much pressure from his, (laughing) from his stakeholders, but that, that is a point well taken. Everything’s about rules. Everything’s about making those rules based on what you’ve observed in the past. And then uh, and then systematic, the second half of  systematic, is relentless execution.

Adam:11:44The most powerful tool and probably the most widely used tool in the quant toolbox is as we, as we started out this idea of back testing or, or simulation and so, we want to spend a little bit of time talking about how we think about back testing or simulations and trying to equip investors who are not sort of neck deep in quant, day to day. Who haven’t put in those 10,000 hours but are still charged with having to evaluate, whether or not a strategy has merit to include in their portfolio.

They have some tools at their disposal to distinguish between good simulation or good quant, and not so good quant or fragile quant. But before we dig into that, I just want to make it clear that we, uh, talked about doing a podcast on that. I told Rodrigo that we’re going to get in here and our heads are going to explode because we spend all our time in this space. So I mean, really the objective here is we could go Marianas Trench deep on this thing.

We’re really just going to go snorkeling in order to just give a broad overview of, of some of the top things to think about for, that anybody can use to good effect.

Rodrigo:12:54So, so Adam, why don’t you talk about your experience, what we wrote in that, in that, in the book of your first foray into quantitative investing and use that as a springboard to understand what, what might be useful. And uh, in back testing, what, what is not useful?

Adam:13:10I think as you said, everybody starts in kind of the same place, where you get access to some data and then you dive into Excel or whatever your quantitative platform is, and you begin to create a wide variety of rules and sometimes those rules are conditional on other rules. So you can imagine ah, a rule where something is in a positive trend based on some kind of indicator and then when it’s in a positive trend and this other thing happens, then you take a trade in this direction or another direction. Oh, and then you layer on something else.

So when it’s in a positive trend and this other thing happens, and the threshold is above a certain threshold, then you take another action and imagine having 30 or 35 different layers of these types of conditionalities. What most people don’t realize is that you’re creating many, many, many multiple universes. Many, many different potential states that that system can be in.

So I go through the example in my, in our book where you’ve got 37 different indicators and let’s just simplify it and say each of those indicators has two potential states, well, if you combine all those indicators together, 37 of them with just two states each, in a conditional way, you end up with a hundred and 37 billion different potential states, for that system. And then something that people don’t realize until later on maybe in their exploration of quant is that sample size makes a really big difference.

And you need to have a meaningful number of samples for each potential state of the system. So now you’ve got 137 billion different states and you need to have a simple rule of thumb in statistics as you should have about 30 observations. So you need 137 billion times 30 total samples. So I mean it quickly. This becomes really, really, really silly, right? And so what you learn early on is that simplicity is, is absolutely the best place to start. If want to have the fewest number of moving parts, the fewest number of degrees of freedom possible so that you’ve got a large enough sample size to make meaningful conclusions.

Rodrigo:15:21Yeah. One of the examples is that I like to give is that first model that I built out, was right 92% of the time, so it had a huge edge. 92% of the time I’m going to be right, but it only happened five times, right? We only observed those 35 parameters hit five times in history, so small sample size, huge edge. In sample that is ah, four Sharpe strategy and then you have these more traditional momentum-trend-value strategies that have thousands of observations but a small edge, 51% of the time, 52% of the time.

And once you go live you find that that one that I created just loses money and the other one is wrong. 50% of the time, roughly actually 49% of the time. But having it that at 51% edge over long periods of time is massive. So it’s just a tough pill to swallow that you’re not going to make all this money in a short period of time. It is a small edge, and a series of small edges pay up over time. That is that it really ends up being as, as, or at least as, as simple as we can make it for now.

Mike:16:28There’s been lots of literature written on, on the point of humans trying to follow algorithms too, and the behavioral challenges that they have with that. And so Rodrigo went through the process of finding something that was right 92% of the time so he could have comfort in doing that. Even if you have that strategy and it goes wrong once, humans have a really hard time following it, even if it was right 9/10. And now you can imagine here, you, you, you know, we’re going to talk a little bit more about ensemble methods and the fact that you know, lower, you know lower probability of success but across many systems is more effective.

But here you have a system that might be 50% or 60% effective and you know, size of win also, um affects that as well as not just the percentage that you’re, that you’re right.

Adam:17:15For sure.

Rodrigo:17:17But so, so now you can imagine you’ve got this system that’s right a little bit more than a coin flip, behaviorally that’s really difficult for, for a human to stick with. And the research on this is, is incredible, right? If, if an algorithm is wrong once, it’s dead to people.

Adam:17:32Well it’s, it’s even simpler than that, right? Some of the psychological literature you’ve got experiments conducted over and over and over again, where you, where you give somebody a loaded die, they know it’s a loaded die, they know that the odds are massively stacked in their favor. You get them to, to throw the die over and over again. Well, if they throw the dye, and they get a losing outcome, they’re less likely to bet the next time. If they get a second losing outcome, they’re even less likely to bet, and they scale back the size of their bet.

So, I mean this is even knowing, what this is in markets, can’t know anything, with a loaded die, you know, over time you’re going to have, you’re going to have winning outcomes. So I mean this is just a ubiquitous facet of human nature that is inescapable, and we need to be aware of.

Mike:18:13And especially when we’re building algorithms and systems like, you know this is this, this is a human frailty that you want to make sure that you’re, you’re aware of, and even though you’re aware of it, you’re still susceptible to it, but you’re, you’re, you’re not letting that impact the build out of your quantitative strategy.

Rodrigo:18:32Yeah. When you’re a quantitative firm, I mean you do enough work that you recognize that it’s all about relentless execution. There’s a ton of simple systematic methodologies out there that are being used by individual investors and individual advisors for their clients. And if you’re new to it, I don’t think most people realize how difficult it is to pull the trigger. Right? For us, it’s second nature. We don’t even pull this trigger anymore. We have a whole operations team that pulls the trigger.

Mike:18:55Oh my God, you’re so right. Remember what year was it in January, was it 16, 2016. January, February. We heard about all of the other sort of 10 month moving average strategies being entirely in cash and they’re intense discomfort with that. And I’m.

Rodrigo:19:16Oh yeah, January, 2016.

Mike:19:17Yeah.

Rodrigo:19:18A bunch of these uh, managers went 100% cash and we talked to them and they’re like, we’re terrified. We’re terrified what’s going to happen here? But I even have a very close to home example. We, when I first got into the business, I worked with an awesome, awesome human being who built his book of business talking about the, uh, what was it that he used, to used the 10 week and 40 week moving average crossover.

He called it the golden cross or whatever. And that’s how, that’s how we, he did everything right? And he used a Nortel example when it crossed over, you would have been out at this price and you would have been back in at this other price. So he goes years and years talking about this. But the reality was that we hadn’t seen a cross over for a few years, a bunch of years, and all of a sudden in September, August, September of ’08 you see the Toronto Stock Exchange crossover a bunch of the stocks that he was wanting to crossover.

I wasn’t working with him at the time. I reached out, and I’m saying, “You’re getting, you’re getting out, right? You’re selling out.” And he said, “You know I’m gonna, I’m gonna, watch it for a bit. I’m not sure this is a confirming signal.” And he never got out and it just, it was that difficult. Right? You can spend your whole career talking about what you’re going to do, but it’s another thing to actually execute on it.

So if you’re going to get into quantitative investing, even if it’s simple models, especially if it’s models that only trigger once in a while, make sure you are ready to pull that trigger when you need to.

Mike:20:36So let’s, let’s jump into how we might improve on. Given the concepts we’ve laid out, how might we improve the opportunity to gain a signal? Let’s dig into the ensemble methods a little bit more. What you guys think?

Adam:20:49Yeah, sure.

Mike:20:50It’s a good, good jumping off point on that.

Rodrigo:20:51That, that, that will also be another hour that will eventually do, but I will. Let’s see if we can get it done. Yeah. Yeah. Let’s talk about it briefly. It kinda, it parlays from the conversation of the 51%. And Adam, I know you’ve done a lot of work on this. Can you give us a brief outline?

Adam:21:07Well, I think the best place to start is, uh, we’ve talked a little bit about this in previous episodes, but it’s worth revisiting Larry Swedroe’s framework for evaluating the most constructive or prospective edges, right? So something that’s pervasive works across different markets, persistent works across different timeframes that it has economic intuitions, so when the chips are down or when your strategy hasn’t been performing well for a while, you have a reason to maintain faith in it, that it’s implementable in practice, uh, survive transaction costs and liquidity and market impact and all that kind of stuff.

And so the question is how do you account for those variables in the back testing process? And it comes down to stuff like, so you’ve got an investment universe that you’re going to use for testing. How about you mix up that investment universe? So you’re going to use REITs, let’s do use different, different REIT indexes. You’re going to use U.S. equities, you are going to use different U.S. equity indexes, you are going to substitute the Nasdaq or the Dow for the S&P 500  or U.S. total stock market.

You’re going to use different tenures of bond indexes. You can use foreign and domestic bond markets. You’re going to use different types of commodity indices, these sorts of things, right? Because you want to make sure that your process is robust to a variety of different specifications. It could be just that you discover a process that is uniquely engineered to the idiosyncratic qualities of how a certain market evolved in the past, and that that is, um, that that will not translate into the future. And so you want to avoid creating strategies that have that level of fragility or, or sensitivity.

Mike:22:44And across how many domains are you doing this? Right? So what timeframes might you be looking at? What metrics might you be looking at?

Adam:22:52Absolutely, or different ways to specify your indicators, right? So you’re going to use momentum, is it six month momentum, is it twelve month momentum? Are you going to…

Rodrigo:23:00Lets define what that means? Right. Let’s think about momentum. So what is it momentum trying to do just as an example? Well, it’s herding behavior. We think that there is a real behavioral reason why people tend to do things that are, go into things that have recently done well and get out of things that recently done poorly. Okay.

Adam:23:18Sure.

Rodrigo:23:19The academic approach to that is to look back, rank asset classes and securities over the last 12 months and then pick the top…

Adam:23:25Skip a month….

Rodrigo:23:26Skip a month and then and then hold it and then rinse and repeat over and over again. Okay. That’s one way that’s herding. One way. Why is that the herding behavior parameter? If we use 20 days does that seem to work? What about a six and a half months or anywhere between …

Mike:23:39137 days?

Rodrigo:23:45Right? So they all seem to survive. And they all seem to have over very long periods, is similar Sharpe ratio.

Mike:23:51So, so let’s dig into this a little bit so, so that someone out there can maybe use this a little bit more in a real example like momentum. So we’re going to look back and we’re going to say we want to, we want to think about momentum in this ensemble framework, and so someone might be able to more concretely grasp what we’re talking about. So I remember growing up in, in this world of finance and reading O’Shaughnessy’s book back in the mid 90s, and he had the Compusat database and I’m pretty sure it was nine month momentum that nine month was the, was the sweet spot and then, and then it became the six month momentum which is the sweet spot and, and I see this pervasive across lots of tests everyone’s using the six month and, why are they using the six month? Why was the nine month to use?

Well, because after the fact it was the best one and so, but that, that doesn’t mean it’s going to persist into the future and we don’t know. We know generally speaking, that momentum exists anywhere from, I don’t know 20 days to 300 days, but it’s going to move around in that area and so you have the opportunity to make some pretty big mistakes if you’re just saying, “I’m going to use six months.” And so maybe you say, “Well, I got you Mike. I know what I’m going to do. I’m going to use three month, six month, nine month and twelve month.”

And it’s like, “Okay,” Well we’ve seen pricing behavior. What about when that one month, that moment when you’re looking back and observing your back test, happens to be 3% to 5% different than the actual meat of the trend, right? What happens when that one observation.

Rodrigo:25:26What happens is all the information in between?

Mike:25:29Right.

Rodrigo:25:30Yeah.

Mike:25:31It’s an outlier. So, so you’re missing all of that, that’s okay. So now I hope people are starting to sense. Oh yeah. Well what about when that happens? Okay, well let, let’s, let’s dig into that a little bit deeper. What kind of momentum are we talking about here? We’re talking about point to point, price momentum. So let’s say something has a zero price for six months and then on one day it gaps up 6%, making it the number one performing asset in your universe. Is that reflective of the returns that have been garnered for clients over that timeframe? Is it reflective of the future?

So maybe one of you guys can jump in and, and talk about how we deal with that and the different types of momentums we might use. Because …

Rodrigo:26:08I mean you just talked about one aspect of the momentum methodology, which is how far back do we go to rank asset class momentum or strategy, security selection momentum? Why aren’t we assuming that momentum is best extracted by looking at the price percentage outcome? Why don’t we rank it based on Sharpe ratios or rank them based on the days they’ve been above a certain trend or so on and so forth, but we have a piece called The Many Faces of Momentum that people can read in the show notes that walks through the many ways of trying to extract…

Again, going back to the fundamental reason why we care about it, humans are going to likely continue to herd. That’s what we want to do. We went, there’s this momentum signal going out into the atmosphere. We want to be able to hug it and as many ways as possible, in contrast to trying to build the best antenna. If there’s a signal you don’t want to build the 6.5 month antenna that recently captured the best performance, and you can show that I have a bigger edge and momentum and somebody else. No, you want a hug it. You want to, you want to do a lot of small edges, a lot of simple antennas that together as ensembles are able to eliminate the error terms and capture the true signal, right? Maximize the signal and minimizing noise. And that’s what we mean by ensemble method.

Adam:27:20I just think it’s important more generally just backing it up a little bit for people to realize that the history that we’ve observed is just one potential trajectory and you shouldn’t get too caught up in the precision of the rules that you’re able to derive from that one historical context. I mean, the idea here is to be generally correct and to avoid the potential for being specifically wrong. I mean we’re trying to connect with. We believe that a certain phenomenon exists. We have a great deal of humility about the way that it exists or the way that we would measure it.

We think there’s a variety of valid ways to measure it, and it’s just more reasonable and more humble, a more humble approach to take all of these different methods to measure it and then aggregate them all together. Um, and you know, as a starting point, just an equal weight. And then, I mean maybe at some point in the future we can talk about, ways to use machine learning to maybe skew towards certain specifications and away from others, that sort of thing. But really it comes down to humility. Trying to catch a phenomenon with a net rather than trying to hit it a square with a sniper rifle.

Rodrigo:28:32Exactly. I think that’s kind of what we wanted to get out of this episode today.

Speaker 2:28:36Yeah.

Rodrigo:28:36Really helped people understand good back testing, bad back testing, ensemble methods, making sure that we’re attacking the problem from humility and not finding the best possible max Sharpe back test that’s gonna kill it and it’s all just, um, uh, you know, reverse engineering.

Mike:28:54Generally correct. Not specifically wrong, right? Uh, what we’re trying to avoid luck, both the good luck and bad luck. I mean, another great example is just days of the month in which you would be balanced monthly systems, monthly systems can have variance in the Sharpe ratio, 50% based on what day of the month you happen to rebalance them.

Rodrigo:29:11Yeah. And we, we covered that in a piece recently called “Same, Same but Different” that I encourage everybody to read.

Mike:29:19Lovely. All right. I hope that helped. As always, if you do have questions on an episode like that where we got into, uh, into things a little bit more deeply and you want to know more, you can always reach out to us. We love that. We love having conversations on this topic as you can, you can tell, and that wraps up.

Adam29:37Thanks guys.

Rodrigo:29:39Thank you for listening to our 12 days of Investment Wisdom mini-series. You will find all the information we highlighted in this episode in the show notes @investresolve.com/12 days. You can also learn more about ReSolve’s approach to investing by going to our website and research blog at investresolve.com, where you will find over 200 articles that cover a wide array of important topics in the area of investing. We also encourage you to engage with the whole team on Twitter by searching the handle @investresolve and following Adam, Mike and myself. If you’re really enjoying this series, please take the time to share us with your friends through email, social media, and if you really learned something new and believe that our series would be helpful to others, we would be incredibly grateful if you could leave us a review on iTunes. Thanks again and see you next time.