When back-testing fails

I just came across some interesting information that led to a bit of an "ah-hah" moment for me, and thought it might be worth mentioning in case anyone can relate to and benefit from what I came across. 

To set the stage before I make my main point, plenty of traders out there tout the value of making data-driven decisions. Document your trades and setups. Study the charts. See what works and what doesn't from historical data. That's all fine and dandy, and are great trading practices, but what happens when we diverge from historical norms? 

To answer this, maybe we can glean something from what insurance companies are apparently doing in the wake of unprecedented fires in California in the past few years. Instead of writing policies based on historical trends/data to determine the probability of fires (and setting insurance rates), they're having to resort to more fundamental analysis. This could involve doing a risk assessment of fire damage based on the current condition of the electrical grid, density of forestry, wind and temperature patterns, dryness of shrubbery, etc. The statisticians usually creating statistical models of the probability of fires based on historical data might be taking more coffee breaks than usual these days. 

Likewise, we often need to adapt to fundamentally shifting conditions in the market and can't simply rely on back-testing scenarios when the scenarios themselves have little historical precedent! How do you model Gamestop price action based on a never-before-seen gamma squeeze (it was only a theoretical possibility), crude oil prices actually going negative last year (again, only a theoretical supply/demand imbalance scenario you might have seen in an economics textbook), or say Covid vaccine companies in a historical context? The answer is you can still model them, but not from a historical or data science approach. You create a new model based on fundamental observations about current conditions, recent factors driving their behavior, and what could impact things in the immediate future. 

Engineers are very familiar with fundamental vs. statistically-driven models. And even fundamentally derived models don't always operate the same way, depending on the region of operation (of a variable like voltage or temperature). So it's kind of funny that many traders only believe in back-testing setups, which is perfectly valid under "normal" conditions, but they would be very well served to gain an appreciation of the non-statistical mindset when it comes to modeling future market behavior.