What we are re-checking in our published backtests
Two problems surfaced in the data behind our published backtests in the last three weeks. Neither has a final answer yet. This note says what each one is, which numbers on this site it can touch, and what happens next. When the answers are in, this page will say what they were, and every affected page will link back here.
Stale prices. On October 4 we audited the newest closing price for every active company in our database. 959 of 5,488 were three or more trading sessions behind the market. For 401 of them the nightly price fetch had stopped asking for prices at all, because a rule we wrote months ago only fetched companies that had been priced within the last 60 days. Once a company fell outside that window it never came back. The fetch was rewritten the same day. It now asks for every active company from its own newest price, retries failures, and reports any company that came back empty.
A backtest that holds a company with a stale price values it at the last close it has, so the error sits in the final weeks of a window, on the holdings that went stale. Every page in our backtests series ends between September 13 and September 30, 2026, inside that period. For screens holding large, heavily traded companies we expect the effect to be small, but expecting is not measuring.
Survivorship comparisons. Three of our pages run the same screen twice, once with delisted companies kept in and once without, and report the difference. On September 21 we rebuilt how market capitalization is calculated across our history. Runs of those screens made after the rebuild do not reproduce the published gaps. On the Piotroski page the published gap is 3.06 points a year in the textbook direction, and later runs put it under a point in the other direction. On the midcap value page, 1.51 points became less than a tenth. The later runs also hold fewer of the companies that later delisted: 9 instead of 14 on Piotroski, and 4 instead of 15 on midcap value.
There are two possible explanations and we do not yet know which is right. The rebuild may have left some delisted companies without a market cap, which would quietly drop them from a screen and make the honest run look more like the survivor-only one. Or the September 13 runs may have included companies they should not have. The first would also affect backtests people run in the app, which is why we are checking it before changing a single published number.
What we changed today. We took the two contested survivorship figures off our homepage and put a dated note on every backtest page. No number on any page has been changed yet.
What happens next. Once the price repair is confirmed and the market-cap question is settled, we will re-run every published backtest over exactly its published dates, so any difference comes from the data and not the calendar. Each page will then print the new figures with a note giving the old ones. The survivorship comparisons go back on the homepage only if the re-run supports them.
We publish our losing backtests because a tool that hides its losses is selling something. The same goes for our mistakes.
Aaron