Rebuilding 253M bars from one source
Until this month, EquationDB's price history came from more than one place. Most of it was consistent, but "most" is not good enough for a database whose job is to tell you what usually happens next: a single bad split factor in one stock's history is enough to invent an event that never happened, or to hide one that did.
So we rebuilt every price from one source, our market data provider, with every symbol treated the same way.
What we rebuilt, and what we kept
The rebuild covered market data only: daily bars, corporate actions, fundamentals and the datasets behind EARNINGS, DIVIDENDS, FINANCIALS and the rest. Everything that belongs to you stayed exactly where it was: accounts, API keys, saved queries, alerts, and the universes, fields and events you defined.
Nothing was deleted either. The old market data was moved aside, so we could compare against it and roll back if the new data failed its checks.
How it works
- The symbol list. We start from the list of actively traded symbols and register each one with its profile: exchange, sector, industry, country and currency. That is about 70,000 symbols.
- Full history, one symbol at a time. For each symbol we load every daily bar since it listed, page by page. With 70,000 symbols that is on the order of 100,000 requests and 253 million bars. The load is resumable: if it stops, running it again picks up where it left off.
- Dividends as factors. Daily bars are stored split-adjusted. Each dividend becomes a factor, applied when you ask for total return with
ADJ total, so the stored prices are never rewritten. We explain why in a separate article. - A check before anything else. Before the rebuilt data replaced the old, we compared total-return series for reference symbols over almost three years. The rule was strict: the largest deviation on any day had to stay within 0.1%, or the rebuild stopped.
- Everything derived is recomputed. Indicators, events, breadth composites and the relationship graph are computed from the new bars, never copied from the old ones.
You can see the result for any stock:
GET AAPL SELECT close LAST 30d
Keeping it correct every night
A one-off rebuild is only as good as the next night's data. After the daily bars and corporate actions arrive, a price integrity job re-fetches a symbol's entire history and replaces it when:
- the symbol had a split or a dividend with an ex-date in the last five days, or
- its last bar is older than the latest session, or
- its latest close differs from the official end-of-day figure by more than 0.5% (an unrecorded split or a correction).
Nothing is deferred to another night: failures are retried within the same run. If something larger is wrong, for example the whole day's data failed to arrive, the job reports it instead of trying to repair thousands of symbols, and we are alerted.
One caveat
Dividend factors are loaded for the symbols that carry live indicators and events, which covers the universes you query most (top500, the ETFs, the reference symbols). For the long tail of other symbols, ADJ total currently returns the split-adjusted price. We will close that gap.
What it means for you
- One consistent history for every symbol, from listing to last night.
- Splits never create false events, because history is adjusted when it is read.
- The same checks run every night, so the history stays consistent as new data arrives.
Read more in Data, adjustments & freshness.