Recently, we introduced readers to the exciting potential of our new Vortex engine for generating base demographic estimates using a wide range of public and private data sources.
The Vortex can be best compared to a centrifuge, where we pour an enormous quantity of both raw and processed data into the machine. It uses a gradient boosted tree technique to learn where noise is and to classify the signals. It is a spatial model, in events in nearby areas are considered in the model. It is also a temporal model, which attempts to isolate time lags between signals and actual change.
A brief, as non-technical as possible, diversion into classic parametric statistics is helpful here. Classic regression models rely on finding consistent relationships between the independent variables (the predictors) and the dependent variable (what we are trying to predict). While these need not be linear, each predictor effectively contributes a consistent percentage of the ‘explanation’ of the predicted value, no matter what its value.
But what happens if that is not true? Take something like using traffic counts to predict retail sales – we are not at all surprised to find that the higher the traffic counts, the higher the sales. However, there is a certain upper limit after which the relationship is likely either flat or inverted, since if there is too much traffic, potential shoppers may be deterred from stopping. A simple model would likely perform well with ‘normal’ traffic volumes but would grossly overpredict sales at locations with very high traffic counts.
Most demographic forecasts are based on regression techniques which are in essence ‘if all else stays the same’ trend forecasts. Vortex takes a very different approach, and several key ideas drive the engine:
- Predict the change in population, not the population itself
- Most areas do not fundamentally change year to year, and we want to focus the most effort into explaining those areas with the most change
- There is no overall formula, just patterns of individually weak signals which occur at different points in time and different points in space
Most demographic change occurs in a small number of areas (historically, 5% of the block groups account for about 55% of the change). Most of the blocks are stable and contribute very little to our understanding of why change happens in the first place. Our model assumes first that no change is normal and attempts instead to find the spatial and temporal relationships that indicate that something abnormal is about to occur.
By examining data both temporally and spatially, we are able to focus on areas which have recently changed and find the signals which existed (in space and time) just prior to that change occurring or while it is ongoing (since many developments take place over several years). Once we find patterns in those, we can look for similar signal sequences – parcel subdivision, evidence of land assembly, postal code changes, new street construction, and building permits, just to name a few.
But we also have to look at constraints – there may be pressure to change, but no land available. If we find a block with strong signals but no capacity, that pressure is pushed into neighboring areas. We also cannot allow local projections to run wild and constrain them to supportable projections at higher order geographies.
Emerging from the boosted tree model are a set of change archetypes, such as greenfield development, gentrification, or stable. Each area is scored on each archetype as a probability and a set of rules are learned over time that allow us to classify blocks according to their expected change archetype, which often is the default ‘nothing to see here’ group.
The strengths of this approach?
First, we can easily add new data sources along the way. It need not be national data to be useful because the models are inherently missing data friendly. Second, performance should continue to improve each year since longer time series result in the ability to better discern long temporal patterns and will give us more examples of change archetypes. Third, it is dynamic and can be used for simulation purposes.
Next week, we will talk about some of the results – the change archetypes – and how they are impacting our 2026B estimates and projections.