Snapshot is the production engine that AGS uses to deliver data to clients, undertake custom analytics, and power our report API. While the name might be relatively new, it is actually a fourth-generation platform.
Nearly four decades ago, I was the sole dissenting voice at Compusearch to the purchase of a gently used, million-dollar VAX from the physics department at the University of Toronto. I argued that we could easily put a single Intel 386 based computer on each desktop and outfit the production department for a fraction of the cost. The Compaq 386, with a massive 640k of memory and a 30gb hard drive, could be yours at that time for about the same price my mother paid for her fully loaded Chrysler New Yorker. Over lunch at the nearby Madison Avenue Pub, I uttered for the first time that fateful phrase “I mean, really, how hard can it be?”
My colleagues – and more importantly, the owner – agreed I should give it a go, but with two stipulations:
- The entire dataset for Canada had to fit on a single 20 mb hard disk
- No more than 30 seconds could elapse between hitting the <Enter> key and hearing the sound of the dot matrix printer.
To fit the Canadian data in a tight space, I devised a simple but computationally expensive compression method and despite excruciatingly slow disks and very little memory, the results were, at the time, quite satisfactory: in the most densely parts of Toronto, we could run a report in about 25 seconds.
In 1992, I was shipped off to Los Angeles to join Urban Decision Systems, where production was being run on a time-shared IBM 360 located somewhere in the San Fernando Valley, with monthly bills that were crippling the company. Within a year, I had replaced that system with a Windows based environment that was the 1990’s version of cloud computing – using some primitive file sharing techniques on a network to coordinate the activities of a wall of five computers that handled the processing, printing, plotting, and even faxing. The actual report generation took about two seconds. Much of the core code was simply ported over from Fortran.
The third generation was written in the late 1990’s to power the Anysite platform, which sported a MapInfo front end utilizing the painfully slow MapBasic environment. We provided a much faster engine known as SDRE (spatial database retrieval engine) that had a tiny footprint and handled all database management and production functions, most standard GIS functions like polygon overlay, contouring, and kriging, but also incorporated an extensive library of spatial and standard statistics functions. For Anysite, it was the equivalent of dropping a 351 V-8 engine into a Ford Falcon.
It remained constrained by late 1990’s storage technology, since everything had to be shipped out on CD-ROM disks, and memory was still very expensive. AGS used SDRE as our production engine for nearly 20 years, but the combination of early COVID boredom and an increasing demand for AGS data to be delivered by API proved its demise.
The fourth generation, now known as Snapshot, emerged in mid-2020 as our new production engine. Nearly all the code was re-engineered to utilize the ridiculous amounts of memory available on your typical laptop computer. No SQL engine for us, it is a compact system consisting of an 800k windows command line program and a few DLLs to handle PDF report generation. Performance was nearly two orders of magnitude faster than the SDRE engine it replaced. Instead of seconds per report, we were now measuring performance in reports per second.
The latest version, currently in testing, will be put into production mid-summer. The primary change is that we have implemented a queue-based multi-threaded model that considerably improves production times for bulk trade area reports and database generation. Our test variable list includes 123 output variables that rely on 423 internal formulas, and benchmark times are very encouraging:
The point of all this isn’t speed per se. Does it really matter that the data for a one-mile radius can be assembled in just over a millisecond? No, but it gives us the ability to put in place so many of the analytic models that were just not practical in years past. Here at AGS, being the gold standard for small area demographics isn’t enough, we want to lead site selection into a new era.
Recent Comments