There’s talk on the street, it sounds so familiar
Great expectations, everybody’s watching you.
People you meet, they all seem to know you
Even your old friends treat you like you’re something new.
(Glenn Frey/David Souther/Don Henley, New Kid in Town, 1976, The Eagles, Hotel California)
All eyes are indeed on the new kid in town, and it seems these days, every company out there is peddling some new AI driven features. Every startup in the industry is using the .ai domain or has worked AI into their company name. Established firms are rushing to implement AI in their environments, often implemented as agents tasked with helping the end user understand the complexity of the underlying data and maps.
Oh look, shiny thing! The last one, ML, is out. AI, is in. But is AI the holy grail amongst shiny things? Time will tell, but at present, it is clearly in an early stage of development. Before it gets to maturity and widespread utility, we are bound to have that wave of crushing disappointment when people realize that it is not the panacea they were hoping for.
At present, getting an AI to successfully complete even a simple analytics task can be an exercise in frustration.
It is like trying to get a toddler to perform a simple task like ‘go get your shoes’. Tell this to a moderately competent adult and they will promptly return to you with a suitable pair of shoes. With the toddler, you most certainly must clarify the instructions substantially if success is to be achieved. As instructed, this child is not coming back anytime soon, if they even leave in the first place. There are many, many ways this prompt can go poorly, and the experienced parent soon learns to phrase this as:
Go to your bedroom and open the closet, where you will find your shoes. Do not play with toys or pet the dog. Pick up both blue running shoes – the ones with the three stripes on them. If they are not in the closet, please look under the bed. If they are not under the bed, go look in the crate the dog sleeps in. Once you have picked up the shoes, immediately, without stopping to play with toys or pet the dog, return to me, with the shoes.
Said differently, if you ask a simple question without providing complete context, your toddler is likely to bring you something other than what you wanted, if indeed they come back. This is not the toddler’s fault and if they are repeatedly assigned similar tasks, they will soon master it. But it is unreasonable to expect the toddler to understand the underlying assumptions behind the simple prompt.
Did I just call AI a toddler? Yes, but please allow me to explain.
Recently, one of our users asked us why our estimate of the population age 19 years in the United States was nearly double that which ChatGPT said it was. So, I fired up my subscription and here is the result:
What’s the problem? It went to the census.gov website and promptly found https://www.census.gov/popclock/data_tables.php?component=pyramid&utm_source=chatgpt.com. Reasonable choice, but what ChatGPT failed to realize (or check) was that the 0.67% male was not the percentage of males age 19, but the percentage of the population age 19 and male. In other words, the population age of 19 is 0.67+0.64, or 1.31% of the approximate population of 340 million.
Here’s the interesting part, which should give you a little hope. The latest iteration gave me this:
Response 1 is the ‘new’ version, and here it did a little extra work and presumably read the footnotes and came up with the right answer. Response 2? It is more precise, but still clearly incorrect. When I questioned it further on its silly error, I was told that in the future, with me, if I asked it to save my preferences, that it would actually find the census table rather than using an estimate like it did. It then actually blamed me by hilariously telling me:
Most of us are trying to get the toddler to drive a car when they have only recently learned to walk. At some point, probably not in the far distant future, our little toddler will be able to drive. The danger is that we are putting passengers in the car today, and the results are surely going to be disastrous. This is not to say that we ought not be trying, but the biggest problem we see is that expectations are being set hopelessly beyond the toddler’s current capabilities. Is AI ready for prime time? No, but it is very good at traversing huge volumes of data to find specific things, you just have to remember to show it what a needle is before we set it off looking in the haystack.