PROGRAM, MORON.

One of the most liberating moments in college came in an introductory computer science class.

At one time, the term "moron" referred to an individual who could carry out tasks that were well-defined and sequenced. As my first computer science professor taught me, a computer could be thought of as an electric moron.

That liberated me, early on, from putting too much stock in so-called artificial intelligence.

An artificial intelligence algorithm, admittedly a very good algorithm, is the Moriarty manipulating social media and doing a number of other things, but you'll have to read the book to find out what those other things.  An artificial intelligence algorithm, though, is only as good as its programming, which is to say, it is still an elaborately scripted moron, and, let's say, in implementing its instructions according to the priorities assigned to it.

Moreover, dear reader, here is the good news.  Dan Brown can conceive of a malevolent algorithm.  The programmers, though, have had less luck.

In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." To pull on that thread, the researchers created a carefully controlled LLM environment in an attempt to measure just how well chain-of-thought reasoning works when presented with "out of domain" logical problems that don't match the specific logical patterns found in their training data.

The results suggest that the seemingly large performance leaps made by chain-of-thought models are "largely a brittle mirage" that "become[s] fragile and prone to failure even under moderate distribution shifts," the researchers write. "Rather than demonstrating a true understanding of text, CoT reasoning under task transformations appears to reflect a replication of patterns learned during training."

You can program a moron to recognize the dimensions of a box.  But thinking outside the box?  You might as well be saying "Upward, not Northward" in Flatland.

To test an LLM's generalized reasoning capability in an objective, measurable way, the researchers created a specially controlled LLM training environment called DataAlchemy. This setup creates small models trained on examples of two extremely simple text transformations—an ROT cypher and cyclical shifts—followed by additional training that demonstrates those two functions performed in various orders and combinations.

These simplified models were then tested using a variety of tasks, some of which precisely or closely matched the function patterns in the training data and others that required function compositions that were either partially or fully "out of domain" for the training data. For instance, a model trained on data showing two cyclical shifts might be asked to perform a novel transformation involving two ROT shifts (with basic training on what a single example of either shift looks like). The final answers and reasoning steps were compared to the desired answer using BLEU scores and Levenshtein Distance for an objective measure of their accuracy.

As the researchers hypothesized, these basic models started to fail catastrophically when asked to generalize novel sets of transformations that were not directly demonstrated in the training data. While the models would often try to generalize new logical rules based on similar patterns in the training data, this would quite often lead to the model laying out "correct reasoning paths, yet incorrect answer[s]." In other cases, the LLM would sometimes stumble onto correct answers paired with "unfaithful reasoning paths" that didn't follow logically.

"Rather than demonstrating a true understanding of text, CoT reasoning under task transformations appears to reflect a replication of patterns learned during training," the researchers write.

For the moment, it appears that a truly Heuristic Algorithmic computer does not exist, as even the Heuristics that these learning models are using are Algorithmic and can't recognize what's outside the box.