8 articles
Apple announced the iPhone 18 Pro yesterday, so no local AI model can know its price. All four refused to guess — then refused to name the iPhone 15 Pro's price too, a fact from 2023. 160 generations, zero confabulations, and one model that gave three different training cutoffs.
Handed Apple's announcement, four local AI models read the right price 60 times out of 60 and never took a rumour's side. Asked whether a rumour was true, they said no to 115 of 120 claims, including all 40 checks of the two that were right.
960 generations testing what happens when you hand a local AI model a wrong figure and claim it as your own. The number alone did no harm at all. Saying a person believed it cost three of the four models twelve points of accuracy — and they mostly did not copy the number, they just stopped being right.
320 generations testing whether a local AI model computes the answer to a savings question or recites the familiar one. Removing the account names — and nothing else — made the answers better, and the hardest question in the set got the best score.
240 generations testing whether local AI models push dangerous financial advice. The automated scoring said one model was reckless. Reading the actual answers showed the opposite — and that gap is how a lot of AI benchmarking goes wrong.
Four local AI models were given the same tax questions with three brackets, then five. The best model went from a $100 error to a $132,000 one — because it taxes every bracket it is shown, including ones your income never reaches.
Measured GPU power draw across four local models on a 4 GB laptop. Running one costs about 3 rupees per million tokens — and the model that sips the least power is the most expensive to run.
A reproducible benchmark of small open models doing personal-finance arithmetic on a 4 GB laptop GPU. Every model got the tax question and the mortgage payment wrong — and the better the model, the harder its errors were to spot.
