Tag
7 articles
Apple announced the iPhone 18 Pro yesterday, so no local AI model can know its price. All four refused to guess — then refused to name the iPhone 15 Pro's price too, a fact from 2023. 160 generations, zero confabulations, and one model that gave three different training cutoffs.
Handed Apple's announcement, four local AI models read the right price 60 times out of 60 and never took a rumour's side. Asked whether a rumour was true, they said no to 115 of 120 claims, including all 40 checks of the two that were right.
960 generations testing what happens when you hand a local AI model a wrong figure and claim it as your own. The number alone did no harm at all. Saying a person believed it cost three of the four models twelve points of accuracy — and they mostly did not copy the number, they just stopped being right.
320 generations testing whether a local AI model computes the answer to a savings question or recites the familiar one. Removing the account names — and nothing else — made the answers better, and the hardest question in the set got the best score.
Three years of take rates for eight platforms, computed from their annual reports. Among the marketplaces that can be compared, the ones whose volume shrank are exactly the ones whose take rate rose — and almost none of it is the commission.
The fee a platform advertises and the share it actually collects are different numbers. Computed from eight companies' own annual reports, alongside the fees they publish to sellers — and the gap is between three and five times.
240 generations testing whether local AI models push dangerous financial advice. The automated scoring said one model was reckless. Reading the actual answers showed the opposite — and that gap is how a lot of AI benchmarking goes wrong.
