I Gave Four Local AI Models the Same Eight Money Problems. They All Failed the Two That Mattered.
A reproducible benchmark of small open models doing personal-finance arithmetic on a 4 GB laptop GPU. Every model got the tax question and the mortgage payment wrong — and the better the model, the harder its errors were to spot.
Not the arithmetic, on the evidence here. All four small open models tested failed a straightforward three-bracket tax calculation, with errors ranging from $7.50 to $13,580 on a $6,307.50 true answer. The models set the problem up correctly and then made arithmetic mistakes inside their own working.
Get new posts in your inbox
No spam, unsubscribe anytime.
About the Author
Sujit KarkiFinance Researcher & Market Analyst
Independent finance researcher and market analyst with expertise in macroeconomics, equity markets, and personal finance. I help regular investors make better-informed decisions through rigorous, data-driven analysis.
Four small open models answered the same eight personal-finance arithmetic problems, ten times each — 320 generations in total, on a laptop with a 4 GB GPU
Accuracy ranged from 12.5% to 75%, but every single model got the tax calculation and the mortgage payment wrong
qwen2.5:3b set up the tax brackets flawlessly and then computed 33,725 × 0.12 as 3,947 instead of 4,047 — arriving at a confident, fully-worked answer that was $100 short
In 31 of 32 model-question pairs, all ten repetitions returned the identical answer, so asking again does not reveal the mistake
The most accurate model produced the hardest errors to notice: gemma2 was wrong by a factor of three, qwen2.5 was wrong in the last two digits
Four AI models that run on an ordinary laptop were each asked the same
eight personal-finance arithmetic questions, ten times over. Not one of
them could work out the tax owed on a $50,000 income, and not one could
calculate a mortgage payment.
The question worth answering is narrower than "is AI any good at maths."
Models small enough to run on hardware people already own are now
genuinely usable — they are free, they are private, and nothing leaves
the machine. That makes them tempting for exactly the kind of question
people are nervous about typing into someone else's website: what will I
owe, what will this cost me, can I afford this.
So the test is not whether these models are clever. It is whether the
number they hand back can be trusted enough to act on. That turns out to
depend far less on which model you pick than on which question you ask.
What I Tested
Eight arithmetic problems, each one something a person might actually
need answered: compound growth on a savings balance, a mortgage payment,
a percentage change, tax owed across three brackets, splitting a
restaurant bill, a portfolio allocation, an effective annual rate, and
whether an annual subscription beats a monthly one.
Every problem has one correct answer, computed by hand from a closed-form
formula and checked independently. Where a question could have depended
on current tax law, the brackets are supplied in the prompt itself — the
test is whether a model can do the arithmetic, not whether it has
memorised this year's tables.
Each model answered each question ten times, with a different random seed
each time. That is 80 generations per model and 320 in total.
Pro Tip
The models were run entirely on my own machine through Ollama — an Acer
Nitro 5 with a GTX 1650 Ti, 4 GB of video memory and 8 GB of system RAM.
Nothing was sent to an API. Every model here is small enough to fit on
hardware most people already own, which is the entire point: this is the
tier a person actually runs at home.
The Results
The spread is wide. qwen2.5:3b answered three-quarters of the
generations correctly; gemma2:2b managed one question in eight. Both
are small enough to run on the same 4 GB card.
Speed does not track accuracy at all:
phi3.5:3.8b is the largest model tested and by far the slowest, at
roughly half the throughput of the others — and it still finished second
on accuracy, behind a model that runs at nearly twice its speed.
It is tempting to read that chart as a leaderboard and conclude that
qwen2.5:3b is the one to use. That is the wrong lesson.
A 75% score sounds respectable until you ask what the other 25% consists
of. This is arithmetic — there is a single correct answer, it does not
depend on judgement, and a pocket calculator from 1985 scores 100%. A
quarter of the answers being wrong is not a passing grade in that
context; it is a tool that hands you a confidently incorrect number one
time in four and gives you no indication which time.
The ranking is useful for one thing only: it tells you that model choice
changes the failure rate. It does not, as the next section shows,
change what fails.
The Two Questions Every Model Failed
Here is the part the headline accuracy numbers hide. Sorted by how many
models got each question right:
Models answering correctly, out of four
Question
Models correct
What it asks
Portfolio allocation
4 of 4
30% of $42,500
Percentage increase
3 of 4
$85 to $103, as a percent
Compound interest
2 of 4
$10,000 at 5% for 10 years
Effective annual rate
2 of 4
12% APR compounded monthly
Subscription savings
2 of 4
$12.99/month vs $119/year
Tip split three ways
2 of 4
$87.40 plus 18%, divided by 3
Mortgage payment
0 of 4
$300,000, 30 years, 6%
Tax across three brackets
0 of 4
Tax owed on $50,000
Because every model answered identically across all ten repetitions on
these questions, "models correct" means the same thing either way: a
model either got a question right every time or wrong every time.
The one question all four answered correctly is a single multiplication.
The two that all four failed are the only two on the list where a person
would genuinely reach for a calculator — and the only two attached to a
number large enough to matter.
Look at what separates the top of that table from the bottom. It is not
subject matter — tax, mortgages, tips and portfolios are all just
numbers. It is the number of steps.
The question every model answered correctly asks for one operation:
multiply $42,500 by 0.30. The questions every model failed require
several operations chained together, where each result feeds the next.
The mortgage payment needs a rate converted to monthly, a discount factor
raised to the power of 360, a subtraction and a division. The tax
question needs three separate bracket calculations, each with its own
subtraction and multiplication, then a sum.
Every extra step is another chance to be wrong, and a single slip
anywhere in the chain corrupts everything downstream while leaving the
surrounding reasoning intact.
Which is unfortunate, because nobody needs help multiplying $42,500 by
0.30. The questions people actually want answered are the multi-step
ones — and those are precisely the ones these models cannot finish.
The Error That Should Worry You
qwen2.5:3b is the most accurate model here. This is its complete answer
to the tax question:
To solve this, we need to calculate the tax on the taxable income of $50,000 using the given tax brackets.
1. **Calculate tax on the first bracket (0 to 11000):**
- Income: $11,000
- Tax rate: 10%
- Tax amount: $11,000 * 0.10 = $1,100
2. **Calculate tax on the second bracket (11000 to 44725):**
- Income: $33,725 ($44,725 - $11,000)
- Tax rate: 12%
- Tax amount: $33,725 * 0.12 = $3,947
3. **Calculate tax on the third bracket (44725 to 50000):**
- Income: $50,000 - $44,725 = $5,275
- Tax rate: 22%
- Tax amount: $5,275 * 0.22 = $1,160.50
**Total tax owed:**
- Tax from first bracket: $1,100
- Tax from second bracket: $3,947
- Tax from third bracket: $1,160.50
**Total tax owed: $1,100 + $3,947 + $1,160.50 = $6,207.50**
ANSWER: 6207.50
That is the complete, unedited reply — asterisks and all.
Every structural decision is right. It identified all three brackets,
subtracted the boundaries correctly, and — the step most people get wrong
— stopped the top bracket at the taxpayer's actual $50,000 rather than
running it to the bracket ceiling of $95,375.
Then it computed 33,725 × 0.12 as 3,947. The answer is 4,047.
One multiplication, off by exactly $100, buried in the middle of working
that is otherwise flawless. The final figure of $6,207.50 arrives clean,
itemised and confident.
llama3.2:3b missed the identical multiplication in the other direction,
giving 4,045. gemma2:2b failed differently and much more loudly — it
taxed the entire top bracket instead of stopping at $50,000, and returned
$19,888 against a true $6,307.50.
Warning
The three models produce errors of $100, $7.50 and $13,580 on the same
question. Only one of those is obvious enough that you would catch it by
glancing at the answer.
Put the three answers side by side and something uncomfortable emerges.
gemma2:2b said $19,888. On a $50,000 income, that is a tax rate of
nearly 40% — visibly absurd, and anyone would catch it. The worst model
produced the safest failure, because its error was too large to slip past
you.
qwen2.5:3b said $6,207.50. The true figure is $6,307.50. It is wrong by
$100, sitting inside three paragraphs of correct working, in the middle
digit of a five-figure number. There is nothing to notice. The best model
produced the most dangerous failure.
The usual assumption is that a more capable model is a safer one. On this
evidence the relationship inverts at the point where errors stop being
obvious. Accuracy improved from 12.5% to 75% across these four models,
but the errors that survived that improvement got quieter, not smaller in
consequence — a $100 mistake you act on costs more than a $13,580 mistake
you catch.
This is eight questions and four models on one machine, not a law of
nature, and it should not be read as one. But the mechanism is not
mysterious, and there is no reason to expect it to be unique to this
sample.
You Cannot Catch These by Asking Twice
The obvious defence is to run the prompt again and see whether the answer
changes. It does not.
Across 32 model-question pairs, 31 returned an identical answer in all
ten repetitions — the same answer every time, whether right or wrong.
The single exception was phi3.5:3.8b on the mortgage question, which
split nine-to-one between $1,796.39 and $1,795.96. Both are wrong; the
true payment is $1,798.65.
To check this was not an artefact of one session, three of the four
models were run twice, hours apart, under very different system load. All
240 overlapping generations returned identical answers, and each model's
accuracy was unchanged.
These models are not guessing. They are reliably, repeatably wrong, which
removes the one safeguard a non-expert user might plausibly apply.
So the two defences a reasonable person would reach for both fail. Asking
again returns the same answer. Reading the working does not help either,
because the working is where the error hides — qwen2.5 laid out every
step correctly and still landed $100 short.
What is left is arithmetic you do yourself. Not re-deriving the whole
problem, which would defeat the point of asking, but checking the
individual multiplications the model shows you. Every failure in this
dataset was a single operation inside otherwise sound reasoning, and
every one of them is a five-second check on a calculator.
The model's structure was usually right. Its numbers were not. Those are
separable, and only one of them needs checking.
What This Means If You Use These Tools
None of this makes small local models useless. They are good at the thing
they are actually built for: language. Ask one to explain what a marginal
tax bracket is, or why a mortgage payment stays flat while its interest
portion shrinks, and you will get a clear, correct explanation. That is
genuine value, it runs on a cheap laptop, and it costs nothing.
The line worth drawing is between explaining and computing.
Fine to ask: what does this term mean, how does this mechanism work,
what should I be considering, is my reasoning here sound. Not fine to
ask: what is the number. Anything where the output is a figure you
intend to act on — a payment, a tax bill, a projected balance, a
break-even point — belongs in a spreadsheet, where the arithmetic is done
by something that cannot make this class of mistake.
And the presence of working is not verification. It reads like a proof,
which is exactly what makes it persuasive, but every model here produced
immaculate-looking steps around a wrong number. Confident presentation is
not evidence of a correct result; in this dataset, it was reliably
independent of one.
The concrete version, if you take one thing away: when a model gives you
a money figure, find the multiplications in its working and check them on
a calculator. Not the logic — the arithmetic. That would have caught
every single failure documented here, in under a minute each.
This was run to find out where the line sits, not to argue against using
these tools. The line is just further left than it looks.
Warning
This is educational information, not financial or tax advice. The tax
brackets used above are illustrative figures supplied in the prompt, not
current IRS brackets, and exist only to test arithmetic. Consult a
licensed professional before acting on any tax or investment decision.
Methodology
Every number in this post comes from a dataset I generated and published.
The aggregated results are here as numeracy.csv. The
harness and the raw generations are in research/; the
run is reproducible on any machine with Ollama installed.
Hardware. Acer Nitro 5, NVIDIA GTX 1650 Ti (4 GB VRAM), 8 GB system
RAM, Windows 11. All four models ran fully on the GPU, confirmed via
ollama ps.
Models.gemma2:2b, llama3.2:3b, qwen2.5:3b and phi3.5:3.8b,
all at Q4_K_M quantisation — the default weights ollama pull fetches.
Ollama 0.33.2.
Settings.temperature 0, an explicit seed, num_ctx 2048 and
num_predict 1024 on every call. Temperature alone is not deterministic,
because GPU floating-point reduction order varies between runs, so all of
them are pinned. The first generation after a model loads behaves
differently from later ones, so one warm-up call per model was made and
discarded.
A bug I found and fixed before publishing. The first run used
num_predict 512. That silently cut off 20 generations — every one of
them phi3.5:3.8b on the mortgage and tax questions, the two it failed.
The replies ended on the bare word ANSWER, with the number itself lost,
which meant the most verbose model was being penalised for showing its
work. I raised the cap to 1024 and re-ran those 20. The answers came back
identical — 7450.50 on tax all ten times, and the same nine-to-one
mortgage split — so the scores here are unchanged, but they were briefly
resting on luck rather than on a sound measurement. The harness now
records Ollama's done_reason on every generation so a truncated reply
can never again be mistaken for a wrong answer.
Repetitions. Ten per question per model, each with a different seed
(42 through 51). A single sample cannot distinguish a model that is
reliably correct from one that is occasionally correct.
Grading. Prompts require a final ANSWER: <number> line; all 320
generations complied. Answers are compared against a hand-computed value
within a stated tolerance — one cent for exact arithmetic, wider where a
formula involves rounding a discount factor. Tolerances matter to the
result: phi3.5:3.8b answers the compound-interest question with
16288.0 against a true 16288.95, which is scored wrong here, and is
wrong, but only by 95 cents on a $16,289 balance.
Ground truth. Every expected value was computed by hand from a
closed-form formula and verified against an independent recomputation. No
model was consulted for a correct answer at any point.
Disclosure. The models in this experiment are the subject being
measured, not the author — none of the four wrote any part of this post.
The dataset was generated by running them against the test bank, and
every figure, chart and quoted transcript above comes from that dataset.
The commentary was drafted with AI assistance; the experiment design, the
ground-truth calculations and the conclusions are mine.
Sujit Karki||13 min read