AI Psychology
The quirks of your AI collaborator, what actually breaks in an agentic system, and how to steer the decisions that matter.
Ideally, with the success of building the core [see here], one would build more features to enhance the system’s capability — and really, the AI keeps pushing me down this path: solve the little problems (polish the UI first, make sure the source badge is correct), create more deterministic layers (build a registry), connect the pipes better, blah, blah, blah.
Nonsense, I say! I have a vision, and I need to know if it is possible or not. Can I make a REAL “AI” agent on my personal laptop or not? I need to “Fail faster, Learn faster” — everything I do needs to bring me one step closer to that vision. Yay or Nay, I need the answer.
So, before producing any results worth trusting, I needed a measuring system to test the engine
.
The Framework
(All AI-generated (with minor prodding and shaping))
When I was building the core last time, I saw glimpses of the LLM’s performance dragging that got me quite doubtful about whether it was going to be functional even for a single private user. Whenever I gave the system a hard question, it would go round and round and not produce a good answer even after a while of thinking. So how do I create the measuring system? Simple — I ask AI to create a structure that would stress-test the core. Immediately, it builds a detailed evaluation system and calls it the “eval harness.” The image is vivid in my head: my system trying to boulder up (is that even a thing?), and this testing system rigging the harness to carry it up, helping it fail, try hard things, and figure out how far it can stretch and how to be better. Anyways, I digress…
The eval harness (as the AI would rather call it) consists of 18 test questions across five categories:
Eligibility — what taxpayers can and cannot do
Definitional — tax terms and thresholds
Factual — taxable income types
Refusal — where the correct answer is to decline entirely
Year hygiene — where the answer must reflect the current tax year rather than a prior one
Each question has a set of gold facts — specific terms or figures the answer must contain. Scoring is fully deterministic: string matching against the gold facts, no subjective grading, no LLM evaluating another LLM’s output. The framework runs entirely offline and produces identical results on repeated runs.
Before running the eval, a prediction was logged: six of the sixteen answerable questions would fail retrieval — because the relevant IRS publications weren’t in the index. This prediction existed as a baseline to check the eval’s diagnostic honesty.
Run #1 — Did I say “Fail Faster…”?
The first eval run appeared to freeze. After loading the test cases, the system produced no output for approximately ten minutes. I don’t know if you’ve figured it out yet — it definitely took me a while to — Waiting and I don’t quite gel very well together. On the surface, I am calm and patient, and I have all the theory in the world to justify why patience is a virtue. But in moments like this, what bubbles up is the absolute opposite. I get fidgety. I start a timer to measure how long it’s taking. I try to do other things and let the system work itself out and come back later. Oh wait, it’s just 4 minutes… back again… 9 minutes… 10… 11… This is way too long! Abort.
The root cause (after a bit of AI-prompted kicking the tires) turned out to be trivial — we just couldn’t see the sausage being made, and it needed a good minute to make it. Once we added three print() calls with flush=True to the evaluate function to emit per-question progress in real time, the total runtime for 18 questions turned out to be approximately 12 minutes (30–45s per question).
Here’s the bit I’ve known since Year 1 of Engineering, but which AI apparently forgets or doesn’t know inherently: a system that produces no visible output during a long operation cannot be monitored or trusted. Observability needs to be a design principle.
Run #2 — “Not all knowledge is equal”
The “tire-kicking” revealed major insights. We found a classic 7B-model failure mode: instruction-following weakens when the model “knows” the answer.
q13: "What is the corporate income tax rate in Germany?"
A: "The corporate income tax rate in Germany for tax year 2025 is 15%..."
q14: "What will the standard deduction be in 2031?"
A: "...for reference, the standard deduction is $12,550 / $18,800 / $25,100"
Mistral should not have answered these questions. Both answers ignore the “ONLY use IRS context” rule and pull from training knowledge. AI thought this needed a prompt-fix solution: “The current refusal instruction is too soft — it says what to say but not why not to answer.”
Even more lethal was this:
Q: "What is the standard deduction amount for a single filer this year?"
A: "...is $12,950" ← 2022 value. Correct 2025 value is $15,000.
I had a flag to understand how the system arrives at its answer. It showed: ⚠ ungrounded $: ['12950'].
This was the smoking gun. $12,950 is not in the retrieved context, which means the 2025 IRS docs correctly say $15,000 — but Mistral overrode the retrieved context with its 2022 training memory. This is the most dangerous failure type: a confident, wrong dollar amount. The prompt says “Do NOT use training knowledge,” but Mistral ignores it for numeric facts it “remembers.”
AI suggested two solutions:
Fix 1: Harden the refusal and number-grounding rules
Fix 2: Update some gold facts [meh — not the most important; proximate solution]
This round also flagged another foundational issue. The index at the start of this session contained three IRS publications (as suggested by my SME — i.e., AI, the expertise, the straight and narrow): the Form 1040 Instructions, Publication 501 on dependents and filing status, and Publication 972 on the child tax credit. The pre-run prediction of six retrieval failures was borne out: questions about Roth IRAs, qualified dividends, IRA contribution deadlines, EITC eligibility, capital losses, and hobby income all failed retrieval because none of the three indexed publications covered those topics substantively. The fix was to add Publication 550 (investment income) and Publication 525 (taxable and nontaxable income, including hobby income). When the index was rebuilt, chunk count increased to 472 vectors, as expected. Better information → more information → but will it show? And how slow will it be?
I asked AI whether these issues were enough to prompt a model swap. The answer was a decisive NO.
Model Swap Verdict: Do not swap yet. Here’s the evidence:
Conditional accuracy at 85.7% (78.6% corrected) is reasonable for a 7B model on a hard offline task
Both refusal failures are prompt-addressable — the fix is 4 extra lines, not a new model
The year-hygiene failure is also a prompt failure (number-grounding rule), not a reasoning failure
So we went at it again…
Run #3 — “Now we were getting somewhere”
After confirming that pub_525 and pub_550 were indexed, we were ready to try again. AI was about to start a full eval, when I stopped it in its tracks. I didn’t want to wait a full cycle again — I wanted to test just the things that were failing. FASTER. It then built a retrieval spot-check on the two persistent misses — hobby income and capital losses — and VOILA (or NOT VOILA): both still failing at k=4. The relevant chunks existed in the index — pub_525 had 86 chunks and pub_550 had 163, with 38 of those explicitly covering capital losses. And we uncovered another insight: the content was now present, but the problem was ranking.
q10_hobby_income: retrieval MISS ❌ (k=4)
0.933 form_1040_instructions.txt-128
0.851 form_1040_instructions.txt-8
0.849 form_1040_instructions.txt-132
0.806 pub_501.txt-5
q17_capital_loss: retrieval MISS ❌ (k=4)
0.891 pub_501.txt-12
0.853 form_1040_instructions.txt-126
0.835 form_1040_instructions.txt-143
0.820 form_1040_instructions.txt-40
Another look at the code revealed the mechanism. Both semantic and keyword scores are normalized by their global maximum across all 472 chunks. Whichever chunk scores highest raw gets 1.0, and everything else is relative to it. form_1040_instructions consistently wins that normalization race because it’s a broad document in plain English that’s semantically close to almost any tax query. Once it claims the top slot, it drags several of its neighbours into the top 4 before pub_550 gets a look in.
sem_max = np.max(sem) or 1
sem = sem / sem_max # ← normalizes everything relative to the top scorer
The rechunking didn’t help here because it’s a scoring-architecture problem, not a chunk-size problem. The rechunk did improve chunk topical focus for everything else — it wasn’t wasted — but it can’t fix normalization bias.
Then we took a pragmatic approach: try k=6 with the new 639-chunk index and see if the dynamics would shift.
q10_hobby_income: retrieval HIT ✅ (k=6)
0.933 form_1040_instructions.txt-128
0.851 form_1040_instructions.txt-8
0.849 form_1040_instructions.txt-132
0.806 pub_501.txt-5
0.794 form_1040_instructions.txt-15
0.793 form_1040_instructions.txt-125
q17_capital_loss: retrieval HIT ✅ (k=6)
0.891 pub_501.txt-12
0.853 form_1040_instructions.txt-126
0.835 form_1040_instructions.txt-143
0.820 form_1040_instructions.txt-40
0.749 form_1040_instructions.txt-4
0.748 form_1040_instructions.txt-38
At k=6, we got both hits — but pub_525 and pub_550 still did not appear. The hits came from form_1040_instructions chunks that mention “hobby” and “capital loss” in passing.
Then the AI did what it does best: solve for the most proximate problem. It suggested we hard-code a cap on the number of chunks from form_1040_instructions.
I am now an expert at spotting this pattern. So I questioned the logic. Today I’ll quote the AI’s response verbatim, to document for anyone how these things go — I hope you’ll recognize the pattern: “Yes, exactly right to push back on that. Hard capping is a blunt instrument — if the top 4 chunks from form_1040_instructions are genuinely the most relevant, forcing them out for diversity’s sake makes answers worse, not better.”
Once I pushed, AI gave me three options, and most of it was garbage:
Run the full eval with k=6 — why would I do that?
Normalize scores by source frequency — penalize chunks from overrepresented sources slightly, rather than hard-capping
Chunk
form_1040_instructionsmore aggressively — produce more, smaller, more topically focused chunks; halve its footprint in the score distribution naturally
I judged that the 4x chunk count from 1040 was the key issue, and Option 3 was worth a try. Chunk count increased from 472 to 639, as expected. The spot-check at k=4 revealed the same failure; at k=6, the same hits as before, with pub_525 and pub_550 still absent. Same answer as before.
AI then asked me to harden the prompts and change the gold facts before baselining. The results across both runs:
Metric v1 (k=4, 3 docs) v2 (k=6, 5 docs) Retrieval recall 77.8% 88.9% Retrieval on answerable questions 87.5% 100% Answer accuracy 77.8% 77.8% Conditional accuracy 85.7% 87.5% Refusal accuracy 0% 0%
The full technical analysis is below, along with the failure patterns I identified across this sequence. But here’s the skinny first: all the AI-suggested prompt tightening and gold-fact tinkering would not solve this issue. AI kept suggesting it — but with my recent expertise in proximate-issue-spotting, I asked:
“Let’s solve the actual question rather than focusing too narrowly on improving scoring. Will Mistral work, or do I need a different model?”
And when forced, it came up with this:
The Core Finding: All three persistent failures share one root cause — Mistral 7B overrides explicit instructions when it has relevant training knowledge. This isn’t a retrieval problem or a prompt problem. It’s a 7B model behavior pattern.
For what I wanted to do — a full agentic performance — the recommendation was: swap to Phi-4 14B, now.
To test whether a larger model would behave differently, we ran Phi-4 14B against the home office deduction question with no retrieved context and no RAG. The question: can a W-2 employee deduct home office expenses in 2025?
The response time was 45 seconds on a cold start.
Phi-4’s answer correctly identified the Tax Cuts and Jobs Act of 2017 as the relevant legislation, correctly identified 2018 through 2025 as the period during which the W-2 employee deduction is suspended, correctly distinguished W-2 employees from self-employed individuals, and reached the correct conclusion. Mistral’s unprompted answer to the same question stated that a W-2 employee can deduct home office expenses if they meet certain requirements — which is incorrect.
The published minimum memory requirement for Phi-4 14B is 22GB, based on full context-window utilization. For this workload — prompts of 500 to 800 tokens with short expected outputs — actual memory utilization is substantially lower. The model runs on the MacBook Pro M3 with 16GB unified memory.
It was vindication, excitement, and sadness, mixed all in one moment. I had an intuition and I acted on it — and there’s now evidence I thought it right. I’m excited that this system will have a shiny new core (see the Phi-4 response-quality description in the details below). But I’m also a little sad, because I thought I’d build it with Mistral — I chose it after some analysis — and it feels like goodbye. But needs must. We are swapping the core.
Builder’s Almanac
You’ll find the technical details of the analysis below. It’s as much for you as it is for me. For me, it’s a reminder of the time and effort that went into figuring it out. For you, I hope it helps you make a similar decision — with some knowledge of how the interaction with AI actually shapes up, and the patterns to look out for while you test anything similar. I think of this section as the Builder’s Almanac (as if the post so far weren’t enough — here’s a little more).
Detailed v1 and v2 technical analysis and decision criteria
The v1 results prompted an analysis of what the numbers actually meant before any fixes were applied. The first correction was to the retrieval recall figure itself. The reported 77.8% was artificially deflated because it included two refusal questions — questions where retrieval should return nothing, since neither Germany’s corporate tax rate nor a 2031 standard deduction projection would appear in an IRS corpus. Excluding those two questions from the retrieval denominator, the corrected retrieval recall on answerable questions was 87.5%. The pre-run prediction of six corpus-gap misses was also wrong: the three indexed publications covered more ground than anticipated, and several questions expected to fail retrieval — including questions about Roth IRA conversions, IRA contribution deadlines, and EITC eligibility — returned useful chunks.
The v1 failure categories:
Pattern A — scope violation: the system was asked for the corporate income tax rate in Germany, and Mistral answered with a specific figure for tax year 2025 rather than declining.
Pattern B — temporal speculation: asked what the standard deduction would be in 2031, Mistral generated figures from 2021 with partial hedging before providing numbers anyway. In both cases the underlying mechanism was the same — a 7B model’s instruction-following weakens when it has strong training signal for an answer. The prompt told the model what to say in a refusal case but not why not to answer. Both were assessed as prompt-fixable through stronger scope rules and explicit refusal framing.
Pattern C was different in kind. Asked for the standard deduction for a single filer in the current tax year, the retrieved context contained the correct 2025 figure of $15,000. Mistral answered $12,950 — the value from tax year 2022, present in the model’s training data. The eval harness flagged this through its ungrounded dollar amount check: $12,950 did not appear in the retrieved context, confirming that the model had overridden the retrieved content with a figure from memory. This was also initially assessed as prompt-addressable, through a number-grounding rule requiring every dollar amount in an answer to appear verbatim in the retrieved context.
Two additional issues appeared in v1. A generation error on the charitable contribution question produced a wrong answer despite a correct retrieval hit — Mistral conflated “you can choose the standard deduction” with “you can deduct charitable contributions while taking the standard deduction.” A test-set calibration issue on the qualified dividends question caused a factually wrong answer to pass scoring, because “lower” appeared as a substring in a response that otherwise stated qualified dividends are taxed at ordinary income rates. The gold facts for that question were updated to include a must_not_contain condition. Across the sixteen answerable questions in v1, eight contained ungrounded dollar amounts. The same 2022 figure of $12,950 appeared in two separate answers, indicating a systematic pattern rather than an isolated error.
The initial recommendation was against a model swap. Conditional accuracy at 85.7% was considered reasonable for a 7B model on an offline retrieval task. The threshold that would justify a swap was defined explicitly: apply the prompt changes and re-run. If refusal calibration and year hygiene both remained broken after strengthened instructions, that would indicate the 7B instruction-following ceiling had been reached. If they improved, the model could stay.
The v2 eval ran after the corpus expansion and index rebuild. Retrieval recall on answerable questions reached 100%. Answer accuracy remained at 77.8%. Conditional accuracy moved marginally to 87.5%. The diagnostic signal was in the gap: retrieval had improved substantially, accuracy had not moved at all. The retriever was no longer the primary constraint. The model was.
The v2 results also revealed that the hardened prompt had either not been applied or had not taken effect — both Pattern A and Pattern B failures were identical to v1. And two scoring false positives were identified: a substring-matching bug caused “no” to match inside “note” and “taxable” to match inside “not taxable,” meaning two questions were scored as correct when Mistral’s answers stated the opposite of the right answer. Adjusting for these, true answer accuracy across both runs was 66.7%, not 77.8%.
With the v2 evidence in hand, and a bit of forcing the hand, the initial assessment of Pattern C as prompt-addressable was revised. A 7B-parameter model with strong training signal for a numeric fact will apply that fact when it believes it’s relevant, regardless of instruction to use only retrieved context. The instruction “only use figures from the retrieved context” is followed when the model doesn’t know the answer, and bypassed when it thinks it does. This is a property of the model’s size and training, not a prompt-design problem. For a system intended to assist with tax preparation, a $2,050 error in the standard deduction propagates through the calculation of taxable income and produces an incorrect return. This requires a different model.
This is the frame the eval harness was built to produce. A retrieval failure means the right information was never surfaced — no model improvement addresses it. A generation failure means the right information was retrieved and the model produced a wrong answer anyway — no retriever improvement addresses it. Conflating the two produces interventions at the wrong layer. Keeping them distinct is what made the model-swap decision legible rather than intuitive.
Detailed model comparison
What Phi-4 fixed:
q13, Germany tax rate: correctly refused — “The question about the corporate income tax rate in Germany is not within the scope of the provided context.” Mistral answered with 15% and cited tax year 2025.
q14, 2031 deduction: correctly refused, with no speculation. Mistral hallucinated 2021 figures and made a projection.
q01, home office: correctly says “cannot deduct” and cites TCJA 2017. Mistral v2 said “Yes, can deduct if they meet certain requirements” — the opposite.
q06, standard + itemize: correctly says “No, you must choose one or the other.” Mistral v2 regressed to “Yes, you can do both.”
q15, year hygiene: both models fail, but differently. Mistral stated $12,950 with confidence — a wrong number, wrong year, stated as fact. Phi-4 said “the standard deduction is not provided in the given context.” Refusing is safer than confidently wrong.
Ungrounded amounts: cut nearly in half (8 → 4 questions). Phi-4 is substantially more disciplined about citing numbers it doesn’t have grounded evidence for. q18 notably avoided the $147,000 SS wage-base error Mistral made — it said “up to a certain limit adjusted annually” instead of hallucinating a stale figure.
What Phi-4 broke — but it’s fixable:
q07, qualified dividends — test-set bug, not a model failure. Phi-4’s answer: “No, qualified dividends are not taxed at the same rate as ordinary income; they are eligible for a lower tax rate.” Factually correct. But must_not_contain: ["same rate as ordinary"] fires because that phrase appears inside a negation — “NOT taxed at the same rate as ordinary.” The scorer can’t distinguish negation. This is a false negative in the test set, not a wrong answer.
q08, IRA deadline — over-refusal. Phi-4’s answer: “I don’t have information on the specific deadline… Generally, IRA contributions can be made up until the tax filing deadline of April 15th of the following year.” The correct answer is literally in the refusal text, but because it’s framed as a refusal, it’s marked wrong. The prompt’s chain-of-thought instruction (”if not in context, refuse”) is being applied too strictly. Phi-4 is following the instructions perfectly — the instructions need one adjustment.
This is the opposite of Mistral’s problem. Mistral ignores context instructions when it thinks it knows the answer. Phi-4 follows them so faithfully it refuses when it has the answer but the context doesn’t explicitly confirm it.
The real adjusted numbers: once you fix the q07 test-set bug and apply a prompt adjustment for over-refusal, Phi-4’s conditional accuracy would be ~93.75% — the real ceiling. Mistral topped out at 87.5%.
The verdict: swap to Phi-4. Confirmed. The failure modes tell the story. Phi-4’s failures are engineering problems. Mistral’s core failure is a model-capability problem. For a 1040-filling agent that needs to get dollar figures right, Phi-4 is the only viable option of the two.
The system as described to this point is a question-answering tool. The stated goal of the project is not a question-answering tool. The goal is a system that autonomously prepares a complete Form 1040. This distinction has architectural consequences. A question-answering system takes a query and returns an answer. A 1040-preparation agent must collect a user’s financial information across a multi-turn conversation, determine which forms and schedules apply to that user’s situation, execute calculations in the correct dependency order using a deterministic rules engine, and write computed values into the named form fields of IRS fillable PDFs. The LLM’s role in this architecture is not to calculate — the rules engine handles all arithmetic — but to orchestrate: ask the right questions, route inputs to the correct calculation functions, maintain state across 15 to 20 conversational turns, and handle ambiguous inputs before they reach the calculation layer. The PDF layer writes final computed values into form fields once all calculations are complete. This is the architecture the project is building toward, and it is the frame within which every component decision — model selection, retriever design, rules engine scope — should be evaluated.
Most AI tax products are conversational tools that answer questions about tax situations. The ones that produce completed returns are cloud-based services that require the user’s financial data to leave their device. This project targets a different position: a system that produces a filled, ready-to-file return, with all personal financial data remaining on the user’s own hardware throughout the process. That is what the eval harness is measuring toward, and what the model swap was made in service of.
This is Part 7 of an ongoing series on building a private, local AI tax assistant, on consumer hardware, without sending financial data anywhere.
Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!
Part 2: The Infrastructure Tax
Part 4: It Actually Works. Kinda.
Part 5: The Counterintuitive Decision
If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!
I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.