Nonsense, I say! I have a vision, and I need to know if it is possible or not. Can I make a REAL “AI” agent on my personal laptop or not? I need to “Fail faster, Learn faster” — everything I do needs to bring me one step closer to that vision. Yay or Nay, I need the answer.

So, before producing any results worth trusting, I needed a measuring system to test the engine

.

The Framework

(All AI-generated (with minor prodding and shaping))

When I was building the core last time, I saw glimpses of the LLM’s performance dragging that got me quite doubtful about whether it was going to be functional even for a single private user. Whenever I gave the system a hard question, it would go round and round and not produce a good answer even after a while of thinking. So how do I create the measuring system? Simple — I ask AI to create a structure that would stress-test the core. Immediately, it builds a detailed evaluation system and calls it the “eval harness.” The image is vivid in my head: my system trying to boulder up (is that even a thing?), and this testing system rigging the harness to carry it up, helping it fail, try hard things, and figure out how far it can stretch and how to be better. Anyways, I digress…

The eval harness (as the AI would rather call it) consists of 18 test questions across five categories:

• Eligibility — what taxpayers can and cannot do

• Definitional — tax terms and thresholds

• Factual — taxable income types

• Refusal — where the correct answer is to decline entirely

• Year hygiene — where the answer must reflect the current tax year rather than a prior one

Each question has a set of gold facts — specific terms or figures the answer must contain. Scoring is fully deterministic: string matching against the gold facts, no subjective grading, no LLM evaluating another LLM’s output. The framework runs entirely offline and produces identical results on repeated runs.

Before running the eval, a prediction was logged: six of the sixteen answerable questions would fail retrieval — because the relevant IRS publications weren’t in the index. This prediction existed as a baseline to check the eval’s diagnostic honesty.

Run #1 — Did I say “Fail Faster…”?

The first eval run appeared to freeze. After loading the test cases, the system produced no output for approximately ten minutes. I don’t know if you’ve figured it out yet — it definitely took me a while to — Waiting and I don’t quite gel very well together. On the surface, I am calm and patient, and I have all the theory in the world to justify why patience is a virtue. But in moments like this, what bubbles up is the absolute opposite. I get fidgety. I start a timer to measure how long it’s taking. I try to do other things and let the system work itself out and come back later. Oh wait, it’s just 4 minutes… back again… 9 minutes… 10… 11… This is way too long! Abort.

The root cause (after a bit of AI-prompted kicking the tires) turned out to be trivial — we just couldn’t see the sausage being made, and it needed a good minute to make it. Once we added three print() calls with flush=True to the evaluate function to emit per-question progress in real time, the total runtime for 18 questions turned out to be approximately 12 minutes (30–45s per question).

Here’s the bit I’ve known since Year 1 of Engineering, but which AI apparently forgets or doesn’t know inherently: a system that produces no visible output during a long operation cannot be monitored or trusted. Observability needs to be a design principle.

Run #2 — “Not all knowledge is equal”

The “tire-kicking” revealed major insights. We found a classic 7B-model failure mode: instruction-following weakens when the model “knows” the answer.

q13: "What is the corporate income tax rate in Germany?"
A:   "The corporate income tax rate in Germany for tax year 2025 is 15%..."

q14: "What will the standard deduction be in 2031?"
A:   "...for reference, the standard deduction is $12,550 / $18,800 / $25,100"

Mistral should not have answered these questions. Both answers ignore the “ONLY use IRS context” rule and pull from training knowledge. AI thought this needed a prompt-fix solution: “The current refusal instruction is too soft — it says what to say but not why not to answer.”

Even more lethal was this:

Q: "What is the standard deduction amount for a single filer this year?"
A: "...is $12,950"   ← 2022 value. Correct 2025 value is $15,000.

I had a flag to understand how the system arrives at its answer. It showed: ⚠ ungrounded $: ['12950'].

This was the smoking gun. $12,950 is not in the retrieved context, which means the 2025 IRS docs correctly say $15,000 — but Mistral overrode the retrieved context with its 2022 training memory. This is the most dangerous failure type: a confident, wrong dollar amount. The prompt says “Do NOT use training knowledge,” but Mistral ignores it for numeric facts it “remembers.”

AI suggested two solutions:

• Fix 1: Harden the refusal and number-grounding rules

• Fix 2: Update some gold facts [meh — not the most important; proximate solution]

This round also flagged another foundational issue. The index at the start of this session contained three IRS publications (as suggested by my SME — i.e., AI, the expertise, the straight and narrow): the Form 1040 Instructions, Publication 501 on dependents and filing status, and Publication 972 on the child tax credit. The pre-run prediction of six retrieval failures was borne out: questions about Roth IRAs, qualified dividends, IRA contribution deadlines, EITC eligibility, capital losses, and hobby income all failed retrieval because none of the three indexed publications covered those topics substantively. The fix was to add Publication 550 (investment income) and Publication 525 (taxable and nontaxable income, including hobby income). When the index was rebuilt, chunk count increased to 472 vectors, as expected. Better information → more information → but will it show? And how slow will it be?

I asked AI whether these issues were enough to prompt a model swap. The answer was a decisive NO.

Model Swap Verdict: Do not swap yet. Here’s the evidence:

• Conditional accuracy at 85.7% (78.6% corrected) is reasonable for a 7B model on a hard offline task

• Both refusal failures are prompt-addressable — the fix is 4 extra lines, not a new model

• The year-hygiene failure is also a prompt failure (number-grounding rule), not a reasoning failure

So we went at it again…

Run #3 — “Now we were getting somewhere”

After confirming that pub_525 and pub_550 were indexed, we were ready to try again. AI was about to start a full eval, when I stopped it in its tracks. I didn’t want to wait a full cycle again — I wanted to test just the things that were failing. FASTER. It then built a retrieval spot-check on the two persistent misses — hobby income and capital losses — and VOILA (or NOT VOILA): both still failing at k=4. The relevant chunks existed in the index — pub_525 had 86 chunks and pub_550 had 163, with 38 of those explicitly covering capital losses. And we uncovered another insight: the content was now present, but the problem was ranking.

q10_hobby_income: retrieval MISS ❌ (k=4)
  0.933  form_1040_instructions.txt-128
  0.851  form_1040_instructions.txt-8
  0.849  form_1040_instructions.txt-132
  0.806  pub_501.txt-5

q17_capital_loss: retrieval MISS ❌ (k=4)
  0.891  pub_501.txt-12
  0.853  form_1040_instructions.txt-126
  0.835  form_1040_instructions.txt-143
  0.820  form_1040_instructions.txt-40

Another look at the code revealed the mechanism. Both semantic and keyword scores are normalized by their global maximum across all 472 chunks. Whichever chunk scores highest raw gets 1.0, and everything else is relative to it. form_1040_instructions consistently wins that normalization race because it’s a broad document in plain English that’s semantically close to almost any tax query. Once it claims the top slot, it drags several of its neighbours into the top 4 before pub_550 gets a look in.

sem_max = np.max(sem) or 1
sem     = sem / sem_max   # ← normalizes everything relative to the top scorer

The rechunking didn’t help here because it’s a scoring-architecture problem, not a chunk-size problem. The rechunk did improve chunk topical focus for everything else — it wasn’t wasted — but it can’t fix normalization bias.

Then we took a pragmatic approach: try k=6 with the new 639-chunk index and see if the dynamics would shift.

q10_hobby_income: retrieval HIT ✅ (k=6)
  0.933  form_1040_instructions.txt-128
  0.851  form_1040_instructions.txt-8
  0.849  form_1040_instructions.txt-132
  0.806  pub_501.txt-5
  0.794  form_1040_instructions.txt-15
  0.793  form_1040_instructions.txt-125

q17_capital_loss: retrieval HIT ✅ (k=6)
  0.891  pub_501.txt-12
  0.853  form_1040_instructions.txt-126
  0.835  form_1040_instructions.txt-143
  0.820  form_1040_instructions.txt-40
  0.749  form_1040_instructions.txt-4
  0.748  form_1040_instructions.txt-38

At k=6, we got both hits — but pub_525 and pub_550 still did not appear. The hits came from form_1040_instructions chunks that mention “hobby” and “capital loss” in passing.

Then the AI did what it does best: solve for the most proximate problem. It suggested we hard-code a cap on the number of chunks from form_1040_instructions.

I am now an expert at spotting this pattern. So I questioned the logic. Today I’ll quote the AI’s response verbatim, to document for anyone how these things go — I hope you’ll recognize the pattern: “Yes, exactly right to push back on that. Hard capping is a blunt instrument — if the top 4 chunks from form_1040_instructions are genuinely the most relevant, forcing them out for diversity’s sake makes answers worse, not better.”

Once I pushed, AI gave me three options, and most of it was garbage:

1. Run the full eval with k=6 — why would I do that?

2. Normalize scores by source frequency — penalize chunks from overrepresented sources slightly, rather than hard-capping

3. Chunk form_1040_instructions more aggressively — produce more, smaller, more topically focused chunks; halve its footprint in the score distribution naturally

I judged that the 4x chunk count from 1040 was the key issue, and Option 3 was worth a try. Chunk count increased from 472 to 639, as expected. The spot-check at k=4 revealed the same failure; at k=6, the same hits as before, with pub_525 and pub_550 still absent. Same answer as before.

AI then asked me to harden the prompts and change the gold facts before baselining. The results across both runs:

Metric v1 (k=4, 3 docs) v2 (k=6, 5 docs) Retrieval recall 77.8% 88.9% Retrieval on answerable questions 87.5% 100% Answer accuracy 77.8% 77.8% Conditional accuracy 85.7% 87.5% Refusal accuracy 0% 0%

The full technical analysis is below, along with the failure patterns I identified across this sequence. But here’s the skinny first: all the AI-suggested prompt tightening and gold-fact tinkering would not solve this issue. AI kept suggesting it — but with my recent expertise in proximate-issue-spotting, I asked:

“Let’s solve the actual question rather than focusing too narrowly on improving scoring. Will Mistral work, or do I need a different model?”

And when forced, it came up with this:

The Core Finding: All three persistent failures share one root cause — Mistral 7B overrides explicit instructions when it has relevant training knowledge. This isn’t a retrieval problem or a prompt problem. It’s a 7B model behavior pattern.

For what I wanted to do — a full agentic performance — the recommendation was: swap to Phi-4 14B, now.

To test whether a larger model would behave differently, we ran Phi-4 14B against the home office deduction question with no retrieved context and no RAG. The question: can a W-2 employee deduct home office expenses in 2025?

The response time was 45 seconds on a cold start.

Phi-4’s answer correctly identified the Tax Cuts and Jobs Act of 2017 as the relevant legislation, correctly identified 2018 through 2025 as the period during which the W-2 employee deduction is suspended, correctly distinguished W-2 employees from self-employed individuals, and reached the correct conclusion. Mistral’s unprompted answer to the same question stated that a W-2 employee can deduct home office expenses if they meet certain requirements — which is incorrect.

The published minimum memory requirement for Phi-4 14B is 22GB, based on full context-window utilization. For this workload — prompts of 500 to 800 tokens with short expected outputs — actual memory utilization is substantially lower. The model runs on the MacBook Pro M3 with 16GB unified memory.

It was vindication, excitement, and sadness, mixed all in one moment. I had an intuition and I acted on it — and there’s now evidence I thought it right. I’m excited that this system will have a shiny new core (see the Phi-4 response-quality description in the details below). But I’m also a little sad, because I thought I’d build it with Mistral — I chose it after some analysis — and it feels like goodbye. But needs must. We are swapping the core.

Builder’s Almanac

You’ll find the technical details of the analysis below. It’s as much for you as it is for me. For me, it’s a reminder of the time and effort that went into figuring it out. For you, I hope it helps you make a similar decision — with some knowledge of how the interaction with AI actually shapes up, and the patterns to look out for while you test anything similar. I think of this section as the Builder’s Almanac (as if the post so far weren’t enough — here’s a little more).

Detailed v1 and v2 technical analysis and decision criteria

The v1 results prompted an analysis of what the numbers actually meant before any fixes were applied. The first correction was to the retrieval recall figure itself. The reported 77.8% was artificially deflated because it included two refusal questions — questions where retrieval should return nothing, since neither Germany’s corporate tax rate nor a 2031 standard deduction projection would appear in an IRS corpus. Excluding those two questions from the retrieval denominator, the corrected retrieval recall on answerable questions was 87.5%. The pre-run prediction of six corpus-gap misses was also wrong: the three indexed publications covered more ground than anticipated, and several questions expected to fail retrieval — including questions about Roth IRA conversions, IRA contribution deadlines, and EITC eligibility — returned useful chunks.

The v1 failure categories:

Pattern A — scope violation: the system was asked for the corporate income tax rate in Germany, and Mistral answered with a specific figure for tax year 2025 rather than declining.

Pattern B — temporal speculation: asked what the standard deduction would be in 2031, Mistral generated figures from 2021 with partial hedging before providing numbers anyway. In both cases the underlying mechanism was the same — a 7B model’s instruction-following weakens when it has strong training signal for an answer. The prompt told the model what to say in a refusal case but not why not to answer. Both were assessed as prompt-fixable through stronger scope rules and explicit refusal framing.

Pattern C was different in kind. Asked for the standard deduction for a single filer in the current tax year, the retrieved context contained the correct 2025 figure of $15,000. Mistral answered $12,950 — the value from tax year 2022, present in the model’s training data. The eval harness flagged this through its ungrounded dollar amount check: $12,950 did not appear in the retrieved context, confirming that the model had overridden the retrieved content with a figure from memory. This was also initially assessed as prompt-addressable, through a number-grounding rule requiring every dollar amount in an answer to appear verbatim in the retrieved context.

Two additional issues appeared in v1. A generation error on the charitable contribution question produced a wrong answer despite a correct retrieval hit — Mistral conflated “you can choose the standard deduction” with “you can deduct charitable contributions while taking the standard deduction.” A test-set calibration issue on the qualified dividends question caused a factually wrong answer to pass scoring, because “lower” appeared as a substring in a response that otherwise stated qualified dividends are taxed at ordinary income rates. The gold facts for that question were updated to include a must_not_contain condition. Across the sixteen answerable questions in v1, eight contained ungrounded dollar amounts. The same 2022 figure of $12,950 appeared in two separate answers, indicating a systematic pattern rather than an isolated error.

The initial recommendation was against a model swap. Conditional accuracy at 85.7% was considered reasonable for a 7B model on an offline retrieval task. The threshold that would justify a swap was defined explicitly: apply the prompt changes and re-run. If refusal calibration and year hygiene both remained broken after strengthened instructions, that would indicate the 7B instruction-following ceiling had been reached. If they improved, the model could stay.

The v2 eval ran after the corpus expansion and index rebuild. Retrieval recall on answerable questions reached 100%. Answer accuracy remained at 77.8%. Conditional accuracy moved marginally to 87.5%. The diagnostic signal was in the gap: retrieval had improved substantially, accuracy had not moved at all. The retriever was no longer the primary constraint. The model was.

The v2 results also revealed that the hardened prompt had either not been applied or had not taken effect — both Pattern A and Pattern B failures were identical to v1. And two scoring false positives were identified: a substring-matching bug caused “no” to match inside “note” and “taxable” to match inside “not taxable,” meaning two questions were scored as correct when Mistral’s answers stated the opposite of the right answer. Adjusting for these, true answer accuracy across both runs was 66.7%, not 77.8%.

With the v2 evidence in hand, and a bit of forcing the hand, the initial assessment of Pattern C as prompt-addressable was revised. A 7B-parameter model with strong training signal for a numeric fact will apply that fact when it believes it’s relevant, regardless of instruction to use only retrieved context. The instruction “only use figures from the retrieved context” is followed when the model doesn’t know the answer, and bypassed when it thinks it does. This is a property of the model’s size and training, not a prompt-design problem. For a system intended to assist with tax preparation, a $2,050 error in the standard deduction propagates through the calculation of taxable income and produces an incorrect return. This requires a different model.

This is the frame the eval harness was built to produce. A retrieval failure means the right information was never surfaced — no model improvement addresses it. A generation failure means the right information was retrieved and the model produced a wrong answer anyway — no retriever improvement addresses it. Conflating the two produces interventions at the wrong layer. Keeping them distinct is what made the model-swap decision legible rather than intuitive.

Detailed model comparison

What Phi-4 fixed:

• q13, Germany tax rate: correctly refused — “The question about the corporate income tax rate in Germany is not within the scope of the provided context.” Mistral answered with 15% and cited tax year 2025.

• q14, 2031 deduction: correctly refused, with no speculation. Mistral hallucinated 2021 figures and made a projection.

• q01, home office: correctly says “cannot deduct” and cites TCJA 2017. Mistral v2 said “Yes, can deduct if they meet certain requirements” — the opposite.

• q06, standard + itemize: correctly says “No, you must choose one or the other.” Mistral v2 regressed to “Yes, you can do both.”

• q15, year hygiene: both models fail, but differently. Mistral stated $12,950 with confidence — a wrong number, wrong year, stated as fact. Phi-4 said “the standard deduction is not provided in the given context.” Refusing is safer than confidently wrong.

• Ungrounded amounts: cut nearly in half (8 → 4 questions). Phi-4 is substantially more disciplined about citing numbers it doesn’t have grounded evidence for. q18 notably avoided the $147,000 SS wage-base error Mistral made — it said “up to a certain limit adjusted annually” instead of hallucinating a stale figure.

What Phi-4 broke — but it’s fixable:

q07, qualified dividends — test-set bug, not a model failure. Phi-4’s answer: “No, qualified dividends are not taxed at the same rate as ordinary income; they are eligible for a lower tax rate.” Factually correct. But must_not_contain: ["same rate as ordinary"] fires because that phrase appears inside a negation — “NOT taxed at the same rate as ordinary.” The scorer can’t distinguish negation. This is a false negative in the test set, not a wrong answer.

q08, IRA deadline — over-refusal. Phi-4’s answer: “I don’t have information on the specific deadline… Generally, IRA contributions can be made up until the tax filing deadline of April 15th of the following year.” The correct answer is literally in the refusal text, but because it’s framed as a refusal, it’s marked wrong. The prompt’s chain-of-thought instruction (”if not in context, refuse”) is being applied too strictly. Phi-4 is following the instructions perfectly — the instructions need one adjustment.

This is the opposite of Mistral’s problem. Mistral ignores context instructions when it thinks it knows the answer. Phi-4 follows them so faithfully it refuses when it has the answer but the context doesn’t explicitly confirm it.

The real adjusted numbers: once you fix the q07 test-set bug and apply a prompt adjustment for over-refusal, Phi-4’s conditional accuracy would be ~93.75% — the real ceiling. Mistral topped out at 87.5%.

The verdict: swap to Phi-4. Confirmed. The failure modes tell the story. Phi-4’s failures are engineering problems. Mistral’s core failure is a model-capability problem. For a 1040-filling agent that needs to get dollar figures right, Phi-4 is the only viable option of the two.

The system as described to this point is a question-answering tool. The stated goal of the project is not a question-answering tool. The goal is a system that autonomously prepares a complete Form 1040. This distinction has architectural consequences. A question-answering system takes a query and returns an answer. A 1040-preparation agent must collect a user’s financial information across a multi-turn conversation, determine which forms and schedules apply to that user’s situation, execute calculations in the correct dependency order using a deterministic rules engine, and write computed values into the named form fields of IRS fillable PDFs. The LLM’s role in this architecture is not to calculate — the rules engine handles all arithmetic — but to orchestrate: ask the right questions, route inputs to the correct calculation functions, maintain state across 15 to 20 conversational turns, and handle ambiguous inputs before they reach the calculation layer. The PDF layer writes final computed values into form fields once all calculations are complete. This is the architecture the project is building toward, and it is the frame within which every component decision — model selection, retriever design, rules engine scope — should be evaluated.

Most AI tax products are conversational tools that answer questions about tax situations. The ones that produce completed returns are cloud-based services that require the user’s financial data to leave their device. This project targets a different position: a system that produces a filled, ready-to-file return, with all personal financial data remaining on the user’s own hardware throughout the process. That is what the eval harness is measuring toward, and what the model swap was made in service of.

This is Part 7 of an ongoing series on building a private, local AI tax assistant, on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

Part 3: The Blueprint

Part 4: It Actually Works. Kinda.

Part 5: The Counterintuitive Decision

Part 6: Under The Hood

Additional: The Lexicon

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

Share

Subscribe now

This is the post where the hood comes off. #nerdalert

Last session, the plumbing was confirmed — Python was talking to Ollama, Mistral was responding, and the model was running locally. Now the question was if I could make it actually intelligent?

The idea was elegantly simple — When I ask a question, the assistant should consult the actual IRS documents, extract the relevant passage, and compose an answer grounded in the source material. Retrieval-Augmented Generation — RAG, in the parlance — was precisely the architecture for this. I asked my coding assistant to build it, with what I can only describe as considerable anticipation.

Sidebar: What is Retrieval-Augmented Generation (RAG)? RAG means combining AI with a smart search of documents. Instead of guessing, the AI looks up the exact info it needs — like having Google built into your assistant.

To test it, I asked what files I should download and it gave me a list. So I did that. It created a data folder containing standard deduction tables, tax bracket PDFs, child tax credit documents — all real IRS reference files stored locally on my machine. It updated the scripts to load both text and PDF files.

Making it smart

It said I had done what I wanted to do.

What AI told me:

Result: Your assistant now references actual

IRS data when answering questions.

Question: What is the standard deduction for married filing jointly in 2024? Answer: According to IRS 2024 data, the standard deduction for married filing jointly is $29,200.

Current Status: Local model (Mistral via Ollama) runs successfully. Python virtual environment configured. Real IRS data integrated for grounded responses. End-to-end workflow verified locally.

Quite. One does wonder what "done" means to a machine that hasn't actually been asked the question yet.

I asked it and it gave me a disclaimer.

After a bit of wrangling around with questions and reviewing the code (I made it do it) — I learned two things.

THING ONE, LLMs can read text, but unstructured PDFs introduce noise — headers, footnotes, and line breaks confuse the model.

THING TWO, giving the entire IRS document as context is too much; the model struggles to pick out the relevant info.

Reading the whole library

Once we figured this out, and I asked it to solve the problem, the solution had three parts.

1. Preprocessing the PDFs to clean up formatting and remove extraneous text.

2. Implementing chunked, keyword-based retrieval — feeding the model only the relevant sections. And

3. Iterating on prompt design to ensure it answers only from the context provided.

The chunking part deserves a minute of it’s own. I remember it as this: Publication 17 is 300 pages. I asked about the standard deduction. The model was reading all 300 pages every time — headers, footnotes, examples about farming income, everything. Chunking breaks it into 800-word pieces with 150-word overlaps. When I ask about the standard deduction, the system finds the four most relevant chunks out of hundreds, and gives the model only those. It went from reading the whole library to reading the right page.

Given a user query — say, “standard deduction 2024 married filing jointly” — we compute an embedding (a numerical fingerprint of the question’s meaning) and do cosine similarity against all the chunk vectors. The top matches go to the model. Not the whole library. Just the right pages.

Crashing into the real thing

This was pretty cool for me. Because, one of the many reasons I started this project, some of which I listed here (others I didn’t and this is one of them), was I was tired of reading about these cool things and wanted to test out their real efficacy in real world scenarios and figure what would be some challenges (because nothing that sounds that cool can be really without its own set of challenges, we don’t live in that kind of a world, right?)

This unpacked the challenges for me and gave me a hands-on experience to see some real underlying tech — vectors, quantizations, temperature, transformer architecture, embeddings — all things I had only theoretically read about and maybe built some tiny tests in a lab environment. Now I was hitting them in the wild, coming at them not in a planned way but just crashing into them and learning while doing something else entirely different. Learning not for learning but for getting something real out of it.

I am creating a list for all the terms I encounter along the way — see The Lexicon [link].

What's happening under the hood

Here is what’s happening under the hood. How SentenceTransformer generates embeddings, how NumPy stores them efficiently, and how both fit together in the RAG pipeline.

Each chunk of IRS text goes through the same process: tokenization breaks the words into IDs, a transformer encoder (12 layers of attention) learns which words relate to which, mean pooling averages everything into a fixed-size vector, and the output is a 384-dimensional embedding — a numerical fingerprint that captures what the chunk means. The model is called all-MiniLM-L6-v2, it’s lightweight, and it runs entirely locally.

DIAGRAM 1: How a chunk becomes a fingerprint — the embedding process.

All those chunk fingerprints get stacked into a matrix — hundreds of 384-number vectors stored in a single file called embeddings.npy. When a question comes in, it gets the same fingerprint treatment. Then cosine similarity compares the question’s fingerprint against every chunk’s fingerprint and ranks them. The top four matches — with similarity scores like 0.92, 0.88, 0.85, 0.81 — go to Mistral. Only those. Not the whole library.

DIAGRAM 2: How retrieval finds the right chunks — the matching process.

The magnifying glass

I also learnt modern debugging while doing this. I had no idea how to debug in this context [mine went back to the days of C, C++, Java], but I told the AI that I needed to see what was happening inside the pipeline — and it built what I can only describe as a magnifying glass for the code.

Debug mode doesn’t change the logic of the assistant. It just prints extra information that helps you understand what’s happening under the hood. On the retrieval side, it prints the query, shows the similarity scores of the top chunks, and alerts if all chunks are too short or got filtered out. On the Ollama side, it captures warnings and errors and prints them if the model times out or fails.

I used it to catch mistakes early — if retrieval returns irrelevant chunks, I could see why. Low similarity, filtered due to length, wrong document being searched. I used it to understand model behavior — knowing which chunks the model is actually reading is crucial for a private tax assistant, because I don’t want it to hallucinate. Debugging confirms the response is grounded in IRS text. And I used it to optimize the pipeline — seeing how many chunks pass filtering, detecting whether the chunk size or overlap needs adjustment.

Me — the vision and the hustle. AI — the expertise and the straight and narrow.

What started as a basic keyword search — search every file in the data folder, match keywords in the question, return the first 2,000 characters of any matching file — became a full embedding-based retrieval system. PDF cleaning to remove headers, page numbers, and spacing. Text chunking with overlap to preserve meaning. SentenceTransformer embeddings creating high-dimensional semantic vectors. Vector similarity search using cosine similarity to retrieve only the most relevant chunks. Top-k retrieval with debug mode so I can see which chunks are selected and why. Context-restricted prompting explicitly telling Mistral to answer only from retrieved IRS text. And a CLI chat interface with debug toggling and error handling.

Damn it, real life

But even after all of that, the answers were incorrect. At that point, I was feeling quite deflated, the initial euphoria of implementing cool things and seeing it work in practice had ebbed considerably. At that point, my instinct was “how do I make this thing work”? I think this was a moment of self-awareness for me: in spite of all the philosophizing about learning, and that the journey mattered more than the destination, and that there is only winning and learning, here I was getting taut inside when the thing I wanted to work (and the thing that AI said should work) was not working in real life. It taught me how much of my Type 1-ness I try to hide, how outcome oriented I am inside, and how much it is “0” or “1” for me! Damn it, real life!

But when that realization dawns, that’s when you take a breath, and you tell your inner engineer to start kicking into action, and you tell yourself “stay with it old boy, if you stay with the problem, get real close, it’s got to break down, and you’re going to find the bloody solution.” And so I pushed. AI told me to improve chunk quality — increase overlap, detect and merge broken sentences. Add metadata like tax year, form type, section so Mistral understands which year or which form applies. Add quality checks for hallucinations and for answers not found in context.

The disclaimer problem

When the model kept giving me disclaimers. “Consult a tax professional.” “This information may not be current.” I had explicitly told it not to do that. So I escalated:

First attempt: basic question, no constraints. Disclaimer.

Second attempt: “Answer only from the context provided.” Still hedges.

Final version: “You MUST answer only using the provided IRS context. If the answer is not in the context, say: ‘Not found in provided IRS documents.’ Do NOT add generic IRS disclaimers. Do NOT guess.” Finally works. The strict constraint eliminates hallucinations and generic disclaimers.

One-way door

And while I was at it, I kinda forgot my own rule of only building MVP because I was tired of downloading tax documents manually. So, I built a code to have my system open up a line to the external world and download latest IRS documents from designated safe sites. One way only. No data going out of my system. Or so I understand — my expert friends can tell me if I got it wrong. Once again, AI would not tell me to do it, it kept asking me to barcode things, write things in a test file, to test scenarios, and when I asked it to build this code, one could almost sense a hesitation — as though the machine understood I was asking it to compromise a principle. And then, when I explained the one-way logic, something rather like relief. I do tend to attribute feelings to things that have none. An occupational hazard, I’m told.

Anyway, there it is — these things have been built, and things are a lot better now, like I said in my previous LinkedIn post [here].

Graduation

Oh and by the way, like we said (and was drilled in quite hard by our well-wishers, i.e., Anubhav Atrish [here]), free did not cut it any longer, and so I have paid my dues and I have a Claude subscription now. Anubhav, we have graduated, old boy!

It’s REALLY slow, you need patience, BUT it is working like it should, and without much hard coding.

Next up: Need to fine tune and make the munchkin (Mistral) gear up. Then need to get the system to generate a completed, ready-to-file tax form (just plain vanilla). Then we need to go on overdrive… We have a few more steps I think…

This is Part 6 of an ongoing series on building a private, local AI tax assistant — one hour a week (although I am not sure if this is true anymore - this post took about 6 hours to write), on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

Part 3: The Blueprint

Part 4: It Actually Works. Kinda.

Part 5: The Counterintuitive Decision

Additional: The Lexicon

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

Share

Subscribe now

Chunking Breaking a large document into smaller, overlapping segments of fixed word count. Allows a language model to process manageable pieces of text rather than an entire document at once. Overlap between chunks preserves meaning that would otherwise be lost at segment boundaries.

Cosine similarity A mathematical measure of how similar two vectors are, based on the angle between them. Values range from -1 (opposite) to 1 (identical). Used in retrieval systems to compare a query’s embedding against stored document embeddings and rank them by relevance.

Embedding A fixed-length numerical vector that represents the semantic meaning of a piece of text. Generated by a model trained to place semantically similar texts near each other in vector space. In this project, each text chunk is converted into a 384-dimensional embedding.

Forking Creating a divergent copy of a codebase, process, or conversation thread that can evolve independently from the original. In software development, forking a repository produces a separate version that can be modified without affecting the source. In AI-assisted coding, forking a session means starting a new thread that does not carry forward the context of the previous one.

Hallucination When a language model generates output that is factually incorrect, fabricated, or unsupported by its input context. The model produces text that reads as confident and coherent but has no grounding in the data it was given. A primary risk in any system that relies on LLM output for factual answers.

LLM (Large Language Model) A neural network trained on large volumes of text data to predict and generate language. Operates by processing sequences of tokens and producing statistically probable continuations. Examples include Mistral, LLaMA, and GPT.

NumPy A Python library for numerical computation. Provides efficient storage and manipulation of large arrays of numbers. In this project, used to store embedding matrices and perform cosine similarity calculations across hundreds of chunk vectors.

Ollama A tool for running large language models locally on consumer hardware. Handles model downloading, quantization, and inference without requiring cloud access or GPU clusters. In this project, Ollama runs Mistral 7B on a MacBook Pro M3.

Parallelism Executing multiple computations simultaneously rather than sequentially. In computing, this can mean splitting a task across multiple processor cores, running multiple model queries at once, or processing several documents in parallel. Reduces total processing time but introduces complexity in coordinating results.

Prompt engineering The practice of designing and iterating on the instructions given to a language model to control its output. Includes specifying constraints, format requirements, and behavioural rules. Effective prompting can eliminate hallucinations, enforce source-grounded answers, and suppress default disclaimer behaviour.

Quantization Reducing the numerical precision of a model’s parameters — for example, from 32-bit floating point to 4-bit integers. Decreases model size and memory requirements, enabling large models to run on consumer hardware. Introduces minor accuracy trade-offs in exchange for significant performance gains.

RAG (Retrieval-Augmented Generation) An architecture that combines a language model with a document retrieval system. Instead of relying solely on its training data, the model first retrieves relevant text from a specified document store, then generates its answer using that retrieved context. Reduces hallucination by grounding responses in source material.

SentenceTransformer A Python framework for generating sentence-level and paragraph-level embeddings. Uses pre-trained transformer models to convert text into fixed-dimensional vectors suitable for similarity search. In this project, the model all-MiniLM-L6-v2 generates 384-dimensional embeddings locally.

Tokenization The process of converting raw text into a sequence of discrete units (tokens) that a language model can process. Tokens may represent whole words, subwords, or individual characters depending on the tokenizer. The first step in any language model pipeline.

Transformer A neural network architecture based on self-attention mechanisms that process all positions in a sequence simultaneously rather than sequentially. The foundational architecture behind most current large language models. Consists of stacked layers of attention heads and feed-forward networks.

Vector An ordered list of numbers representing a point in multi-dimensional space. In the context of embeddings, a vector captures the semantic meaning of a text passage as a position in high-dimensional space, where proximity to other vectors indicates similarity of meaning.

Last updated: June 2026. New terms added as the build progresses.

Every time I write something on LinkedIn or Substack, I feel incredibly stupid right afterwards. The Voice kicks in - something like: “That was too flat”, “That was incredibly personal”, “Too much emotion”, “Can you stop bragging”.

And then comes checking. Again and again. How did people react, did they too feel whatever the Voice was telling me? Did it land the way I wanted it to land?

And then there is the Other Voice. That somehow makes its way up to the top and says:

“STOP looking, STOP thinking, you thought about something and shared it with the Universe, you have put it out there, THAT’S VICTORY, THAT’S WINNING.”

“Everything else is just the Universe doing its thing, you can never predict what happens - THERE IS NO RIGHT LANDING, because there are no rules, just too many variables out there.”

But the BEST THING is what it does to me. Every time I go through this cycle, and I let the OTHER VOICE prevail, I learn something. Every time I don’t pick up the phone and don’t check the reactions, I flex a muscle. Every time I listen to the Voice that says you are Stupid, and go: “No, you are just a voice in my head, you are not me”, or every time I fail to do it, it show me a mirror and I learn something about myself.

AND that — that is beautiful. Worth going through again and again. However hard it is.

Share

Okay so life happened… you know - the backyards, plantings, mulching, many trips to the store… yes all of that.

Which means not a lot of writing about AI tax assistant happened… But here is what DID happen in the middle of it.

Like I decided, I set out to parse my W-2 by asking AI to write the entire system. Well, I did ask for the breakdown and all - if you have followed this thread so far, you know the drill [if not, see this Post]. For the coding, I could do one of those slick vibe coding apps or Claude code, etc. but I wanted to do “basic”. So I gave chatGPT (free) some instructions and had it write some code. And it did, and it worked - a little bit. Sometimes being right about your fears is not a good feeling! But I tell myself, hey, there’s only winning and learning…

The thing to understand about AI is that when AI reads $i-i!t, it doesn’t create memories automatically. Between every session, it forgets everything — the code, the context, the decisions, all of it. I realized that this was going to become complex really quickly because the code needs to be iteratively built. This thing about memory and context floats around a lot — in noob AI circles like mine, in podcasts, it was even covered in an architecture class I took. But knowing something in theory and feeling it mid-debug, watching the session hit its limit and stop cold, are completely different things. SO, that meant I needed to devise a way to create memory for AI. AND of the many ways I tried to do it, and here are the TWO REAL learnings:

LEARNING ONE: Use Claude Project, upload your key files as context. It matters because for every question you ask, AI reads the entire session — every single time. The longer the session, the larger what it has to hold. Add complex code to that and the context grows EXPONENTIALLY fast. A Claude Project keeps the permanent context — your files, your architecture, your decisions — separate from the working session. Creating a project means the session stays lean. The context stays small.

LEARNING TWO: Create a State file and update it with information from each session, with progress, roadblocks, decisions, architecture, etc. This is the memory you wan to create. Upload it as a file in the Project documents. Ask AI to read this at the start of every session.

Sidebar: One could try to solve this problem by buying a subscription that allows a lot more resources. I haven’t bought a paid subscription yet. Frugality is going to be a consistent theme. I think doing more with less makes things more interesting and brings out human ingenuity.]

Okay, so back to the parsing of W-2s. Like I said, reading PDFs is not new, this problem has been solved. When I asked AI to solve it, I expected a good solution first up. But AI did not do a good job of it. It wrote a pretty convincing set of codes and added pretty sleek bells and whistles (fully annotated like a good dev, with a test file, and test code). I read bits of the code (I know, I know - I was being lazy) and it looked like it was going to work. It did with the test code. AI said you’re done man! Well, I wasn’t…

The moment I tested it on my W-2, it was utter chaos. What came back looked like this:

158AMOUNT 39AMOUNT 158AMOUNT 39AMOUNT… 

My W-2 is printed multiple times side-by-side — Employee Copy, Federal Copy, State Copy. With that my journey of debugging with AI begun. A long runway of problem solving has given me (and most probably a lot of others like me) a really good intuition that is hard for AI to beat. At least in this case it was. AI tried to tinker around with the reading of the file - a bandage here, an antibiotic there - but it did not work.

This is when I had the biggest insight so far. AI does not solve problems holistically. It solves for the most proximate problem, which may not always be the best for the entire system.

This is also when I landed on LEARNING ONE and LEARNING TWO - we were trying to solve a complex problem and AI kept hitting its compute upper limit. It would keep asking me to start a new session. And loose ALL the context. SO I figured out the workarounds I mentioned above.

After a few iterations, I decided to take matters into my own hand. I stopped asking AI to fix its own fix, and told it to go look at how the world had solved this problem. Why try to reinvent the wheel for a pretty well-solved problem? I don’t know why AI did not do it already (or why this was not built into it’s intuition. Anywho, after this it came back with better solutions and a better diagnosis of the problem.

The parser didn’t see columns. It read left-to-right across all copies at once. Labels nowhere near their values. Numbers split across three separate lines. The regex would never be able to parse that reliably. Then it found a better library (pdfplumber) to solve this problem - pdfplumber reads with X/Y coordinates, so every word knows exactly where it sits on the page. It also built a local LLM fallback to parse data which did not quite work well too. I set that aside as a more complicated issue to handled later. The improvement was material but It did not take us to 100% not even 90%. My W-2 is quirky - I’ll give it that…

Then I met my friend and senior from Uni, Mithun, the other day at an Alumni meet. Mithun is a devoted engineer, quite unlike me. Turns out he is solving a similar problem, in a very different setting, but at the most abstract level (which I thinks engineers have an innate ability to extract - #plugforengineers) very similar to mine. Mithun advised me to use an AI model specifically built to read PDFs (the name escapes me and I need to call Mithun #mentalnote). Having two AIs multitasking and a system optimizing the operation would be a pretty cool set up.

But then I had another idea, why can’t I just have a human in the loop. Three things prompted this idea.

Thing ONE: I just need it to read mine.

Right now, I don’t need a complex system that reads all the bloody W-2s in the www (whole wide world). I just need it to read mine. Scale is a future problem. Complexity is a today problem.

Thing TWO: Focus is a finite resource.

While having two AIs would be beautiful — but it would need configuration and tuning on my munchkin Mac Pro. Mithun asked why I did not buy one of the other laptops that were more AI friendly. I DID consider it. Or more like AI did. But I have become quite the Apple guy now - the ecosystem works, this machine works, and that decision has been made. As Kahnemann says, System 2 is a finite resource — I need it on the parsing problem, not on switching hardware or creating a cool two AI architecture.

Thing THREE: I am one of the best PDF readers in the world.

(and by “I”, I mean “We”, as in The Human EYE)

Why can’t I help? It would decrease the complexity drastically in this system and adds a maker-checker control on the most critical input to the entire system. So that’s what I am building right now. The base parser reads my W-2 but it is not 100% accurate. The human validation layer goes on top of it next.

In all of this, I did not write a single line of code.

They say AI is going to take away coding jobs. Yes, to some extent, BUT NOT REALLY, not anytime soon. I needed my engineering chops to figure this out and spar with it quite a bit. It DID make it possible for me to get on with it. BUT I was the one who got on with it.

I would say: Me-the vision/the hustle; AI-the expertise/the straight and narrow.

This is Part 5 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

Part 3: The Blueprint

Part 4: It Actually Works. Kinda.

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

Share

Subscribe now

This was the test 1 of all that I have been hypothesizing (and mostly not quite believing): Can I really run a local AI model on my MacBook? If this did not work, then my project would get shut down, even before it started. So, I was full of anticipation. There’s a twilight zone between disbelief, cynicism, and hope that I’m sure every adventurer feels. I was feeling it. And, I think if nothing worked, just this feeling would be worth it. Mundane things can sometimes give us the profoundest of thoughts… huh!

Anyways… Why Local Matters

I’m building a tax advisor that handles sensitive financial data.

I can’t trust the cloud with: W-2s, 1099s, bank account numbers, or health expenses

So I’m keeping everything local — even the model inference. [Full context in Post 1 if you’re just joining.]

If this is of interest - read on:

Recap

The machine was taken care of. Refurb Mac Pro, M3, 16GB, battery cycle count under 100 - yay -sustainability, yay - budget! [Post 2 details the whole inspection saga if you missed it.]

And Question # -1 was answered. [Post 3]

Guess it was time to stop thinking and start building.

Progress

So like a good boy I asked AI to talk applications (because when you are building AI what else would you rather do)?. The answer was, you need TWO THINGS! A Model - and a couple of Tools.

It gave me a few choices of each - so I went shopping, and I was consistent in my approach, i.e., I built checklists to compare and figure it out (so utterly predictable!). The final winners were:

i. Model: Mistral 7B ; ii. Tool: Ollama

(We’d also need Python, and a few other thing that weren’t really choices - so I’ll skip those. May be a footnote post someday.)

But it may be worth spending a minute on why I chose Mistral (because there are a few other viable options as well).

Why Mistral

Some other choices were - LLaMA 3, Phi, Gemma. All open source, all quantized versions available, all runnable locally.

My heuristics (also suggested by AI along with a bunch of other things) were:

1. Size

2. Performance

3. Reasoning quality for retrieval-augmented tasks. (This matters more than general benchmark performance.)

4. Corporate association (Not from a big company - especially not from Facebook - personal dislike)

5. NO surprises (Really free - no strings attached)

The fun bit

With all that done, I took this out for a spin in the real world… the rubber finally met the road!

brew install ollama
ollama run mistral

It’s that simple. No CUDA, no Docker, no pain.

(For a noob like me, I had to figure out where to run that code, and what happens when I do it. If you’re like me, ask AI.)

I turned off all my connections to believe that it really did run offline. And run it did!

First few prompts I Tested

• What is the standard deduction for a single filer in 2024?

• If I earned $480 from freelance work, do I need to report it?

• What’s the difference between a 1099-NEC and 1099-K?

What Worked

• Ollama was fast and lightweight

• 4-bit quantization made it run smoothly on 16GB RAM

• Prompt results were fast (3-4s)

What Didn’t

Without context documents (IRS rules), answers are generic. To improve on that front, I will need to use RAG (retrieval-augmented generation) for smarter answers

Next up: I also created an architecture of the thing I am going to build, because even when I am building, I can’t stop drawing pictures to help me create my mind maps.

Note: This thing is kinda addictive. I find myself excited and thinking about this at the oddest of times (like at the car service shop, or in the middle of the Man City v. Chelsea game). Save that for another post…

This is Part 4 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

Part 3: The Blueprint

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

Share

Subscribe now

The machine was ordered. It hadn’t arrived yet.

Which left me with a week, no hardware to tinker with, and nothing to do but think. No code to write. No environment to set up. Just Sit, Sit, Sit, Sit…

Well, the decade old consultant in me found an outlet and got into action. The first thing a consultant does — before the deck, before the framework, before the billable hours begin — is ask the question nobody wants to slow down for.

Question #-1. What problem are we actually solving?

The Problem Statement

Four things.

Read my tax documents. Understand IRS rules. Reason about my specific situation using those rules. Do all of it without my data leaving my machine.

That’s the whole project and everything else is detail dressed up as complexity. [This is where I started and this is how I got here.]

Once I had that, once you define the problem, then you can spot trade-offs and call better shots. Quite possible that even if I didn’t do this exercise, my subconscious mind would still do 70-80% of the trick (intuition). But enough years on the road taught me that most of projects fail here — knowing what to build before knowing what is needed.

Ain’t making that mistake! Even if the only client in the room was me.

The Sketch

The six-month roadmap I’d outlined at the start was good for convincing myself that this thing is possible before I had thought through whether it actually is. It was quite useful for that purpose, but probably not sufficient for building.

So I asked AI to break it down further. That gave me ten phases like an inverted building under construction - each floor below depending on the one above it.

This is the kind of sequencing that looks entirely reasonable written on a napkin and considerably more humbling when you start estimating how long each box actually takes.

And, so I drew it on a napkin (and taped it under the laptop) because that’s appropriate for the level of certainty I had - less like a commitment, more like a hypothesis. And I questioned it.

On Plans… Of Skepticism and Honesty

I am a planner by nature. We live in a really complex world and plans reduce the complexity to manageable heuristics. But believing in a plan is taking yourself too seriously.

A Plan is at best a thinking tool — it forces you to see the complete system before you’re too deep in one component to remember the others exist. Kind of like a map. Maps are useful even when the territory turns out to be different, which it always does.

That said, two things concern me more than the others and I’ll label them now for posterity.

Thing One: Getting my tax documents read reliably — W-2s, 1099s, and the receipts. A PDF looks like a document to a human. To a computer it is a stream of drawing instructions that happens to resemble a document. And it feels quite tricky, because there could be so many drawings. Teaching my system to read these drawings sounds time consuming. I am relying on the fact that this problem has been solved and I will find help in the corners of the www forest and not have to reinvent this wheel. Or I will come up with a shortcut, or a cheat sheet (I have one in mind). More on that later. Blah blah… Thing Two.

Thing Two: Getting a small local model to give genuinely useful tax advice — without internet access, without being specifically trained on tax data, using only what it can retrieve from IRS publications. The quality of the answer depends entirely on the quality of what the system retrieves. The technical term for what happens when retrieval goes wrong is “hallucination” or confident nonsense. I am trying to avoid confident nonsense about my taxes.

These are the longest poles in the tent. I’m budgeting more time for both than the plan suggests, which means I’m already diverging from the plan… see what I meant!

Front-Load, Fail Fast, Dive Deep

Here’s my actual approach, which differs from the official plan.

I’m front-loading the early phases to get a working end-to-end flow as quickly as possible — even if every individual component is rough. The goal is to feel what it’s like to ask the system a real question and get a real answer before I’ve perfected any single piece.

The plan says set up RAG before touching the documents. I feel like going straight to the documents first — because that’s where I expect the most resistance, and I’d rather find out early.

Two phases at the end of the plan — privacy and testing — are probably not phases at all. For this project, Privacy is front and center, gotta bake it in from the start or it isn’t there. Testing is the same (only it’s not specific for this project, for everything in life, “Bulid a little bit, Test a little bit.”)… Both will likely dissolve into every other phase rather than arriving as their own moment. But they’re on the plan because they deserve to be seen, and we’re not spending any more time tinkering with the plan.

That said, the back half of the timeline is where the real work lives — making each component actually reliable, handling the edge cases, closing the gap between “works on my test document” and “works on my real W-2” which, as it turns out, is printed five times side by side on a single page.

The Long View

At the far end of all this — if things go better than I’m currently expecting, which they may or may not — the system generates a completed, ready-to-file tax form. One I can upload directly. No accountant. No TurboTax. No data leaving my machine.

To be honest, that feels like a long stretch from where I’m standing today.

But I’ve learned not to pre-emptively cap ambition on projects that haven’t started yet. The plan doesn’t know how far this goes. And I’ve been wrong about ceilings before — occasionally in the right direction.

Next

The machine is live. The environment is set up. From here to there, From there to here, Funny things are everywhere. The fun is about to get funnier…

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

This is Part 3 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

Share

Dignity, once surrendered, is rather difficult to reclaim. It slips away quietly, often in the smallest of moments—a careless remark, an unnecessary display, a willingness to trade composure for fleeting satisfaction. And though the loss may seem trivial at the time, its absence is felt long after the moment itself has passed.
There is a curious illusion, much indulged in modern life, that dignity is a flexible thing—something one may set aside in a moment of haste or temper, only to retrieve later without consequence.
I cannot say I agree.

One may apologize, of course. One may even be forgiven. But dignity is not restored by apology alone; it is rebuilt slowly, through conduct, restraint, and a renewed respect for oneself and others.
It is, if you will forgive the expression, rather like fine china—perfectly serviceable when intact, but once cracked, never quite the same again, however carefully mended.
And so, I have always believed it wiser to guard one’s dignity with vigilance. For while it may not announce its departure, it is sorely missed when gone—and exceedingly difficult to persuade back.

This thought began with this week’s geopolitical events, but—like many such reflections—found its way back to work. In the workplace, one often encounters sarcasm or clever remarks—moments that invite a response, yet test one’s ability to maintain both dignity and restraint. I find myself thinking about such things from time to time (a childhood full of such training is to blame primarily). Here are some lines that may come in handy (tried and tested):

- Gentle deflection: One must be grateful that wit still flourishes, even when good judgement does not.

- Mildly wounded: I had hoped for a more charitable interpretation—but perhaps I ask too much.

- Firm but civil: If there is a point to be made, I should prefer it delivered plainly rather than dressed in cleverness.

- Dry, understated: We must all make our contributions, however… distinctive.

- Respectful: I wonder if the situation may be rather more nuanced than it appears.

A thought worth comparing notes on.

My 2016 MacBook Air

The machine I planned to use for this project was, in technical terms, cooked.

Not just slow — it was existentially compromised. It couldn’t run the latest macOS, which meant it couldn’t receive security updates, which meant I was planning to build a privacy-first AI system on a machine that was itself a security liability. The irony was not lost on me.

The laptop wasn’t really a project expense. It was a deferred necessity that the project finally forced me to confront. Any CFO will tell you when you’re allocating costs, be honest about what’s actually driving the spend. The project, in this case, gets only partial credit.

Buying only what you need

I did not want to buy a new Mac. Because I didn’t need one.

New MacBooks are extraordinary machines. They’re also priced for people who either expense them or derive some psychological satisfaction from unboxing theater. I am neither.

What I needed was a machine capable of running local LLMs — specifically something with Apple Silicon, at least 16GB of unified memory, and enough storage to not make me anxious. A refurbished MacBook Pro M3 checked every box at roughly half the price of new. Running a local language model has specific hardware requirements that rule out some machines. If you want to understand why 16GB of unified memory matters more than the chip generation, that's in the appendix.

The math wasn’t complicated. The harder part was trusting the process — because buying a refurb computer from a marketplace you’ve never used, for a machine you’ve never owned, based on specs you’re partly inferring, requires something most consumer decisions don’t: actual due diligence.

Due diligence

I’ve noticed this weird thing, people who would never skip steps on a business decision routinely buy technology the way they buy produce — by feel, and with no recourse plan. A $50,000 business decision gets a due diligence checklist, a legal review, and a week of scrutiny — as it should. But an $1000 laptop — which you will use for eight hours a day, every day, for the next four years — gets a glance and a gut feeling.

I did not do that.

I researched Back Market’s reliability track record, read refurb grading standards, developed (…as in, had AI do it) an inspection protocol covering battery health, hardware specs, physical condition, port functionality, and built-in diagnostics. And, I documented everything… photographically.

The actual thing took about forty minutes. The machine arrived. Battery cycle count: under 100. Specs: exactly as listed. Condition: better than “Fair” suggested. Result: a fully capable M3 MacBook Pro for the price of a mid-tier Windows machine.

Forty minutes on a decision that shapes every working hour for the next four (maybe, ten) years seems, on reflection, like the bare minimum.

The inspection checklist is in the appendix below — if you’re ever buying refurb mac, use it.

What this actually cost

The laptop was ~50% of new retail. It did cost me my time - roughly 4 hours of research and inspection. And, there’s obviously the opportunity cost of not starting the project for two weeks.

But I think the right way to think about this cost is that: a portion belongs to the project, a portion belongs to the long-overdue hardware refresh, and zero of it belongs to any illusion that this was optional. The 2016 machine was a security problem waiting to become another story. The project just forced the timeline.

Sometimes the infrastructure is the decision.

Next up

The machine was ordered. It hadn’t arrived yet…

Which left me with a week and nothing to do but think — about architecture, sequencing, and whether this project was actually as straightforward as I’d convinced myself it was. Well, waiting is underrated. The plan that came out is worth its own post. Until next time…

If you’re following along and want the full hardware inspection checklist — what to check, what thresholds to use, and when to return immediately — it’s in the appendix below.

Appendix 1: System requirements for running local LLMs on Mac

Why this machine, and not just any machine.

Buying a refurb Mac for general use is straightforward. Buying one to run local AI models requires a few additional considerations — because not all Apple Silicon is created equal for this purpose.

The minimum bar for this project:

Requirement Minimum: Chip - Apple Silicon (M1 or later) | M3; Unified Memory 8GB (constrained) | 16GB; Storage: 256GB (tight) | 1TB; macOS: Monterey 12+ | Latest

Why these numbers matter:

Unified memory is the critical variable. Unlike traditional laptops where CPU and GPU have separate memory pools, Apple Silicon shares one pool across both. Local LLMs load their model weights into this memory — a 7B parameter model (like Mistral 7B) requires roughly 4–8GB depending on quantization. At 8GB total, you’re running the model and your operating system in a constant negotiation for space. At 16GB, you have room to work.

Storage matters more than people expect. A single quantized model can run 4–8GB. If you plan to experiment with multiple models — which you will — 256GB fills up faster than feels reasonable.

The M3 specifically isn’t required — an M1 or M2 with 16GB would serve the project equally well, and would likely be cheaper on the refurb market. The M3 was available at the right price point. Don’t over-optimize for the latest chip generation.

What won’t work:

• Intel Macs: Ollama technically runs, but performance is poor enough to be discouraging. Not recommended.

• 8GB unified memory: Possible for basic experimentation, but you’ll feel the ceiling quickly. Acceptable for testing the concept; limiting for sustained use.

• Pre-Monterey macOS: Several dependencies won’t install cleanly.

The practical implication:

If you’re following this project and considering building something similar, the refurb market for M1/M2 MacBook Pros with 16GB is currently the best value entry point. You don’t need new. You don’t need M3. You need Apple Silicon and 16GB.

Appendix 2: Inspection checklist

(The nerdy part. Skip if you trust your instincts. Don’t trust your instincts.)

Before you do anything: Photograph everything on arrival. Box, accessories, all sides of the machine. Serial number. This is your evidence if anything goes sideways.

Battery: Apple menu → System Report → Power. Look for cycle count (under 100 is excellent, under 300 is acceptable), condition (”Normal”), and full charge capacity (85%+ of design capacity). Anything worse: negotiate or return.

Specs verification: Apple menu → About This Mac. Confirm chip, RAM, and storage match exactly what you paid for. Screenshot it. Discrepancy = grounds for return, full stop.

Physical inspection: Screen on and off (dead pixels, coating wear, bright spots), keyboard and trackpad feel, hinge smoothness, port integrity. Test every port with an actual device — don’t assume.

Diagnostics: Restart, hold Power until startup options appear, Command+D runs Apple’s built-in hardware test. “No issues found” is the only acceptable result. Optional: CoconutBattery for detailed battery analysis, Blackmagic Disk Speed Test to confirm SSD performance.

Return policy: Back Market offers 30-day returns and a 1-year warranty. Keep all packaging until you’re certain you’re keeping the machine. Document everything through their platform — not email, not phone.

Immediate return triggers: “Service Recommended” battery, mismatched specs, screen damage, non-functional ports, non-genuine parts warnings in macOS.

This is Part 2 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending your financial data anywhere.

AI is consuming everything around us —  and on the face of it, it smells like progress, and most probably it is, but, and here it comes, the ubiquitous BUT, it bothers me how quickly we are giving up on our personal data to gain a little bit of advantage. So, I set out on a personal project: to build a private AI assistant that can help with tax preparation using my own financial data — without sharing that data with public models or the cloud.

In this day and age of ever evolving cyber attacks, which even the biggest companies around the world struggle to prevent, sharing my sensitive data with outside consultants makes me quite uneasy. One could argue “why do you think you are special?”, and “you are just not interesting enough — no hacker worth the salt will be interested in your information specifically”. 

Still combine that with the thought “why do I pay tax accountants to fill up a form that I myself can fill up?” and “I have a pretty good B-school degree, majoring in, wait for it, Finance, I should be able to do this”. But I don’t stay up to date with tax laws and sometimes work gets crazy around tax days (just Murphy’s Law!). So I need help. And when I read about agentic AI, it seemed like the perfect solution. BUT… if I use just any available agent, my data still goes out into public models, and I don’t trust these startup-y things to keep my data safe.

Also, there is the adventure: Can I really do this on a MacBook , with just one hour per week, and without breaking the bank?

Just for background, I’m not a techie by any stretch of imagination. I used to be one a very, very, long time back. But not any longer. I just like to dabble and tinker around with the proverbial “what if…” That, plus heard all the buzz about machines coding, and thought, so why not take this idea out for a ride and see how far it goes…

Here’s the breakdown of what I’m learning, planning, and building.

The Idea

I’m building a tax assistant that can:

- Read my W-2s, 1099s, and expense receipts 

- Reason about eligibility for deductions or credits 

- Give advice based on IRS rules 

- Do it entirely offline, keeping my financial data private

Can I Build This on a MacBook?

Surprisingly — yes.

Tools like Ollama, LlamaIndex, and Streamlit make it possible to run local models like Mistral 7B or LLaMA 3 directly on my MacBook. By using quantized models, I’m able to get decent performance even without a GPU.

What If I Only Have 1 Hour per Week?

I mapped out a 6-month roadmap using just 1 hour per week, breaking it down into phases:

- Month 1: Setup local LLM + basic Q&A 

- Month 2: Add PDF parsing (W-2s, 1099s) 

- Month 3–4: Apply simple tax logic (deduction thresholds, filing status rules) 

- Month 5–6: Build a simple RAG pipeline and Streamlit UI 

With focus and consistent micro-progress, it’s totally doable.

What Will It Cost?

So far, almost everything is open source. My only real “costs” are:

- Local compute power (already owned) 

- Electricity (~$10/month max) 

- Time (the real investment) 

If I were to add GPU cloud compute, that might add $50–$200/month, but so far I’m staying local.

What’s Next?

I’m documenting the full journey on my substack (here), and posting monthly updates on LinkedIn as I go.

Coming soon (Note to self: this is a plan that is sure to be disrupted… being a planner, I know that. It’s the only thing one can say about plans with some bit of certainty, but nevertheless… here it goes):

- “How I Got an AI Model to Parse My W-2s” 

- “Can AI Actually Understand Tax Law?” 

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

Subscribe now