The Blueprint
Or: what you think about when the laptop hasn’t arrived yet and you have nothing better to do...
The machine was ordered. It hadn’t arrived yet.
Which left me with a week, no hardware to tinker with, and nothing to do but think. No code to write. No environment to set up. Just Sit, Sit, Sit, Sit…
Well, the decade old consultant in me found an outlet and got into action. The first thing a consultant does — before the deck, before the framework, before the billable hours begin — is ask the question nobody wants to slow down for.
Question #-1. What problem are we actually solving?
The Problem Statement
Four things.
Read my tax documents. Understand IRS rules. Reason about my specific situation using those rules. Do all of it without my data leaving my machine.
That’s the whole project and everything else is detail dressed up as complexity. [This is where I started and this is how I got here.]
Once I had that, once you define the problem, then you can spot trade-offs and call better shots. Quite possible that even if I didn’t do this exercise, my subconscious mind would still do 70-80% of the trick (intuition). But enough years on the road taught me that most of projects fail here — knowing what to build before knowing what is needed.
Ain’t making that mistake! Even if the only client in the room was me.
The Sketch
The six-month roadmap I’d outlined at the start was good for convincing myself that this thing is possible before I had thought through whether it actually is. It was quite useful for that purpose, but probably not sufficient for building.
So I asked AI to break it down further. That gave me ten phases like an inverted building under construction - each floor below depending on the one above it:
This is the kind of sequencing that looks entirely reasonable written on a napkin and considerably more humbling when you start estimating how long each box actually takes.
And, so I drew it on a napkin (and taped it under the laptop) because that’s appropriate for the level of certainty I had - less like a commitment, more like a hypothesis. And I questioned it. I’ll leave that for the next post…
On Plans… Of Skepticism and Honesty
I am a planner by nature— the kind with a tender heart that likes certainty. We live in a really complex world and plans reduce the complexity to manageable heuristics. But believing in a plan is taking yourself too seriously.
A Plan is at best a thinking tool — it forces you to see the complete system before you’re too deep in one component to remember the others exist. Kind of like a map. Maps are useful even when the territory turns out to be different, which it always does.
That said, two things concern me more than the others and I’ll label them now for posterity.
Thing One: Getting my tax documents read reliably — W-2s, 1099s, and the receipts. A PDF looks like a document to a human. To a computer it is a stream of drawing instructions that happens to resemble a document. And it feels quite tricky, because there could be so many drawings. Teaching my system to read these drawings sounds time consuming. I am relying on the fact that this problem has been solved and I will find help in the corners of the www forest and not have to reinvent this wheel. Or I will come up with a shortcut, or a cheat sheet (I have one in mind). More on that later. Blah blah… Thing Two.
Thing Two: Getting a small local model to give genuinely useful tax advice — without internet access, without being specifically trained on tax data, using only what it can retrieve from IRS publications. The quality of the answer depends entirely on the quality of what the system retrieves. The technical term for what happens when retrieval goes wrong is “hallucination” or confident nonsense. I am trying to avoid confident nonsense about my taxes.
These are the longest poles in the tent. I’m budgeting more time for both than the plan suggests, which means I’m already diverging from the plan… see what I meant!
Front-Load, Fail Fast, Dive Deep
Here’s my actual approach, which differs from the official plan.
I’m front-loading the early phases to get a working end-to-end flow as quickly as possible — even if every individual component is rough. The goal is to feel what it’s like to ask the system a real question and get a real answer before I’ve perfected any single piece.
The plan says set up RAG before touching the documents. I feel like going straight to the documents first — because that’s where I expect the most resistance, and I’d rather find out early.
Two phases at the end of the plan — privacy and testing — are probably not phases at all. For this project, Privacy is front and center, gotta bake it in from the start or it isn’t there. Testing is the same (only it’s not specific for this project, for everything in life, “Bulid a little bit, Test a little bit.”)… Both will likely dissolve into every other phase rather than arriving as their own moment. But they’re on the plan because they deserve to be seen, and we’re not spending any more time tinkering with the plan.
That said, the back half of the timeline is where the real work lives — making each component actually reliable, handling the edge cases, closing the gap between “works on my test document” and “works on my real W-2” which, as it turns out, is printed five times side by side on a single page.
The Long View
At the far end of all this — if things go better than I’m currently expecting, which they may or may not — the system generates a completed, ready-to-file tax form. One I can upload directly. No accountant. No TurboTax. No data leaving my machine.
To be honest, that feels like a long stretch from where I’m standing today.
But I’ve learned not to pre-emptively cap ambition on projects that haven’t started yet. The plan doesn’t know how far this goes. And I’ve been wrong about ceilings before — occasionally in the right direction.
Next
The machine is live. The environment is set up. From here to there, From there to here, Funny things are everywhere. The fun is about to get funnier…
I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.
This is Part 3 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending financial data anywhere.
Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!
Part 2: The Infrastructure Tax
If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!