This was the test 1 of all that I have been hypothesizing (and mostly not quite believing): Can I really run a local AI model on my MacBook? If this did not work, then my project would get shut down, even before it started. So, I was full of anticipation. There’s a twilight zone between disbelief, cynicism, and hope that I’m sure every adventurer feels. I was feeling it. And, I think if nothing worked, just this feeling would be worth it. Mundane things can sometimes give us the profoundest of thoughts… huh!

Anyways… Why Local Matters

I’m building a tax advisor that handles sensitive financial data.

I can’t trust the cloud with: W-2s, 1099s, bank account numbers, or health expenses

So I’m keeping everything local — even the model inference. [Full context in Post 1 if you’re just joining.]

If this is of interest - read on:

Recap

The machine was taken care of. Refurb Mac Pro, M3, 16GB, battery cycle count under 100 - yay -sustainability, yay - budget! [Post 2 details the whole inspection saga if you missed it.]

And Question # -1 was answered. [Post 3]

Guess it was time to stop thinking and start building.

Progress

So like a good boy I asked AI to talk applications (because when you are building AI what else would you rather do)?. The answer was, you need TWO THINGS! A Model - and a couple of Tools.

It gave me a few choices of each - so I went shopping, and I was consistent in my approach, i.e., I built checklists to compare and figure it out (so utterly predictable!). The final winners were:

i. Model: Mistral 7B ; ii. Tool: Ollama

(We’d also need Python, and a few other thing that weren’t really choices - so I’ll skip those. May be a footnote post someday.)

But it may be worth spending a minute on why I chose Mistral (because there are a few other viable options as well).

Why Mistral

Some other choices were - LLaMA 3, Phi, Gemma. All open source, all quantized versions available, all runnable locally.

My heuristics (also suggested by AI along with a bunch of other things) were:

1. Size

2. Performance

3. Reasoning quality for retrieval-augmented tasks. (This matters more than general benchmark performance.)

4. Corporate association (Not from a big company - especially not from Facebook - personal dislike)

5. NO surprises (Really free - no strings attached)

The fun bit

With all that done, I took this out for a spin in the real world… the rubber finally met the road!

brew install ollama
ollama run mistral

It’s that simple. No CUDA, no Docker, no pain.

(For a noob like me, I had to figure out where to run that code, and what happens when I do it. If you’re like me, ask AI.)

I turned off all my connections to believe that it really did run offline. And run it did!

First few prompts I Tested

• What is the standard deduction for a single filer in 2024?

• If I earned $480 from freelance work, do I need to report it?

• What’s the difference between a 1099-NEC and 1099-K?

What Worked

• Ollama was fast and lightweight

• 4-bit quantization made it run smoothly on 16GB RAM

• Prompt results were fast (3-4s)

What Didn’t

Without context documents (IRS rules), answers are generic. To improve on that front, I will need to use RAG (retrieval-augmented generation) for smarter answers

Next up: I also created an architecture of the thing I am going to build, because even when I am building, I can’t stop drawing pictures to help me create my mind maps.

Note: This thing is kinda addictive. I find myself excited and thinking about this at the oddest of times (like at the car service shop, or in the middle of the Man City v. Chelsea game). Save that for another post…

This is Part 4 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

Part 3: The Blueprint

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

Share

Subscribe now

The machine was ordered. It hadn’t arrived yet.

Which left me with a week, no hardware to tinker with, and nothing to do but think. No code to write. No environment to set up. Just Sit, Sit, Sit, Sit…

Well, the decade old consultant in me found an outlet and got into action. The first thing a consultant does — before the deck, before the framework, before the billable hours begin — is ask the question nobody wants to slow down for.

Question #-1. What problem are we actually solving?

The Problem Statement

Four things.

Read my tax documents. Understand IRS rules. Reason about my specific situation using those rules. Do all of it without my data leaving my machine.

That’s the whole project and everything else is detail dressed up as complexity. [This is where I started and this is how I got here.]

Once I had that, once you define the problem, then you can spot trade-offs and call better shots. Quite possible that even if I didn’t do this exercise, my subconscious mind would still do 70-80% of the trick (intuition). But enough years on the road taught me that most of projects fail here — knowing what to build before knowing what is needed.

Ain’t making that mistake! Even if the only client in the room was me.

The Sketch

The six-month roadmap I’d outlined at the start was good for convincing myself that this thing is possible before I had thought through whether it actually is. It was quite useful for that purpose, but probably not sufficient for building.

So I asked AI to break it down further. That gave me ten phases like an inverted building under construction - each floor below depending on the one above it:

This is the kind of sequencing that looks entirely reasonable written on a napkin and considerably more humbling when you start estimating how long each box actually takes.

And, so I drew it on a napkin (and taped it under the laptop) because that’s appropriate for the level of certainty I had - less like a commitment, more like a hypothesis. And I questioned it. I’ll leave that for the next post…

On Plans… Of Skepticism and Honesty

I am a planner by nature— the kind with a tender heart that likes certainty. We live in a really complex world and plans reduce the complexity to manageable heuristics. But believing in a plan is taking yourself too seriously.

A Plan is at best a thinking tool — it forces you to see the complete system before you’re too deep in one component to remember the others exist. Kind of like a map. Maps are useful even when the territory turns out to be different, which it always does.

That said, two things concern me more than the others and I’ll label them now for posterity.

Thing One: Getting my tax documents read reliably — W-2s, 1099s, and the receipts. A PDF looks like a document to a human. To a computer it is a stream of drawing instructions that happens to resemble a document. And it feels quite tricky, because there could be so many drawings. Teaching my system to read these drawings sounds time consuming. I am relying on the fact that this problem has been solved and I will find help in the corners of the www forest and not have to reinvent this wheel. Or I will come up with a shortcut, or a cheat sheet (I have one in mind). More on that later. Blah blah… Thing Two.

Thing Two: Getting a small local model to give genuinely useful tax advice — without internet access, without being specifically trained on tax data, using only what it can retrieve from IRS publications. The quality of the answer depends entirely on the quality of what the system retrieves. The technical term for what happens when retrieval goes wrong is “hallucination” or confident nonsense. I am trying to avoid confident nonsense about my taxes.

These are the longest poles in the tent. I’m budgeting more time for both than the plan suggests, which means I’m already diverging from the plan… see what I meant!

Front-Load, Fail Fast, Dive Deep

Here’s my actual approach, which differs from the official plan.

I’m front-loading the early phases to get a working end-to-end flow as quickly as possible — even if every individual component is rough. The goal is to feel what it’s like to ask the system a real question and get a real answer before I’ve perfected any single piece.

The plan says set up RAG before touching the documents. I feel like going straight to the documents first — because that’s where I expect the most resistance, and I’d rather find out early.

Two phases at the end of the plan — privacy and testing — are probably not phases at all. For this project, Privacy is front and center, gotta bake it in from the start or it isn’t there. Testing is the same (only it’s not specific for this project, for everything in life, “Bulid a little bit, Test a little bit.”)… Both will likely dissolve into every other phase rather than arriving as their own moment. But they’re on the plan because they deserve to be seen, and we’re not spending any more time tinkering with the plan.

That said, the back half of the timeline is where the real work lives — making each component actually reliable, handling the edge cases, closing the gap between “works on my test document” and “works on my real W-2” which, as it turns out, is printed five times side by side on a single page.

The Long View

At the far end of all this — if things go better than I’m currently expecting, which they may or may not — the system generates a completed, ready-to-file tax form. One I can upload directly. No accountant. No TurboTax. No data leaving my machine.

To be honest, that feels like a long stretch from where I’m standing today.

But I’ve learned not to pre-emptively cap ambition on projects that haven’t started yet. The plan doesn’t know how far this goes. And I’ve been wrong about ceilings before — occasionally in the right direction.

Next

The machine is live. The environment is set up. From here to there, From there to here, Funny things are everywhere. The fun is about to get funnier…

I. Thinking on strategy, innovation, and philosophy — for people who think seriously about how to build things and make decisions.

This is Part 3 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending financial data anywhere.

Part 1: Building a Private AI Tax Assistant: In public, on a MacBook!

Part 2: The Infrastructure Tax

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

Share

Dignity, once surrendered, is rather difficult to reclaim. It slips away quietly, often in the smallest of moments—a careless remark, an unnecessary display, a willingness to trade composure for fleeting satisfaction. And though the loss may seem trivial at the time, its absence is felt long after the moment itself has passed.
There is a curious illusion, much indulged in modern life, that dignity is a flexible thing—something one may set aside in a moment of haste or temper, only to retrieve later without consequence.
I cannot say I agree.

One may apologize, of course. One may even be forgiven. But dignity is not restored by apology alone; it is rebuilt slowly, through conduct, restraint, and a renewed respect for oneself and others.
It is, if you will forgive the expression, rather like fine china—perfectly serviceable when intact, but once cracked, never quite the same again, however carefully mended.
And so, I have always believed it wiser to guard one’s dignity with vigilance. For while it may not announce its departure, it is sorely missed when gone—and exceedingly difficult to persuade back.

This thought began with this week’s geopolitical events, but—like many such reflections—found its way back to work. In the workplace, one often encounters sarcasm or clever remarks—moments that invite a response, yet test one’s ability to maintain both dignity and restraint. I find myself thinking about such things from time to time (a childhood full of such training is to blame primarily). Here are some lines that may come in handy (tried and tested):

- Gentle deflection: One must be grateful that wit still flourishes, even when good judgement does not.

- Mildly wounded: I had hoped for a more charitable interpretation—but perhaps I ask too much.

- Firm but civil: If there is a point to be made, I should prefer it delivered plainly rather than dressed in cleverness.

- Dry, understated: We must all make our contributions, however… distinctive.

- Respectful: I wonder if the situation may be rather more nuanced than it appears.

A thought worth comparing notes on.

My 2016 MacBook Air

The machine I planned to use for this project was, in technical terms, cooked.

Not just slow — it was existentially compromised. It couldn’t run the latest macOS, which meant it couldn’t receive security updates, which meant I was planning to build a privacy-first AI system on a machine that was itself a security liability. The irony was not lost on me.

The laptop wasn’t really a project expense. It was a deferred necessity that the project finally forced me to confront. Any CFO will tell you when you’re allocating costs, be honest about what’s actually driving the spend. The project, in this case, gets only partial credit.

Buying only what you need

I did not want to buy a new Mac. Because I didn’t need one.

New MacBooks are extraordinary machines. They’re also priced for people who either expense them or derive some psychological satisfaction from unboxing theater. I am neither.

What I needed was a machine capable of running local LLMs — specifically something with Apple Silicon, at least 16GB of unified memory, and enough storage to not make me anxious. A refurbished MacBook Pro M3 checked every box at roughly half the price of new. Running a local language model has specific hardware requirements that rule out some machines. If you want to understand why 16GB of unified memory matters more than the chip generation, that's in the appendix.

The math wasn’t complicated. The harder part was trusting the process — because buying a refurb computer from a marketplace you’ve never used, for a machine you’ve never owned, based on specs you’re partly inferring, requires something most consumer decisions don’t: actual due diligence.

Due diligence

I’ve noticed this weird thing, people who would never skip steps on a business decision routinely buy technology the way they buy produce — by feel, and with no recourse plan. A $50,000 business decision gets a due diligence checklist, a legal review, and a week of scrutiny — as it should. But an $1000 laptop — which you will use for eight hours a day, every day, for the next four years — gets a glance and a gut feeling.

I did not do that.

I researched Back Market’s reliability track record, read refurb grading standards, developed (…as in, had AI do it) an inspection protocol covering battery health, hardware specs, physical condition, port functionality, and built-in diagnostics. And, I documented everything… photographically.

The actual thing took about forty minutes. The machine arrived. Battery cycle count: under 100. Specs: exactly as listed. Condition: better than “Fair” suggested. Result: a fully capable M3 MacBook Pro for the price of a mid-tier Windows machine.

Forty minutes on a decision that shapes every working hour for the next four (maybe, ten) years seems, on reflection, like the bare minimum.

The inspection checklist is in the appendix below — if you’re ever buying refurb mac, use it.

What this actually cost

The laptop was ~50% of new retail. It did cost me my time - roughly 4 hours of research and inspection. And, there’s obviously the opportunity cost of not starting the project for two weeks.

But I think the right way to think about this cost is that: a portion belongs to the project, a portion belongs to the long-overdue hardware refresh, and zero of it belongs to any illusion that this was optional. The 2016 machine was a security problem waiting to become another story. The project just forced the timeline.

Sometimes the infrastructure is the decision.

Next up

The machine was ordered. It hadn’t arrived yet…

Which left me with a week and nothing to do but think — about architecture, sequencing, and whether this project was actually as straightforward as I’d convinced myself it was. Well, waiting is underrated. The plan that came out is worth its own post. Until next time…

If you’re following along and want the full hardware inspection checklist — what to check, what thresholds to use, and when to return immediately — it’s in the appendix below.

Appendix 1: System requirements for running local LLMs on Mac

Why this machine, and not just any machine.

Buying a refurb Mac for general use is straightforward. Buying one to run local AI models requires a few additional considerations — because not all Apple Silicon is created equal for this purpose.

The minimum bar for this project:

Requirement Minimum: Chip - Apple Silicon (M1 or later) | M3; Unified Memory 8GB (constrained) | 16GB; Storage: 256GB (tight) | 1TB; macOS: Monterey 12+ | Latest

Why these numbers matter:

Unified memory is the critical variable. Unlike traditional laptops where CPU and GPU have separate memory pools, Apple Silicon shares one pool across both. Local LLMs load their model weights into this memory — a 7B parameter model (like Mistral 7B) requires roughly 4–8GB depending on quantization. At 8GB total, you’re running the model and your operating system in a constant negotiation for space. At 16GB, you have room to work.

Storage matters more than people expect. A single quantized model can run 4–8GB. If you plan to experiment with multiple models — which you will — 256GB fills up faster than feels reasonable.

The M3 specifically isn’t required — an M1 or M2 with 16GB would serve the project equally well, and would likely be cheaper on the refurb market. The M3 was available at the right price point. Don’t over-optimize for the latest chip generation.

What won’t work:

• Intel Macs: Ollama technically runs, but performance is poor enough to be discouraging. Not recommended.

• 8GB unified memory: Possible for basic experimentation, but you’ll feel the ceiling quickly. Acceptable for testing the concept; limiting for sustained use.

• Pre-Monterey macOS: Several dependencies won’t install cleanly.

The practical implication:

If you’re following this project and considering building something similar, the refurb market for M1/M2 MacBook Pros with 16GB is currently the best value entry point. You don’t need new. You don’t need M3. You need Apple Silicon and 16GB.

Appendix 2: Inspection checklist

(The nerdy part. Skip if you trust your instincts. Don’t trust your instincts.)

Before you do anything: Photograph everything on arrival. Box, accessories, all sides of the machine. Serial number. This is your evidence if anything goes sideways.

Battery: Apple menu → System Report → Power. Look for cycle count (under 100 is excellent, under 300 is acceptable), condition (”Normal”), and full charge capacity (85%+ of design capacity). Anything worse: negotiate or return.

Specs verification: Apple menu → About This Mac. Confirm chip, RAM, and storage match exactly what you paid for. Screenshot it. Discrepancy = grounds for return, full stop.

Physical inspection: Screen on and off (dead pixels, coating wear, bright spots), keyboard and trackpad feel, hinge smoothness, port integrity. Test every port with an actual device — don’t assume.

Diagnostics: Restart, hold Power until startup options appear, Command+D runs Apple’s built-in hardware test. “No issues found” is the only acceptable result. Optional: CoconutBattery for detailed battery analysis, Blackmagic Disk Speed Test to confirm SSD performance.

Return policy: Back Market offers 30-day returns and a 1-year warranty. Keep all packaging until you’re certain you’re keeping the machine. Document everything through their platform — not email, not phone.

Immediate return triggers: “Service Recommended” battery, mismatched specs, screen damage, non-functional ports, non-genuine parts warnings in macOS.

This is Part 2 of an ongoing series on building a private, local AI tax assistant — one hour a week, on consumer hardware, without sending your financial data anywhere.

AI is consuming everything around us —  and on the face of it, it smells like progress, and most probably it is, but, and here it comes, the ubiquitous BUT, it bothers me how quickly we are giving up on our personal data to gain a little bit of advantage. So, I set out on a personal project: to build a private AI assistant that can help with tax preparation using my own financial data — without sharing that data with public models or the cloud.

In this day and age of ever evolving cyber attacks, which even the biggest companies around the world struggle to prevent, sharing my sensitive data with outside consultants makes me quite uneasy. One could argue “why do you think you are special?”, and “you are just not interesting enough — no hacker worth the salt will be interested in your information specifically”. 

Still combine that with the thought “why do I pay tax accountants to fill up a form that I myself can fill up?” and “I have a pretty good B-school degree, majoring in, wait for it, Finance, I should be able to do this”. But I don’t stay up to date with tax laws and sometimes work gets crazy around tax days (just Murphy’s Law!). So I need help. And when I read about agentic AI, it seemed like the perfect solution. BUT… if I use just any available agent, my data still goes out into public models, and I don’t trust these startup-y things to keep my data safe.

Also, there is the adventure: Can I really do this on a MacBook , with just one hour per week, and without breaking the bank?

Just for background, I’m not a techie by any stretch of imagination. I used to be one a very, very, long time back. But not any longer. I just like to dabble and tinker around with the proverbial “what if…” That, plus heard all the buzz about machines coding, and thought, so why not take this idea out for a ride and see how far it goes…

Here’s the breakdown of what I’m learning, planning, and building.

The Idea

I’m building a tax assistant that can:

- Read my W-2s, 1099s, and expense receipts 

- Reason about eligibility for deductions or credits 

- Give advice based on IRS rules 

- Do it entirely offline, keeping my financial data private

Can I Build This on a MacBook?

Surprisingly — yes.

Tools like Ollama, LlamaIndex, and Streamlit make it possible to run local models like Mistral 7B or LLaMA 3 directly on my MacBook. By using quantized models, I’m able to get decent performance even without a GPU.

What If I Only Have 1 Hour per Week?

I mapped out a 6-month roadmap using just 1 hour per week, breaking it down into phases:

- Month 1: Setup local LLM + basic Q&A 

- Month 2: Add PDF parsing (W-2s, 1099s) 

- Month 3–4: Apply simple tax logic (deduction thresholds, filing status rules) 

- Month 5–6: Build a simple RAG pipeline and Streamlit UI 

With focus and consistent micro-progress, it’s totally doable.

What Will It Cost?

So far, almost everything is open source. My only real “costs” are:

- Local compute power (already owned) 

- Electricity (~$10/month max) 

- Time (the real investment) 

If I were to add GPU cloud compute, that might add $50–$200/month, but so far I’m staying local.

What’s Next?

I’m documenting the full journey on my substack (here), and posting monthly updates on LinkedIn as I go.

Coming soon (Note to self: this is a plan that is sure to be disrupted… being a planner, I know that. It’s the only thing one can say about plans with some bit of certainty, but nevertheless… here it goes):

- “How I Got an AI Model to Parse My W-2s” 

- “Can AI Actually Understand Tax Law?” 

If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!

Subscribe now