Building a Private AI Tax Assistant: In public, on a MacBook!
Why on earth? Is It Possible? How Long Will It Take? What Will It Cost?
WHY on earth would you want to do that?
AI is consuming everything around us — and on the face of it, it smells like progress, and most probably it is, but, and here it comes, the ubiquitous BUT, it bothers me how quickly we are giving up on our personal data to gain a little bit of advantage. So, I set out on a personal project: to build a private AI assistant that can help with tax preparation using my own financial data — without sharing that data with public models or the cloud.
In this day and age of ever evolving cyber attacks, which even the biggest companies around the world struggle to prevent, sharing my sensitive data with outside consultants makes me quite uneasy. One could argue “why do you think you are special?”, and “you are just not interesting enough — no hacker worth the salt will be interested in your information specifically”.
Still combine that with the thought “why do I pay tax accountants to fill up a form that I myself can fill up?” and “I have a pretty good B-school degree, majoring in, wait for it, Finance, I should be able to do this”. But I don’t stay up to date with tax laws and sometimes work gets crazy around tax days (just Murphy’s Law!). So I need help. And when I read about agentic AI, it seemed like the perfect solution. BUT… if I use just any available agent, my data still goes out into public models, and I don’t trust these startup-y things to keep my data safe.
Also, there is the adventure: Can I really do this on a MacBook , with just one hour per week, and without breaking the bank?
Just for background, I’m not a techie by any stretch of imagination. I used to be one a very, very, long time back. But not any longer. I just like to dabble and tinker around with the proverbial “what if…” That, plus heard all the buzz about machines coding, and thought, so why not take this idea out for a ride and see how far it goes…
Here’s the breakdown of what I’m learning, planning, and building.
The Idea
I’m building a tax assistant that can:
- Read my W-2s, 1099s, and expense receipts
- Reason about eligibility for deductions or credits
- Give advice based on IRS rules
- Do it entirely offline, keeping my financial data private
Can I Build This on a MacBook?
Surprisingly — yes.
Tools like Ollama, LlamaIndex, and Streamlit make it possible to run local models like Mistral 7B or LLaMA 3 directly on my MacBook Pro (M3, 16GB RAM). By using quantized models, I’m able to get decent performance even without a GPU.
What If I Only Have 1 Hour per Week?
I mapped out a 6-month roadmap using just 1 hour per week, breaking it down into phases:
- Month 1: Setup local LLM + basic Q&A
- Month 2: Add PDF parsing (W-2s, 1099s)
- Month 3–4: Apply simple tax logic (deduction thresholds, filing status rules)
- Month 5–6: Build a simple RAG pipeline and Streamlit UI
With focus and consistent micro-progress, it’s totally doable.
What Will It Cost?
So far, almost everything is open source. My only real “costs” are:
- Local compute power (already owned)
- Electricity (~$10/month max)
- Time (the real investment)
If I were to add GPU cloud compute, that might add $50–$200/month, but so far I’m staying local.
What’s Next?
I’m documenting the full journey on my substack (here), and posting monthly updates on LinkedIn as I go.
Coming soon (Note to self: this is a plan that is sure to be disrupted… being a planner, I know that. It’s the only thing one can say about plans with some bit of certainty, but nevertheless… here it goes):
- “How I Got an AI Model to Parse My W-2s”
- “Why Privacy-First AI Matters (And What I’m Doing About It)”
- “Can AI Actually Understand Tax Law?”
If you’re building something similar or have any questions/ideas to share, I’d love to hear from you. Cheers!