This was written with assistance and editing by A.I. Obvi.

Building a Real Local AI Assistant: What Happened When I Stopped Chasing Models

A few months ago, I decided I wanted to build a local AI assistant.

At the time, my reasoning seemed straightforward. I already paid for ChatGPT and used it daily, but I was increasingly interested in the idea of running models on my own hardware. Privacy was part of the appeal. Independence was another. Mostly, though, I wanted to understand what all the excitement around local AI was really about.

Like a lot of people, I started by reading articles, watching YouTube videos, and browsing Reddit threads. The advice was remarkably consistent. Install Ollama. Download a model. Open a chat interface. Ask a few questions. Congratulations—you now have your own private ChatGPT. Technically, that advice wasn't wrong. It was just incomplete.

What I eventually discovered is that running a model locally is the easiest part of the entire process. Building something genuinely useful is a completely different challenge.

At the center of my home lab is a server I call Strongbox. It's not an especially exotic machine by AI enthusiast standards. It runs Ubuntu, has an RTX 3060 with 12 GB of VRAM, and hosts a growing collection of services I've accumulated over the years. Immich, OpenWebUI, Docker containers, automation tools, development projects, and various experiments all live there.

Strongbox wasn't built for AI. It simply evolved into the place where most of my projects eventually ended up.

When I started exploring local models, I assumed the primary challenge would be finding the right one. Every discussion online seemed to revolve around model comparisons. One week everyone was talking about DeepSeek. The next week it was Qwen. Then Gemma. Then some newly released reasoning model that had climbed a benchmark leaderboard. I fell into the same trap everyone else does. I downloaded models. Then I downloaded more models. Then I downloaded models that were supposed to replace the models I'd downloaded a week earlier.

At one point my Ollama instance looked less like a carefully designed system and more like a museum of AI releases. Every benchmark result convinced me that the next download would finally be the one that made everything click.

What surprised me wasn't how different the models were. It was how similar the overall experience felt.

Some models were undeniably smarter than others. Some wrote better code. Some reasoned more effectively. Some produced more polished answers. Yet after the novelty wore off, I found myself paying attention to something that benchmark charts never seem to capture: waiting.

A model might generate a slightly better answer, but if it took forty seconds to get there, the experience often felt worse. I'd ask a question, stare at a spinner, and eventually realize I could have opened a browser tab and found the answer myself. The more I used local AI, the more I realized that responsiveness mattered far more than I expected.

Eventually, I stopped obsessing over benchmark scores and started paying attention to which models I actually enjoyed using. The winners weren't necessarily the smartest models. They were the ones that responded quickly enough to stay out of my way.

At the time, I thought this was the major lesson of the project. It wasn't.

The real lesson arrived later, and it took me much longer to recognize.

The turning point came while I was working on an unrelated infrastructure project. I needed information about one of my servers and instinctively asked a model. The answer wasn't useful. Not because the model was incapable of answering the question, but because it didn't know anything about my environment. It didn't know which machine hosted a particular service. It didn't know the IP address of my Hubitat hub. It didn't know where I had deployed a project six months earlier. It didn't know which decisions I had already made and which ideas I had already abandoned. The more I thought about it, the more obvious the problem became. The issue wasn't intelligence, The issue was memory.

For years I've accumulated technical knowledge about my own environment. Server addresses, Docker configurations, troubleshooting notes, project decisions, networking quirks, automation rules, and countless other details exist somewhere in notes, documentation, terminal history, or my own head. None of that information is particularly difficult, but finding it repeatedly is frustrating. I wasn't looking for an AI that could explain quantum mechanics, I wanted an AI that could tell me where I put something. That realization changed the entire direction of the project. Instead of trying to build a better chatbot, I started trying to build a memory system.

Once I began viewing the problem through that lens, the architecture emerged naturally. OpenWebUI remained the front end because I liked the interface. Ollama continued to provide local models. Beyond that, however, the focus shifted almost entirely toward retrieval and memory.

After experimenting with different approaches, I settled on a stack built around Mem0, Qdrant, and local embeddings generated by Nomic Embed Text. A small API sat in the middle and exposed a handful of endpoints for storing and retrieving information. The individual components weren't particularly groundbreaking, but together they formed something much more valuable than a standalone language model.

For the first time, the assistant could access information that mattered to me.

Of course, building the system turned out to be far messier than drawing the architecture diagram.

There were networking issues that made no sense until they suddenly did. Containers that could reach one service but not another. Configuration mistakes that consumed entire evenings. Health checks that reported everything was fine when it clearly wasn't. Looking back through my notes, I probably spent more time troubleshooting Docker networking than I did configuring the actual AI components.

That's another thing many local AI articles leave out. The screenshots usually show the finished result. They don't show the three-hour debugging session that preceded it.

But even after the infrastructure was working, another challenge emerged.

The models didn't always behave the way I expected.

I had built memory tools that could search and store information, and I assumed the models would naturally use them. Instead, I discovered that tool calling is one of the least discussed and most important aspects of building useful AI systems.

Sometimes a model would call the correct tool.

Sometimes it would explain how the tool worked instead.

Sometimes it would generate code that theoretically called the tool.

Sometimes it would confidently answer a question without performing a search at all.

The memory system worked. The tools worked. The API worked.

The unpredictable part was the model.

That experience taught me something I hadn't fully appreciated before: intelligence and reliability are not the same thing. A model can be extremely capable while still being difficult to integrate into a larger workflow. Once you move beyond simple chat conversations and start building systems, consistency often becomes more important than raw capability.

By that point, the project barely resembled the one I had imagined when I started.

I thought I was building a local AI assistant.

What I was actually building was a personal knowledge system that happened to use AI as an interface.

The distinction sounds subtle, but it changed how I think about the entire field.

Early in the project, I spent most of my time comparing models. Today, I spend much more time thinking about retrieval, context, memory, and workflow design. Models still matter, of course, but they no longer occupy the center of the conversation. Once a model reaches a certain level of competence, the quality of the surrounding system often determines whether it's useful.

A model that scores slightly higher on a benchmark is interesting.

A model that remembers your infrastructure, understands your projects, recalls decisions you've made, and helps you continue work you started last month is genuinely valuable.

That's the difference I was searching for all along, even if I didn't realize it at the time.

Looking back, the biggest surprise is that this wasn't really an AI project. At least, not in the way I expected. The most difficult problems had very little to do with inference, model selection, or GPU performance. They involved organizing information, designing interfaces between systems, deciding what should be remembered, and figuring out how to retrieve it effectively when needed.

Those are fundamentally systems problems.

And systems problems are where useful software gets built.

The local AI community spends an enormous amount of time discussing which model is best. After months of experimentation, I've become much more interested in a different question: what does the model actually know, and how easily can it access the information that matters?

For my own work, that question has turned out to be far more important than any benchmark score.

The models got me interested in local AI.

Memory is what made it useful.