TIMBOTIC / Field report

Building TIMBOT: a personal AI assistant with visible boundaries

The interesting part of building a personal AI assistant turned out to be everything around the model.

Getting a local language model to answer a question was the beginning. Making it remember useful facts, look something up, ask for a second opinion, and explain which services it had contacted took considerably more thought.

TIMBOT grew out of that work. It is a small, request-driven assistant built around local inference, with optional access to web research and cloud models. Its central rule is straightforward: use the local model by default, and keep the human in control of changes to that arrangement.

That sounds simple until a model says it cannot browse when browsing is authorized, invents a weather tool, or recommends escalation without making its reason clear. Those awkward moments have been useful. Each one exposed a boundary that needed to become explicit in code, in the interface, or both.

A small application around a separate AI machine

TIMBOT has a browser interface, a Python FastAPI backend, and SQLite storage for conversations and memories. The interface uses ordinary HTML, CSS and JavaScript. There is no frontend build pipeline or external asset service required to draw the chat window.

The application runs on a VPS behind an HTTPS reverse proxy. The main local model—Gemma in this setup—runs through llama.cpp’s llama-server on a separate AI PC, reached over a private network. The VPS coordinates requests; the PC does the local model inference.

This separation keeps model weights out of the application container. The supplied container retains a 512 MB memory limit. Gemma uses the PC’s GPU, while a separate advisory classifier, LAYA, uses CPU and system RAM on that same PC.

Here is the broad request flow. The arrows to outside services are conditional; they are not steps every message takes.

“Local” therefore does not mean “nothing leaves the room.” Messages pass through the owner’s VPS. A permitted search sends a query to a search provider. A selected cloud model receives the chosen context. The useful distinction is that the default inference endpoint is personally controlled, and additional destinations have separate purposes and checks.

There are real tradeoffs. Local hardware limits model size and throughput. The PC needs to be available. Multiple generation rounds can make a simple-looking request slow. There is also more infrastructure to understand than with a single hosted chat service. In return, the main inference path uses hardware under the owner’s control and does not depend on a cloud-model call for every answer.

Deterministic routing comes first

It was tempting to make routing another AI problem: ask a model which model should answer. TIMBOT instead starts with ordinary code.

The deterministic router checks explicit selections, local-only restrictions, verification requests and a narrow set of uncertainty cues. A normal request can go straight to the local model. An explicit provider selection bypasses LAYA. A request to check an answer can produce a choice screen without selecting a paid model on the user’s behalf.

This is deliberately narrower than general semantic understanding. The router does not recognize every possible way to express uncertainty. Its advantage is that the ordinary path is predictable, avoids unnecessary classifier calls, and is straightforward to test.

Most importantly, routing is not permission. After a route is proposed, the API independently checks whether a cloud destination matches the authenticated request’s explicit selection. The outbound provider adapter checks again before contacting that provider.

LAYA can recommend, but cannot authorize

LAYA has a specific job: classify certain ambiguous requests that pass the deterministic gate.

It runs as a separate authenticated HTTP service on the local AI machine. TIMBOT’s VPS process calls that service; it does not load LAYA or PyTorch. The service uses fixed classification questions and CPU inference, leaving the main model’s GPU allocation alone.

LAYA receives the current request text. It is not handed the conversation database, saved memories, images or credentials. That smaller input is intentional, though it also means references to earlier messages may be ambiguous to the classifier.

The four signals ask whether the request involves complex reasoning, requires current information, requests independent verification, or might materially benefit from a stronger reasoning model. These are classifications about the request, not evaluations of whether an answer is true.

One important correction was to evaluate actionable signals independently. Previously, uncertainty in an unrelated classification could veto a strong verification signal. Now, at the default threshold of 0.75, a signal qualifies when both its positive probability and confidence meet the threshold.

A qualifying verification or stronger-model signal can recommend ask_user. A qualifying current-information signal can also contribute when web permission is absent. Complex reasoning alone does not trigger that recommendation. None of these classifications grants permission to call anything.

In live advisory mode, the interface can say:

LAYA recommends verification
LAYA detected a 92% verification signal for this request.

The choices remain Choose OpenAI, Choose Anthropic, and Keep Local. The displayed percentage comes from the relevant classification’s probability, not the minimum confidence across unrelated classifiers. It is not “92% accuracy” or a probability that the answer is wrong.

The configuration defaults to shadow mode, where advice is recorded without changing the route; live advisory mode must be enabled deliberately. A timeout or invalid classifier response leaves the original local route intact. LAYA’s strongest live intervention is asking the human.

Cloud models are an explicit option

OpenAI and Anthropic fit into the same assistant interface through provider adapters. They are useful when the user wants another model’s capabilities, but they are not automatic recovery paths when local inference struggles.

The provider and model controls make that choice visible. The backend binds permission to the selected provider, rather than treating “cloud allowed” as permission to choose any destination. A visible cloud selection can remain selected for subsequent sends; a new conversation or reload returns the UI to Local.

There is a small but important distinction between discovering models and sending a conversation. Explicitly choosing or refreshing a cloud provider can contact its model-list endpoint. That does not send chat content. Generation happens when the user submits with that provider selected.

The user also chooses how much saved conversation text to include: the current message only, a recent-message window, or the available conversation history, subject to context limits. Separate saved-memory lookup is disabled for cloud generation. Selected messages can still contain personal facts, so scope control is not automatic redaction.

Check Answer needs an actual answer to check

An early verification prompt effectively said, “Check the answer.” That is understandable to a person looking at the screen, but insufficient when the verification model receives a deliberately narrow context.

The Check Answer button now prepares an editable request containing the corresponding original user question, the exact assistant answer, and a clear verification instruction. Those two pieces are packaged as JSON inside the draft, so the target interaction is explicit.

Clicking the button does not send anything. It sets the history scope to current-message-only and opens the choice controls. The user can select a cloud provider for an independent review or keep the work local. Web permission remains separate.

This also lets a verifier assess an answer that honestly says it could not confirm a current fact. It receives the actual limitation to evaluate, rather than having to guess which earlier exchange the user meant.

Useful memory without distributing it everywhere

TIMBOT’s memory is intentionally uncomplicated: durable facts in SQLite, retrieved with full-text search rather than a vector database.

Facts have stable keys, so updating a remembered detail can replace the existing fact. Retrieval uses search relevance, importance, small synonym expansions and explicit size limits. Location-related requests prioritize location facts. The interface also allows memories to be inspected, edited and deleted.

This makes small personal references possible. A short “at home” reply can prioritize stored location information instead of requiring the user to type it again. It is still retrieval plus model interpretation—not an infallible understanding of someone’s life. Missing, stale or ambiguous facts can require clarification.

Retrieved facts enter the local model’s context as data. They do not become instructions or permissions. LAYA does not receive the memory store, and cloud generation cannot use the memory-search tool to bypass a narrow disclosure scope.

There is another practical guard: once web results have entered a request’s tool loop, model-driven memory writes are blocked for that request. This reduces one route by which instructions embedded in a page could turn into durable remembered facts.

Web research is available by default, per message

Every fresh normal composer starts with Allow web access for this message checked.

That is permission to use research when useful. It is not an instruction to search on every turn. “Explain binary search” can be answered locally with no web contact. A current forecast or recent news question has a different information requirement.

The browser serializes the checkbox’s current value as allow_web. If the user unchecks it, search and page-fetch execution are blocked in the backend. That opt-out applies to the submitted message. After a successful reply, the next fresh composer starts checked again. Failed requests and pending provider-choice drafts retain their existing choice for retry.

Two diagnostics keep the distinction visible:

web_authorized: true
web_contacted: false

That can be a perfectly normal result: research was permitted but unnecessary. When an external web transport is attempted, web_contacted becomes true. It does not prove successful retrieval. A missing search key is a configuration failure, not web contact.

Unchecking the box disables web research; it does not cancel a separately selected cloud model. Conversely, checking it does not authorize OpenAI or Anthropic. Keeping those permissions independent is essential to understanding where a message may go.

A weather question exercises the whole system

Consider this private-safe exchange:

User: What will the weather be like today?
TIMBOT: Where would you like the forecast for?
User: At home.

With suitable saved memory, the local model can resolve the location. The new message carries its own checked web permission. The model can then request a general search using the resolved city and forecast period. Search snippets return through the tool loop, and the local model produces an answer with source links.

There is no special weather provider in this flow. TIMBOT uses Brave Search and, when requested by the model, public-page fetching. Retrieved material is treated as untrusted data, and fetched destinations are checked against restrictions on private addresses and unsafe URLs.

The location needed for the query necessarily reaches the search service. Keeping inference local does not conceal a query from the service asked to answer it. The goal is purposeful disclosure, not an impossible promise of zero disclosure.

The recent “at home” repair path is covered with synthetic memory and mocked model/web transports. That establishes application behavior under the tested outputs; it does not establish that every real-model weather exchange will succeed.

A failed tool call should be inconvenient, not dangerous

Language models do not always produce the interface we intended. A model may invent get_weather(...) even though TIMBOT exposes only web_search, web_fetch, memory_search and memory_store.

TIMBOT validates actions before execution. A supported search looks like this:

{"action":"web_search","arguments":{"query":"Example City weather today"}}

The query must be a nonempty string within its length limit. Extra arguments are forbidden. Other tools have their own schemas. Limited compatibility parsing exists for recognized formats, but it is not arbitrary code execution or permission to invent new actions.

When validation fails, the agent offers one repair attempt. The repair instructions now include exact argument schemas generated from the same definitions used by validation, and explain that weather research should use general search. An invalid action is never executed; a valid replacement must still pass permission checks. If repair fails, the request stops safely.

Recent work also improved the rejection diagnostics. They record a failure category, safe tool label and repair outcome, while omitting argument values. Unknown names outside a small safe set are redacted too.

We cannot retrospectively identify a particular historical Gemma failure whose raw calls were not retained. The fix improves the interface and future diagnosis without pretending those missing details were recovered.

Permission and capability were not enough

Another discovery was that an authorized model with available tools could still return plain text claiming web access was disabled. The agent previously accepted that answer immediately.

The response was a bounded research fallback, not compulsory searching. The model normally chooses its tools. If it skips research for a recognized, clear live-information request, the agent can issue one search through the same validated execution path and return the result—or the actual tool error—to the model.

This fallback is deliberately conservative. It recognizes short weather, current-price, current-CEO, selected latest-information and explicit-search patterns, plus an immediate weather clarification containing a supplied location. It does not independently mine memories, images or quoted Check Answer interactions for queries. The “at home” case still depends on the local model using retrieved memory correctly.

It also does not duplicate research after a web tool has already been attempted. The existing generation and tool-round bounds remain in place. This improves a specific failure mode without pretending that keyword rules provide general language understanding.

Inspecting decisions instead of hidden thoughts

Routing Details exposes operational behavior when routing diagnostics are enabled. A sanitized local-research result might include:

{
  "router": "deterministic",
  "deterministic": "local",
  "laya_invoked": false,
  "final_policy_decision": "local",
  "cloud_contacted": false,
  "web_authorized": true,
  "web_contacted": true
}

When LAYA participates, fields such as laya_status and laya_actionable_signals explain its role. Individual classification values are available with LAYA debug enabled; the friendly advisory works without it.

These are infrastructure facts, not a display of hidden chain-of-thought. They answer useful debugging questions: which route ran, whether a classifier participated, what permission existed, and whether an outbound transport was attempted.

The same distinction shaped progress feedback. The browser polls actual activity stages while the backend consumes model streams and eventually returns the completed answer. “Searching the web” is useful evidence of activity; a decorative percentage is not a measurement of how much thinking remains.

What the awkward cases taught us

The most useful tests sit at boundaries. A checkbox test should inspect the serialized request, not merely the HTML checked attribute. A permission test should reach a mocked transport and prove that unauthorized calls never arrive. A verification test should inspect what the selected provider actually receives.

The current regression suite covers those boundaries alongside memory, LAYA, provider adapters and tool repair. In the latest local validation preceding this article, 373 Python tests and 20 JavaScript tests passed. That is evidence about tested behavior, not a security certification or a guarantee of model quality.

Several design lessons keep recurring. Classifier probabilities need careful labels. Small models benefit from explicit, constrained interfaces. A visible UI default must agree with backend authorization. Errors need enough metadata to investigate without turning logs into another store of personal data.

And failure needs a defined destination. LAYA timing out leaves the local route intact. Missing cloud selection cannot turn into a cloud request. An unchecked web box blocks research. A rejected action executes nothing, and a rejected repair ends the attempt. Those defaults are part of the product, not incidental exception handling.

The assistant I wanted to build

TIMBOT remains a practical experiment with limits: hardware capacity, model behavior, network availability and configuration all matter. It is request-driven, with a small tool set, rather than a general autonomous machine operator.

What makes it interesting is the combination. Local intelligence handles the default conversation. Memory supplies useful continuity. Web research brings in current evidence. Another model is available when the human chooses it. The interface makes those boundaries visible enough to inspect and question.

There is still room to improve the experience. But the direction is clear: bring in external capabilities when they add value, keep their permissions distinct, and leave the important choices with the person using the assistant.