Ollama (fully local)

Ollama runs AI models on your own computer. With AnythingLLM as the chat window, your question, your books and the answer never leave the PC.

Ollama is the engine that runs the model. It has no chat window that can use MCP tools, so a chat app sits in front of it. These steps use AnythingLLM, a free desktop app for Windows and Mac that works with both Ollama and MCP. The other way to stay local is LM Studio, which runs models by itself: see the LM Studio guide.

1. Ollama and a model

1. Install Ollama from ollama.com.
2. Open a terminal (PowerShell on Windows, Terminal on a Mac) and get a model that can use tools. qwen3:8b is about 5 GB:

ollama pull qwen3:8b

A computer with 16 GB of memory runs an 8-billion-parameter model comfortably. Larger models answer better if your computer has the memory for them.

Ollama also offers cloud models. Their names end in -cloud and they run on Ollama’s servers, not yours. For fully local answers, pick a model without -cloud.

2. AnythingLLM

1. Install AnythingLLM Desktop from anythingllm.com and open it.
2. Choose Ollama as the LLM provider (in the first-run setup, or later under Settings, AI Providers, LLM) and pick the model you pulled. AnythingLLM finds Ollama on the same computer by itself.
3. If @agent later says it has no model, open the workspace’s settings, then Agent Configuration, and choose Ollama and the same model there.

AnythingLLM sends anonymous usage counts by default. If you want no traffic from it at all, switch telemetry off in its privacy settings.

3. Add MCP for Tally

The easy way: open MCP for Tally, go to Connect AI apps and click Add beside AnythingLLM. The app writes the settings below for you, keeps a backup of the file, and never touches your other entries.

Then, in AnythingLLM, open Settings, Agent Skills and press Refresh. mcpfortally shows under MCP servers. By hand, put this in AnythingLLM’s MCP file and press Refresh:

{
  "mcpServers": {
    "mcpfortally": {
      "command": "C:\\Users\\<you>\\AppData\\Local\\Programs\\MCP for Tally\\runtime\\node.exe",
      "args": ["C:\\Users\\<you>\\AppData\\Local\\Programs\\MCP for Tally\\app\\mcpfortally.js", "mcp"]
    }
  }
}

{
  "mcpServers": {
    "mcpfortally": {
      "command": "/Applications/MCP for Tally.app/Contents/MacOS/mcpfortally",
      "args": ["mcp"]
    }
  }
}

4. Ask

In a workspace chat, start the message with @agent, so the model may use tools: @agent Which companies are open in Tally? Without @agent, AnythingLLM chats without tools.

If the model ignores the tools or stops halfway, give it a larger context window in AnythingLLM’s Ollama settings (16,000 tokens or more), or try a larger model. Local models do best with one question at a time.

Another chat app that runs on Ollama and can add an MCP server by command can be set up the same way, with the settings on Any other MCP app.

Something not working? Write to [email protected] with the app’s Copy diagnostics text. It never includes figures from your books.