Local AI server

1. What you set up

A model server with Ollama on Ubuntu 24.04, with two models: gpt-oss:20b writes the answers and the queries, bge-m3 finds content by meaning. The products connect to it over the network; one server can serve all three.

2. Hardware

An NVIDIA GPU with at least 16 GB of memory for gpt-oss:20b — 24 GB leaves room for bge-m3 next to it. 32 GB of RAM, 8 cores and 60 GB of free disk. Without a GPU the models run too, but an answer takes minutes rather than seconds — fine for a first look, not for daily work.

The AI server can be the same machine as the products or a machine of its own. A machine of its own keeps the GPU's load away from the applications.

3. The GPU driver

On a fresh Ubuntu 24.04 install the recommended NVIDIA driver and restart:

sudo apt update
sudo ubuntu-drivers install
sudo reboot

After the restart the GPU and its memory are listed by:

nvidia-smi

4. Ollama

Ollama installs itself as a systemd service that starts with the machine:

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama

Then fetch the models:

ollama pull gpt-oss:20b
ollama pull bge-m3
ollama list

5. Reachable for the products — and for nobody else

Ollama listens on 127.0.0.1:11434. If the products run on the same machine, there is nothing to do. On a machine of its own, let it listen on the network:

sudo systemctl edit ollama

# insert between the comment lines:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=24h"

sudo systemctl restart ollama

Ollama has no sign-in of its own. Open the port only to the servers running the products — never to the internet:

sudo ufw allow from 10.0.0.21 to any port 11434 proto tcp
sudo ufw enable

Across networks, put a reverse proxy with sign-in or a VPN in front of it; the products can send a user and password or an API key. OLLAMA_KEEP_ALIVE keeps the models loaded, so the first question of the morning does not wait for them.

6. Connecting the products

In each product: System settings › AI assistant.

“Test the connection” must name the models on offer, an answer from the model and meaning vectors of 1024 values. Then switch the assistant on and save. An address other than the product's own server has to be confirmed once by an administrator; the confirmation is in the audit log.

Who may use the assistant is set under General permissions (“Use the AI assistant”). The background worker then builds the search index; the AI settings show when it last went round.

7. Checking and keeping it current

Whether the server answers, from a product's server:

curl http://<AI server>:11434/api/tags

While a question is answered, nvidia-smi shows the GPU at work. Ollama is updated by running the install line again; a model by pulling it again. The test catalogue in the AI settings shows, by numbers, whether a new model answers better than the old one.

8. Without your own server

The products can also use Anthropic's models in their cloud. Questions and the content found then leave the house; the products say so on the settings page and start the assistant only after an administrator's confirmation.