LLM Server Hosting

What is LLM server hosting and who is it for?

LLM server hosting means renting hardware to run open-weight language models on a GPU server under your control instead of a third-party AI API. Because the model runs on your server, the text, documents and customer data you send do not go to another company’s infrastructure, and cost is set by server resources rather than per request.

Your own LLM server stands out in these scenarios:

• Assistants that work with confidential data: question-answering (RAG) systems over contracts, health, finance or HR documents
• Internal chat interface: offering staff a private chat assistant through an interface such as Open WebUI
• High-volume automation: repetitive work such as thousands of product descriptions, email classification, summarisation and translation
• Software teams: local code models for code completion and code review
• R&D and fine-tuning: LoRA / QLoRA fine-tuning with your own data and model comparisons

A CPU alone falls short for these workloads; inference speed is determined by the graphics card’s memory and memory bandwidth. This page covers which model fits on which server and how to run it in production. For a broader look at GPU hardware, see our GPU server hosting range.

How much VRAM does each model need?

You estimate the VRAM a model needs roughly as number of parameters × bytes per parameter, then add a KV cache for context and working headroom. At 16-bit (FP16 or BF16) precision each parameter takes 2 bytes, so an 8-billion-parameter model needs about 16 GB for the weights alone. Quantised to 4 bits, the same model drops to about 5 GB.

The item most often forgotten is the KV cache. For every token in the conversation the model stores key and value vectors per layer; this memory grows linearly with context length and the number of concurrent requests. In a system that works with long documents or serves many users at once, the KV cache can take as much space as the model weights.

Practical rules:

• Add at least 10–30% to the weight figure in the table for a single user and medium-length context.
• If the model does not fit fully in VRAM, some layers can be offloaded to the CPU and system memory (partial GPU offload in llama.cpp), but speed drops noticeably.
• If you are planning a multi-user API, aim to fit not just the model but “model + target context length × concurrent requests” into VRAM.

You do not have to use the full context length stated in the model card; in production, limiting context to what you need frees room for more concurrent requests on the same GPU.

What is quantisation and how does it affect model quality?

Quantisation is the technique of storing model weights at lower precision, such as 8, 5 or 4 bits instead of 16, which reduces memory needs and in most cases response generation time. In return, some model accuracy is lost; at 8 bits this loss is usually too small to notice, but it becomes noticeable below 4 bits.

A general roadmap: start with the 4-bit version of your target model and measure quality with your own sample questions. If the result is not good enough, try the 5- or 8-bit version of the same model or the 4-bit version of a larger model. In most business scenarios, the 4-bit version of a large model performs better than the 16-bit version of a small one. For precision tasks such as maths, code and structured output (JSON), always verify the effect of low-bit quantisation with your own tests.

Should you use Ollama, llama.cpp or vLLM?

Ollama is the right choice for single-user experiments, development and small internal teams; llama.cpp for controlling the hardware in fine detail and running GGUF models flexibly; and vLLM for a production API that answers many concurrent requests with high throughput. All three offer an OpenAI-compatible API, so moving your application from one to another is easy.

By default vLLM reserves most of the GPU memory up front (--gpu-memory-utilization) and manages the KV cache in that space; if you will run another model on the same GPU, you need to lower this ratio.

How many users can an LLM server serve?

How many users an LLM server can serve depends on model size, quantisation level, context length and the batching ability of the inference software together; giving a single number would be misleading. The right method is to load-test with your own prompt and response lengths.

Separate two measures when planning:

• Time to first token: how long the user waits for the response to start. Long prompts and long documents increase this time.
• Tokens per second: the streaming speed of the response. For a single user, GPU memory bandwidth decides; with many users, total throughput matters.

Servers with continuous batching such as vLLM process several requests together on the same GPU and can multiply total throughput compared with a single user; in return, each user’s streaming speed drops somewhat. Ollama offers limited parallelism with default settings and is sufficient for small internal teams.

Ways to increase capacity: choose a smaller or quantised model, limit context length, move to a GPU with more VRAM, or spread requests across several GPU servers. For latency-sensitive applications it also matters that the server is close to your users; for users in Türkiye and the surrounding region, a server hosted in Türkiye reduces network latency.

How important are CPU, RAM and disk besides the GPU?

In LLM inference the GPU does the real work, but if system memory, disk and CPU are poorly chosen, you cannot use the GPU’s full power. Model loading time and layers that spill over to the CPU in particular depend on these components.

• System memory (RAM): The model file passes through system memory on its way to the GPU; having more system memory than the file size of the largest model you use prevents problems during loading. If you will run some layers on the CPU, you need this memory on top.
• Disk: Even the 4-bit version of a 70-billion-parameter model is tens of gigabytes; trying several models and versions needs hundreds of gigabytes. NVMe storage noticeably shortens the time to read a model from disk and load it into memory.
• CPU: Tokenisation, request handling, document processing for RAG and embedding generation use the CPU; in a multi-user API, enough cores prevent a bottleneck.
• Network: Model downloads and heavy API traffic need high bandwidth; the 600 Gbit/s total upstream capacity of the LIONFIN infrastructure leaves network headroom for downloading large model files.

Compare the GPU memory, CPU, system memory and disk values of the plans together in the package table on this page; do not decide by looking at the GPU model alone.

How do you prepare an LLM server?

The basic order for preparing an LLM server is: verify the NVIDIA driver and CUDA, install the inference software, download the model and expose the API securely. On LIONFIN GPU servers, plans marked with a CUDA-ready image in the package table come with the driver installed; if you install your own image, install the NVIDIA driver and CUDA first.

1. Verify the GPU: The nvidia-smi command should show the GPU, the driver version and free memory.
2. Container layer (optional): If you install Docker and the NVIDIA Container Toolkit, you can run models inside containers with the --gpus all parameter.
3. Install the inference software: Ollama for a quick start, vLLM for a production API.
4. Choose and download the model: ollama pull in Ollama, vllm serve with the Hugging Face model name in vLLM. Some models require accepting a licence before download.
5. Protect the API: Make the software listen only on 127.0.0.1, put an authenticating Nginx reverse proxy in front of it and enable HTTPS.
6. Add an interface: For team use, you can install a chat interface such as Open WebUI on the same server.

Defining the system as a systemd service or with Docker Compose ensures the model comes back up automatically when the server restarts. If you prefer containers, see Docker hosting.

How do you build a RAG system with your own documents?

RAG (Retrieval-Augmented Generation) is the method of finding relevant passages in your own documents and adding them to the prompt before asking the model, so it answers with your company knowledge without retraining. In most enterprise LLM projects, RAG is the approach to try before fine-tuning.

A RAG system has four parts:

1. Document processing: PDF, Word and web pages are converted to text and split into meaningful chunks.
2. Embedding: Each chunk is turned into a numeric vector with a small embedding model. Embedding models are much smaller than language models and can run alongside them on the same GPU.
3. Vector database: The vectors are stored in PostgreSQL’s pgvector extension or a vector database such as Qdrant.
4. Response generation: The chunks closest to the user’s question are found and given to the language model as context.

RAG quality is often decided not by the model but by chunking and retrieval quality: chunks that are too large fill the context, chunks that are too small break up meaning. If you work with documents in several languages, make sure the embedding model is multilingual. Because RAG increases context length and therefore KV cache needs, size VRAM for long prompts. Remember to include the vector database’s persistent data in your regular backup plan.

How do you ensure security and data privacy on an LLM server?

The most common security mistake on an LLM server is leaving the inference API open to the internet without authentication. Ollama’s API has no authentication by default and listens only on the local address; once it is opened to all interfaces with OLLAMA_HOST, anyone who finds your server can use your GPU for free, delete your models or download them.

• Local listening: Ollama, vLLM and llama-server should listen only on 127.0.0.1; expose only an authenticated reverse proxy over HTTPS.
• API key: Use vLLM’s --api-key parameter or validate keys at the proxy layer.
• Firewall: Keep ports 11434, 8000 and 8080 closed externally; leave only SSH and 443 open.
• Logs: Prompt and response logs may contain personal data; decide how long they are kept and who can access them.
• Prompt injection: In systems that use RAG and tools, limit the model’s permissions against instructions coming from document content; the model should not have direct rights to write to databases or send email.

With a model running on your own server, the data you send is not passed to a third-party AI service; this simplifies assessments around issues such as cross-border transfers of personal data.

Your own LLM server, an AI API or cloud GPU?

Your own LLM server makes more sense when you have continuous, predictable usage, data privacy requirements and a need to customise open models; commercial AI APIs make more sense at low, irregular volume or when you need the most powerful closed models. Hourly cloud GPUs suit short training runs and experiments.

To decide, estimate your monthly token volume and test with a small pilot whether an open model can do the same job. If quality is sufficient in the pilot and volume is sustained, a fixed-cost server often becomes the better deal. The dedicated GPU server or cloud GPU page compares the two in detail. For other GPU workloads such as image and video processing, see GPU render server.

What are the most common problems on an LLM server?

The most common problems on LLM servers are insufficient VRAM, the model running on the CPU instead of the GPU, driver and CUDA version mismatches, and a slow first request. The first diagnostic step is always to check the nvidia-smi output: it shows whether GPU memory is full and which process is using the GPU.

• “CUDA out of memory”: The model, context length and concurrent requests do not fit in VRAM. Choose a lower-bit version, lower --max-model-len in vLLM or stop other processes running on the GPU.
• Very slow responses, idle GPU: The model is running on the CPU; the driver is not loaded, the container was started without GPU access, or layers were moved to the CPU because the model does not fit in VRAM. In Ollama, the ollama ps output shows whether the model is running on GPU or CPU.
• Driver / CUDA mismatch: If the inference software was built against a newer CUDA version than the driver supports, the software does not see the GPU.
• The first request is slow: The model is being loaded from disk into memory. In Ollama you can extend how long the model stays in memory with OLLAMA_KEEP_ALIVE.
• Poor answer quality in a given language: The model you chose may be weak in that language; compare models with strong multilingual training and higher-bit versions.

If you suspect a hardware issue, attaching the nvidia-smi output when opening a support ticket from the customer panel speeds up diagnosis.