How to self-host an LLM on a private GPU server

Running a language model on hardware you rent is the only way to be certain your prompts are not logged, retained, or read by anyone but you. Every hosted API — however good its privacy policy — receives every token you send, and that policy can change without your involvement. Self-hosting moves the boundary: the weights sit on a disk you control, inference happens in VRAM you rented by the hour, and nothing leaves the machine unless you send it somewhere. The software side has become genuinely easy. What still catches people out is sizing the card before ordering it, and remembering that an open inference port is an open inference port.

What self-hosting actually buys you — and what it does not

Be precise about the benefit, because "private AI" is sold loosely. A model on your own GPU changes some things completely and other things not at all.

• Your prompts and outputs stay on the machine — no provider-side logging, no retention window, no dataset built from what you asked.
• No account, no per-key rate limit, and no policy layer between you and the weights. The model you deployed is the model you get.
• Predictable cost at volume. Past a few million tokens a day a rented card is cheaper than per-token pricing, and the price does not move when a provider revises its list.
• It does not make a weak model strong. An open-weights model on your own hardware is still that model — self-hosting buys control, not frontier capability.
• It does not hide anything from the host. Anyone with physical access to a running machine can in principle read its memory, which is a question about the host, not about the model.

VRAM is the whole game: sizing a model to a card

Almost every failed first attempt is a VRAM arithmetic error. Weights, KV cache and working room all have to fit in the card at once, and there is no graceful degradation — you get an out-of-memory error a few seconds into loading. The rule of thumb is workable: at 16-bit precision a model needs roughly 2 GB of VRAM per billion parameters, about 1 GB at 8-bit, and roughly 0.6 GB at 4-bit. Then add the KV cache, which grows with context length and with concurrent requests, and which is what actually bites you in production rather than in testing.

• A 7B to 8B model: about 16 GB at FP16, or comfortably under 8 GB at 4-bit. This is the class that fits an RTX A4000 16 GB with room to spare.
• A 13B to 14B model: roughly 28 GB at FP16, around 9 GB at 4-bit. An RTX 4090 24 GB runs it quantised with plenty of context left over.
• A 30B to 34B model: around 68 GB at FP16, close to 20 GB at 4-bit — the natural home for an RTX 5090 32 GB.
• A 70B model: about 140 GB at FP16, roughly 40 GB at 4-bit. That means an A100 or H100 80 GB, and at full precision it means more than one card.
• Long context multiplies the KV cache, not the weights. A 128k-token window on a large model can want tens of gigabytes on its own — budget for it before you order, not after.

Picking the GPU: from an A4000 to an H100, and when hourly beats monthly

Every plan on the GPU server range is a whole physical card with all of its VRAM — no MIG partitioning, no time-sharing — so the number on the spec sheet is the number you can actually fill. Match the card to the model you intend to serve, not to the model you might try one day.

• RTX A4000 16 GB at $89/mo — 8 vCPU and 64 GB RAM. Enough for a quantised 7B to 13B assistant, an embeddings service, or a classification pipeline that runs all day.
• RTX 4090 24 GB at $189/mo — 16 vCPU, 128 GB and 2 TB NVMe. The best value per token for 13B to 34B models at 4-bit, and the usual choice for a small production endpoint.
• RTX 5090 32 GB at $279/mo — 24 vCPU and 192 GB. Those extra 8 GB are often the difference between a 34B model that fits with real context and one that does not.
• A100 80 GB at $690/mo — 32 vCPU, 256 GB and 4 TB NVMe. HBM2e bandwidth and enough memory for a 70B at 4-bit with a long window, or for heavily batched serving.
• H100 80 GB at $1190/mo — 48 vCPU, 384 GB, 8 TB NVMe and a 10 Gbps port. FP8 support and HBM3 bandwidth make it the only sensible choice for serious throughput or for training.

Billing is hourly or monthly from the same prepaid balance, and that changes the arithmetic more than people expect. An evaluation that takes two afternoons on an H100 costs the hours it ran, not a month. A chat endpoint that has to answer at three in the morning wants a monthly box. If you need several cards in one machine, or a GPU sitting beside a lot of local storage, a dedicated server is the better shape — and the general case for paying in crypto is covered in renting a GPU server with crypto.

Step by step: from a crypto top-up to a running endpoint

1. 01Create an account with a burner emailAn email and a password. No name, phone or ID, so nothing on our side ties the model you run to you.
2. 02Top up your balance in cryptoFund a prepaid balance with Bitcoin, Monero or any of 8 coins. It never expires and is never frozen.
3. 03Deploy a GPU instance on a CUDA imagePick the card, a region and a CUDA-ready image with recent drivers and PyTorch. The box is ready in minutes.
4. 04Confirm the card is visible before you pull anythingA single nvidia-smi call reports the GPU, the driver version and the free VRAM. It is a ten-second check that saves an hour of confusion.
5. 05Pull the weights onto local NVMeFetch the model into the instance's own disk rather than a network mount. Weights run to tens of gigabytes and you will load them more than once.
6. 06Start the server bound to localhostLaunch vLLM or Ollama on 127.0.0.1, confirm one completion locally, and only then decide how you will reach it from outside.

vLLM, Ollama or llama.cpp: choosing the serving stack

Three stacks cover almost every case, and the right one depends on how many requests the endpoint will see rather than on which is technically most impressive.

• Ollama — the shortest path from nothing to a working endpoint. One command pulls a quantised model and serves it behind an OpenAI-compatible API. Ideal for one user, a prototype, or a private assistant.
• vLLM — the production answer. Continuous batching and paged attention let a single card serve many concurrent requests at several times the throughput of a naive loop. That is the difference between an endpoint for you and an endpoint for an application.
• llama.cpp — the pragmatist. GGUF quantisations down to 4-bit and below, CPU offload for layers that will not fit, and the lowest memory floor of the three. It is what makes an oversized model run at all on an undersized card.
• TGI, SGLang and TensorRT-LLM are faster still in specific shapes. Reach for them once you have measured a bottleneck, not before.

All three expose an OpenAI-compatible chat-completions route, so application code written against a commercial API usually needs one base-URL change and nothing else. That compatibility is the practical reason self-hosting has stopped being a project and become a configuration choice.

Keeping the endpoint private: never expose the inference port

An inference server with no authentication is the modern equivalent of an open database. Scanners find new IPs within hours, and an exposed endpoint means somebody else is spending your GPU hours — or reading whatever your application sends through it.

If the endpoint genuinely has to be public — a product rather than a private assistant — terminate TLS at a reverse proxy, require a bearer token you rotate, and rate-limit per key. Putting that proxy on a separate small VPS and leaving the GPU box reachable only through the tunnel is the cleaner arrangement. It is the same reasoning as hosting a website without KYC: the machine that faces the internet should hold as little as possible.

Fine-tuning: when an afternoon on a rented card beats a monthly bill

Fine-tuning is the strongest argument for renting rather than subscribing. A LoRA or QLoRA run on a 7B to 13B model — a few thousand examples, a handful of epochs — finishes in hours on a 4090 and costs exactly the hours it ran. The adapter that comes out is a few hundred megabytes, so you can keep the checkpoint, destroy the instance, and later load that adapter onto a smaller card for serving. Full fine-tunes of a 70B model are a different budget and want an A100 or H100 with room for optimiser state, but they are still measured in days rather than in a contract.

The workflow hourly billing enables is simple: deploy, train, copy the adapter off the box, destroy it. You pay for the compute you used and there is no idle card in between. Keep the dataset on the instance's NVMe while you work and pull it back out when you are finished — nothing you leave on a rented machine should ever be the only copy.

The honest privacy trade-off of a GPU you do not own

Self-hosting removes the largest and most concrete leak: a third party that receives every prompt, keeps it for some retention window, and decides on its own what to do with it. On a rented card the model and your data live inside a machine that answers only to your key. What renting cannot remove is the host itself. Anyone with physical access to a running server can in principle read its memory, and full-disk encryption on a remote machine protects you against a stolen drive, not a live one.

So the useful question is not "can the host see it" but "does the host know who I am". No-KYC hosting answers that one: an email and a password, no verification, nothing on file to leak or to subpoena. Paying from a crypto balance removes the bank. How far you take the payment side is a threat-model choice — Bitcoin is pseudonymous, not anonymous, so coins traced back to a verified exchange still lead somewhere, while topping up in Monero closes that gap at the protocol level.

Mistakes that quietly waste GPU hours

• Sizing on the weights alone and forgetting the KV cache, so the model loads happily and then dies on the first long request.
• Serving at FP16 on a card that would run the same model at 4-bit with three times the context, out of a quality worry nobody actually measured.
• Downloading forty gigabytes of weights onto a network mount, then wondering why every restart takes twenty minutes.
• Leaving the server bound to 0.0.0.0 "just to test from my laptop", which on a public IP is found the same day.
• Paying monthly for a card that sits idle twenty hours a day, when the workload was always a batch job.
• Benchmarking with one request at a time, concluding vLLM is not worth the trouble, and missing that its whole advantage appears under concurrency.

None of these are exotic failures. They are what happens when the card is chosen before the model is. Decide the model and the quantisation first, do the VRAM arithmetic honestly, keep the port closed, and a self-hosted LLM is one of the least demanding things you can leave running.