LLMs - in Review (2024)
LLMs - in Review (2024)
December 2024
2024 was the first year where I took LLMs seriously. I successfully hosted a
Llama 70b parameter model in production which was used as with
continue.dev for a self-hosted co-pilot
replacement, along with a code autocomplete like
Qwen Coder or
Deepseek, these were fine replacements and
surprisingly robust.
Huggingface’s TGI
along with Triton Server
were the main heroes for this project, (Triton was used to serve onnx models
for embeddings) though I’ve yet to find a “good” embedding model. At this stage
in time, most of the vector database solutions “feel” the same and can all
seemingly be trivially hosted via Kubernetes.
Tools used:
- Huggingface TGI (LLMs)
- Triton Inference Server (embeddings/
onnxmodels)
Optimising usage of LLMs at scale is still a massive challenge, particularly when opening up for general access. The space is still new so there are still lots of patterns especially related to agentic patterns which need to be explored. On that particular note, this particular blog post from Anthropic is something I’m looking at working on in the new year. At the same time, I’ve been experimenting with mlflow for LLM evaluations which has worked reasonably well, though I’ve pretty disappointed at mlflow from a self-hosting perspective, especially if only a tracking server is required. Maybe this is an open source project for the future.