Stories

Completed digest stories linked to this service.

• Local LLM serving on 24GB GPUs: vLLM scales, llama.cpp/Ollama survive spills

  2026-07-05

  A new benchmark shows vLLM crushes throughput on a 24GB GPU but hard-OOMs once models spill to RAM, while llam...
• One-command vLLM server on Hugging Face Jobs (OpenAI-compatible, pay-per-second)

  2026-06-26

  Hugging Face Jobs now lets you launch a private, OpenAI-compatible vLLM endpoint with a single command, no ser...
• kube-llmops brings one-chart, cloud-agnostic LLM serving to any Kubernetes clust...

  2026-06-09

  An open-source project, kube-llmops, packages end-to-end LLM serving and ops for any Kubernetes cluster in a s...
• Local LLM agents are crossing the usability gap — if you own the infra

  2026-05-29

  Open‑weight models hosted with vLLM can run real agentic workloads — but only if you add explicit state, prove...
• OpenAI ships Apache-licensed gpt-oss models you can run off-API, unlocking real ...

  2026-05-13

  OpenAI released Apache-licensed gpt-oss models you can run on your own hardware instead of through the OpenAI ...
• MCP is turning into the observability and control plane for AI agents — but it s...

  2026-04-16

  AI agents are pushing observability and APIs toward MCP-driven, kernel-level telemetry while exposing fresh se...
• KV-cache compression upends LLM serving economics: 6x memory cut, no retrain

  2026-04-12

  Google’s TurboQuant claims 6x KV‑cache compression for LLM inference with no retraining, turning memory‑bound ...
• Agentic coding grows up: open‑weights MiniMax M2.7 meets Grok’s tool‑calling wor...

  2026-04-12

  Open-weights MiniMax M2.7 and xAI’s tool-calling Grok push agentic coding from demos to production workflows. ...
• LLMOps Part 14: Practical LLM Serving and vLLM in Production

  2026-03-29

  A new LLMOps chapter explains how to serve models in production and walks through practical trade-offs, includ...
• The practical playbook for faster, cheaper LLM inference: vLLM, KV caches, and d...

  2026-03-22

  A hands-on deep dive shows how to speed up and scale LLM inference with vLLM, KV caching, and modern attention...

Resources

Links to check for updates: homepage, feed, or git repo.