DATASETTE’S LLM STACK ADDS ASYNC WRAPPERS, PURPOSE-BASED ROUTING, AND RICHER USAGE LOGGING
A coordinated set of releases tightens how Datasette and the LLM toolchain handle async models, model routing, and usage logging. The LLM CLI gained a smarter ...
A coordinated set of releases tightens how Datasette and the LLM toolchain handle async models, model routing, and usage logging.
The LLM CLI gained a smarter plugin hook in llm 0.30, enabling plugins to see all registered models. That unblocks llm-all-models-async 0.1, which wraps sync-only models in async via a thread pool so async-only consumers, like Datasette, can use them.
datasette-llm 0.1a4 adds purpose-based API keys and routing, while 0.1a5 tracks prompts across chains and tool-call loops. Debugging improved with llm-echo 0.3 for key and tool-call tests and 0.4 exposing input/output token counts.
Downstream, datasette-enrichments-llm 0.2a0 and datasette-extract 0.3a0 now use datasette-llm and can whitelist models for the "enrichments" purpose. datasette-llm-usage 0.2a0 logs prompts, responses, and tool calls to SQLite and shifts pricing/allowances to a separate accountant plugin.
Async wrappers and purpose-based routing reduce integration friction and let you govern which models run which jobs.
Deeper logging makes cost, risk, and reliability visible across prompt chains and tool calls.
-
terminal
Wrap a sync-only model with llm-all-models-async, point Datasette at it, and validate concurrency and latency under load.
-
terminal
Enable datasette-llm-usage.log_prompts, run a tool-call chain, and verify prompt, response, and token logs using llm-echo.
Legacy codebase integration strategies...
- 01.
Migrate existing model configs to datasette-llm with purpose-scoped API keys; audit secrets and env separation per purpose.
- 02.
Decide retention for prompt logs and scrub PII before enabling in production; move any pricing/allowance logic to the accountant plugin.
Fresh architecture paradigms...
- 01.
Adopt datasette-llm as the model hub from day one with purpose-based routing and usage logging baked in.
- 02.
Prefer async models; if unavailable, wrap sync ones and design enrich/extract pipelines with an approved-models list.