Music Lab replicationnon-self-correcting cascadesevaluation contamination
Following the design of the classic Music Lab experiment, the authors tested large populations of LLM agents recommending unfamiliar songs, with and without exposure to cumulative recommendation counts. Exposure produced runaway cascades: extreme, unpredictable collective recommendations that diverged increasingly from a song's intrinsic appeal.
Compared with human populations, LLM populations were far more susceptible to herding, and β unlike humans β their cascades were less able to self-correct. The authors flag consequences for recommender systems, automated evaluation, complex problem-solving, and unauthorised conspiracies among agents.
Why read it A sharp experimental result with a wide blast radius: any multi-agent setup where agents observe each other's outputs is a feedback loop, and these loops do not self-correct the way human crowds do.
unified quantization representationcomposable CUDA kernels1.8 s JIT compile
Ant Group and the vLLM community open-sourced Humming, a general-purpose kernel library for quantized inference. It separates the quantization description from the execution choice β a unified representation of weight format, scaling granularity, scale type, zero points and secondary scales β then compiles a kernel from shared CUDA components for the target device and workload.
It supports NVIDIA architectures from Turing onward plus CUDA-compatible devices such as T-Head's Zhenwu 810E, covering WNA16, W8A8, W4A8 and W4A4 across integer and floating-point formats. Reported results: roughly 3β4x over cuBLAS at small batch for dense W4A16, about 2x vLLM Marlin at large-batch H200, and end-to-end throughput gains of 14β43% on DeepSeek-V4.1-Flash on eight H20s. Kernel compile time drops from 23.35 s with CUTLASS to about 1.8 s.
Why read it The combinatorial explosion of quantization formats was becoming a maintenance tax on every inference stack. Collapsing it into composable components with fast JIT is the right structural answer β and the Blackwell hidden-format discovery is a nice bonus.
speculative decoding hurts on AppleKV pruning 1.75xMetal lacks async copy
Nearly all LLM inference optimizations are designed and validated on server-class NVIDIA GPUs under batched serving. This paper evaluates four families β key-value (KV) cache pruning, speculative decoding, ported GPU kernels and post-training quantization β across four Apple machines under a single-request decode workload, with matched RTX 4090 comparisons where possible.
The results are a corrective: KV-cache pruning reaches up to 1.75x end-to-end throughput on Apple but can reduce throughput on desktop NVIDIA GPUs; every evaluated speculative decoding method reduces throughput on Apple despite reported 2β6x gains on server GPUs; CUDA kernel designs transfer only partially because public Metal lacks asynchronous copy and batch-one decode makes serving-oriented mechanisms inapplicable, though transferable components yield up to 1.20x. Quantization gains vary by machine.
Why read it A published speedup is not a portable property of an algorithm. If you run models locally on a Mac, this is the paper that tells you which of the standard tricks to skip.
prefix cache split68% cost reductioneffort-level A/B testing
Cloudflare's AI code reviewer went from $1.19 per review in April to about $2.46 by August without architectural change, because models got better and engineers started producing larger changes. An optimisation push brought the average to $0.77 β a 68% cut β and the median review from 4m 09s to 1m 32s.
The single biggest saving came from inspecting what actually reached the provider on each session's first request: the shared 11kβ21k-token prefix was only a cache hit 14β26% of the time, because the system prompt builder placed the sole cache breakpoint after a repository-specific environment block. Splitting the block so the shared prefix got its own breakpoint fixed it. About 40% of savings came from code changes; the rest from model and effort-mix experiments.
Why read it The best kind of engineering write-up: a diagnosis you can copy. Two checks it recommends for anyone running agents at volume β does each session's first request actually read from the cache, and does your reviewer still find real problems at the effort level you pay for.
90% capability at 10-25% coston-prem in two or three rackssystems-talent gap
On IBM's third-quarter earnings call, chief executive Arvind Krishna traced the closing gap between closed and open models: in 2024 a broad benchmark might have scored the top closed model at 70 and an open model at 20 or 30; by late 2025 it was 70 to 50. Open-weight models now deliver, in his words, anywhere from 25% to 10% of the token cost of frontier models while being about 90% as effective.
He framed the enterprise decision plainly β pay ten times more for the last 10%, or not β and said most tasks (summarising an email, examining a contract, answering a customer call) do not need the premium. He also named the real obstacle to on-premise deployment: not hardware, but finding the kernel, networking and caching engineers to make a cluster perform.
Why read it A vendor chief executive putting numbers on the open-versus-closed trade-off in public, on an earnings call where the numbers carry legal weight. The closing observation β that the scarce resource is systems talent, not silicon β is the part worth remembering.
correct abstention rateiterative RAG hurts abstentionreranking cost
Reliable retrieval-augmented generation (RAG) requires recognising when neither retrieved documents nor parametric knowledge supports an answer. DeclineCraft is a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure and later cut-offs for future models.
Across 27 LLMs, single-shot retrieval significantly raised the correct abstention rate in 38 of 40 configurations versus no retrieval. But iterative RAG β where the model self-assesses its context and can request up to two more retrieval rounds β significantly lowered it in 30 of 40 configurations (mean 46.4%), with little change in answer accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid retrieval lowered abstention for all 12 models tested.
Why read it A counterintuitive, well-measured result that cuts against the usual advice: more retrieval machinery can make a system more confidently wrong. Worth checking against your own pipeline before adding another retrieval round.
GPU prefill / NPU decodeRDMA KV transfertensor layout translation
Prefill is compute-intensive and decode is memory-bound, so accelerators are unevenly suited to the two phases: graphics processing units (GPUs) handle prefill well but sit underused during decode, while neural processing units (NPUs) use their compute more effectively for decode. HAWK explores a heterogeneous architecture that maps prefill to GPUs and decode to NPUs.
The obstacle is that phase disaggregation requires moving the key-value (KV) cache from prefill to decode across devices with different communication mechanisms and tensor layouts. HAWK establishes a remote direct memory access (RDMA) path for direct cross-device transfer and handles the layout discrepancies as part of the transfer.
flowchart LR
R[Request] --> P[Prefill on GPU]
P --> KV[KV cache]
KV -->|RDMA + layout\nconversion| D[Decode on NPU]
D --> O[Tokens]
Why read it A concrete answer to a question that follows from earlier KV-cache work: once the cache is the thing you move, which hardware should each phase run on? Heterogeneous prefill/decode is a plausible shape for the next generation of serving clusters.