The Feed Got Fast. Then It Got Noisy.
In February I wrote Splitting the Pipeline. Our AI feed updated three times a day and felt dead between runs. We split it into a fast Tier-1 snapshot every 30 minutes and a slower Tier-0 ranking pass, then blended both in the API so fresh stories showed up at the top right away.
For a few months, that solved the problem I was looking at.
By October I was opening my own site in the morning and skimming past half of it. One day the feed showed seven separate cards for the same DeepSeek V4.1-Flash release, all from a single Google News query, each with a slightly different headline and publisher. Anthropic’s “2026 Usage Policy update” sat in the feed for days while the engineering posts barely showed up. Another slot went to a search result whose body was mostly a photo caption.
I pulled the last 201 builds across eight days to see whether I was imagining it. There were 4,824 slots but only 382 unique URLs. 87% of items repeated from the previous build, and 21% were older than 48 hours. Anthropic’s usage-policy post appeared 88 times across those eight days, while its engineering blog got 13 slots, all for the same article.
Two tiers meant two different feeds
The February architecture looked clean on a diagram because each box had a clear job.
Figure 1. Tier-1 fed the API directly and also fed Tier-0. What landed on the page was the union of two filters.
In practice, the two tiers had different definitions of a good story. Tier-1 scored items on freshness, source weight, and a few keyword checks so new links could reach the top within minutes. Tier-0 ran the full ranking hours later. Because the API blended both snapshots whenever a reader loaded the page, a fresh search hit could jump to the top before the full ranking ever saw it.
Search feeds made that gap worse. I had added them to catch model launches early, which put raw Google News queries in the same pool as hand-picked engineering blogs. Company blogs carried a high trust weight, so a policy announcement from a frontier lab could hold a top slot for days. And the ranker started from scratch on every run, with no memory of how many times it had already shown me the same link that week.
Rules cleared the repeats, not the fluff
Before touching a model, I put plain rules in front of the ranker.
Every source got a role: official lab blog, practitioner, aggregator, community, search, release feed, or research. I split official lab blogs into engineering posts on one side and newsroom updates on the other, and capped the newsroom side at one slot. The minimum Anthropic slot now counts only engineering posts. Search results and company marketing posts went into quarantine unless the title showed a concrete release or technical signal. Coverage of the same event got merged into one story, and a story that had already appeared in recent editions took a growing penalty.
That stopped a single launch from taking seven cards and cleared out the obvious repeats, but rules still cannot read an article. A regex can catch a “Claude Code vs Codex” comparison post or an IPO rumor. It has no way to tell a real search scoop from an SEO rewrite, or an agent evaluation paper from a clinical benchmark that happens to have “eval” in the title. When a lab ships something that changes how agents talk to each other, a keyword check cannot separate that post from a routine customer story.
Tuning source weights and blocklists any further would only move those mistakes around. Something in the pipeline had to read the excerpt and decide whether it was worth a platform engineer’s time.
Benchmarking a judge on 50 borderline items
Before wiring a model into production, I built a 50-item benchmark from the eight kinds of content that kept causing trouble: company engineering posts, company marketing, practitioner analysis, SEO-style search results, business gossip from search, off-topic science, research papers, and release notes. Each item has a title, a source, a 2,500-character excerpt, a keep or drop label, and a finer grade from “must read” down to “junk”.
I loaded the sample with the cases that had fooled my rules. The keep side included a company announcement that changes agent interoperability, a genuine search scoop, and a launch-day serving post. The drop side included a funding story written like a technical post, a clinical question-answering benchmark, and a tool comparison post. I labeled them with help from my coding agent, so the labels are provisional. Fifty items will not prove a judge is good, though it is plenty to catch a broken one.
For the judge I used Jev, a decision model from TypeSafe called through OpenRouter. Jev takes a list of typed questions and returns a probability for each yes/no question (or a distribution over choices), so I get numbers back directly without parsing prose.
I compared three setups on the same 50 items.
| Metric | Four questions + category | One question | Rule-based baseline |
|---|---|---|---|
| AUC | 0.969 | 0.965 | 0.871 |
| F1 at best threshold | 0.927 | 0.878 | 0.829 |
| Precision | 0.950 | 0.900 | 0.850 |
| Recall | 0.905 | 0.857 | 0.809 |
All 50 requests cost $0.0027 and took about 440ms each.
One question hid four judgments
Asking only “does this belong in the daily brief?” already beat the rules by about five F1 points. The mistakes it made were specific:
- It kept a paper on a low-resource language because the method was rigorous, even though the topic is off-mission for platform engineers.
- It kept a funding story because the text quoted a real usage number.
- It dropped “OpenAI models are coming to Google Cloud” with a score of 0.12, because the wording read like a routine partnership press release rather than a change in where models can be deployed.
One number was standing in for four separate judgments. So I split the question: does it belong in the brief, is it about building or operating AI systems, is it a routine release note, and is it corporate news with no concrete change. A fifth, multiple-choice question assigns a topic for the editors downstream.
Multiplying the four probabilities together made things worse and dropped F1 to 0.810. Keeping the overall “belongs in the brief” score as the primary signal and using the other three as caps worked much better: a high release-note or marketing score, or a low platform-relevance score, pulls the main score down to the floor. That brought F1 to 0.927.
A hard filter emptied the feed
With 0.95 precision, the obvious move was a hard cutoff. A cheap first pass narrows the day’s items to about 140 candidates, and anything the judge marks as drop gets thrown out.
When I wired that in and ran a full build, the feed came out worse. The filter removed about 60% of the candidates, and narrow categories like agent tooling releases dropped to zero. Because the ranker still had category quotas and an edition size to fill, it reached back into the rejected pile and took the least-bad leftovers. Duplicates went down, but filler went up.
Scoring one article at a time tells you nothing about the rest of the day’s pool, whether there are three good engineering posts waiting or thirty, or how many release notes the page already has. That balance belongs in the ranker.
So I turned the hard cutoff off and fed the judge’s score into the ranking as one weight among several. High scores get a bounded boost, while low scores take a penalty and cannot fill optional slots. On a quiet day the edition stops early instead of scraping the bottom of the pool.
One builder replaced two tiers
At a fraction of a cent per run, the judge made the fast unjudged tier pointless. I deleted the Tier-1 build and its 213 MB of snapshots, and the API now reads only the finished edition.
Figure 2. Rules clean up first. The judge reads about 140 candidates, reuses earlier verdicts, and hands a score to the ranker.
Every two hours the pipeline now runs a single path: collect, apply the editorial rules, narrow to about 140 candidates, judge them, rank, and publish. Getting that loop to hold up in production took three smaller fixes.
Fetch the body when the teaser is thin
My first live test gave a strong Cloudflare post about inference costs a low score. The excerpt I had sent was mostly navigation menu text. The benchmark had 2,500 characters of real article body, while production feeds often give a one-line teaser. Now, when the teaser is too short, the pipeline fetches up to 2,500 characters of the actual page before calling the judge.
Commit past verdicts to the repo
Temporary caches do not survive between runs on a fresh build machine, so I commit the judge’s verdicts alongside the data. Each verdict is keyed by the item, the model, the question wording, and the prompt, and kept for 14 days. If any of those change, the item is judged again.
The first cold run judged 140 candidates (93 keep, 31 drop, 16 review) and produced a 22-item edition. The next run reused 138 of them and made only 2 new calls.
Keep release notes in their own lane
Routine release notes are exempt from the judge’s penalty. A version bump can stay out of the main brief and still show up in the releases view.
Editor decks read the same verdicts
The daily recap, weekly recap, Playbook, wiki, and Foundations pages are written by editor agents. They now see the judge’s verdict and topic for every article, and they skip articles the judge already dropped.
At first I planned to hand them the 140 judged candidates directly, until I looked at what was inside that pool. It covers only a single two-hour window, has no article text, and is heavily skewed toward papers (in one run, 100 of the 140 candidates came from arXiv). So the decks keep reading the finished editions with the verdicts attached. The judge’s topic replaces the old catch-all “Industry News” bucket, and the wiki and Foundations skip routine release notes.
If the judge is unavailable or an item was never judged, both the feed and the decks continue without it.
Where this leaves the feed
The February split solved the problem I was measuring at the time, which was latency. Eight months later, the problem was that two paths meant two standards. Running everything through one builder and one judge means every card on the front page clears the same bar, and on a quiet news day the feed can stop at 18 items instead of padding itself to 24.