AutomationIntegrations
Tech Radar: pulling signal out of a developer community feed
- What this was
- My own service: it collects messages from a professional community, filters the noise with explainable rules and assembles a digest. Running on my server.
- My role
- Framing, architecture, development, operations — alone
- Period
- 2026, ongoing
- Outcome
- 7,627 messages from 581 authors archived; the service runs unattended.
Context and original problem
A developer community discussing new libraries, tools and API pricing. The value is real but spread across hundreds of messages a day, and reading everything by hand is not an option.
Business task
Miss nothing useful without reading everything. Get a short daily summary instead of a feed, and be able to explain why a message made the cut.
Constraints and unknowns
- One server, 2 CPU / 4 GB, no GPU
- The messages belong to other people: the archive stays private, only links and fragments surface
- It has to run unattended — I am busy with other work
- No paid APIs
My responsibility
Everything: collection via Telethon, database schema, selection rules, digest, the systemd unit, tests and operations.
Key decisions and trade-offs
- The raw archive is untouchable: processing never deletes from items, and every derived verdict lives separately in item_analysis. Rules can be rewritten and replayed over the whole history.
- Deterministic rules first, an LLM only after. Rules are cheap, reproducible and explain themselves with a list of reasons; a model on two cores without a GPU is slow and cannot.
- The LLM sits behind an ItemAnalyzer interface rather than inside the processing code. Adding a local model means another implementation of the same protocol, not a rewrite.
- Incremental collection by cursor: a restart does not re-read history.
- Observability from day one — every collector run is recorded with its status, including abnormal termination.
What changed
- 7,627 messages from 581 authors collected between March and August 2026
- Deduplication by normalised text and by canonicalised links, with tracking parameters stripped
- The digest reports a score, a category and the list of reasons a message was selected
- The service runs under systemd and survives a server reboot with no manual step
- 1,148 lines of my own code, four external dependencies, three test suites
Disclosure and outcome boundaries
Next step for a similar task
A similar task: extract signal from a stream of messages, emails or inbound requests. We would start by defining what counts as signal.