Feed Collection
Last updated
This page explains how Collector manages feeds and turns RSS into the news list.
Feed Management
| Method | Command | Behavior |
|---|---|---|
NewCollector(db) |
— | Creates a collector with a 30-second HTTP timeout |
Add(link) |
add |
Returns on an empty string; otherwise InsertFeed(), re-enabling an existing URL |
Remove(link) |
rm/remove |
RemoveFeed(), setting dismiss to 1 |
List() |
config |
Returns active feeds ordered by creation time |
Fetch and Parse
GetNews() processes each active feed:
| Step | Behavior |
|---|---|
| Fetch | HTTP GET, then decode into model.RSS with encoding/xml; on failure, log and skip the feed |
| Source name | <channel><title>; falls back to the feed URL when empty |
| Deduplication | Keyed on the trimmed <link>, across all feeds |
| Time window | Items published more than 72 hours ago are skipped |
| Cleanup | Strip HTML tags from title and description and unescape HTML entities |
| Sorting | Newest first |
Only the RSS 2.0 <rss><channel><item> structure is parsed; Atom feeds are not supported.
Date Parsing
pubDate is tried against these layouts in order and converted to UTC:
| Layout |
|---|
Mon,02 Jan 2006 15:04:05 -0700 |
Mon, 02 Jan 2006 15:04:05 -0700 |
Mon, 2 Jan 2006 15:04:05 -0700 |
2006-01-02T15:04:05Z07:00 |
2006-01-02 15:04:05 |
time.RFC1123Z, time.RFC1123, time.RFC3339 |
An empty or unparseable pubDate uses the current time; the Refresh Cycle restores the stored time once the article is in the database.
Data Model
| Type | Fields |
|---|---|
model.RSS |
Channel |
model.Channel |
Title, Description, Item []Item |
model.Item |
Title, Description, Link, PubDate |
model.News |
Title, Content (RSS description), Source, URL, PublishedAt, FullContent, Author, WordCount (the last three are pointers, nil until extracted) |