Document behaviour, invariants and architecture

- expand README: post cadence, architecture, config reference, ops,
  troubleshooting, limitations
- add AGENTS.md with hard rules and domain gotchas
- comment the non-obvious parts: window clamping, reminder fallback,
  series vs occurrence, outbox lifecycle, image failure handling
This commit is contained in:
2026-11-18 16:45:00 +00:00
parent bdd41929d4
commit 751fa518e6
10 changed files with 413 additions and 83 deletions
+165 -78
View File
@@ -1,9 +1,9 @@
# calbot — iCalendar → Signal
Watches an iCalendar feed (built for The Events Calendar / MCR Commons) and posts
events to a Signal group, with the event's poster image attached.
Posts look like this:
Polls an iCalendar feed and posts events to a Signal group, with the event's
poster image attached. Built for [MCR Commons](https://commonscalendar.uk) (The
Events Calendar), but works with any feed that exposes `UID`, `SUMMARY`,
`DTSTART`, `DTEND` and optionally `ATTACH`, `LOCATION`, `URL`.
```
Folk Night
@@ -13,46 +13,81 @@ Old Print Works B12 9AH
https://commonscalendar.uk/event/folk-night/
```
## How it behaves
## Behaviour
- **Announcements** — a newly seen event is posted within one refresh interval,
deferred to the next posting window (`08:00–10:00` or `18:00–21:00` by default).
- **Reminders** — every event is posted again 3 days before it starts. If the
event was added less than 3 days ahead, the reminder goes out on the morning
of the event instead. An imminent event breaks the posting window rather than
arriving late.
- **Images** — taken from the feed's `ATTACH` property (The Events Calendar
emits the featured image), falling back to the event page's JSON-LD / `og:image`.
Every post carries an image.
- **Recurring events** — the feed pre-expands occurrences, but announcements are
deduplicated per series, so adding a weekly event produces one announcement.
- **First run** — seeds silently (`announce.backfill = false`); reminders still
work for everything already in the feed.
- **Watchdog** — messages you on Signal if the feed stops working, parses to
nothing, or sending keeps failing.
Each event produces **at most two posts**; every post carries the poster image
when one exists.
## Setup
| Post | When |
|---|---|
| Announcement | As soon as the event is first seen in the feed |
| Reminder | 3 days before it starts |
| Reminder (fallback) | 09:00 on the day, if the event was added under 3 days out |
### 1. Link a Signal account
Posts are deferred to the next preferred window — **08:00–10:00** or
**18:00–21:00** local time — unless waiting would mean posting after the event
starts (or within an hour of it), in which case they go out immediately.
Linking as a secondary device is the quickest route: the bot posts **as your own
number**. For a separate bot identity you must register a dedicated number
instead (see `signal-cli-rest-api` docs).
Everything due in the same window is **batched into one message**. That is also
what merges the "new event" and "today" posts for an event added on the day.
Other rules:
- **First run seeds silently** (`announce.backfill = false`), so enabling the
bot does not dump the whole calendar into the group. Reminders still work for
everything already in the feed.
- **Recurring events are announced once per series**, not once per occurrence.
The feed pre-expands recurrences; announcements are keyed on the series.
- **Events that disappear from the feed** (cancelled) have their pending posts
dropped. Events beyond the feed's coverage are left alone — The Events
Calendar only exports ~30 upcoming events.
- **The watchdog** messages your own number ("Note to Self") if the feed keeps
failing, parses to nothing, or sending fails repeatedly.
## How it works
```
feed (ICS) ──► parse/expand ──► SQLite ──► dispatch ──► signal-cli-rest-api
│ │
reminder_plans + outbox Signal group
```
One tick every 30 s:
1. **Refresh** — conditional GET (ETag/Last-Modified, every `refresh_minutes`).
The feed is parsed and expanded, then reconciled with the database. New
event rows get reminder plans; a new series gets an announcement.
2. **Dispatch** — every outbox row whose time has come is grouped by event,
rendered, and sent as one message.
3. **Heartbeat** — optional once-a-day "alive" note.
Scheduling decisions are persisted, so restarts and feed edits cannot reorder
or duplicate posts. Delivery is at-least-once: a crash between sending and
committing can duplicate exactly one post.
## Quick start
Requirements: Docker (and Docker Compose). A Signal account to link.
### 1. Start signal-cli-rest-api and link a device
```bash
mkdir -p signal-data data cache
docker compose up -d signal-api
```
Open <http://localhost:8080/v1/qrcodelink?device_name=calbot> and scan the QR
code from **Signal → Settings → Linked devices**.
Open <http://localhost:8080/v1/qrcodelink?device_name=calbot> and scan it from
**Signal → Settings → Linked devices**.
> Linking as a secondary device means the bot posts **as your own number**.
> For a separate identity, register a dedicated number instead (see the
> `signal-cli-rest-api` docs on `/v1/register`).
Fetch the linked number and the group ID:
```bash
# the linked number:
curl -s http://localhost:8080/v1/accounts
# groups for that number — copy the internal_id (starts with "group."):
curl -s http://localhost:8080/v1/groups/+447700900000 | jq
curl -s http://localhost:8080/v1/groups/+447700900000 | jq -r '.[].internal_id'
```
### 2. Configure
@@ -60,67 +95,119 @@ curl -s http://localhost:8080/v1/groups/+447700900000 | jq
```bash
cp config.example.toml config.toml
chmod 600 config.toml
$EDITOR config.toml
```
Fill in `signal.number`, `signal.group_id`, and `calendar.url`. `webcal://`
URLs are accepted and fetched over https.
At minimum fill in `signal.number`, `signal.group_id` (including the `group.`
prefix) and `calendar.url`. `webcal://` URLs are accepted and fetched over
https.
### 3. Check, preview, run
### 3. Verify, preview, run
```bash
# is the account linked and the config valid?
docker compose run --rm bot python -m calbot --check
# print the rendered posts for the feed without sending anything
docker compose run --rm bot python -m calbot --dump
# start the bot
docker compose run --rm bot python -m calbot --check # config + account
docker compose run --rm bot python -m calbot --dump # rendered posts, no sending
docker compose up -d bot
docker compose logs -f bot
```
`--once` runs a single tick and exits.
## CLI
## Install by hand
| Command | Purpose |
|---|---|
| `calbot --config config.toml` | Run the loop |
| `calbot --once` | Run a single tick and exit |
| `calbot --dump` | Print rendered posts for the feed; uses a temporary DB, sends nothing |
| `calbot --check` | Validate config and confirm the Signal account is linked |
All commands accept `--log-level` (or `LOG_LEVEL`).
## Configuration
`config.example.toml` is commented and complete. The essentials:
| Key | Default | Meaning |
|---|---|---|
| `signal.api_url` | — | `http://signal-api:8080` inside Compose |
| `signal.number` | — | The linked account's number, international format |
| `signal.group_id` | — | `internal_id` from `/v1/groups/<number>`, starts with `group.` |
| `signal.text_mode` | `normal` | `styled` enables Signal markup |
| `calendar.url` | — | ICS feed (webcal accepted) |
| `calendar.timezone` | `Europe/London` | Used for windows, hashtags and display |
| `calendar.refresh_minutes` | `15` | Feed poll interval |
| `calendar.window_days` | `60` | How far ahead to keep events |
| `posting.windows` | `08:00-10:00`, `18:00-21:00` | Preferred send times |
| `posting.urgent_guard_minutes` | `60` | Break a window when the event is this close |
| `announce.enabled` | `true` | Post when a new series appears |
| `announce.max_days_ahead` | `30` | Ignore announcements further out |
| `announce.backfill` | `false` | Announce everything on first run |
| `reminders.enabled` | `true` | Post before events |
| `reminders.lead_times` | `["3d"]` | Any number of leads (`"3d"`, `"2h"`, `"90m"`) |
| `reminders.fallback_to_day_of` | `true` | Day-of reminder for late additions |
| `reminders.day_of_time` | `09:00` | When that day-of reminder goes out |
| `images.enabled` | `true` | Attach posters |
| `images.max_pixels` | `1600` | Longest edge after resizing |
| `images.max_bytes` | `400000` | JPEG size target |
| `images.max_images_per_post` | `3` | Cap for batched posts |
| `images.scrape_event_page` | `true` | Fetch pages for venue/`og:image` |
| `format.show_*` | all `true` | Toggle each line of the post |
| `format.shorten_links` | `false` | Shorten links via TinyURL |
| `watchdog.enabled` | `true` | Operator alerts |
| `watchdog.notify_number` | own number | Where alerts go |
| `watchdog.feed_failures` | `3` | Consecutive failures before alerting |
| `watchdog.daily_heartbeat` | `false` | Daily "alive" message |
## Operations
| Task | How |
|---|---|
| Follow logs | `docker compose logs -f bot` |
| Apply config changes | edit `config.toml`, then `docker compose restart bot` |
| Update code | `git pull && docker compose up -d --build` |
| Reset duplicate/seen state | stop bot, delete `data/calbot.sqlite3`, start (re-seeds silently) |
| Clear image cache | `rm -rf cache/images` |
| Back up | `signal-data/` (linked-device keys) and `config.toml` |
**Back up `signal-data/`.** It contains the linked-device credentials. Losing it
means re-scanning the QR code; if the phone is re-registered, the link breaks
and must be re-paired.
## Troubleshooting
| Symptom | Check |
|---|---|
| `Signal account … not found` | The QR was not scanned, or `signal.number` is wrong (`curl /v1/accounts`). |
| Nothing posts | `--dump` works? Then check windows and whether the first run only seeded. Logs show planned `send_at` only at debug level — check `outbox` in SQLite if needed. |
| Posts appear as you | Expected with a linked device; use a dedicated number for a bot identity. |
| No venue line | The feed has no `LOCATION` and the event page has no JSON-LD venue. |
| No image | Event has no featured image and no `og:image`, or the download exceeded the 20 MB cap. Text still posts. |
| Duplicate post after a crash | At-least-once delivery; expected and bounded to one. |
| Watchdog alert about an empty feed | The site returned an empty calendar; stored events are left untouched. |
| `404` from the feed | The Events Calendar export URL changed; test it in a browser first. |
## Limitations
- Signal has no bot API; this relies on `signal-cli` and an unofficial REST
wrapper. Account linking can break when the phone is re-registered.
- Signal message text is not styled by default; hashtags are plain text (Signal
does not linkify them).
- The feed is a snapshot (TEC caps it at ~30 events), not a full history. There
is no reliable cancellation signal beyond "it vanished".
- Rescheduled events keep the same post ID, so they get a fresh reminder but no
new announcement.
- No lockfile yet: `Dockerfile` runs `pip install .`, so builds are not
byte-reproducible.
## Development
```bash
python -m venv .venv && . .venv/bin/activate
pip install -e '.[dev]'
calbot --config config.toml --dump
calbot --config config.toml
```
## Files that matter
| Path | Purpose |
|---|---|
| `config.toml` | numbers, group, feed, behaviour (never committed) |
| `signal-data/` | linked-device keys — **back this up**, losing it means re-linking |
| `data/calbot.sqlite3` | event/outbox/delivery state (safe to delete to re-seed) |
| `cache/images/` | downscaled posters |
Deleting `data/calbot.sqlite3` resets dedupe state: on the next run everything
is seeded silently again, so you will not get a flood, but new-series detection
restarts from that moment.
## Notes and limits
- Signal has no bot API; this uses [`signal-cli`](https://github.com/AsamK/signal-cli)
via [`signal-cli-rest-api`](https://github.com/bbernhard/signal-cli-rest-api).
- Delivery is **at-least-once**. A crash between sending and recording can
duplicate one post.
- The feed is capped by The Events Calendar (30 events by default) and may omit
past events; the bot only knows what the feed contains.
- Venue/postcode come from the ICS `LOCATION` when present (this feed often has
none), then from the event page's structured data, then from the description.
## Tests
```bash
pip install -e '.[dev]'
pytest
```
Fixtures are stored as `tests/fixtures/*.ics.txt` so that `*.ics` can stay
gitignored.
Tests are offline. They inject a fake clock, sender, scraper and image service,
and use `httpx.MockTransport` for the feed. Calendar fixtures live in
`tests/fixtures/*.ics.txt` — `.ics` is gitignored so raw feeds can never be
committed.
See `AGENTS.md` for the invariants and gotchas before making changes.