Drop the "You are reading content from Scuttlebutt" banner

Upstream ssb-viewer printed it above every page, aimed at people who had
arrived at a stray scuttlebutt message and needed context. On cust.ooo the
context is the site itself, so it was explaining the wrong thing.

Removes the function, all four call sites and the now-dead .top-tip CSS.

Adds a "Changing how pages look" section to the README, since the real
question was where this lives. There is no template directory - pages are
built as strings in render.js with one <style> block - so the answer is
"grep for the text you can see", and the table maps each visible piece to the
function that emits it. It also records the two things that will bite an
editor: hyperscript silently drops attributes it mistakes for DOM properties
(why loading="lazy" uses setAttribute), and only text passed through h() is
escaped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0192zBTNZKZn5svyJ5HTnYds

.gitignore

@@ -1,2 +1,2 @@
 node_modules/
-package-lock.json
+.deployed-commit

README.md

@@ -1,117 +1,105 @@
-# ssb-viewer
+# custo-viewer
 
-HTTP server for read-only views of SSB content. Serves content as web pages or as scripts for embedding in other web pages.
+The web view behind **[www.cust.ooo](https://www.cust.ooo)** — a fork of
+[ssb-viewer](https://gitlab.com/dwynen/ssb-viewer) that renders custo items from a
+Scuttlebutt pub.
 
-## Install & Run
+Upstream's own docs are kept verbatim in [UPSTREAM-README.md](UPSTREAM-README.md).
 
-As a sbot plugin:
+## What custo adds
-```sh
-mkdir -p ~/.ssb/node_modules
-cd ~/.ssb/node_modules
-git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
-npm install
-sbot plugins.enable ssb-viewer
-# restart sbot
-```
 
-Or standalone:
+| | |
-```sh
+|---|---|
-git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
+| `/items` | One merged grid of every item the pub knows about, across all kiosks. New kiosks appear by being followed — no code change. |
-npm install
+| known-blob gate | `serveBlob` only serves blobs referenced by a feed we replicate, so the node never hands out a stranger's cached blob. |
-./bin.js
+| exit on dead sbot | `bin.js` exits when its muxrpc handle dies, instead of holding the port open and hanging every request forever. |
-```
+| `blob-wanter.js` | Standing `blobs.want` orders for our own feeds' blobs, replacing what `ssb-blobs` `sympathy` used to do before it was turned off. |
 
-## Usage
+## The data model
 
-To view a thread as a web page, navigate to a url like `http://localhost:8807/%MSGID`.
+An **item** is an ordinary `type: "post"` carrying custo's own fields:
 
-To embed a thread into another web page, load it as follows:
-
-```html
-<script src="http://localhost:8807/%MSGID.js"></script>
-```
-
-To add more than the base styles, you can also load `http://localhost:8807/static/nicer.css`.
-
-## Routes
-
-- `/%msgid`: web page showing a message thread
-- `/%msgid.js`: script to embed a message thread
-- `/%msgid.json`: message thread as JSON
-- `/&feedid`: web page showing a complete feed
-- `/user-feed/&feedid`: web page showing messages from followed users and channels of a feed
-- `/channel/#channel`: web page showing messages in a specific channel
-
-### Query options
-
-- `noroot`: don't include the root message in the thread
-- `base=...`: base url for links that ssb-viewer can handle
-- `msg_base=...`: base url for links to messages
-- `feed_base=...`: base url for links to feeds
-- `blob_base=...`: base url for links to blobs
-- `img_base=...`: base url for embedded blobs (images)
-- `emoji_base=...`: base url for emoji images
-
-The `*_base` query options overwrite the defaults set in the config.
-The `base` option is a fallback instead of specifying the URLs separately.
-The base options are mostly useful for embedding, where the script is embedded
-on a different origin than where ssb-viewer is running. However, you may not
-need them, as the ssb-viewer embed script will detect the base where it is
-included from.
-
-## Config
-
-To change `ssb-viewer`'s default options, edit your `~/.ssb/config`, to have
-properties like the following:
 ```json
-{
+{ "type": "post", "custodisco": "true", "nft": "mint",
-  "viewer": {
+  "text": "![photo.jpg](&…sha256)\n\n…\n\n a #custodisco item ",
-    "port": 8807,
+  "mentions": [{ "name": "photo.jpg", "type": "image/jpeg", "link": "&…sha256" }] }
-    "host": "::"
-  }
-}
 ```
-You can also pass these as command-line options to `./bin.js` or `sbot` as,
-e.g. `--viewer.port 8807`.
 
-- `viewer.port`: port for the server to listen on. default: `8807`
+A **transfer** of custody is a reply to that message:
-- `viewer.host`: host address for the server to listen on. default: `::`
-- `viewer.base`: default base url for links that ssb-viewer can handle
-- `viewer.msg_base`: base url for links to ssb messages
-- `viewer.feed_base`: base url for links to ssb feeds
-- `viewer.blob_base`: base url for links to ssb blobs
-- `viewer.img_base`: base url for embedded blobs (images)
-- `viewer.emoji_base`: base url for emoji images
-- `viewer.require_opt_in`: whether to serve content from feeds that have not published a `publicWebHosting` `about` message. default: `true`
-- `viewer.disallowRobots`: whether to direct search engines to not index the site. default: `true`
 
-## References
+```json
+{ "type": "post", "custodisco": "true", "nft": "give",
+  "target": "@…ed25519", "root": "%…sha256", "branch": "%…sha256" }
+```
 
-- Concept: [ssb-porthole][]
+Two things to know before writing a query against this:
-- UI ideas: [sdash][], [patchbay][]
-- Server techniques: [ssb-web-server][], [ssb-ws][], [git-ssb-web][]
 
+- `custodisco` is the **string** `"true"`, not a boolean.
+- **No message ever sets `content.channel`.** The `#custodisco` hashtag exists only in
+  post text. Anything keyed on the channel index will silently return nothing — which is
+  why `/channel/custodisco` renders an empty page.
 
-[ssb-porthole]: %cgkDJXsh6pO5m458B3ngEro+U0qUMGTY1TRGTZOP6lQ=.sha256
+Messages are published by the kiosks (`/home/trav/custodisco-kiosk/ssb-post.sh`), not by
-[patchbay]: %s9mSFATE4RGyJx9wgH22lBrvD4CgUQW4yeguSWWjtqc=.sha256
+this viewer. The viewer is read-only.
-[sdash]: %qrU04j9vfUJKfq1rGZrQ5ihtSfA4ilfY3wLy7xFv0xk=.sha256
-[git-ssb-web]: %q5d5Du+9WkaSdjc8aJPZm+jMrqgo0tmfR+RcX5ZZ6H4=.sha256
-[ssb-web-server]: %gYctTCrA06BhAGGvQ6PJ0H2eCCQLj1iEsmfn8SD5+nk=.sha256
-[ssb-ws]: %tFjo5SoD+Y0SaB5vqZYppmoPmv9LKB5wMPl96qtu4qk=.sha256
 
-## License
+## Changing how pages look
 
-Copyright (c) 2016-2017 Secure Scuttlebutt Consortium
+Everything visual is in **`render.js`**. There is no template directory and no CSS
+files to hunt through — pages are built as strings and there is exactly one `<style>`
+block, near the middle of the file.
 
-This program is free software: you can redistribute it and/or modify
+Find things by searching for the text you can see on the page. `grep -n` on a phrase
-it under the terms of the GNU Affero General Public License as
+from the rendered page lands you on the line that produces it:
-published by the Free Software Foundation, either version 3 of the
-License, or (at your option) any later version.
 
-This program is distributed in the hope that it will be useful,
+```bash
-but WITHOUT ANY WARRANTY; without even the implied warranty of
+grep -n "Join Scuttlebutt" render.js
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+```
-GNU Affero General Public License for more details.
 
-You should have received a copy of the GNU Affero General Public License
+The pieces, in the order you meet them on a page:
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+| What you see | Where |
+|---|---|
+| `<title>` and the whole `<head>` | `wrapPage()` |
+| "custo items", the count, the search box | `itemsHeader()` |
+| an item card: photo, caption, kiosk, time | `renderItemCard()` |
+| "older items" / "show everything" | `itemsFooter()` |
+| the "Join Scuttlebutt now" button | `callToAction()` |
+| licence, repo link, deployed commit | the `footer` constant |
+| all colours, spacing, the grid itself | the `styles` template string |
+
+The grid rules are the `.item-*` selectors in `styles`. Card width is one number —
+`minmax(190px, 1fr)` in `main.item-grid` — raise it for fewer, larger cards.
+
+Two gotchas worth knowing before you edit:
+
+- **Attributes set through `h()` can vanish.** hyperscript assigns anything that looks
+  like a DOM property rather than an attribute, and `outerHTML` then drops it. That is
+  why `loading="lazy"` is applied with `media.setAttribute(...)` instead. If an
+  attribute you added does not appear in the output, this is why.
+- **Text passed to `h()` is escaped; strings concatenated into `wrap()` are not.**
+  Anything derived from a message must go through `h()`.
+
+To see a change: edit, then from the ops repo
+
+```bash
+bin/deploy-viewer.sh --yes
+```
+
+which syncs, restarts the viewer and checks `/items` answers before it returns. Commit
+and push separately — the deploy only moves files.
+
+## Running it
+
+`bin.js` connects to a local `ssb-server` and serves on `conf.viewer.port` (8807).
+See `UPSTREAM-README.md` for the plugin-vs-standalone options.
+
+## Deployment
+
+Deployed by rsync from a laptop, not by `git pull` — the server holds no push
+credentials, and one deploy path is better than two. The script, the systemd units, the
+nginx config, and the disaster-recovery runbook all live in the ops repo alongside this
+one (`documents/custo/ssb-viewer`), which is the source of truth for the server itself.
+
+**Never commit secrets here.** Keys and server config belong in the ops repo.
+
+AGPL-3.0+, inherited from upstream.

UPSTREAM-README.md

@@ -0,0 +1,128 @@
+# ssb-viewer
+
+HTTP server for read-only views of SSB content. Serves content as web pages or as scripts for embedding in other web pages.
+
+## Install & Run
+
+Before you install or upgrade ssb-viewer, make sure the plugin is disabled.
+Otherwise sbot will crash while you install, because it tries to execute half-compiled JS.
+If you're running the install inside the same docker container as sbot, this will kill your build
+and leave you in a broken state where sbot doesn't start anymore.
+
+As a sbot plugin:
+```sh
+mkdir -p ~/.ssb/node_modules
+cd ~/.ssb/node_modules
+
+# for a new installation:
+git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
+
+# for an upgrade:
+cd ssb-viewer && git pull
+
+npm install
+sbot plugins.enable ssb-viewer
+# restart sbot
+```
+
+Or standalone:
+```sh
+git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
+npm install
+./bin.js
+```
+
+## Usage
+
+To view a thread as a web page, navigate to a url like `http://localhost:8807/%MSGID`.
+
+To embed a thread into another web page, load it as follows:
+
+```html
+<script src="http://localhost:8807/%MSGID.js"></script>
+```
+
+To add more than the base styles, you can also load `http://localhost:8807/static/nicer.css`.
+
+## Routes
+
+- `/%msgid`: web page showing a message thread
+- `/%msgid.js`: script to embed a message thread
+- `/%msgid.json`: message thread as JSON
+- `/&feedid`: web page showing a complete feed
+- `/user-feed/&feedid`: web page showing messages from followed users and channels of a feed
+- `/channel/#channel`: web page showing messages in a specific channel
+
+### Query options
+
+- `noroot`: don't include the root message in the thread
+- `base=...`: base url for links that ssb-viewer can handle
+- `msg_base=...`: base url for links to messages
+- `feed_base=...`: base url for links to feeds
+- `blob_base=...`: base url for links to blobs
+- `img_base=...`: base url for embedded blobs (images)
+- `emoji_base=...`: base url for emoji images
+
+The `*_base` query options overwrite the defaults set in the config.
+The `base` option is a fallback instead of specifying the URLs separately.
+The base options are mostly useful for embedding, where the script is embedded
+on a different origin than where ssb-viewer is running. However, you may not
+need them, as the ssb-viewer embed script will detect the base where it is
+included from.
+
+## Config
+
+To change `ssb-viewer`'s default options, edit your `~/.ssb/config`, to have
+properties like the following:
+```json
+{
+  "viewer": {
+    "port": 8807,
+    "host": "::"
+  }
+}
+```
+You can also pass these as command-line options to `./bin.js` or `sbot` as,
+e.g. `--viewer.port 8807`.
+
+- `viewer.port`: port for the server to listen on. default: `8807`
+- `viewer.host`: host address for the server to listen on. default: `::`
+- `viewer.base`: default base url for links that ssb-viewer can handle
+- `viewer.msg_base`: base url for links to ssb messages
+- `viewer.feed_base`: base url for links to ssb feeds
+- `viewer.blob_base`: base url for links to ssb blobs
+- `viewer.img_base`: base url for embedded blobs (images)
+- `viewer.emoji_base`: base url for emoji images
+- `viewer.require_opt_in`: whether to serve content from feeds that have not published a `publicWebHosting` `about` message. default: `true`
+- `viewer.disallowRobots`: whether to direct search engines to not index the site. default: `true`
+
+## References
+
+- Concept: [ssb-porthole][]
+- UI ideas: [sdash][], [patchbay][]
+- Server techniques: [ssb-web-server][], [ssb-ws][], [git-ssb-web][]
+
+
+[ssb-porthole]: %cgkDJXsh6pO5m458B3ngEro+U0qUMGTY1TRGTZOP6lQ=.sha256
+[patchbay]: %s9mSFATE4RGyJx9wgH22lBrvD4CgUQW4yeguSWWjtqc=.sha256
+[sdash]: %qrU04j9vfUJKfq1rGZrQ5ihtSfA4ilfY3wLy7xFv0xk=.sha256
+[git-ssb-web]: %q5d5Du+9WkaSdjc8aJPZm+jMrqgo0tmfR+RcX5ZZ6H4=.sha256
+[ssb-web-server]: %gYctTCrA06BhAGGvQ6PJ0H2eCCQLj1iEsmfn8SD5+nk=.sha256
+[ssb-ws]: %tFjo5SoD+Y0SaB5vqZYppmoPmv9LKB5wMPl96qtu4qk=.sha256
+
+## License
+
+Copyright (c) 2016-2020 Secure Scuttlebutt Consortium
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

bin.js

@@ -1,6 +1,49 @@
 #!/usr/bin/env node
 
+// ssb-viewer connects to sbot exactly once and hands the handle to index.js,
+// which captures it in a closure behind http.createServer().listen(). There is
+// no reconnect. So when sbot dies -- which on this box means "gets OOM-killed",
+// after which run-server.sh restarts it -- the viewer keeps port 8807 open
+// while holding a dead muxrpc connection, and never answers another request.
+//
+// That is the silent wedge. The fix is not to reconnect (index.js can't take a
+// new handle) but to exit loudly, so the `while true` loop in run-ssb-viewer.sh
+// restarts us with a fresh connection.
+
+var PING_INTERVAL_MS = 60000
+var PING_TIMEOUT_MS = 20000
+
+function bail (why) {
+  console.error('[viewer] ' + new Date().toISOString() + ' ' + why + ' — exiting so the run loop restarts us')
+  process.exit(1)
+}
+
 require('ssb-client')(function (err, sbot, config) {
-  if (err) throw err
+  // ssb-client already wraps this as "could not connect to sbot", so don't
+  // prefix it again.
+  if (err) return bail(String(err.message || err))
+
+  // muxrpc emits 'closed' when the underlying stream ends
+  // (see node_modules/muxrpc/index.js). This is the fast path.
+  sbot.on('closed', function () {
+    bail('sbot connection closed')
+  })
+
+  // Belt and braces: a half-open TCP connection can leave 'closed' unfired
+  // while calls silently hang forever. Poll a cheap RPC to catch that case.
+  if (typeof sbot.whoami === 'function') {
+    var timer = setInterval(function () {
+      var timeout = setTimeout(function () {
+        bail('sbot did not answer whoami within ' + (PING_TIMEOUT_MS / 1000) + 's')
+      }, PING_TIMEOUT_MS)
+      sbot.whoami(function (err) {
+        clearTimeout(timeout)
+        if (err) bail('sbot whoami failed: ' + (err.message || err))
+      })
+    }, PING_INTERVAL_MS)
+    // Don't hold the event loop open on our account.
+    timer.unref()
+  }
+
   require('.').init(sbot, config)
 })

blob-wanter.js

@@ -0,0 +1,214 @@
+#!/usr/bin/env node
+//
+// blob-wanter.js — issue standing blobs.want() calls for blobs referenced by
+// the feeds this node replicates.
+//
+// WHY THIS EXISTS
+// ---------------
+// Nothing in ssb-server scans replicated messages and fetches the blobs they
+// mention. `blobs.want` is called from exactly one place in the whole install
+// (ssb-ws/blobs.js), and only in response to an inbound HTTP request.
+//
+// So how did friends' images ever arrive? Via ssb-blobs `sympathy` (default 3):
+// when a peer publishes a blob it calls blobs.push(), broadcasting a pretend
+// "want" at hop -1; we adopt that want out of sympathy and fetch. That is the
+// SAME code path that let 83 strangers fill this disk with 31k blobs we never
+// asked for. Turning sympathy off without a replacement would also stop our own
+// friends' images arriving.
+//
+// This is the replacement, and it is strictly better for our situation:
+//
+//   * want() has no timeout and no expiry (ssb-blobs inject.js:312), and
+//     createWantStream ships the whole want map to every peer on connect.
+//     So an explicit want is a STANDING ORDER: the moment a feed that has been
+//     dark for months reappears, our want goes out and the blob transfers.
+//     Sympathy only fires if that peer happens to re-announce.
+//
+//   * It wants ONLY blobs our own feeds reference. Strangers get nothing.
+//
+// IMPORTANT: sbot's want map is in-memory (`var want = {}`), so it is lost on
+// every sbot restart -- which on this box means every OOM kill. This process
+// exits when its sbot connection drops and is restarted by its run loop, which
+// re-runs the backfill and re-arms every want. The two lifetimes are deliberately
+// tied together.
+//
+// Usage:
+//   ./blob-wanter.js              # backfill, then follow the log live
+//   ./blob-wanter.js --dry-run    # report what it would want, change nothing
+//   ./blob-wanter.js --once       # backfill only, then exit
+
+var fs = require('fs')
+var path = require('path')
+var pull = require('pull-stream')
+var ssbKeys = require('ssb-keys')
+
+var DRY_RUN = process.argv.indexOf('--dry-run') !== -1
+var ONCE = process.argv.indexOf('--once') !== -1
+
+// sha256 of the empty string. ssb-blobs special-cases this and never stores it,
+// so asking for it is pointless noise.
+var EMPTY_HASH = '&47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=.sha256'
+var BLOB_RE = /&[A-Za-z0-9+/]{43}=\.sha256/g
+// If we ever want more than this, something is wrong -- log loudly rather than
+// silently opening tens of thousands of muxrpc requests on a 469MB box.
+var SANITY_CAP = 2000
+
+function log () {
+  var msg = '[blob-wanter] ' + new Date().toISOString() + ' ' +
+    Array.prototype.slice.call(arguments).join(' ')
+  console.log(msg)
+}
+
+function bail (why) {
+  console.error('[blob-wanter] ' + new Date().toISOString() + ' ' + why +
+    ' — exiting so the run loop restarts us')
+  process.exit(1)
+}
+
+// Pull every blob ref out of a message, including inside private messages we
+// can unbox. Regex over the serialised content catches mentions, markdown
+// image links, and ad-hoc fields alike.
+function blobRefsIn (msg, keys) {
+  var content = msg && msg.value && msg.value.content
+  if (!content) return []
+
+  if (typeof content === 'string') {
+    if (!/\.box\d*$/.test(content)) return []   // encrypted but not for us
+    try {
+      var unboxed = ssbKeys.unbox(content, keys)
+      if (!unboxed) return []
+      content = unboxed
+    } catch (e) {
+      return []
+    }
+  }
+
+  var found
+  try {
+    found = JSON.stringify(content).match(BLOB_RE)
+  } catch (e) {
+    return []
+  }
+  return found || []
+}
+
+require('ssb-client')(function (err, sbot, config) {
+  if (err) return bail(String(err.message || err))
+
+  sbot.on('closed', function () { bail('sbot connection closed') })
+
+  var keys = ssbKeys.loadOrCreateSync(path.join(config.path, 'secret'))
+
+  // `seen` must be marked SYNCHRONOUSLY on entry to consider(). Deduping inside
+  // the async has() callback would let the same ref spawn many in-flight has()
+  // calls, and would push duplicate callbacks into ssb-blobs' waiting[id] array
+  // (a real leak, since those callbacks are never freed until the blob arrives).
+  var seen = Object.create(null)
+  var wanted = Object.create(null)
+  var stats = { scanned: 0, refs: 0, already: 0, wanted: 0, arrived: 0 }
+
+  // has() is async, so the backfill stream ends long before the lookups finish.
+  // Track them so --once/--dry-run can report real numbers instead of zeros.
+  var pending = 0
+  var onDrained = null
+  function settle () {
+    if (pending === 0 && onDrained) { var f = onDrained; onDrained = null; f() }
+  }
+
+  function consider (ref, why) {
+    if (ref === EMPTY_HASH) return
+    if (seen[ref]) return
+    seen[ref] = true
+    stats.refs++
+
+    pending++
+    sbot.blobs.has(ref, function (err, has) {
+      pending--
+      if (err) { log('has() failed for', ref, '-', err.message || err); return settle() }
+      if (has) { stats.already++; return settle() }
+
+      if (Object.keys(wanted).length >= SANITY_CAP) {
+        log('WARN hit sanity cap of', SANITY_CAP, 'outstanding wants — not adding', ref)
+        return settle()
+      }
+
+      wanted[ref] = true
+      stats.wanted++
+
+      if (DRY_RUN) {
+        log('DRY-RUN would want', ref, '(' + why + ')')
+        return settle()
+      }
+
+      log('want', ref, '(' + why + ')')
+      // The callback fires only if/when the blob actually arrives. If the peer
+      // holding it never appears it simply never fires -- that is the standing
+      // order working as intended, not a leak.
+      sbot.blobs.want(ref, function (err) {
+        if (err) return log('want failed for', ref, '-', err.message || err)
+        stats.arrived++
+        log('ARRIVED', ref)
+      })
+      settle()
+    })
+  }
+
+  function handle (msg, why) {
+    if (!msg || !msg.value) return
+    stats.scanned++
+    blobRefsIn(msg, keys).forEach(function (ref) { consider(ref, why) })
+  }
+
+  log('connected to sbot' + (DRY_RUN ? ' (DRY RUN — nothing will be wanted)' : ''))
+  log('backfilling from the local log…')
+
+  pull(
+    sbot.createLogStream({ keys: true, values: true }),
+    pull.drain(
+      function (msg) { handle(msg, 'backfill') },
+      function (err) {
+        if (err) return bail('backfill stream failed: ' + (err.message || err))
+
+        // Wait for the in-flight has() lookups before reporting or exiting.
+        onDrained = function () {
+          log('backfill done —',
+              'messages=' + stats.scanned,
+              'unique_blobrefs=' + stats.refs,
+              'already_have=' + stats.already,
+              'newly_wanted=' + stats.wanted)
+
+          if (ONCE || DRY_RUN) {
+            log('exiting (' + (DRY_RUN ? '--dry-run' : '--once') + ')')
+            return process.exit(0)
+          }
+          startLive()
+        }
+        settle()
+      }
+    )
+  )
+
+  function startLive () {
+    log('following the log live; standing wants re-arm on every restart')
+    pull(
+      sbot.createLogStream({ keys: true, values: true, live: true, old: false }),
+      pull.drain(
+        function (msg) { handle(msg, 'live') },
+        function (err) {
+          bail('live stream ended' + (err ? ': ' + (err.message || err) : ''))
+        }
+      )
+    )
+
+    // Periodic heartbeat so the log shows the process is alive and what it is
+    // holding, without needing to attach to the screen.
+    setInterval(function () {
+      log('stats —',
+          'msgs=' + stats.scanned,
+          'unique_refs=' + stats.refs,
+          'have=' + stats.already,
+          'standing_wants=' + stats.wanted,
+          'arrived=' + stats.arrived)
+    }, 3600000).unref()
+  }
+})

index.js

@@ -24,13 +24,16 @@ var {
   renderShowAll,
   renderRssItem,
   wrapRss,
+  renderItemGrid,
 } = require('./render')
+var getFollows = require('./lib/follows')
+var createItemsHandler = require('./items')
 
 var appHash = hash([fs.readFileSync(__filename)])
 
 var urlIdRegex = /^(?:\/(([%&@]|%25|%26|%40)(?:[A-Za-z0-9\/+]|%2[Ff]|%2[Bb]){43}(?:=|%3[Dd])\.(?:sha256|ed25519))(?:\.([^?]*))?|(\/.*?))(?:\?(.*))?$/
 
-var zeros = new Buffer(24); zeros.fill(0)
+var zeros = Buffer.alloc(24); zeros.fill(0)
 
 function hash(arr) {
   return arr.reduce(function (hash, item) {
@@ -44,6 +47,43 @@ exports.version = require('./package').version
 
 exports.init = function (sbot, config) {
   var conf = config.viewer || {}
+
+  // --- known-blob gate -----------------------------------------------------
+  // This node used to cache blobs on behalf of strangers (ssb-blobs `sympathy`
+  // defaults to 3), and serveBlob would hand any of them to anyone who knew the
+  // hash. sympathy is 0 now and the cache has been pruned, so everything we hold
+  // is referenced by a feed we replicate. This keeps that true even if a stray
+  // blob ever lands.
+  //
+  // The local log is small (~360KB), so a regex scan is cheap. It is refreshed
+  // periodically rather than kept live: being a few minutes stale can only delay
+  // a legitimate image, never serve one that is not ours.
+  var knownBlobs = null   // null = not loaded yet -> fail open, never 404 everything
+  var logOffsetPath = path.join(config.path, 'flume', 'log.offset')
+
+  function refreshKnownBlobs() {
+    fs.readFile(logOffsetPath, function (err, buf) {
+      if (err) {
+        console.error('[viewer] blob gate: could not read log.offset:', err.message)
+        return   // keep whatever set we already had
+      }
+      var found = buf.toString('binary').match(/&[A-Za-z0-9+/]{43}=\.sha256/g) || []
+      var next = Object.create(null)
+      found.forEach(function (id) { next[id] = true })
+      if (knownBlobs === null) {
+        console.log('[viewer] blob gate active:', Object.keys(next).length, 'known blobs')
+      }
+      knownBlobs = next
+    })
+  }
+  refreshKnownBlobs()
+  setInterval(refreshKnownBlobs, 5 * 60 * 1000).unref()
+
+  // serveBlob is a free function outside this closure, so hand the check over.
+  sbot.isKnownBlob = function (id) {
+    if (knownBlobs === null) return true
+    return !!knownBlobs[id]
+  }
   var port = conf.port || 8807
   var host = conf.host || config.host || '::'
 
@@ -75,6 +115,16 @@ exports.init = function (sbot, config) {
   var getAbout = memo({cache: lru(100)}, require('./lib/about'), sbot)
   var serveAcmeChallenge = require('ssb-acme-validator')(sbot)
 
+  var serveItems = createItemsHandler({
+    sbot: sbot,
+    defaultOpts: defaultOpts,
+    addAuthorAbout: addAuthorAbout,
+    renderItemGrid: renderItemGrid,
+    wrapPage: wrapPage,
+    respond: respond,
+    toPull: toPull,
+  })
+
   http.createServer(serve).listen(port, host, function () {
     if (/:/.test(host)) host = '[' + host + ']'
     console.log('[viewer] Listening on http://' + host + ':' + port)
@@ -90,6 +140,7 @@ exports.init = function (sbot, config) {
     if (m[4] === '/robots.txt') return serveRobots(req, res, conf)
     if (req.url.startsWith('/static/')) return serveStatic(req, res, m[4])
     if (req.url.startsWith('/emoji/')) return serveEmoji(req, res, m[4])
+    if (m[4] === '/items') return serveItems(req, res, m[5])
     if (req.url.startsWith('/user-feed/')) return serveUserFeed(req, res, m[4])
     else if (req.url.startsWith('/channel/')) return serveChannel(req, res, m[4])
     else if (req.url.startsWith('/.well-known/acme-challenge')) return serveAcmeChallenge(req, res)
@@ -109,6 +160,23 @@ exports.init = function (sbot, config) {
     return respond(res, 404, 'Not found')
   }
 
+  // The bare id-lookup form this used to render was a dead end for anyone
+  // arriving without an id already in hand. Keep the ?id= redirect, then show
+  // the item grid instead of an empty box.
+  function serveHome(req, res, query) {
+    var q = query ? qs.parse(query) : {}
+    var id = asLink(q.id)
+    if (id) {
+      res.writeHead(303, {
+        Location: '/' + (
+          id[0] === '#' ? 'channel/' + id.substr(1) :
+          refs.isMsgId(id) ? encodeURIComponent(id) : id)
+      })
+      return res.end()
+    }
+    return serveItems(req, res, query)
+  }
+
   function serveFeed(req, res, feedId, ext) {
     console.log('serving feed: ' + feedId)
 
@@ -189,40 +257,12 @@ exports.init = function (sbot, config) {
     var feedId = url.substring(url.lastIndexOf('user-feed/')+10, 100)
     console.log('serving user feed: ' + feedId)
 
-    var following = []
-    var channelSubscriptions = []
-    
     getAbout(feedId, function (err, about) {
-      pull(
+      getFollows(sbot, feedId, function (err, sets) {
-        sbot.createUserStream({ id: feedId }),
+        if (err) return respond(res, 500, err.stack || err)
-        pull.filter((msg) => {
+        serveFeeds(req, res, sets.following, sets.channelSubscriptions, feedId,
-          return !msg.value ||
-            msg.value.content.type == 'contact' ||
-            (msg.value.content.type == 'channel' &&
-             typeof msg.value.content.subscribed != 'undefined')
-        }),
-        pull.collect(function (err, msgs) {
-          msgs.forEach((msg) => {
-            if (msg.value.content.type == 'contact')
-            {
-              if (msg.value.content.following)
-                following[msg.value.content.contact] = 1
-              else
-                delete following[msg.value.content.contact]
-            }
-            else // channel subscription
-            {
-              if (msg.value.content.subscribed)
-                channelSubscriptions[msg.value.content.channel] = 1
-              else
-                delete channelSubscriptions[msg.value.content.channel]
-            }
-          })
-          
-          serveFeeds(req, res, following, channelSubscriptions, feedId,
                    'user feed ' + (about ? about.name : ''))
       })
-      )
     })
   }
 
@@ -474,6 +514,12 @@ function serveBlob(req, res, sbot, id, query) {
   var etag = id + (unbox || '')
 
   if (req.headers['if-none-match'] === etag) return respond(res, 304)
+  // Only serve blobs referenced by a feed we replicate. Anything else is not
+  // ours to hand out, so treat it as absent rather than confirm we hold it.
+  if (typeof sbot.isKnownBlob === 'function' && !sbot.isKnownBlob(id)) {
+    return respond(res, 404, 'Not found')
+  }
+
   sbot.blobs.has(id, function (err, has) {
     if (err) {
       if (/^invalid/.test(err.message)) return respond(res, 400, err.message)
@@ -483,13 +529,13 @@ function serveBlob(req, res, sbot, id, query) {
 
     var unboxKey
     if (unbox) {
-      try { unboxKey = new Buffer(unbox, 'base64') }
+      try { unboxKey = Buffer.from(unbox, 'base64') }
       catch(e) { return respond(res, 400, err.message) }
       if (unboxKey.length !== 32) return respond(res, 400, 'Bad blob key')
     }
 
     res.writeHead(200, {
-      'Cache-Control': 'public, max-age=315360000',
+      'Cache-Control': 'public, max-age=315360000, immutable',
       'etag': etag
     })
 
@@ -570,32 +616,6 @@ function asLink(id) {
   if (refs.isLink(id)) return id
 }
 
-function serveHome(req, res, query, conf) {
-  var q = query ? qs.parse(query) : {}
-  var id = asLink(q.id)
-  if (id) {
-    res.writeHead(303, {
-      Location: '/' + (
-        id[0] === '#' ? 'channel/' + id.substr(1) :
-        refs.isMsgId(id) ? encodeURIComponent(id) : id)
-    })
-    return res.end()
-  }
-  res.writeHead(200, {
-    'Content-Type': 'text/html'
-  })
-  pull(
-    pull.once(h('form', {method: 'get', action: ''},
-      h('input', {name: 'id', placeholder: 'id', size: 60, value: q.id || ''}), ' ',
-      h('input', {type: 'submit', value: 'Go'})
-    ).outerHTML),
-    wrapPage('ssb-viewer'),
-    toPull(res, function (err) {
-      if (err) console.error('[viewer]', err)
-    })
-  )
-}
-
 function serveRobots(req, res, conf) {
   var disallow = conf.disallowRobots == null ? true : conf.disallowRobots
   res.end('User-agent: *\n'

items.js

@@ -0,0 +1,162 @@
+var pull = require('pull-stream')
+var paramap = require('pull-paramap')
+var qs = require('querystring')
+var getFollows = require('./lib/follows')
+
+// One merged view of every custo item the pub knows about.
+//
+// The selection rule is deliberately "whatever this pub follows", not a list of
+// kiosk feed ids: adding a kiosk means following it from the pub, and its items
+// appear here on the next cache miss. No code change, no edit to cust.ooo.
+//
+// ITEMS ONLY. A `nft: "give"` message is a change of custody, not a thing - it
+// has no photo of its own and describes an item already on the page, so showing
+// it here listed the same object twice. Custody is answered one level down: the
+// card links to the item's thread, where the mint and every hand-off since are
+// in order.
+//
+// Two facts about the data that this file depends on, both verified against the
+// live log and both easy to get wrong:
+//   * `custodisco` is the STRING "true", not a boolean.
+//   * no custo message has ever set `content.channel`, so the channel index is
+//     useless here and the filter has to look at content fields directly.
+
+var CACHE_MS = 60 * 1000
+var PAGE_SIZE = 60
+
+var IMAGE_MD = /!\[[^\]]*\]\((&[A-Za-z0-9+/]{43}=\.sha256)\)/
+
+// The item's photo: a mints's first image mention, falling back to the markdown
+// embed in the text for messages whose mentions array is missing or unhelpful.
+function itemPhoto (c) {
+  if (c && Array.isArray(c.mentions)) {
+    for (var i = 0; i < c.mentions.length; i++) {
+      var m = c.mentions[i]
+      if (!m || typeof m.link !== 'string' || m.link[0] !== '&') continue
+      if (typeof m.type === 'string' && !/^image\//.test(m.type)) continue
+      return m.link
+    }
+  }
+  var md = IMAGE_MD.exec(String((c && c.text) || ''))
+  return md ? md[1] : null
+}
+
+// Strip the parts of the post text that are furniture rather than description:
+// the image embed itself, and the kiosk's " a #custodisco item " sign-off.
+function itemCaption (c) {
+  return String((c && c.text) || '')
+    .replace(/!\[[^\]]*\]\([^)]*\)/g, ' ')
+    .replace(/\s*a\s+#custodisco\s+item\s*$/i, ' ')
+    .replace(/\s+/g, ' ')
+    .trim()
+}
+
+// Every custodisco message carries nft: "mint" or "give" - checked across all
+// 407 of them, none lacks the field - so matching "mint" is exact rather than
+// merely excluding gives.
+function isItem (msg) {
+  var c = msg && msg.value && msg.value.content
+  return !!c && typeof c === 'object' &&
+    c.custodisco === 'true' && c.nft === 'mint'
+}
+
+// ctx supplies the pieces that live in index.js's init closure:
+//   sbot, defaultOpts, addAuthorAbout, respond, toPull, renderItemGrid, wrapPage
+module.exports = function createItemsHandler (ctx) {
+  var cache = null   // { at, msgs, feeds }
+
+  // A full log scan takes well under a second on this node (11k messages,
+  // 360 KB), so this cache exists to keep a page refresh from redoing it, not
+  // because the scan is expensive.
+  function load (cb) {
+    if (cache && Date.now() - cache.at < CACHE_MS) return cb(null, cache)
+
+    ctx.sbot.whoami(function (err, feed) {
+      if (err) return cb(err)
+      var me = feed.id
+      getFollows(ctx.sbot, me, function (err, sets) {
+        if (err) return cb(err)
+        var authors = sets.following
+        authors[me] = true   // the pub's own items count too
+
+        pull(
+          ctx.sbot.createLogStream({ reverse: true }),
+          pull.filter(function (msg) {
+            return isItem(msg) && authors[msg.value.author] === true
+          }),
+          pull.collect(function (err, msgs) {
+            if (err) return cb(err)
+            // Log order is arrival order, which diverges from publish order
+            // whenever an old feed is backfilled. Sort so "newest first" and
+            // the ?before cursor both mean the same thing.
+            msgs.sort(function (a, b) { return b.value.timestamp - a.value.timestamp })
+            // Count the feeds that actually contributed an item, not the size
+            // of the follow set: the pub follows itself and publishes no items,
+            // so the follow set overstates it by one and would keep drifting
+            // as feeds are followed for other reasons.
+            var contributing = Object.create(null)
+            msgs.forEach(function (m) { contributing[m.value.author] = true })
+
+            cache = {
+              at: Date.now(),
+              msgs: msgs,
+              feeds: Object.keys(contributing).length
+            }
+            cb(null, cache)
+          })
+        )
+      })
+    })
+  }
+
+  function addItemContext (msg, cb) {
+    var c = msg.value.content
+    msg.itemPhoto = itemPhoto(c)
+    msg.itemCaption = itemCaption(c)
+    cb(null, msg)
+  }
+
+  return function serveItems (req, res, query) {
+    var q = query ? qs.parse(query) : {}
+    var showAll = 'showAll' in q
+    var before = q.before ? Number(q.before) : null
+
+    load(function (err, data) {
+      if (err) return ctx.respond(res, 500, err.stack || err)
+
+      var msgs = data.msgs
+      if (before) {
+        msgs = msgs.filter(function (m) { return m.value.timestamp < before })
+      }
+      var page = showAll ? msgs : msgs.slice(0, PAGE_SIZE)
+      var last = page[page.length - 1]
+      var older = (!showAll && msgs.length > page.length && last)
+        ? last.value.timestamp
+        : null
+
+      res.writeHead(200, { 'Content-Type': 'text/html' })
+      pull(
+        pull.values(page),
+        paramap(ctx.addAuthorAbout, 8),
+        paramap(addItemContext, 8),
+        pull(
+          ctx.renderItemGrid(ctx.defaultOpts, {
+            total: data.msgs.length,
+            shown: page.length,
+            feeds: data.feeds,
+            older: older,
+            showAll: showAll
+          }),
+          ctx.wrapPage('items')
+        ),
+        ctx.toPull(res, function (err) {
+          if (err) console.error('[viewer]', err)
+        })
+      )
+    })
+  }
+}
+
+module.exports.itemPhoto = itemPhoto
+module.exports.itemCaption = itemCaption
+module.exports.isItem = isItem

lib/follows.js

@@ -0,0 +1,38 @@
+var pull = require('pull-stream')
+
+// Reduce a feed's own contact and channel-subscription messages into the sets
+// they describe. createUserStream runs oldest-first, so a later unfollow
+// correctly undoes an earlier follow.
+//
+// Both /user-feed/ and /items are built on this: "who does this feed listen
+// to" is the whole selection rule, which is what lets a new kiosk show up by
+// being followed rather than by being added to a list somewhere.
+module.exports = function getFollows (sbot, feedId, cb) {
+  var following = Object.create(null)
+  var channelSubscriptions = Object.create(null)
+
+  pull(
+    sbot.createUserStream({ id: feedId }),
+    pull.filter(function (msg) {
+      var c = msg && msg.value && msg.value.content
+      if (!c || typeof c !== 'object') return false   // also skips private (string) content
+      return c.type === 'contact' ||
+        (c.type === 'channel' && typeof c.subscribed !== 'undefined')
+    }),
+    pull.drain(function (msg) {
+      var c = msg.value.content
+      if (c.type === 'contact') {
+        if (!c.contact) return
+        if (c.following) following[c.contact] = true
+        else delete following[c.contact]
+      } else {
+        if (!c.channel) return
+        if (c.subscribed) channelSubscriptions[c.channel] = true
+        else delete channelSubscriptions[c.channel]
+      }
+    }, function (err) {
+      if (err) return cb(err)
+      cb(null, { following: following, channelSubscriptions: channelSubscriptions })
+    })
+  )
+}

package.json

@@ -1,9 +1,14 @@
 {
-  "name": "ssb-viewer",
+  "name": "custo-viewer",
   "version": "1.0.0",
-  "description": "serve ssb threads as (embeddable) web pages",
+  "description": "custo's web view of scuttlebutt - the item feed behind www.cust.ooo",
   "main": "index.js",
   "bin": "bin.js",
+  "homepage": "https://git.autonomic.zone/trav/custo-viewer",
+  "repository": {
+    "type": "git",
+    "url": "ssh://git@git.autonomic.zone:2222/trav/custo-viewer.git"
+  },
   "dependencies": {
     "asyncmemo": "^1.0.0",
     "emoji-named-characters": "^1.0.2",
@@ -15,7 +20,7 @@
     "pull-cat": "^1.1.11",
     "pull-paramap": "^1.2.1",
     "pull-stream": "^3.5.0",
-    "ssb-acme-validator": "http://localhost:8989/blobs/get/&MgpztmIbg8wShqDXBcKt0w78qwcpNe8qb4n3fsveve8=.sha256",
+    "ssb-acme-validator": "file:./ssb-acme-validator.tar.gz",
     "ssb-client": "^4.5.2",
     "ssb-marked": "^0.7.3",
     "ssb-ref": "^2.9.0",
@@ -26,6 +31,6 @@
   "devDependencies": {
     "tape": "^4.6.2"
   },
-  "author": "cel",
+  "author": "trav (fork of ssb-viewer by cel)",
   "license": "AGPL-3.0+"
 }

prune-blobs.js

@@ -0,0 +1,163 @@
+#!/usr/bin/env node
+//
+// prune-blobs.js — delete cached blobs that none of our own feeds reference.
+//
+// CONTEXT
+// -------
+// ssb-blobs `sympathy` (default 3) made this node fetch and cache blobs on
+// behalf of any peer up to 3 network hops away. With port 8008 open to the
+// world, that turned it into a free CDN: 31,283 blobs / 4.6 GB, of which only
+// 267 are referenced by any message we hold.
+//
+// Once sympathy is 0 the inflow stops, but the existing hoard stays on disk --
+// and ssb-viewer will serve any of it by hash to anyone who asks. This removes
+// it.
+//
+// SAFETY
+// ------
+//   * Dry run by default. Pass --delete to actually remove anything.
+//   * Every deleted id is written to a manifest first. Blobs are content-
+//     addressed, so anything deleted in error can be re-requested with
+//     `ssb-server blobs.want <id>` if a peer still has it.
+//   * The keep set is built from the raw log, so it does not depend on sbot
+//     being up or on any index being in sync.
+//
+// Blobs are stored by multiblob at blobs/<alg>/<hex[0:2]>/<hex[2:]>, where hex
+// is the base64 blob hash decoded to hex (see multiblob/index.js toPath).
+
+var fs = require('fs')
+var path = require('path')
+
+var DELETE = process.argv.indexOf('--delete') !== -1
+var HOME = process.env.HOME || '/home/cyberian'
+var LOG_OFFSET = path.join(HOME, '.ssb/flume/log.offset')
+var BLOB_DIR = path.join(HOME, '.ssb/blobs/sha256')
+var PUSH_DIR = path.join(HOME, '.ssb/blobs_push')
+var MANIFEST = path.join(HOME, 'logs/pruned-blobs.txt')
+
+var BLOB_RE = /&[A-Za-z0-9+/]{43}=\.sha256/g
+// sha256 of the empty string; ssb-blobs special-cases it and never stores it.
+var EMPTY_HASH = '&47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=.sha256'
+
+function idToHex (id) {
+  return Buffer.from(id.slice(1).replace(/\.sha256$/, ''), 'base64').toString('hex')
+}
+function hexToId (hex) {
+  return '&' + Buffer.from(hex, 'hex').toString('base64') + '.sha256'
+}
+function human (bytes) {
+  var u = ['B', 'KB', 'MB', 'GB']
+  var i = 0
+  while (bytes >= 1024 && i < u.length - 1) { bytes /= 1024; i++ }
+  return bytes.toFixed(1) + ' ' + u[i]
+}
+
+// ---- keep set -------------------------------------------------------------
+var keep = Object.create(null)
+keep[EMPTY_HASH] = true
+
+var log = fs.readFileSync(LOG_OFFSET).toString('binary')
+var refs = log.match(BLOB_RE) || []
+refs.forEach(function (r) { keep[r] = true })
+var fromLog = Object.keys(keep).length - 1
+
+// Anything queued for push to the network is ours by definition. The leveldb
+// is locked by the running sbot, so scrape the ids out of its files instead of
+// opening it.
+var fromPush = 0
+try {
+  fs.readdirSync(PUSH_DIR).forEach(function (f) {
+    if (!/\.(log|ldb)$/.test(f)) return
+    var buf = fs.readFileSync(path.join(PUSH_DIR, f)).toString('binary')
+    ;(buf.match(BLOB_RE) || []).forEach(function (r) {
+      if (!keep[r]) { keep[r] = true; fromPush++ }
+    })
+  })
+} catch (e) {
+  console.error('warning: could not scan blobs_push:', e.message)
+}
+
+// ---- walk the blob store --------------------------------------------------
+var kept = { n: 0, bytes: 0 }
+var doomed = []
+var orphans = []
+
+fs.readdirSync(BLOB_DIR).forEach(function (sub) {
+  var subdir = path.join(BLOB_DIR, sub)
+  var st
+  try { st = fs.statSync(subdir) } catch (e) { return }
+  if (!st.isDirectory()) return
+
+  fs.readdirSync(subdir).forEach(function (name) {
+    var full = path.join(subdir, name)
+    var size
+    try { size = fs.statSync(full).size } catch (e) { return }
+
+    var id
+    try {
+      id = hexToId(sub + name)
+    } catch (e) {
+      orphans.push(full)
+      return
+    }
+
+    if (keep[id]) { kept.n++; kept.bytes += size }
+    else doomed.push({ id: id, path: full, size: size })
+  })
+})
+
+var doomedBytes = doomed.reduce(function (a, b) { return a + b.size }, 0)
+
+console.log('keep set:      ' + (fromLog + fromPush) +
+            ' (' + fromLog + ' referenced in the log, ' + fromPush + ' from blobs_push)')
+console.log('blobs on disk: ' + (kept.n + doomed.length))
+console.log('  keeping:     ' + kept.n + '  (' + human(kept.bytes) + ')')
+console.log('  removing:    ' + doomed.length + '  (' + human(doomedBytes) + ')')
+if (orphans.length) console.log('  unparseable: ' + orphans.length + ' (left alone)')
+
+// A keep-set entry we do not actually have on disk is fine -- blob-wanter has a
+// standing want for it. Just report so the numbers reconcile.
+var missing = Object.keys(keep).filter(function (id) {
+  if (id === EMPTY_HASH) return false
+  var hex = idToHex(id)
+  return !fs.existsSync(path.join(BLOB_DIR, hex.substring(0, 2), hex.substring(2)))
+})
+if (missing.length) {
+  console.log('  note: ' + missing.length + ' referenced blob(s) are not on disk; ' +
+              'blob-wanter holds standing wants for them')
+}
+
+if (!DELETE) {
+  console.log('\nDRY RUN — nothing deleted. Re-run with --delete to apply.')
+  process.exit(0)
+}
+
+// ---- delete ---------------------------------------------------------------
+// Manifest first, so a crash mid-delete still leaves a record of what went.
+fs.mkdirSync(path.dirname(MANIFEST), { recursive: true })
+var out = fs.createWriteStream(MANIFEST, { flags: 'a' })
+out.write('# pruned ' + new Date().toISOString() + ' — ' + doomed.length +
+          ' blobs, ' + human(doomedBytes) + '\n')
+out.write('# re-request any of these with: ssb-server blobs.want <id>\n')
+doomed.forEach(function (d) { out.write(d.id + '\t' + d.size + '\n') })
+out.end()
+
+var removed = 0, failed = 0, freed = 0
+doomed.forEach(function (d) {
+  try { fs.unlinkSync(d.path); removed++; freed += d.size }
+  catch (e) { failed++ }
+})
+
+// Clean up the now-empty two-char shard directories.
+fs.readdirSync(BLOB_DIR).forEach(function (sub) {
+  var subdir = path.join(BLOB_DIR, sub)
+  try {
+    if (fs.statSync(subdir).isDirectory() && fs.readdirSync(subdir).length === 0) {
+      fs.rmdirSync(subdir)
+    }
+  } catch (e) { /* not empty, or vanished */ }
+})
+
+console.log('\nremoved ' + removed + ' blobs, freed ' + human(freed) +
+            (failed ? ', ' + failed + ' failed' : ''))
+console.log('manifest: ' + MANIFEST)

render.js

@@ -1,4 +1,6 @@
+var fs = require('fs')
 var path = require('path')
+var proc = require('child_process')
 var pull = require("pull-stream")
 var marked = require("ssb-marked")
 var htime = require("human-time")
@@ -6,6 +8,7 @@ var emojis = require("emoji-named-characters")
 var cat = require("pull-cat")
 var h = require('hyperscript')
 var refs = require('ssb-ref')
+var pkg = require('./package')
 
 var emojiDir = path.join(require.resolve("emoji-named-characters"), "../pngs")
 
@@ -14,6 +17,7 @@ exports.MdRenderer = MdRenderer
 exports.renderEmoji = renderEmoji
 exports.formatMsgs = formatMsgs
 exports.renderThread = renderThread
+exports.renderItemGrid = renderItemGrid
 exports.renderAbout = renderAbout
 exports.renderShowAll = renderShowAll
 exports.renderRssItem = renderRssItem
@@ -52,6 +56,18 @@ MdRenderer.prototype.image = function(href, title, text) {
              { type: 'image/svg+xml',
                data: href,
                alt: text }).outerHTML
+  else if (/^video:/.test(text))
+    return h('video', {
+      controls: 'controls',
+      src: this.opts.blob_base + href,
+      title: title || undefined
+    }).outerHTML
+  else if (/^audio:/.test(text))
+    return h('audio', {
+      controls: 'controls',
+      src: this.opts.blob_base + href,
+      title: title || undefined
+    }).outerHTML
   else
     return h('img',
              { src: this.opts.img_base + href,
@@ -104,24 +120,31 @@ function wrap(before, after) {
   }
 }
 
+// ---------------------------------------------------------------------------
+// Page furniture. These are the bits of chrome wrapped around every page, and
+// they are the things most likely to want changing.
+//
+//   callToAction()  the "Join Scuttlebutt now" button at the foot of a page
+//   footer          just below it: licence, repo name, deployed commit
+//   styles          the single <style> block; .item-* rules are the grid
+//   itemsHeader()   the /items title, count and search box
+//
+// Removed already: toolTipTop(), which printed "You are reading content from
+// Scuttlebutt" above every page. It is in the git history if it is ever wanted
+// back.
+// ---------------------------------------------------------------------------
+
 function callToAction() {
   return h('a.call-to-action',
            { href: 'https://www.scuttlebutt.nz' },
            'Join Scuttlebutt now').outerHTML
 }
 
-function toolTipTop() {
-  return h('span.top-tip',
-           'You are reading content from ',
-           h('a', { href: 'https://www.scuttlebutt.nz' },
-             'Scuttlebutt')).outerHTML
-}
-
 function renderAbout(opts, about, showAllHTML = "") {
   if (about.publicWebHosting === false || (about.publicWebHosting == null && opts.requireOptIn)) {
     return pull(
       pull.map(renderMsg.bind(this, opts, '')),
-      wrap(toolTipTop() + '<main>', '</main>' + callToAction())
+      wrap('<main>', '</main>' + callToAction())
     )
   }
 
@@ -129,7 +152,7 @@ function renderAbout(opts, about, showAllHTML = "") {
   figCaption.innerHTML = 'Feed of ' + escape(about.name) + '<br>' + marked(String(about.description || ''), opts.marked)
   return pull(
     pull.map(renderMsg.bind(this, opts, '')),
-    wrap(toolTipTop() + '<main>' +
+    wrap('<main>' +
          h('article',
            h('header',
              h('figure',
@@ -146,7 +169,7 @@ function renderAbout(opts, about, showAllHTML = "") {
 function renderThread(opts, id, showAllHTML = "") {
   return pull(
     pull.map(renderMsg.bind(this, opts, id)),
-    wrap(toolTipTop() + '<main>', 
+    wrap('<main>', 
          showAllHTML + '</main>' + callToAction())
   )
 }
@@ -157,17 +180,43 @@ function renderRssItem(opts) {
   )
 }
 
+// Which commit is actually running. Deploys are rsync, not `git pull`, so the
+// checkout on the server keeps whatever HEAD it was cloned at and `git
+// rev-parse` there would name a commit that has nothing to do with these files.
+// bin/deploy-viewer.sh writes the real one to .deployed-commit; fall back to
+// git only when running straight from a working copy.
+const gitHead = (function () {
+  try {
+    var stamped = fs.readFileSync(path.join(__dirname, '.deployed-commit'), 'utf8').trim()
+    if (stamped) return stamped
+  } catch (e) { /* not a deploy, ask git */ }
+  return proc.spawnSync('git', ['rev-parse', 'HEAD'], {
+    encoding: 'utf8',
+    cwd: __dirname
+  }).stdout.trim()
+}())
+const gitHeadShort = gitHead && gitHead.substr(0, 7)
+const commitUrl = pkg.homepage &&
+  pkg.homepage.replace(/\/+$/, '') + '/commit/' + gitHead
+const gitLink = !gitHead ? '' :
+  !commitUrl ? `<code title="${gitHead}">${gitHeadShort}</code>` :
+  `<a href="${commitUrl}" title="${gitHead}"><code>${gitHeadShort}</code></a>`
+
+const footer = `
+<div class=footer>AGPLv3 &copy; <a href="${pkg.homepage}">${pkg.name}</a> ${gitLink}</div>
+`
+
 function wrapPage(id) {
   return wrap(
     "<!doctype html><html><head>" +
       "<meta charset=utf-8>" +
       "<title>" +
-      id + " | ssb-viewer" +
+      id + " | custo" +
       "</title>" +
       '<meta name=viewport content="width=device-width,initial-scale=1">' +
       styles +
       "</head><body>",
-    "</body></html>"
+    footer + "\n</body></html>"
   )
 }
 
@@ -197,15 +246,6 @@ var styles = `
     }
     a { color: #364fc7; }
 
-    .top-tip, .top-tip a {
-      color: #868e96;
-    }
-    .top-tip {
-      text-align: center;
-      display: block;
-      margin-bottom: 10px;
-      font-size: 14px;
-    }
     main { margin: 0 auto; max-width: 40rem; }
     main article:first-child { border-radius: 3px 3px 0 0; }
     main article:last-child { border-radius: 0 0 3px 3px; }
@@ -289,7 +329,7 @@ var styles = `
       text-align: center;
       text-decoration: none;
       margin-top: 20px;
-      margin-bottom: 60px;
+      margin-bottom: 30px;
       background-color: #5c7cfa;
       padding: 15px 0;
       color: #edf2ff;
@@ -303,6 +343,71 @@ var styles = `
     .attending {
       text-align: center;
     }
+    .footer {
+      text-align: center;
+      margin-bottom: 10px;
+      font-size: 14px;
+      color: #868e96;
+    }
+
+    .items-header { max-width: 60rem; margin: 0 auto 24px; text-align: center; }
+    .items-header h1 { margin: 0 0 4px; font-size: 1.6em; letter-spacing: 0.04em; }
+    .items-count { margin: 0 0 14px; color: #868e96; font-size: 14px; }
+    .items-search input[type=submit] { cursor: pointer; }
+
+    main.item-grid {
+      max-width: 60rem;
+      display: grid;
+      grid-template-columns: repeat(auto-fill, minmax(190px, 1fr));
+      gap: 14px;
+    }
+    /* the single-column rules above target articles; cards are anchors */
+    .item-card {
+      display: flex;
+      flex-direction: column;
+      background-color: white;
+      border-radius: 3px;
+      box-shadow: 0 1px 3px #949494;
+      overflow: hidden;
+      text-decoration: none;
+      color: #212529;
+    }
+    .item-card:hover { box-shadow: 0 2px 8px #6c757d; }
+    .item-media { position: relative; }
+    .item-photo {
+      display: block;
+      width: 100%;
+      aspect-ratio: 1 / 1;
+      object-fit: cover;
+      background-color: #e9ecef;
+    }
+    .item-photo-missing {
+      display: flex;
+      align-items: center;
+      justify-content: center;
+      color: #adb5bd;
+      font-size: 12px;
+      text-align: center;
+    }
+    .item-photo-missing::after { content: "photo not fetched yet"; }
+    .item-meta { padding: 10px 12px; display: flex; flex-direction: column; gap: 3px; }
+    .item-caption {
+      line-height: 1.35em;
+      display: -webkit-box;
+      -webkit-line-clamp: 3;
+      -webkit-box-orient: vertical;
+      overflow: hidden;
+    }
+    .item-by { color: #495057; font-size: 13px; }
+    .item-meta time { color: #868e96; font-size: 12px; }
+    .items-more {
+      max-width: 60rem;
+      margin: 20px auto 0;
+      text-align: center;
+      display: flex;
+      gap: 20px;
+      justify-content: center;
+    }
   </style>
 `
 
@@ -597,3 +702,87 @@ function renderShowAll(showAll, url) {
   if (!showAll)
     return '<br>' + h('a', { href : url + '?showAll' }, 'Show whole feed').outerHTML
 }
+
+// ---------------------------------------------------------------------------
+// The custo item grid (/items)
+//
+// Cards, not articles: this is a collection of things, and a single-column feed
+// of 300 photos reads like a mailing list rather than an archive. Items only -
+// who holds an item is answered by its thread, one click in, not by an overlay
+// on the grid.
+// ---------------------------------------------------------------------------
+
+function renderItemGrid(opts, meta) {
+  return pull(
+    pull.map(renderItemCard.bind(this, opts)),
+    wrap(itemsHeader(opts, meta) + '<main class="item-grid">',
+         '</main>' + itemsFooter(opts, meta) + callToAction())
+  )
+}
+
+function itemsHeader(opts, meta) {
+  var counted = meta.total === 1 ? '1 item' : meta.total + ' items'
+  var feeds = meta.feeds === 1 ? '1 feed' : meta.feeds + ' feeds'
+  return h('header.items-header',
+           h('h1', 'custo items'),
+           h('p.items-count', counted + ' from ' + feeds),
+           h('form.items-search', { method: 'get', action: opts.base },
+             h('input', { name: 'id', size: 34,
+                          placeholder: 'paste a message or feed id' }), ' ',
+             h('input', { type: 'submit', value: 'Go' }))
+          ).outerHTML
+}
+
+function itemsFooter(opts, meta) {
+  var links = []
+  if (meta.older)
+    links.push(h('a', { href: opts.base + 'items?before=' + meta.older }, 'older items'))
+  if (!meta.showAll && meta.total > meta.shown)
+    links.push(h('a', { href: opts.base + 'items?showAll' }, 'show everything'))
+  if (!links.length) return ''
+  return h('div.items-more', links).outerHTML
+}
+
+function renderItemCard(opts, msg) {
+  var c = msg.value.content
+
+  var media
+  if (msg.itemPhoto) {
+    media = h('img.item-photo', { src: opts.img_base + msg.itemPhoto, alt: '' })
+    // hyperscript assigns these as DOM properties, which outerHTML then drops,
+    // so they have to be set as attributes explicitly.
+    media.setAttribute('loading', 'lazy')
+    media.setAttribute('decoding', 'async')
+    // 267 blobs are held against 301 mints, so some photos genuinely are not
+    // here yet. Fall back to the placeholder rather than a broken-image icon.
+    media.setAttribute('onerror',
+      "this.className='item-photo item-photo-missing';this.removeAttribute('src')")
+  } else {
+    media = h('div.item-photo.item-photo-missing')
+  }
+
+  var caption = msg.itemCaption || itemFallbackCaption(c)
+
+  // Links to the item's own thread: the mint, and every hand-off since.
+  return h('a.item-card',
+           { href: opts.base + encodeURIComponent(msg.key) },
+           h('div.item-media', media),
+           h('div.item-meta',
+             h('span.item-caption', caption),
+             h('span.item-by', msg.author && msg.author.name),
+             itemTime(msg))
+          ).outerHTML
+}
+
+function itemFallbackCaption(c) {
+  return String(c.text || '').replace(/\s+/g, ' ').trim().substr(0, 140) || 'untitled'
+}
+
+// Like msgTimestamp, but without the nested <a> \u2014 the whole card is a link.
+function itemTime(msg) {
+  var date = new Date(msg.value.timestamp)
+  var isoStr = date.toISOString()
+  return h('time.ssb-timestamp',
+           { datetime: isoStr, title: isoStr },
+           formatDate(date))
+}

run-blob-wanter.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# run-blob-wanter.sh
+# Same shape as run-server.sh / run-ssb-viewer.sh: keep it alive, and let the
+# process exit cleanly when its sbot connection dies.
+#
+# Restarting on sbot loss is not just tolerated here, it is REQUIRED: sbot's
+# want map is in-memory and is wiped on every sbot restart. Coming back up
+# re-runs the backfill and re-arms every standing want.
+while true; do
+	NODE_OPTIONS="--dns-result-order=ipv4first --max-old-space-size=128" ./blob-wanter.js
+	echo "Restarting blob-wanter at $(date)"
+	sleep 5
+done

run-ssb-viewer.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+# run-ssb-viewer.sh
+while true; do
+	NODE_OPTIONS="--dns-result-order=ipv4first" timeout 6h ./bin.js 
+	echo "Restarting ssb-viewer at $(date)"
+	sleep 2 
+done

startup-howto.md

@@ -0,0 +1 @@
+NODE_OPTIONS="--dns-result-order=ipv4first" ./bin.js