Fork ssb-viewer as custo-viewer

Rebrand package.json (name, homepage, repository) so render.js's page footer,
which is built from pkg.homepage + git HEAD, points at this repo rather than
upstream's ssb:// URL.

Track package-lock.json instead of ignoring it. It was gitignored upstream, but
this is a deployed application on a 469MB box running node 18 — a reproducible
dependency tree is the difference between RESTORE.md working and not.

Upstream's README is kept verbatim as UPSTREAM-README.md; the new one documents
the custo data model, including the two things that will otherwise cost someone
an afternoon: custodisco is the string "true", and no message ever sets
content.channel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0192zBTNZKZn5svyJ5HTnYds

.gitignore

@@ -1,2 +1 @@
 node_modules/
-package-lock.json

README.md

@@ -1,117 +1,59 @@
-# ssb-viewer
+# custo-viewer
 
-HTTP server for read-only views of SSB content. Serves content as web pages or as scripts for embedding in other web pages.
+The web view behind **[www.cust.ooo](https://www.cust.ooo)** — a fork of
+[ssb-viewer](https://gitlab.com/dwynen/ssb-viewer) that renders custo items from a
+Scuttlebutt pub.
 
-## Install & Run
+Upstream's own docs are kept verbatim in [UPSTREAM-README.md](UPSTREAM-README.md).
 
-As a sbot plugin:
-```sh
-mkdir -p ~/.ssb/node_modules
-cd ~/.ssb/node_modules
-git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
-npm install
-sbot plugins.enable ssb-viewer
-# restart sbot
-```
+## What custo adds
 
-Or standalone:
-```sh
-git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
-npm install
-./bin.js
-```
+| | |
+|---|---|
+| `/items` | One merged grid of every item the pub knows about, across all kiosks. New kiosks appear by being followed — no code change. |
+| known-blob gate | `serveBlob` only serves blobs referenced by a feed we replicate, so the node never hands out a stranger's cached blob. |
+| exit on dead sbot | `bin.js` exits when its muxrpc handle dies, instead of holding the port open and hanging every request forever. |
+| `blob-wanter.js` | Standing `blobs.want` orders for our own feeds' blobs, replacing what `ssb-blobs` `sympathy` used to do before it was turned off. |
 
-## Usage
+## The data model
 
-To view a thread as a web page, navigate to a url like `http://localhost:8807/%MSGID`.
+An **item** is an ordinary `type: "post"` carrying custo's own fields:
 
-To embed a thread into another web page, load it as follows:
-
-```html
-<script src="http://localhost:8807/%MSGID.js"></script>
-```
-
-To add more than the base styles, you can also load `http://localhost:8807/static/nicer.css`.
-
-## Routes
-
-- `/%msgid`: web page showing a message thread
-- `/%msgid.js`: script to embed a message thread
-- `/%msgid.json`: message thread as JSON
-- `/&feedid`: web page showing a complete feed
-- `/user-feed/&feedid`: web page showing messages from followed users and channels of a feed
-- `/channel/#channel`: web page showing messages in a specific channel
-
-### Query options
-
-- `noroot`: don't include the root message in the thread
-- `base=...`: base url for links that ssb-viewer can handle
-- `msg_base=...`: base url for links to messages
-- `feed_base=...`: base url for links to feeds
-- `blob_base=...`: base url for links to blobs
-- `img_base=...`: base url for embedded blobs (images)
-- `emoji_base=...`: base url for emoji images
-
-The `*_base` query options overwrite the defaults set in the config.
-The `base` option is a fallback instead of specifying the URLs separately.
-The base options are mostly useful for embedding, where the script is embedded
-on a different origin than where ssb-viewer is running. However, you may not
-need them, as the ssb-viewer embed script will detect the base where it is
-included from.
-
-## Config
-
-To change `ssb-viewer`'s default options, edit your `~/.ssb/config`, to have
-properties like the following:
 ```json
-{
-  "viewer": {
-    "port": 8807,
-    "host": "::"
-  }
-}
+{ "type": "post", "custodisco": "true", "nft": "mint",
+  "text": "![photo.jpg](&…sha256)\n\n…\n\n a #custodisco item ",
+  "mentions": [{ "name": "photo.jpg", "type": "image/jpeg", "link": "&…sha256" }] }
 ```
-You can also pass these as command-line options to `./bin.js` or `sbot` as,
-e.g. `--viewer.port 8807`.
 
-- `viewer.port`: port for the server to listen on. default: `8807`
-- `viewer.host`: host address for the server to listen on. default: `::`
-- `viewer.base`: default base url for links that ssb-viewer can handle
-- `viewer.msg_base`: base url for links to ssb messages
-- `viewer.feed_base`: base url for links to ssb feeds
-- `viewer.blob_base`: base url for links to ssb blobs
-- `viewer.img_base`: base url for embedded blobs (images)
-- `viewer.emoji_base`: base url for emoji images
-- `viewer.require_opt_in`: whether to serve content from feeds that have not published a `publicWebHosting` `about` message. default: `true`
-- `viewer.disallowRobots`: whether to direct search engines to not index the site. default: `true`
+A **transfer** of custody is a reply to that message:
 
-## References
+```json
+{ "type": "post", "custodisco": "true", "nft": "give",
+  "target": "@…ed25519", "root": "%…sha256", "branch": "%…sha256" }
+```
 
-- Concept: [ssb-porthole][]
-- UI ideas: [sdash][], [patchbay][]
-- Server techniques: [ssb-web-server][], [ssb-ws][], [git-ssb-web][]
+Two things to know before writing a query against this:
 
+- `custodisco` is the **string** `"true"`, not a boolean.
+- **No message ever sets `content.channel`.** The `#custodisco` hashtag exists only in
+  post text. Anything keyed on the channel index will silently return nothing — which is
+  why `/channel/custodisco` renders an empty page.
 
-[ssb-porthole]: %cgkDJXsh6pO5m458B3ngEro+U0qUMGTY1TRGTZOP6lQ=.sha256
-[patchbay]: %s9mSFATE4RGyJx9wgH22lBrvD4CgUQW4yeguSWWjtqc=.sha256
-[sdash]: %qrU04j9vfUJKfq1rGZrQ5ihtSfA4ilfY3wLy7xFv0xk=.sha256
-[git-ssb-web]: %q5d5Du+9WkaSdjc8aJPZm+jMrqgo0tmfR+RcX5ZZ6H4=.sha256
-[ssb-web-server]: %gYctTCrA06BhAGGvQ6PJ0H2eCCQLj1iEsmfn8SD5+nk=.sha256
-[ssb-ws]: %tFjo5SoD+Y0SaB5vqZYppmoPmv9LKB5wMPl96qtu4qk=.sha256
+Messages are published by the kiosks (`/home/trav/custodisco-kiosk/ssb-post.sh`), not by
+this viewer. The viewer is read-only.
 
-## License
+## Running it
 
-Copyright (c) 2016-2017 Secure Scuttlebutt Consortium
+`bin.js` connects to a local `ssb-server` and serves on `conf.viewer.port` (8807).
+See `UPSTREAM-README.md` for the plugin-vs-standalone options.
 
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as
-published by the Free Software Foundation, either version 3 of the
-License, or (at your option) any later version.
+## Deployment
 
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
+Deployed by rsync from a laptop, not by `git pull` — the server holds no push
+credentials, and one deploy path is better than two. The script, the systemd units, the
+nginx config, and the disaster-recovery runbook all live in the ops repo alongside this
+one (`documents/custo/ssb-viewer`), which is the source of truth for the server itself.
 
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
+**Never commit secrets here.** Keys and server config belong in the ops repo.
+
+AGPL-3.0+, inherited from upstream.

UPSTREAM-README.md

@@ -0,0 +1,128 @@
+# ssb-viewer
+
+HTTP server for read-only views of SSB content. Serves content as web pages or as scripts for embedding in other web pages.
+
+## Install & Run
+
+Before you install or upgrade ssb-viewer, make sure the plugin is disabled.
+Otherwise sbot will crash while you install, because it tries to execute half-compiled JS.
+If you're running the install inside the same docker container as sbot, this will kill your build
+and leave you in a broken state where sbot doesn't start anymore.
+
+As a sbot plugin:
+```sh
+mkdir -p ~/.ssb/node_modules
+cd ~/.ssb/node_modules
+
+# for a new installation:
+git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
+
+# for an upgrade:
+cd ssb-viewer && git pull
+
+npm install
+sbot plugins.enable ssb-viewer
+# restart sbot
+```
+
+Or standalone:
+```sh
+git clone ssb://%MeCTQrz9uszf9EZoTnKCeFeIedhnKWuB3JHW2l1g9NA=.sha256 ssb-viewer && cd ssb-viewer
+npm install
+./bin.js
+```
+
+## Usage
+
+To view a thread as a web page, navigate to a url like `http://localhost:8807/%MSGID`.
+
+To embed a thread into another web page, load it as follows:
+
+```html
+<script src="http://localhost:8807/%MSGID.js"></script>
+```
+
+To add more than the base styles, you can also load `http://localhost:8807/static/nicer.css`.
+
+## Routes
+
+- `/%msgid`: web page showing a message thread
+- `/%msgid.js`: script to embed a message thread
+- `/%msgid.json`: message thread as JSON
+- `/&feedid`: web page showing a complete feed
+- `/user-feed/&feedid`: web page showing messages from followed users and channels of a feed
+- `/channel/#channel`: web page showing messages in a specific channel
+
+### Query options
+
+- `noroot`: don't include the root message in the thread
+- `base=...`: base url for links that ssb-viewer can handle
+- `msg_base=...`: base url for links to messages
+- `feed_base=...`: base url for links to feeds
+- `blob_base=...`: base url for links to blobs
+- `img_base=...`: base url for embedded blobs (images)
+- `emoji_base=...`: base url for emoji images
+
+The `*_base` query options overwrite the defaults set in the config.
+The `base` option is a fallback instead of specifying the URLs separately.
+The base options are mostly useful for embedding, where the script is embedded
+on a different origin than where ssb-viewer is running. However, you may not
+need them, as the ssb-viewer embed script will detect the base where it is
+included from.
+
+## Config
+
+To change `ssb-viewer`'s default options, edit your `~/.ssb/config`, to have
+properties like the following:
+```json
+{
+  "viewer": {
+    "port": 8807,
+    "host": "::"
+  }
+}
+```
+You can also pass these as command-line options to `./bin.js` or `sbot` as,
+e.g. `--viewer.port 8807`.
+
+- `viewer.port`: port for the server to listen on. default: `8807`
+- `viewer.host`: host address for the server to listen on. default: `::`
+- `viewer.base`: default base url for links that ssb-viewer can handle
+- `viewer.msg_base`: base url for links to ssb messages
+- `viewer.feed_base`: base url for links to ssb feeds
+- `viewer.blob_base`: base url for links to ssb blobs
+- `viewer.img_base`: base url for embedded blobs (images)
+- `viewer.emoji_base`: base url for emoji images
+- `viewer.require_opt_in`: whether to serve content from feeds that have not published a `publicWebHosting` `about` message. default: `true`
+- `viewer.disallowRobots`: whether to direct search engines to not index the site. default: `true`
+
+## References
+
+- Concept: [ssb-porthole][]
+- UI ideas: [sdash][], [patchbay][]
+- Server techniques: [ssb-web-server][], [ssb-ws][], [git-ssb-web][]
+
+
+[ssb-porthole]: %cgkDJXsh6pO5m458B3ngEro+U0qUMGTY1TRGTZOP6lQ=.sha256
+[patchbay]: %s9mSFATE4RGyJx9wgH22lBrvD4CgUQW4yeguSWWjtqc=.sha256
+[sdash]: %qrU04j9vfUJKfq1rGZrQ5ihtSfA4ilfY3wLy7xFv0xk=.sha256
+[git-ssb-web]: %q5d5Du+9WkaSdjc8aJPZm+jMrqgo0tmfR+RcX5ZZ6H4=.sha256
+[ssb-web-server]: %gYctTCrA06BhAGGvQ6PJ0H2eCCQLj1iEsmfn8SD5+nk=.sha256
+[ssb-ws]: %tFjo5SoD+Y0SaB5vqZYppmoPmv9LKB5wMPl96qtu4qk=.sha256
+
+## License
+
+Copyright (c) 2016-2020 Secure Scuttlebutt Consortium
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as
+published by the Free Software Foundation, either version 3 of the
+License, or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

bin.js

@@ -1,6 +1,49 @@
 #!/usr/bin/env node
 
+// ssb-viewer connects to sbot exactly once and hands the handle to index.js,
+// which captures it in a closure behind http.createServer().listen(). There is
+// no reconnect. So when sbot dies -- which on this box means "gets OOM-killed",
+// after which run-server.sh restarts it -- the viewer keeps port 8807 open
+// while holding a dead muxrpc connection, and never answers another request.
+//
+// That is the silent wedge. The fix is not to reconnect (index.js can't take a
+// new handle) but to exit loudly, so the `while true` loop in run-ssb-viewer.sh
+// restarts us with a fresh connection.
+
+var PING_INTERVAL_MS = 60000
+var PING_TIMEOUT_MS = 20000
+
+function bail (why) {
+  console.error('[viewer] ' + new Date().toISOString() + ' ' + why + ' — exiting so the run loop restarts us')
+  process.exit(1)
+}
+
 require('ssb-client')(function (err, sbot, config) {
-  if (err) throw err
+  // ssb-client already wraps this as "could not connect to sbot", so don't
+  // prefix it again.
+  if (err) return bail(String(err.message || err))
+
+  // muxrpc emits 'closed' when the underlying stream ends
+  // (see node_modules/muxrpc/index.js). This is the fast path.
+  sbot.on('closed', function () {
+    bail('sbot connection closed')
+  })
+
+  // Belt and braces: a half-open TCP connection can leave 'closed' unfired
+  // while calls silently hang forever. Poll a cheap RPC to catch that case.
+  if (typeof sbot.whoami === 'function') {
+    var timer = setInterval(function () {
+      var timeout = setTimeout(function () {
+        bail('sbot did not answer whoami within ' + (PING_TIMEOUT_MS / 1000) + 's')
+      }, PING_TIMEOUT_MS)
+      sbot.whoami(function (err) {
+        clearTimeout(timeout)
+        if (err) bail('sbot whoami failed: ' + (err.message || err))
+      })
+    }, PING_INTERVAL_MS)
+    // Don't hold the event loop open on our account.
+    timer.unref()
+  }
+
   require('.').init(sbot, config)
 })

blob-wanter.js

@@ -0,0 +1,214 @@
+#!/usr/bin/env node
+//
+// blob-wanter.js — issue standing blobs.want() calls for blobs referenced by
+// the feeds this node replicates.
+//
+// WHY THIS EXISTS
+// ---------------
+// Nothing in ssb-server scans replicated messages and fetches the blobs they
+// mention. `blobs.want` is called from exactly one place in the whole install
+// (ssb-ws/blobs.js), and only in response to an inbound HTTP request.
+//
+// So how did friends' images ever arrive? Via ssb-blobs `sympathy` (default 3):
+// when a peer publishes a blob it calls blobs.push(), broadcasting a pretend
+// "want" at hop -1; we adopt that want out of sympathy and fetch. That is the
+// SAME code path that let 83 strangers fill this disk with 31k blobs we never
+// asked for. Turning sympathy off without a replacement would also stop our own
+// friends' images arriving.
+//
+// This is the replacement, and it is strictly better for our situation:
+//
+//   * want() has no timeout and no expiry (ssb-blobs inject.js:312), and
+//     createWantStream ships the whole want map to every peer on connect.
+//     So an explicit want is a STANDING ORDER: the moment a feed that has been
+//     dark for months reappears, our want goes out and the blob transfers.
+//     Sympathy only fires if that peer happens to re-announce.
+//
+//   * It wants ONLY blobs our own feeds reference. Strangers get nothing.
+//
+// IMPORTANT: sbot's want map is in-memory (`var want = {}`), so it is lost on
+// every sbot restart -- which on this box means every OOM kill. This process
+// exits when its sbot connection drops and is restarted by its run loop, which
+// re-runs the backfill and re-arms every want. The two lifetimes are deliberately
+// tied together.
+//
+// Usage:
+//   ./blob-wanter.js              # backfill, then follow the log live
+//   ./blob-wanter.js --dry-run    # report what it would want, change nothing
+//   ./blob-wanter.js --once       # backfill only, then exit
+
+var fs = require('fs')
+var path = require('path')
+var pull = require('pull-stream')
+var ssbKeys = require('ssb-keys')
+
+var DRY_RUN = process.argv.indexOf('--dry-run') !== -1
+var ONCE = process.argv.indexOf('--once') !== -1
+
+// sha256 of the empty string. ssb-blobs special-cases this and never stores it,
+// so asking for it is pointless noise.
+var EMPTY_HASH = '&47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=.sha256'
+var BLOB_RE = /&[A-Za-z0-9+/]{43}=\.sha256/g
+// If we ever want more than this, something is wrong -- log loudly rather than
+// silently opening tens of thousands of muxrpc requests on a 469MB box.
+var SANITY_CAP = 2000
+
+function log () {
+  var msg = '[blob-wanter] ' + new Date().toISOString() + ' ' +
+    Array.prototype.slice.call(arguments).join(' ')
+  console.log(msg)
+}
+
+function bail (why) {
+  console.error('[blob-wanter] ' + new Date().toISOString() + ' ' + why +
+    ' — exiting so the run loop restarts us')
+  process.exit(1)
+}
+
+// Pull every blob ref out of a message, including inside private messages we
+// can unbox. Regex over the serialised content catches mentions, markdown
+// image links, and ad-hoc fields alike.
+function blobRefsIn (msg, keys) {
+  var content = msg && msg.value && msg.value.content
+  if (!content) return []
+
+  if (typeof content === 'string') {
+    if (!/\.box\d*$/.test(content)) return []   // encrypted but not for us
+    try {
+      var unboxed = ssbKeys.unbox(content, keys)
+      if (!unboxed) return []
+      content = unboxed
+    } catch (e) {
+      return []
+    }
+  }
+
+  var found
+  try {
+    found = JSON.stringify(content).match(BLOB_RE)
+  } catch (e) {
+    return []
+  }
+  return found || []
+}
+
+require('ssb-client')(function (err, sbot, config) {
+  if (err) return bail(String(err.message || err))
+
+  sbot.on('closed', function () { bail('sbot connection closed') })
+
+  var keys = ssbKeys.loadOrCreateSync(path.join(config.path, 'secret'))
+
+  // `seen` must be marked SYNCHRONOUSLY on entry to consider(). Deduping inside
+  // the async has() callback would let the same ref spawn many in-flight has()
+  // calls, and would push duplicate callbacks into ssb-blobs' waiting[id] array
+  // (a real leak, since those callbacks are never freed until the blob arrives).
+  var seen = Object.create(null)
+  var wanted = Object.create(null)
+  var stats = { scanned: 0, refs: 0, already: 0, wanted: 0, arrived: 0 }
+
+  // has() is async, so the backfill stream ends long before the lookups finish.
+  // Track them so --once/--dry-run can report real numbers instead of zeros.
+  var pending = 0
+  var onDrained = null
+  function settle () {
+    if (pending === 0 && onDrained) { var f = onDrained; onDrained = null; f() }
+  }
+
+  function consider (ref, why) {
+    if (ref === EMPTY_HASH) return
+    if (seen[ref]) return
+    seen[ref] = true
+    stats.refs++
+
+    pending++
+    sbot.blobs.has(ref, function (err, has) {
+      pending--
+      if (err) { log('has() failed for', ref, '-', err.message || err); return settle() }
+      if (has) { stats.already++; return settle() }
+
+      if (Object.keys(wanted).length >= SANITY_CAP) {
+        log('WARN hit sanity cap of', SANITY_CAP, 'outstanding wants — not adding', ref)
+        return settle()
+      }
+
+      wanted[ref] = true
+      stats.wanted++
+
+      if (DRY_RUN) {
+        log('DRY-RUN would want', ref, '(' + why + ')')
+        return settle()
+      }
+
+      log('want', ref, '(' + why + ')')
+      // The callback fires only if/when the blob actually arrives. If the peer
+      // holding it never appears it simply never fires -- that is the standing
+      // order working as intended, not a leak.
+      sbot.blobs.want(ref, function (err) {
+        if (err) return log('want failed for', ref, '-', err.message || err)
+        stats.arrived++
+        log('ARRIVED', ref)
+      })
+      settle()
+    })
+  }
+
+  function handle (msg, why) {
+    if (!msg || !msg.value) return
+    stats.scanned++
+    blobRefsIn(msg, keys).forEach(function (ref) { consider(ref, why) })
+  }
+
+  log('connected to sbot' + (DRY_RUN ? ' (DRY RUN — nothing will be wanted)' : ''))
+  log('backfilling from the local log…')
+
+  pull(
+    sbot.createLogStream({ keys: true, values: true }),
+    pull.drain(
+      function (msg) { handle(msg, 'backfill') },
+      function (err) {
+        if (err) return bail('backfill stream failed: ' + (err.message || err))
+
+        // Wait for the in-flight has() lookups before reporting or exiting.
+        onDrained = function () {
+          log('backfill done —',
+              'messages=' + stats.scanned,
+              'unique_blobrefs=' + stats.refs,
+              'already_have=' + stats.already,
+              'newly_wanted=' + stats.wanted)
+
+          if (ONCE || DRY_RUN) {
+            log('exiting (' + (DRY_RUN ? '--dry-run' : '--once') + ')')
+            return process.exit(0)
+          }
+          startLive()
+        }
+        settle()
+      }
+    )
+  )
+
+  function startLive () {
+    log('following the log live; standing wants re-arm on every restart')
+    pull(
+      sbot.createLogStream({ keys: true, values: true, live: true, old: false }),
+      pull.drain(
+        function (msg) { handle(msg, 'live') },
+        function (err) {
+          bail('live stream ended' + (err ? ': ' + (err.message || err) : ''))
+        }
+      )
+    )
+
+    // Periodic heartbeat so the log shows the process is alive and what it is
+    // holding, without needing to attach to the screen.
+    setInterval(function () {
+      log('stats —',
+          'msgs=' + stats.scanned,
+          'unique_refs=' + stats.refs,
+          'have=' + stats.already,
+          'standing_wants=' + stats.wanted,
+          'arrived=' + stats.arrived)
+    }, 3600000).unref()
+  }
+})

index.js

@@ -30,7 +30,7 @@ var appHash = hash([fs.readFileSync(__filename)])
 
 var urlIdRegex = /^(?:\/(([%&@]|%25|%26|%40)(?:[A-Za-z0-9\/+]|%2[Ff]|%2[Bb]){43}(?:=|%3[Dd])\.(?:sha256|ed25519))(?:\.([^?]*))?|(\/.*?))(?:\?(.*))?$/
 
-var zeros = new Buffer(24); zeros.fill(0)
+var zeros = Buffer.alloc(24); zeros.fill(0)
 
 function hash(arr) {
   return arr.reduce(function (hash, item) {
@@ -44,6 +44,43 @@ exports.version = require('./package').version
 
 exports.init = function (sbot, config) {
   var conf = config.viewer || {}
+
+  // --- known-blob gate -----------------------------------------------------
+  // This node used to cache blobs on behalf of strangers (ssb-blobs `sympathy`
+  // defaults to 3), and serveBlob would hand any of them to anyone who knew the
+  // hash. sympathy is 0 now and the cache has been pruned, so everything we hold
+  // is referenced by a feed we replicate. This keeps that true even if a stray
+  // blob ever lands.
+  //
+  // The local log is small (~360KB), so a regex scan is cheap. It is refreshed
+  // periodically rather than kept live: being a few minutes stale can only delay
+  // a legitimate image, never serve one that is not ours.
+  var knownBlobs = null   // null = not loaded yet -> fail open, never 404 everything
+  var logOffsetPath = path.join(config.path, 'flume', 'log.offset')
+
+  function refreshKnownBlobs() {
+    fs.readFile(logOffsetPath, function (err, buf) {
+      if (err) {
+        console.error('[viewer] blob gate: could not read log.offset:', err.message)
+        return   // keep whatever set we already had
+      }
+      var found = buf.toString('binary').match(/&[A-Za-z0-9+/]{43}=\.sha256/g) || []
+      var next = Object.create(null)
+      found.forEach(function (id) { next[id] = true })
+      if (knownBlobs === null) {
+        console.log('[viewer] blob gate active:', Object.keys(next).length, 'known blobs')
+      }
+      knownBlobs = next
+    })
+  }
+  refreshKnownBlobs()
+  setInterval(refreshKnownBlobs, 5 * 60 * 1000).unref()
+
+  // serveBlob is a free function outside this closure, so hand the check over.
+  sbot.isKnownBlob = function (id) {
+    if (knownBlobs === null) return true
+    return !!knownBlobs[id]
+  }
   var port = conf.port || 8807
   var host = conf.host || config.host || '::'
 
@@ -474,6 +511,12 @@ function serveBlob(req, res, sbot, id, query) {
   var etag = id + (unbox || '')
 
   if (req.headers['if-none-match'] === etag) return respond(res, 304)
+  // Only serve blobs referenced by a feed we replicate. Anything else is not
+  // ours to hand out, so treat it as absent rather than confirm we hold it.
+  if (typeof sbot.isKnownBlob === 'function' && !sbot.isKnownBlob(id)) {
+    return respond(res, 404, 'Not found')
+  }
+
   sbot.blobs.has(id, function (err, has) {
     if (err) {
       if (/^invalid/.test(err.message)) return respond(res, 400, err.message)
@@ -483,13 +526,13 @@ function serveBlob(req, res, sbot, id, query) {
 
     var unboxKey
     if (unbox) {
-      try { unboxKey = new Buffer(unbox, 'base64') }
+      try { unboxKey = Buffer.from(unbox, 'base64') }
       catch(e) { return respond(res, 400, err.message) }
       if (unboxKey.length !== 32) return respond(res, 400, 'Bad blob key')
     }
 
     res.writeHead(200, {
-      'Cache-Control': 'public, max-age=315360000',
+      'Cache-Control': 'public, max-age=315360000, immutable',
       'etag': etag
     })
 

package.json

@@ -1,9 +1,14 @@
 {
-  "name": "ssb-viewer",
+  "name": "custo-viewer",
   "version": "1.0.0",
-  "description": "serve ssb threads as (embeddable) web pages",
+  "description": "custo's web view of scuttlebutt - the item feed behind www.cust.ooo",
   "main": "index.js",
   "bin": "bin.js",
+  "homepage": "https://git.autonomic.zone/trav/custo-viewer",
+  "repository": {
+    "type": "git",
+    "url": "ssh://git@git.autonomic.zone:2222/trav/custo-viewer.git"
+  },
   "dependencies": {
     "asyncmemo": "^1.0.0",
     "emoji-named-characters": "^1.0.2",
@@ -15,7 +20,7 @@
     "pull-cat": "^1.1.11",
     "pull-paramap": "^1.2.1",
     "pull-stream": "^3.5.0",
-    "ssb-acme-validator": "http://localhost:8989/blobs/get/&MgpztmIbg8wShqDXBcKt0w78qwcpNe8qb4n3fsveve8=.sha256",
+    "ssb-acme-validator": "file:./ssb-acme-validator.tar.gz",
     "ssb-client": "^4.5.2",
     "ssb-marked": "^0.7.3",
     "ssb-ref": "^2.9.0",
@@ -26,6 +31,6 @@
   "devDependencies": {
     "tape": "^4.6.2"
   },
-  "author": "cel",
+  "author": "trav (fork of ssb-viewer by cel)",
   "license": "AGPL-3.0+"
 }

prune-blobs.js

@@ -0,0 +1,163 @@
+#!/usr/bin/env node
+//
+// prune-blobs.js — delete cached blobs that none of our own feeds reference.
+//
+// CONTEXT
+// -------
+// ssb-blobs `sympathy` (default 3) made this node fetch and cache blobs on
+// behalf of any peer up to 3 network hops away. With port 8008 open to the
+// world, that turned it into a free CDN: 31,283 blobs / 4.6 GB, of which only
+// 267 are referenced by any message we hold.
+//
+// Once sympathy is 0 the inflow stops, but the existing hoard stays on disk --
+// and ssb-viewer will serve any of it by hash to anyone who asks. This removes
+// it.
+//
+// SAFETY
+// ------
+//   * Dry run by default. Pass --delete to actually remove anything.
+//   * Every deleted id is written to a manifest first. Blobs are content-
+//     addressed, so anything deleted in error can be re-requested with
+//     `ssb-server blobs.want <id>` if a peer still has it.
+//   * The keep set is built from the raw log, so it does not depend on sbot
+//     being up or on any index being in sync.
+//
+// Blobs are stored by multiblob at blobs/<alg>/<hex[0:2]>/<hex[2:]>, where hex
+// is the base64 blob hash decoded to hex (see multiblob/index.js toPath).
+
+var fs = require('fs')
+var path = require('path')
+
+var DELETE = process.argv.indexOf('--delete') !== -1
+var HOME = process.env.HOME || '/home/cyberian'
+var LOG_OFFSET = path.join(HOME, '.ssb/flume/log.offset')
+var BLOB_DIR = path.join(HOME, '.ssb/blobs/sha256')
+var PUSH_DIR = path.join(HOME, '.ssb/blobs_push')
+var MANIFEST = path.join(HOME, 'logs/pruned-blobs.txt')
+
+var BLOB_RE = /&[A-Za-z0-9+/]{43}=\.sha256/g
+// sha256 of the empty string; ssb-blobs special-cases it and never stores it.
+var EMPTY_HASH = '&47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=.sha256'
+
+function idToHex (id) {
+  return Buffer.from(id.slice(1).replace(/\.sha256$/, ''), 'base64').toString('hex')
+}
+function hexToId (hex) {
+  return '&' + Buffer.from(hex, 'hex').toString('base64') + '.sha256'
+}
+function human (bytes) {
+  var u = ['B', 'KB', 'MB', 'GB']
+  var i = 0
+  while (bytes >= 1024 && i < u.length - 1) { bytes /= 1024; i++ }
+  return bytes.toFixed(1) + ' ' + u[i]
+}
+
+// ---- keep set -------------------------------------------------------------
+var keep = Object.create(null)
+keep[EMPTY_HASH] = true
+
+var log = fs.readFileSync(LOG_OFFSET).toString('binary')
+var refs = log.match(BLOB_RE) || []
+refs.forEach(function (r) { keep[r] = true })
+var fromLog = Object.keys(keep).length - 1
+
+// Anything queued for push to the network is ours by definition. The leveldb
+// is locked by the running sbot, so scrape the ids out of its files instead of
+// opening it.
+var fromPush = 0
+try {
+  fs.readdirSync(PUSH_DIR).forEach(function (f) {
+    if (!/\.(log|ldb)$/.test(f)) return
+    var buf = fs.readFileSync(path.join(PUSH_DIR, f)).toString('binary')
+    ;(buf.match(BLOB_RE) || []).forEach(function (r) {
+      if (!keep[r]) { keep[r] = true; fromPush++ }
+    })
+  })
+} catch (e) {
+  console.error('warning: could not scan blobs_push:', e.message)
+}
+
+// ---- walk the blob store --------------------------------------------------
+var kept = { n: 0, bytes: 0 }
+var doomed = []
+var orphans = []
+
+fs.readdirSync(BLOB_DIR).forEach(function (sub) {
+  var subdir = path.join(BLOB_DIR, sub)
+  var st
+  try { st = fs.statSync(subdir) } catch (e) { return }
+  if (!st.isDirectory()) return
+
+  fs.readdirSync(subdir).forEach(function (name) {
+    var full = path.join(subdir, name)
+    var size
+    try { size = fs.statSync(full).size } catch (e) { return }
+
+    var id
+    try {
+      id = hexToId(sub + name)
+    } catch (e) {
+      orphans.push(full)
+      return
+    }
+
+    if (keep[id]) { kept.n++; kept.bytes += size }
+    else doomed.push({ id: id, path: full, size: size })
+  })
+})
+
+var doomedBytes = doomed.reduce(function (a, b) { return a + b.size }, 0)
+
+console.log('keep set:      ' + (fromLog + fromPush) +
+            ' (' + fromLog + ' referenced in the log, ' + fromPush + ' from blobs_push)')
+console.log('blobs on disk: ' + (kept.n + doomed.length))
+console.log('  keeping:     ' + kept.n + '  (' + human(kept.bytes) + ')')
+console.log('  removing:    ' + doomed.length + '  (' + human(doomedBytes) + ')')
+if (orphans.length) console.log('  unparseable: ' + orphans.length + ' (left alone)')
+
+// A keep-set entry we do not actually have on disk is fine -- blob-wanter has a
+// standing want for it. Just report so the numbers reconcile.
+var missing = Object.keys(keep).filter(function (id) {
+  if (id === EMPTY_HASH) return false
+  var hex = idToHex(id)
+  return !fs.existsSync(path.join(BLOB_DIR, hex.substring(0, 2), hex.substring(2)))
+})
+if (missing.length) {
+  console.log('  note: ' + missing.length + ' referenced blob(s) are not on disk; ' +
+              'blob-wanter holds standing wants for them')
+}
+
+if (!DELETE) {
+  console.log('\nDRY RUN — nothing deleted. Re-run with --delete to apply.')
+  process.exit(0)
+}
+
+// ---- delete ---------------------------------------------------------------
+// Manifest first, so a crash mid-delete still leaves a record of what went.
+fs.mkdirSync(path.dirname(MANIFEST), { recursive: true })
+var out = fs.createWriteStream(MANIFEST, { flags: 'a' })
+out.write('# pruned ' + new Date().toISOString() + ' — ' + doomed.length +
+          ' blobs, ' + human(doomedBytes) + '\n')
+out.write('# re-request any of these with: ssb-server blobs.want <id>\n')
+doomed.forEach(function (d) { out.write(d.id + '\t' + d.size + '\n') })
+out.end()
+
+var removed = 0, failed = 0, freed = 0
+doomed.forEach(function (d) {
+  try { fs.unlinkSync(d.path); removed++; freed += d.size }
+  catch (e) { failed++ }
+})
+
+// Clean up the now-empty two-char shard directories.
+fs.readdirSync(BLOB_DIR).forEach(function (sub) {
+  var subdir = path.join(BLOB_DIR, sub)
+  try {
+    if (fs.statSync(subdir).isDirectory() && fs.readdirSync(subdir).length === 0) {
+      fs.rmdirSync(subdir)
+    }
+  } catch (e) { /* not empty, or vanished */ }
+})
+
+console.log('\nremoved ' + removed + ' blobs, freed ' + human(freed) +
+            (failed ? ', ' + failed + ' failed' : ''))
+console.log('manifest: ' + MANIFEST)

render.js

@@ -1,4 +1,6 @@
+var fs = require('fs')
 var path = require('path')
+var proc = require('child_process')
 var pull = require("pull-stream")
 var marked = require("ssb-marked")
 var htime = require("human-time")
@@ -6,6 +8,7 @@ var emojis = require("emoji-named-characters")
 var cat = require("pull-cat")
 var h = require('hyperscript')
 var refs = require('ssb-ref')
+var pkg = require('./package')
 
 var emojiDir = path.join(require.resolve("emoji-named-characters"), "../pngs")
 
@@ -52,6 +55,18 @@ MdRenderer.prototype.image = function(href, title, text) {
              { type: 'image/svg+xml',
                data: href,
                alt: text }).outerHTML
+  else if (/^video:/.test(text))
+    return h('video', {
+      controls: 'controls',
+      src: this.opts.blob_base + href,
+      title: title || undefined
+    }).outerHTML
+  else if (/^audio:/.test(text))
+    return h('audio', {
+      controls: 'controls',
+      src: this.opts.blob_base + href,
+      title: title || undefined
+    }).outerHTML
   else
     return h('img',
              { src: this.opts.img_base + href,
@@ -157,6 +172,21 @@ function renderRssItem(opts) {
   )
 }
 
+const gitHead = proc.spawnSync('git', ['rev-parse', 'HEAD'], {
+  encoding: 'utf8',
+  cwd: __dirname
+}).stdout.trim()
+const gitHeadShort = gitHead && gitHead.substr(0, 7)
+const commitUrl = pkg.homepage &&
+  pkg.homepage.replace(/\/+$/, '') + '/commit/' + gitHead
+const gitLink = !gitHead ? '' :
+  !commitUrl ? `<code title="${gitHead}">${gitHeadShort}</code>` :
+  `<a href="${commitUrl}" title="${gitHead}"><code>${gitHeadShort}</code></a>`
+
+const footer = `
+<div class=footer>AGPLv3 &copy; <a href="${pkg.homepage}">${pkg.name}</a> ${gitLink}</div>
+`
+
 function wrapPage(id) {
   return wrap(
     "<!doctype html><html><head>" +
@@ -167,7 +197,7 @@ function wrapPage(id) {
       '<meta name=viewport content="width=device-width,initial-scale=1">' +
       styles +
       "</head><body>",
-    "</body></html>"
+    footer + "\n</body></html>"
   )
 }
 
@@ -289,7 +319,7 @@ var styles = `
       text-align: center;
       text-decoration: none;
       margin-top: 20px;
-      margin-bottom: 60px;
+      margin-bottom: 30px;
       background-color: #5c7cfa;
       padding: 15px 0;
       color: #edf2ff;
@@ -303,6 +333,12 @@ var styles = `
     .attending {
       text-align: center;
     }
+    .footer {
+      text-align: center;
+      margin-bottom: 10px;
+      font-size: 14px;
+      color: #868e96;
+    }
   </style>
 `
 

run-blob-wanter.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# run-blob-wanter.sh
+# Same shape as run-server.sh / run-ssb-viewer.sh: keep it alive, and let the
+# process exit cleanly when its sbot connection dies.
+#
+# Restarting on sbot loss is not just tolerated here, it is REQUIRED: sbot's
+# want map is in-memory and is wiped on every sbot restart. Coming back up
+# re-runs the backfill and re-arms every standing want.
+while true; do
+	NODE_OPTIONS="--dns-result-order=ipv4first --max-old-space-size=128" ./blob-wanter.js
+	echo "Restarting blob-wanter at $(date)"
+	sleep 5
+done

run-ssb-viewer.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+# run-ssb-viewer.sh
+while true; do
+	NODE_OPTIONS="--dns-result-order=ipv4first" timeout 6h ./bin.js 
+	echo "Restarting ssb-viewer at $(date)"
+	sleep 2 
+done

startup-howto.md

@@ -0,0 +1 @@
+NODE_OPTIONS="--dns-result-order=ipv4first" ./bin.js