Brings together small and dispersed streams of web content from different applications and websites together in a single large stream.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Aadil Ayub 3055ee37df
Merge pull request 'adjust templates to new taxonomy' (#43) from new-taxonomy into main
6 months ago
konfluks adjust templates to new taxonomy 9 months ago
.gitignore feat: pull posts from mastodon 1 year ago add agplv3 license 1 year ago fix links 11 months ago
konfluks.svg additional gardening for konfluks rename 11 months ago
poetry.lock Revert "feed: move to saneyaml" 1 year ago
pyproject.toml additional gardening for konfluks rename 11 months ago additional gardening for konfluks rename 11 months ago

Konfluks logo is a stylized and schematic representation of a drainage basin


A drainage basin is a geographical feature that collects all precipitation in an area, first in to smaller streams and finally together in to the large river. Similarly, Konfluks can bring together small and dispersed streams of web content from different applications and websites together in a single large stream.

Specifically, Konfluks turns Peertube videos, iCal calendar events, other websites through their RSS and OPDS feeds and Mastodon posts under a hashtag in to Hugo page bundles. This allows one to publish from diverse sources to a single stream.

Konfluks was first made by Roel Roscam Abbing as part of, together with ruangrupa and Autonomic.


Konfluks tries to act as a mirror representation of the input sources. That means that whenever something remote is deleted, changed or becomes unavailable, it is also changed or deleted by Konfluks.

Konfluks tries to preserve intention. That means the above, but also by requiring explicit ways of publishing.

Konfluks works by periodically polling the remote sources, taking care not to duplicate work. It caches files, asks for last-modified headers, and skips things it has already. This makes every poll as fast and as light as possible.

Konfluks is written for clarity, not brevity nor cleverness.

Konfluks is extendable, a work in progress and a messy undertaking.

High-level overview

Konfluks consists of different Python scripts which each poll a particular service, say, a Peertube server, to download information and convert it in to Hugo Page Bundles

Each script part of Konfluks will essentially to the following:

  • Parse a source and request posts/updates/videos/a feed

    • Taking care of publish ques
  • Create a Hugo post for each item returned, by:

    • Making a folder per post in the output directory
    • Formatting post metadata as Hugo Post Frontmatter in a file called
    • Grabbing local copies of media and saving them in the post folder
    • Adding the post content to
    • According to jinja2 templates (see konfluks/templates/)

The page bundles created, where possible, are given human friendly names.

Here is a typical output structure:

  user@server: ~/konfluks/output: tree tv/
  ├── forum-27an-mother-earth-353f93f3-5fee-49d6-b71d-8aef753f7041
  │   ├── 86ccae63-3df9-443c-91f3-edce146055db.jpg
  │   └──
  ├── keroncong-tugu-cafrinho-live-at-ruru-gallery-ruangrupa-jakarta-19-august-2014-e6d5bb2a-d77f-4a00-a449-992a579c8c0d
  │   ├── 32291aa2-a391-4219-a413-87521ff373ba.jpg
  │   └──
  ├── lecture-series-1-camp-notes-on-education-8d54d3c9-0322-42af-ab6e-e954d251e076
  │   ├── 0f3c835b-42c2-48a3-a2a3-a75ddac8688a.jpg
  │   └──


Install poetry:

curl -sSL | python -

We use Poetry because it locks the dependencies all the way down and makes it easier to manage installation & maintenance in the long-term. Then install the dependencies & have them managed by Poetry:

poetry install

Each script requires some environment variables to run, you can see the latest deployment configuration over here, look for the values under the environment: ... stanza.

All scripts have an entrypoint described in the pypoetry.toml which you can run via poetry run .... For example, if you want to run the konfluks/ script, you'd do:

mkdir -p testdir
export OUTPUT_DIR=/testdir
poetry run konfluks-vid

Run poetry run poetry2setup > if updating the poetry dependencies. This allows us to run pip install . in the deployment and Pip will understand that it is just a regular Python package. If adding a new cli command, extend pyproject.toml with a new [tool.poetry.scripts] entry.