Write a script to generate abra-apps.json #121

New Issue

Closed

opened 2021-03-22 22:21:24 +00:00 by decentral1se · 5 comments

decentral1se commented 2021-03-22 22:21:24 +00:00

Owner

Copy Link

Part of coop-cloud/organising#51.

We discussed Python for this. We can re-use the installer logic to host this at https://abra-apps.cloud.autonomic.zone. This will be downloaded by abra to learn about what apps, versions and other metadata are available.

The script needs to read the ~/.abra/apps/... contents and generate something that looks like the JSON example in coop-cloud/organising#51 (comment). It needs to read tags and compose.yml files to get the version listings as well as the README.md metdata values for each app.

Here is a script to pull all the apps down btw.

1006c8a58d/scripts/clone.py

Part of https://git.autonomic.zone/coop-cloud/organising/issues/51. We discussed Python for this. We can re-use the [installer](https://git.autonomic.zone/coop-cloud/abra/src/branch/main/installer) logic to host this at https://abra-apps.cloud.autonomic.zone. This will be downloaded by `abra` to learn about what apps, versions and other metadata are available. The script needs to read the `~/.abra/apps/...` contents and generate something that looks like the JSON example in https://git.autonomic.zone/coop-cloud/organising/issues/51#issuecomment-4448. It needs to read tags and `compose.yml` files to get the version listings as well as the `README.md` metdata values for each app. Here is a script to pull all the apps down btw. > https://git.autonomic.zone/coop-cloud/gardening/src/commit/1006c8a58d1717b8969d677ea4148016e71126c9/scripts/clone.py

decentral1se added this to the Beta release milestone 2021-03-22 22:21:24 +00:00

decentral1se added the

enhancement

label 2021-03-22 22:21:24 +00:00

decentral1se referenced this issue from a commit 2021-03-26 00:14:34 +00:00

First stab at the apps JSON generation script

decentral1se commented 2021-04-02 14:09:27 +00:00

Author

Owner

Copy Link

FYI here is the useful output of skopeo when inspecting an image:

(.venv) ➜  src (main) ✔ skopeo inspect docker://gitea/gitea:latest
{
    "Name": "docker.io/gitea/gitea",
    "Digest": "sha256:b3a9c4bc88bf492423b7e59c158ee4d204027c8e659ec8cd5cafff213ddfb599",
    "RepoTags": [
        "1-linux-amd64",
        "1-linux-arm64",
        "1.0.0",
        "1.0.1",
        "1.0.2",
        "1.0",
        "1.1.0",
        "1.1.1",
        "1.1.2",
        "1.1.3",
        "1.1.4",
        "1.1",
        "1.10-linux-amd64",
        "1.10-linux-arm64",
        "1.10.0-linux-amd64",
        "1.10.0-linux-arm64",
        "1.10.0-rc1-linux-amd64",
        "1.10.0-rc1-linux-arm64",
        "1.10.0-rc1",
        "1.10.0-rc2-linux-amd64",
        "1.10.0-rc2-linux-arm64",
        "1.10.0-rc2",
        "1.10.0",
        "1.10.1-linux-amd64",
        "1.10.1-linux-arm64",
        "1.10.1",
        "1.10.2-linux-amd64",
        "1.10.2-linux-arm64",
        "1.10.2",
        "1.10.3-linux-amd64",
        "1.10.3-linux-arm64",
        "1.10.3",
        "1.10.4-linux-amd64",
        "1.10.4-linux-arm64",
        "1.10.4",
        "1.10.5-linux-amd64",
        "1.10.5-linux-arm64",
        "1.10.5",
        "1.10.6-linux-amd64",
        "1.10.6-linux-arm64",
        "1.10.6",
        "1.10",
        "1.11-linux-amd64",
        "1.11-linux-arm64",
        "1.11.0-linux-amd64",
        "1.11.0-linux-arm64",
        "1.11.0-rc1-linux-amd64",
        "1.11.0-rc1-linux-arm64",
        "1.11.0-rc1",
        "1.11.0-rc2-linux-amd64",
        "1.11.0-rc2-linux-arm64",
        "1.11.0-rc2",
        "1.11.0",
        "1.11.1-linux-amd64",
        "1.11.1-linux-arm64",
        "1.11.1",
        "1.11.2-linux-amd64",
        "1.11.2-linux-arm64",
        "1.11.2",
        "1.11.3-linux-amd64",
        "1.11.3-linux-arm64",
        "1.11.3",
        "1.11.4-linux-amd64",
        "1.11.4-linux-arm64",
        "1.11.4",
        "1.11.5-linux-amd64",
        "1.11.5-linux-arm64",
        "1.11.5",
        "1.11.6-linux-amd64",
        "1.11.6-linux-arm64",
        "1.11.6",
        "1.11.7-linux-amd64",
        "1.11.7-linux-arm64",
        "1.11.7",
        "1.11.8-linux-amd64",
        "1.11.8-linux-arm64",
        "1.11.8",
        "1.11",
        "1.12-linux-amd64",
        "1.12-linux-arm64",
        "1.12.0-linux-amd64",
        "1.12.0-linux-arm64",
        "1.12.0-rc1-linux-amd64",
        "1.12.0-rc1-linux-arm64",
        "1.12.0-rc1",
        "1.12.0-rc2-linux-amd64",
        "1.12.0-rc2-linux-arm64",
        "1.12.0-rc2",
        "1.12.0",
        "1.12.1-linux-amd64",
        "1.12.1-linux-arm64",
        "1.12.1",
        "1.12.2-linux-amd64",
        "1.12.2-linux-arm64",
        "1.12.2",
        "1.12.3-linux-amd64",
        "1.12.3-linux-arm64",
        "1.12.3",
        "1.12.4-linux-amd64",
        "1.12.4-linux-arm64",
        "1.12.4",
        "1.12.5-linux-amd64",
        "1.12.5-linux-arm64",
        "1.12.5",
        "1.12.6-linux-amd64",
        "1.12.6-linux-arm64",
        "1.12.6",
        "1.12",
        "1.13-linux-amd64",
        "1.13-linux-arm64",
        "1.13.0-linux-amd64",
        "1.13.0-linux-arm64",
        "1.13.0-rc1-linux-amd64",
        "1.13.0-rc1-linux-arm64",
        "1.13.0-rc1",
        "1.13.0-rc2-linux-amd64",
        "1.13.0-rc2-linux-arm64",
        "1.13.0-rc2",
        "1.13.0",
        "1.13.1-linux-amd64",
        "1.13.1-linux-arm64",
        "1.13.1",
        "1.13.2-linux-amd64",
        "1.13.2-linux-arm64",
        "1.13.2",
        "1.13.3-linux-amd64",
        "1.13.3-linux-arm64",
        "1.13.3",
        "1.13.4-linux-amd64",
        "1.13.4-linux-arm64",
        "1.13.4",
        "1.13.5-linux-amd64",
        "1.13.5-linux-arm64",
        "1.13.5",
        "1.13.6-linux-amd64",
        "1.13.6-linux-arm64",
        "1.13.6",
        "1.13",
        "1.14.0-dev",
        "1.14.0-rc1-linux-amd64-rootless",
        "1.14.0-rc1-linux-amd64",
        "1.14.0-rc1-linux-arm64-rootless",
        "1.14.0-rc1-linux-arm64",
        "1.14.0-rc1-rootless",
        "1.14.0-rc1",
        "1.14.0-rc2-linux-amd64-rootless",
        "1.14.0-rc2-linux-amd64",
        "1.14.0-rc2-linux-arm64-rootless",
        "1.14.0-rc2-linux-arm64",
        "1.14.0-rc2-rootless",
        "1.14.0-rc2",
        "1.2.0-rc1",
        "1.2.0-rc2",
        "1.2.0-rc3",
        "1.2.0-rc4",
        "1.2.0",
        "1.2.1",
        "1.2.2",
        "1.2.3",
        "1.2",
        "1.3.0-rc1",
        "1.3.0-rc2",
        "1.3.0",
        "1.3.1",
        "1.3.2",
        "1.3.3",
        "1.3",
        "1.4.0-rc1",
        "1.4.0-rc2",
        "1.4.0-rc3",
        "1.4.0",
        "1.4.1",
        "1.4.2",
        "1.4.3",
        "1.4",
        "1.5.0-dev",
        "1.5.0-rc1",
        "1.5.0-rc2",
        "1.5.0",
        "1.5.1",
        "1.5.2",
        "1.5.3",
        "1.5",
        "1.6.0-dev",
        "1.6.0-rc1",
        "1.6.0-rc2",
        "1.6.0",
        "1.6.1",
        "1.6.2",
        "1.6.3",
        "1.6.4",
        "1.6",
        "1.7.0-dev",
        "1.7.0-rc1",
        "1.7.0-rc2",
        "1.7.0-rc3",
        "1.7.0",
        "1.7.1",
        "1.7.2",
        "1.7.3",
        "1.7.4",
        "1.7.5",
        "1.7.6",
        "1.7",
        "1.8.0-rc1",
        "1.8.0-rc2",
        "1.8.0-rc3",
        "1.8.0",
        "1.8.1",
        "1.8.2",
        "1.8.3",
        "1.8",
        "1.9-linux-amd64",
        "1.9-linux-arm64",
        "1.9.0",
        "1.9.1",
        "1.9.2-linux-amd64",
        "1.9.2-linux-arm64",
        "1.9.2",
        "1.9.3-linux-amd64",
        "1.9.3-linux-arm64",
        "1.9.3",
        "1.9.4-linux-amd64",
        "1.9.4-linux-arm64",
        "1.9.4",
        "1.9.5-linux-amd64",
        "1.9.5-linux-arm64",
        "1.9.5",
        "1.9.6-linux-amd64",
        "1.9.6-linux-arm64",
        "1.9.6",
        "1.9",
        "1",
        "latest-rootless",
        "latest",
        "linux-amd64-rootless",
        "linux-amd64",
        "linux-arm64-rootless",
        "linux-arm64"
    ],
    "Created": "2021-04-01T20:41:21.960935882Z",
    "DockerVersion": "19.03.8",
    "Labels": {
        "maintainer": "maintainers@gitea.io",
        "org.opencontainers.image.created": "2021-04-01T20:37:39Z",
        "org.opencontainers.image.revision": "bc1f2117f1be539e971c90877658a8e456ce9e25",
        "org.opencontainers.image.source": "https://github.com/go-gitea/gitea.git",
        "org.opencontainers.image.url": "https://github.com/go-gitea/gitea"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15",
        "sha256:294b90a41b52b847fd117f0c7c49da453c677a4abf8aa0911ff94e5c413ce515",
        "sha256:aee94aa25f967c6459386aa9f58bd28f8948efee5afdae4a7e4fe961048003d6",
        "sha256:24ae5978ee66f0e0a57a5e69a41913cc92f410eb48cb5dabc2e99e02d5e9c2b6",
        "sha256:a394877e26399263d3bf185dee711c758b4046de0706dba0e108934b136dd3d7",
        "sha256:44106a6814bfda7a49188b0d587346f22337b8efa71243aa3ea515ddd694ec56",
        "sha256:ef213b39f224518de5a03b92cdcbc83bedbda9bdf36cbe28e7676517104cf881"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "USER=git",
        "GITEA_CUSTOM=/data/gitea"
    ]
}

FYI here is the useful output of `skopeo` when inspecting an image: ``` (.venv) ➜ src (main) ✔ skopeo inspect docker://gitea/gitea:latest { "Name": "docker.io/gitea/gitea", "Digest": "sha256:b3a9c4bc88bf492423b7e59c158ee4d204027c8e659ec8cd5cafff213ddfb599", "RepoTags": [ "1-linux-amd64", "1-linux-arm64", "1.0.0", "1.0.1", "1.0.2", "1.0", "1.1.0", "1.1.1", "1.1.2", "1.1.3", "1.1.4", "1.1", "1.10-linux-amd64", "1.10-linux-arm64", "1.10.0-linux-amd64", "1.10.0-linux-arm64", "1.10.0-rc1-linux-amd64", "1.10.0-rc1-linux-arm64", "1.10.0-rc1", "1.10.0-rc2-linux-amd64", "1.10.0-rc2-linux-arm64", "1.10.0-rc2", "1.10.0", "1.10.1-linux-amd64", "1.10.1-linux-arm64", "1.10.1", "1.10.2-linux-amd64", "1.10.2-linux-arm64", "1.10.2", "1.10.3-linux-amd64", "1.10.3-linux-arm64", "1.10.3", "1.10.4-linux-amd64", "1.10.4-linux-arm64", "1.10.4", "1.10.5-linux-amd64", "1.10.5-linux-arm64", "1.10.5", "1.10.6-linux-amd64", "1.10.6-linux-arm64", "1.10.6", "1.10", "1.11-linux-amd64", "1.11-linux-arm64", "1.11.0-linux-amd64", "1.11.0-linux-arm64", "1.11.0-rc1-linux-amd64", "1.11.0-rc1-linux-arm64", "1.11.0-rc1", "1.11.0-rc2-linux-amd64", "1.11.0-rc2-linux-arm64", "1.11.0-rc2", "1.11.0", "1.11.1-linux-amd64", "1.11.1-linux-arm64", "1.11.1", "1.11.2-linux-amd64", "1.11.2-linux-arm64", "1.11.2", "1.11.3-linux-amd64", "1.11.3-linux-arm64", "1.11.3", "1.11.4-linux-amd64", "1.11.4-linux-arm64", "1.11.4", "1.11.5-linux-amd64", "1.11.5-linux-arm64", "1.11.5", "1.11.6-linux-amd64", "1.11.6-linux-arm64", "1.11.6", "1.11.7-linux-amd64", "1.11.7-linux-arm64", "1.11.7", "1.11.8-linux-amd64", "1.11.8-linux-arm64", "1.11.8", "1.11", "1.12-linux-amd64", "1.12-linux-arm64", "1.12.0-linux-amd64", "1.12.0-linux-arm64", "1.12.0-rc1-linux-amd64", "1.12.0-rc1-linux-arm64", "1.12.0-rc1", "1.12.0-rc2-linux-amd64", "1.12.0-rc2-linux-arm64", "1.12.0-rc2", "1.12.0", "1.12.1-linux-amd64", "1.12.1-linux-arm64", "1.12.1", "1.12.2-linux-amd64", "1.12.2-linux-arm64", "1.12.2", "1.12.3-linux-amd64", "1.12.3-linux-arm64", "1.12.3", "1.12.4-linux-amd64", "1.12.4-linux-arm64", "1.12.4", "1.12.5-linux-amd64", "1.12.5-linux-arm64", "1.12.5", "1.12.6-linux-amd64", "1.12.6-linux-arm64", "1.12.6", "1.12", "1.13-linux-amd64", "1.13-linux-arm64", "1.13.0-linux-amd64", "1.13.0-linux-arm64", "1.13.0-rc1-linux-amd64", "1.13.0-rc1-linux-arm64", "1.13.0-rc1", "1.13.0-rc2-linux-amd64", "1.13.0-rc2-linux-arm64", "1.13.0-rc2", "1.13.0", "1.13.1-linux-amd64", "1.13.1-linux-arm64", "1.13.1", "1.13.2-linux-amd64", "1.13.2-linux-arm64", "1.13.2", "1.13.3-linux-amd64", "1.13.3-linux-arm64", "1.13.3", "1.13.4-linux-amd64", "1.13.4-linux-arm64", "1.13.4", "1.13.5-linux-amd64", "1.13.5-linux-arm64", "1.13.5", "1.13.6-linux-amd64", "1.13.6-linux-arm64", "1.13.6", "1.13", "1.14.0-dev", "1.14.0-rc1-linux-amd64-rootless", "1.14.0-rc1-linux-amd64", "1.14.0-rc1-linux-arm64-rootless", "1.14.0-rc1-linux-arm64", "1.14.0-rc1-rootless", "1.14.0-rc1", "1.14.0-rc2-linux-amd64-rootless", "1.14.0-rc2-linux-amd64", "1.14.0-rc2-linux-arm64-rootless", "1.14.0-rc2-linux-arm64", "1.14.0-rc2-rootless", "1.14.0-rc2", "1.2.0-rc1", "1.2.0-rc2", "1.2.0-rc3", "1.2.0-rc4", "1.2.0", "1.2.1", "1.2.2", "1.2.3", "1.2", "1.3.0-rc1", "1.3.0-rc2", "1.3.0", "1.3.1", "1.3.2", "1.3.3", "1.3", "1.4.0-rc1", "1.4.0-rc2", "1.4.0-rc3", "1.4.0", "1.4.1", "1.4.2", "1.4.3", "1.4", "1.5.0-dev", "1.5.0-rc1", "1.5.0-rc2", "1.5.0", "1.5.1", "1.5.2", "1.5.3", "1.5", "1.6.0-dev", "1.6.0-rc1", "1.6.0-rc2", "1.6.0", "1.6.1", "1.6.2", "1.6.3", "1.6.4", "1.6", "1.7.0-dev", "1.7.0-rc1", "1.7.0-rc2", "1.7.0-rc3", "1.7.0", "1.7.1", "1.7.2", "1.7.3", "1.7.4", "1.7.5", "1.7.6", "1.7", "1.8.0-rc1", "1.8.0-rc2", "1.8.0-rc3", "1.8.0", "1.8.1", "1.8.2", "1.8.3", "1.8", "1.9-linux-amd64", "1.9-linux-arm64", "1.9.0", "1.9.1", "1.9.2-linux-amd64", "1.9.2-linux-arm64", "1.9.2", "1.9.3-linux-amd64", "1.9.3-linux-arm64", "1.9.3", "1.9.4-linux-amd64", "1.9.4-linux-arm64", "1.9.4", "1.9.5-linux-amd64", "1.9.5-linux-arm64", "1.9.5", "1.9.6-linux-amd64", "1.9.6-linux-arm64", "1.9.6", "1.9", "1", "latest-rootless", "latest", "linux-amd64-rootless", "linux-amd64", "linux-arm64-rootless", "linux-arm64" ], "Created": "2021-04-01T20:41:21.960935882Z", "DockerVersion": "19.03.8", "Labels": { "maintainer": "maintainers@gitea.io", "org.opencontainers.image.created": "2021-04-01T20:37:39Z", "org.opencontainers.image.revision": "bc1f2117f1be539e971c90877658a8e456ce9e25", "org.opencontainers.image.source": "https://github.com/go-gitea/gitea.git", "org.opencontainers.image.url": "https://github.com/go-gitea/gitea" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15", "sha256:294b90a41b52b847fd117f0c7c49da453c677a4abf8aa0911ff94e5c413ce515", "sha256:aee94aa25f967c6459386aa9f58bd28f8948efee5afdae4a7e4fe961048003d6", "sha256:24ae5978ee66f0e0a57a5e69a41913cc92f410eb48cb5dabc2e99e02d5e9c2b6", "sha256:a394877e26399263d3bf185dee711c758b4046de0706dba0e108934b136dd3d7", "sha256:44106a6814bfda7a49188b0d587346f22337b8efa71243aa3ea515ddd694ec56", "sha256:ef213b39f224518de5a03b92cdcbc83bedbda9bdf36cbe28e7676517104cf881" ], "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "USER=git", "GITEA_CUSTOM=/data/gitea" ] } ```

decentral1se commented 2021-04-02 14:57:19 +00:00

Author

Owner

Copy Link

Ok, here is the status:

• https://abra-apps.cloud.autonomic.zone
• https://git.autonomic.zone/coop-cloud/abra#generating-a-new-abra-apps-json
• https://git.autonomic.zone/coop-cloud/abra#abra-apps-cloud-autonomic-zone

Remaining things to do are:

• Tag each app and re-run the script to check it works
• Things to keep in mind:
  • it parses the features correctly
  • it can parse the versions of each git tag

The script should be assumed to largely broken / missing edge case coverage :)

Will go slow adding to it. It's modelled off the Gitea app README and setup.

Where things could be improved:

• Add logging to the script
• Add some sort of "start from where left off" because it is a slow script
• Check if we can speed up skopeo in some way (slowest part)

Ok, here is the status: - https://abra-apps.cloud.autonomic.zone - https://git.autonomic.zone/coop-cloud/abra#generating-a-new-abra-apps-json - https://git.autonomic.zone/coop-cloud/abra#abra-apps-cloud-autonomic-zone Remaining things to do are: - Tag each app and re-run the script to check it works - Things to keep in mind: - it parses the features correctly - it can parse the versions of each git tag The script should be assumed to largely broken / missing edge case coverage :) Will go slow adding to it. It's modelled off the Gitea app README and setup. Where things could be improved: - Add logging to the script - Add some sort of "start from where left off" because it is a slow script - Check if we can speed up skopeo in some way (slowest part)

decentral1se referenced this issue 2021-04-02 15:00:20 +00:00

Setup Drone CI cronjob to build & publish abra-apps.json #125

decentral1se commented 2021-04-02 15:01:46 +00:00

Author

Owner

Copy Link

Given that #125 might be the repeating generation logic, it would be great if we could pass a specific app name to the script to generate only the values needed for that app and then it could read and update the existing JSON file. This would allow us to update specific apps on the fly when we need and we're in a rush.

Given that https://git.autonomic.zone/coop-cloud/abra/issues/125 might be the repeating generation logic, it would be great if we could pass a specific app name to the script to generate only the values needed for that app and then it could read and update the existing JSON file. This would allow us to update specific apps on the fly when we need and we're in a rush.

decentral1se referenced this issue from coop-cloud/organising 2021-04-02 15:19:54 +00:00

Ensure users know what is happening with updated versions coming down the pipes #55

3wordchant commented 2021-04-02 15:23:08 +00:00

Owner

Copy Link

Also wondering if we can easily make this output the Markdown for the app catalogue page, and drop the separate app-catalogue.sh?

Also wondering if we can easily make this output the Markdown for [the app catalogue page](https://docs.cloud.autonomic.zone/apps/), and drop the separate `app-catalogue.sh`?

❤️ 1

decentral1se commented 2021-04-02 19:06:47 +00:00

Author

Owner

Copy Link

OK, this is probably good enough for now. If we wanna do further work, we can make tickets for that. I've released a new version which shows that the README parsing works across all the app repos. Also, I've fixed the versioning handling to match what we first imagined (versions contains a dict where all the keys are version values, not lists of service versions, those are now nested).

OK, this is probably good enough for now. If we wanna do further work, we can make tickets for that. I've released a new version which shows that the README parsing works across all the app repos. Also, I've fixed the versioning handling to match what we first imagined (`versions` contains a dict where all the keys are version values, not lists of service versions, those are now nested).

❤️ 1

decentral1se closed this issue 2021-04-02 19:06:47 +00:00

This repo is archived. You cannot comment on issues.

No Branch/Tag Specified

Branches Tags

main

requirements-install-script

prefer-fast-option

add-bump-logic

prompt-cleanup

fix-pwgen

fix-subcommand-select

digest-version

backup_restore

command_help_2

merge-logging

monorepo

10.0.0

9.0.0

8.0.1

8.0.0

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

checkout

0.6.0

0.5.0

0.4.1

0.4.0

0.3.1

0.3.0

0.2.0

0.1.2

0.1.1

0.1.0

Labels

Clear labels   

breaking-change

Anything that breaks the existing API

  

bug

Something is not working

  

CI/CD

Anything to do with testing and infra

  

design

Anything to with UX of this tool

  

documentation

Documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

plugin

Related to the plugin architecture

  

question

More information is needed

  

secrets

Everything to do with docker secrets

  

shell-completion

To do with shell completion

  

versioning

All the things to do with versioning apps

  

wontfix

This won't be fixed

No Label

breaking-change

bug

CI/CD

design

documentation

duplicate

enhancement

help wanted

invalid

plugin

question

secrets

shell-completion

versioning

wontfix

Milestone

Clear milestone

No items

No Milestone

Beta release

Assignees

Clear assignees

No Assignees

2 Participants

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/abra#121

No description provided.