Making ensure_stack_deployed more reliable #165

New Issue

decentral1se · 2021-05-17T10:39:40Z

decentral1se commented

2021-05-17 10:39:40 +00:00

See 4e84664310/abra (L766-L808). I've seen this not really work a bunch of times. I am thinking it might be worth dropping into Python for this, in a separately written utility which can be vendored into ~/.abra/vendor for use by abra. Then we can cover all the nitty-gritty edge cases of tracking the deployment state and ensuring what happened with the deployment. This should be handled by swarmkit but it hasn't been implemented in years so it is unlikely to happen soon at current status. Not knowing if what you ran a deploy on actually succeeded or not is a huge issue.

See https://git.autonomic.zone/coop-cloud/abra/src/commit/4e84664310c7a8c495c544a2828de54e14825baf/abra#L766-L808. I've seen this not really work a bunch of times. I am thinking it might be worth dropping into Python for this, in a separately written utility which can be vendored into `~/.abra/vendor` for use by `abra`. Then we can cover all the nitty-gritty edge cases of tracking the deployment state and ensuring what happened with the deployment. This should be handled by swarmkit but it hasn't been implemented in years so it is unlikely to happen soon at current status. Not knowing if what you ran a deploy on actually succeeded or not is a huge issue.

decentral1se added this to the Beta release milestone 2021-05-17 10:39:40 +00:00

decentral1se added the

enhancement

label 2021-05-17 10:39:40 +00:00

decentral1se commented

2021-05-17 10:41:11 +00:00

The main edge case I've seen is that the new container is deployed successfully but it immediately goes into a flip-flop of up/down/up/down status with obvious error messages in the logs.

Sometimes, you haven't configured a secret or some config issue and the container doesn't even start. There are no container logs and it is really hard to know what went wrong (in some cases, there are simply no logs at all!).

The main edge case I've seen is that the new container is deployed successfully but it immediately goes into a flip-flop of up/down/up/down status with obvious error messages in the logs. Sometimes, you haven't configured a secret or some config issue and the container doesn't even start. There are no container logs and it is really hard to know what went wrong (in some cases, there are simply no logs at all!).

decentral1se referenced this issue from a commit

2021-05-31 21:23:29 +00:00

Add --no-state-poll

decentral1se commented

2021-05-31 21:28:14 +00:00

Seems like we could use the approach of https://stackoverflow.com/a/57403566. We could first list all container hashes and then feed them into a Python script which asynchronously watches all events for those containers and if it sees die/kill/fail events then it could report failure. It could also wait a few seconds once it comes up to make sure the container isn't flapping.

decentral1se referenced this issue

2021-05-31 21:56:00 +00:00

Reconsidering domain polling for now #169

decentral1se referenced this issue

2021-05-31 22:06:19 +00:00

Add `abra app <app> health` to check health of app #117

decentral1se referenced this issue from coop-cloud/organising

2021-06-04 14:28:11 +00:00

App CI/CD test suite brainstorming #69

decentral1se commented

2021-06-07 08:53:33 +00:00

Actually, I take it all back. This can be done in Bash 🐊 We can improve the current implementation. Just need to handle the service container id shuffling and checking the state correctly. If you run docker events when running a deploy you can see the events listing. Those are also viewable in 5a5494a9a7/api/types.pb.go (L421-L428)

Actually, I take it all back. This can be done in Bash 🐊 We can improve the current implementation. Just need to handle the service container id shuffling and checking the state correctly. If you run `docker events` when running a deploy you can see the events listing. Those are also viewable in https://github.com/docker/swarmkit/blob/5a5494a9a7b408b790533a5e4e1cb43ca1c32aad/api/types.pb.go#L421-L428

❤️ 1

decentral1se referenced this issue

2021-06-08 09:14:50 +00:00

Prefer --fast for skipping all checks #175

decentral1se self-assigned this 2021-06-08 12:34:16 +00:00

decentral1se referenced this issue

2021-06-09 08:32:58 +00:00

Make ensure_stack_deployed more reliable #177

decentral1se referenced this issue from a commit

2021-06-27 19:01:30 +00:00

`ensure_stack_deployed` is now somewhat more reliable

decentral1se referenced this issue from a commit