godotenv multiline env vars and unexpected tokens #6

New Issue

decentral1se · 2021-07-20T22:50:08Z

decentral1se commented

2021-07-20 22:50:08 +00:00

While learning/testing abra app ls I ran into the issue mention in with multiline env vars. The following diff seemed to work, pointing to the mentioned fork temporarily. A few env files have multi-line requirements, so it seems we need this.

diff --git a/go.mod b/go.mod
index 4d92852..143782d 100644
--- a/go.mod
+++ b/go.mod
@@ -2,6 +2,9 @@ module coopcloud.tech/abra
 
 go 1.16
 
+// TODO: remove when https://github.com/joho/godotenv/pull/118 is merged
+replace github.com/joho/godotenv => github.com/x1unix/godotenv v1.3.1-0.20200910042738-acd8c1e858a6
+
 require (
        github.com/containerd/containerd v1.5.3 // indirect
        github.com/docker/cli v20.10.7+incompatible

Also, I raised https://github.com/joho/godotenv/issues/150 for another issue. Maybe we need to publish a patch to fix this also?

Godotenv seems quite nice but also kinda buggy. Don't see any alternative right now.

While learning/testing `abra app ls` I ran into the issue mention in with multiline env vars. The following diff seemed to work, pointing to the mentioned fork temporarily. A few env files have multi-line requirements, so it seems we need this. ```diff diff --git a/go.mod b/go.mod index 4d92852..143782d 100644 --- a/go.mod +++ b/go.mod @@ -2,6 +2,9 @@ module coopcloud.tech/abra go 1.16 +// TODO: remove when https://github.com/joho/godotenv/pull/118 is merged +replace github.com/joho/godotenv => github.com/x1unix/godotenv v1.3.1-0.20200910042738-acd8c1e858a6 + require ( github.com/containerd/containerd v1.5.3 // indirect github.com/docker/cli v20.10.7+incompatible ``` Also, I raised https://github.com/joho/godotenv/issues/150 for another issue. Maybe we need to publish a patch to fix this also? Godotenv seems quite nice but also kinda buggy. Don't see any alternative right now.

decentral1se added the

help wanted

label 2021-07-20 22:50:08 +00:00

decentral1se commented

2021-07-20 23:50:59 +00:00

Oh wait, https://github.com/spf13/viper also supports dotenv format... however, it doesn't support multi-line env vars 😢 I see they are using the underyling https://github.com/subosito/gotenv library which would have to implement multi-line support too.

I wonder is this a sign to migrate to YAML? I was able to parse the following using viper:

WORDPRESS_CONFIG_EXTRA: |
  define('MULTISITE', true);
  define('SUBDOMAIN_INSTALL', true);
  define('DOMAIN_CURRENT_SITE', '${DOMAIN}');
  define('PATH_CURRENT_SITE', '/');
  define('SITE_ID_CURRENT_SITE', 1);
  define('BLOG_ID_CURRENT_SITE', 1);
  define('FORCE_SSL_ADMIN', true );
  define('COOKIE_DOMAIN', $_SERVER['HTTP_HOST']);"

func main() {
	v := viper.New()
	v.AddConfigPath(".")
	v.SetConfigType("yml")
	v.SetConfigFile(".env.yml")
	if err := v.ReadInConfig(); err != nil {
		fmt.Printf("Error reading config file, %s", err)
	}
	fmt.Println(os.ExpandEnv(v.GetString("WORDPRESS_CONFIG_EXTRA")))
}

Oh wait, https://github.com/spf13/viper also supports `dotenv` format... however, it doesn't support multi-line env vars 😢 I see they are using the underyling https://github.com/subosito/gotenv library which would have to implement multi-line support too. I wonder is this a sign to migrate to YAML? I was able to parse the following using viper: ```yaml WORDPRESS_CONFIG_EXTRA: | define('MULTISITE', true); define('SUBDOMAIN_INSTALL', true); define('DOMAIN_CURRENT_SITE', '${DOMAIN}'); define('PATH_CURRENT_SITE', '/'); define('SITE_ID_CURRENT_SITE', 1); define('BLOG_ID_CURRENT_SITE', 1); define('FORCE_SSL_ADMIN', true ); define('COOKIE_DOMAIN', $_SERVER['HTTP_HOST']);" ``` ```golang func main() { v := viper.New() v.AddConfigPath(".") v.SetConfigType("yml") v.SetConfigFile(".env.yml") if err := v.ReadInConfig(); err != nil { fmt.Printf("Error reading config file, %s", err) } fmt.Println(os.ExpandEnv(v.GetString("WORDPRESS_CONFIG_EXTRA"))) } ```

decentral1se commented

2021-07-21 00:11:21 +00:00

This may be going on a total tangent but I could imagine we could re-work some of the format of the .env file to be less confusing and based on bash-isms? For example, how secrets are managed.

From our Gitea configs:

SECRET_INTERNAL_TOKEN_VERSION=v1 # length=105
SECRET_DB_PASSWORD_VERSION=v1
SECRET_DB_ROOT_PASSWORD_VERSION=v1
SECRET_JWT_SECRET_VERSION=v1 # length=43
SECRET_SECRET_KEY_VERSION=v1 # length=64
SECRET_SMTP_PASSWORD_VERSION=v1

Could be:

secrets:
  internal_token:
    version: v1
    length: 105
  db_password: v1
  db_root_password: v1
  jwt_secret:
    version: v1
    length: 43
  secret_key:
    version: v1
    length: 63
  smtp_password: v1

Which would be more intuitive since you use those names in the compose.yml file. Then we could change lines like name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION} to something less verbose like name: ${DB_PASSWORD_SECRET} because we have more programmatic control on the loading?

We had originally went with env format because it was easier but maybe it is worth reconsidering this now?

This may be going on a total tangent but I could imagine we could re-work some of the format of the `.env` file to be less confusing and based on bash-isms? For example, how secrets are managed. From our Gitea configs: ``` SECRET_INTERNAL_TOKEN_VERSION=v1 # length=105 SECRET_DB_PASSWORD_VERSION=v1 SECRET_DB_ROOT_PASSWORD_VERSION=v1 SECRET_JWT_SECRET_VERSION=v1 # length=43 SECRET_SECRET_KEY_VERSION=v1 # length=64 SECRET_SMTP_PASSWORD_VERSION=v1 ``` Could be: ```yaml secrets: internal_token: version: v1 length: 105 db_password: v1 db_root_password: v1 jwt_secret: version: v1 length: 43 secret_key: version: v1 length: 63 smtp_password: v1 ``` Which would be more intuitive since you use those names in the `compose.yml` file. Then we could change lines like `name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION}` to something less verbose like `name: ${DB_PASSWORD_SECRET}` because we have more programmatic control on the loading? We had originally went with env format because it was easier but maybe it is worth reconsidering this now?

roxxers commented

2021-07-21 06:34:52 +00:00

I think moving to Viper and then doing what makes sense for the situation is the best way forward sindi it seems to support everything. I will move it to Viper and then we can work on the config later.

🚀 1

decentral1se commented

2021-07-21 10:00:59 +00:00

Out-band-logs lead to a switch to koanf due to:

https://github.com/knadh/koanf/wiki/Comparison-with-spf13-viper

Out-band-logs lead to a switch to koanf due to: > https://github.com/knadh/koanf/wiki/Comparison-with-spf13-viper

roxxers commented

2021-07-21 11:09:13 +00:00

I consider this fixed for now with 0242dfcb0f. Wanna make sure I am not jumping the gun on that. When it comes to like what to do with dealing with apps .env files I think we can make a new issue for this if thats ok with you @decentral1se?

I consider this fixed for now with 0242dfcb0f83e3ca906461b0b3e55f1832d5d528. Wanna make sure I am not jumping the gun on that. When it comes to like what to do with dealing with apps .env files I think we can make a new issue for this if thats ok with you @decentral1se?

decentral1se commented

2021-07-21 11:25:39 +00:00

💯

decentral1se closed this issue

2021-07-21 11:25:39 +00:00

3wordchant commented

2021-07-21 11:38:43 +00:00

@decentral1se @roxxers I think I'm sold on the benefits of YAML -- the one thing we'll lose is the ability to pick up the config files using vanilla Docker, because AFAIK Docker only supports env files.

Maybe we don't need to give as much assurance to folks for the "whoops Co-op Cloud disappeared" situation now that we're becoming more of a thing? But if that still does feel useful, maybe the solution is a new command that can export an app config in env format?

@decentral1se @roxxers I think I'm sold on the benefits of YAML -- the one thing we'll lose is the ability to pick up the config files using vanilla Docker, because AFAIK Docker only supports env files. Maybe we don't need to give as much assurance to folks for the "whoops Co-op Cloud disappeared" situation now that we're becoming more of a thing? But if that still does feel useful, maybe the solution is a new command that can export an app config in env format?

decentral1se referenced this issue from coop-cloud/organising

2021-07-21 14:25:04 +00:00

The idea of using yaml instead of env file format for app env config #105

This repo is archived. You cannot comment on issues.

No Label