Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exported vars are not evaluated recursively in the context of nested params #862

Open
civitaspo opened this issue Sep 14, 2018 · 2 comments · May be fixed by #1827
Open

Exported vars are not evaluated recursively in the context of nested params #862

civitaspo opened this issue Sep 14, 2018 · 2 comments · May be fixed by #1827
Labels
enhancement

Comments

@civitaspo
Copy link
Contributor

summary

Exported vars are not evaluated recursively in the context of nested params

the dig file to reproduce

_export:
  a: aaa
  b: bbb
  c: ccc

+a:
  _export:
    d: ${a}-${b}-${c}

  +b:
    _export:
      e: ${d}
      f:
        g: ${d}
    +c:
      echo>: ${d}
    +d:
      echo>: ${e}
    +e:
      echo>: ${f.g}

the result of the dig file

You can see that only the nested param ${f.g} is not evaluated recursively.

$ digdag run ./var.dig --no-save
2018-09-14 13:18:02 +0900: Digdag v0.9.27
2018-09-14 13:18:03 +0900 [WARN] (main): Using a new session time 2018-09-14T00:00:00+00:00.
2018-09-14 13:18:03 +0900 [INFO] (main): Starting a new session project id=1 workflow name=var session_time=2018-09-14T00:00:00+00:00
2018-09-14 13:18:04 +0900 [INFO] (0016@[0:default]+var+a+b+c): echo>: aaa-bbb-ccc
aaa-bbb-ccc
2018-09-14 13:18:05 +0900 [INFO] (0016@[0:default]+var+a+b+d): echo>: aaa-bbb-ccc
aaa-bbb-ccc
2018-09-14 13:18:05 +0900 [INFO] (0016@[0:default]+var+a+b+e): echo>: ${a}-${b}-${c}
${a}-${b}-${c}
Success.
  * Use --session <daily | hourly | "yyyy-MM-dd[ HH:mm:ss]"> to not reuse the last session time.
  * Use --rerun, --start +NAME, or --goal +NAME argument to rerun skipped tasks.

Ref
@civitaspo
Copy link
Contributor Author

This issue is happend when operators requires nested params.

_export:
  staging_bucket: mybucket
  project: aaa
  workspace_s3_uri_prefix: s3://${staging_bucket}/${project}/${session_uuid}

+create_cluster:
  emr_fleet.create_cluster>:
  ...<snip>...
  bootstrap_acitons:
    - name: "install datadog"
      script: ${workspace_s3_uri_prefix}/install_dd_agent.sh
      content: |
        ...<snip>...

In the above case, the script option gets s3://${staging_bucket}/${project}/${session_uuid}/install_dd_agent.sh, not like s3://mybucket/aaa/28d1743c-2f66-44a2-983d-c14a8f133auc/install_dd_agent.sh.

This was referenced Oct 23, 2018
Merged
Merged
Merged
@civitaspo
Copy link
Contributor Author

I implemented param_eval> operator as a workaround for this issue.

@chezou chezou mentioned this issue Feb 8, 2019
Open
@civitaspo civitaspo mentioned this issue Aug 5, 2020
Open
@ryokotmng ryokotmng mentioned this issue Feb 16, 2024
Closed
@ryokotmng ryokotmng linked a pull request Feb 26, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Eval nested params ryokotmng/digdag
2 participants
@muga @civitaspo