-
-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Description
Hi, I use athena.add_partition>: to make partition in hourly. So calc current date & hour from session_time by digdag's feature.I guess other ops such as ctas , query ,,, supports nested variables, but add_partition (and maybe drop_partiton) doesn't work well.
In the my example.dig, thrown exception about s3 path, but in our production environment, the partition was created with the value that ${moment(session_time...} was not expanded.
example.dig
If you set ${ny_dt} insted of ${curr_dt} at partition_kv: work normaly.
_export:
plugin:
repositories:
- file://${repos}
# - https://jitpack.io
dependencies:
- pro.civitaspo:digdag-operator-athena:0.4.0
athena:
auth_method: profile
curr_dt: ${moment(session_time).format("YYYY-MM-DD")}
ny_dt: 2020-01-01
+step7:
athena.ctas>: select 1 as a, 2 as b, '${ny_dt}' as dt union all select 4 as a, 5 as b, '${curr_dt}' as dt
database: ${database}
table: hoge
location: ${output}
partitioned_by: [dt]
# +step8:
# athena.drop_partition>:
# database: ${database}
# table: hoge
# partition_kv:
# dt: "${curr_dt}"
+step9:
athena.add_partition>:
database: ${database}
table: hoge
partition_kv:
dt: "${curr_dt}"error
* +example+step9:
Illegal character in path at index 37: s3://imaoka-oregon-athena/digdag/dt=${moment(session_time).format("YYYY-MM-DD")}/ (illegal argument)
Metadata
Metadata
Assignees
Labels
No labels
