Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Discuss how drivers and executors will pick up new tokens from the token refresh server #534

Open
@kimoonkim

Description

@kimoonkim

#453 is implementing the HDFS token refresh server which will obtain brand new tokens when prior tokens completely expire after 7 days. For each supported job, the refresh server will write back the new token to the associated K8s secrets as an additional data item. The job's driver and executors should detect the new token and load it into the JVMs so they can continue to access the secure HDFS.

We should discuss how exactly this can be done. I can imagine two approaches:

  1. If K8s secret mounting supports this (does it?), the new token will appear as a new file in the mount point directory of the secret volume. Then, the driver and executors will periodically scan the directory for a new file and load it into memory.
  2. The driver and executors use a K8s watcher for the secret and find the update event. And use K8s API to read the new data item containing the new token. This requires executors also to use K8s API client and service account, which is a new behavior.

I personally prefer (1), if it is possible.

One related note is that there is an existing hook in the base class SparkHadoopUtil both for the driver and executor for supporting this. We just need to subclass the base class and implement (1) or (2) in the subclass:

 /**
   * Start a thread to periodically update the current user's credentials with new credentials so
   * that access to secured service does not fail.
   */
  private[spark] def startCredentialUpdater(conf: SparkConf) {}

Thoughts? Concerns?

@ifilonenko @liyinan926

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions