This repository was archived by the owner on Jan 9, 2020. It is now read-only.
This repository was archived by the owner on Jan 9, 2020. It is now read-only.
Could not find or load main class SparkPi #632
Open
Description
Hi,
I was trying to run sparkpi application on k8s. when I use spark-submit to submit the spark job,
$ spark-2.3.1-bin-hadoop2.7/bin/spark-submit \
--master k8s://<k8s-url> \
--deploy-mode cluster \
--name spark-pi \
--class SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=<spark-image> \
local:///vagrant/softwares/sparkpi_2.11-1.0.jar
I am getting the following exception.
exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java - Dspark.kubernetes.executor.podNamePrefix=spark-pi-d9a15eabd9a9360b863577bed1f247a4 -Dspark.app.id=spark-f0dc3926565f4cd59a48459b367a6460 -Dspark.master=k8s://<k8s-url> -Dspark.app.name=spark-pi -Dspark.kubernetes.container.image=spark-image -Dspark.driver.host=spark-pi-d9a15eabd9a9360b863577bed1f247a4-driver-svc.default.svc -Dspark.executor.instances=5 -Dspark.submit.deployMode=cluster -Dspark.driver.blockManager.port=7079 -Dspark.kubernetes.driver.pod.name=spark-pi-d9a15eabd9a9360b863577bed1f247a4-driver -Dspark.jars=<spark-pi-jar> -Dspark.driver.port=7078 -cp ':/opt/spark/jars/*:/vagrant/softwares/sparkpi_2.11-1.0.jar:/vagrant/softwares/sparkpi_2.11-1.0.jar' -Xms1g -Xmx1g -Dspark.driver.bindAddress=<ip> SparkPi
Error: Could not find or load main class SparkPi
Could anyone help me.
Thanks
Metadata
Metadata
Assignees
Labels
No labels