java.lang.NoClassDefFoundError: org/apache/spark/deploy/SparkSubmit

J'essaye de lancer un peu de scala jar via l'interface web hue. Lors de la présentation du travail, il est échoue avec l'erreur de folloring:java.lang.NoClassDefFoundError: org/apache/spark/deploy/SparkSubmit

java.lang.NoClassDefFoundError: org/apache/spark/deploy/SparkSubmit

et le journal complet:

2017-08-24 08:31:49,438 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]:start:] Start action [[email protected]:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 
2017-08-24 08:31:49,438 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]:start:] [***[email protected]:start:***]Action status=DONE 
2017-08-24 08:31:49,438 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]:start:] [***[email protected]:start:***]Action updated in DB! 
2017-08-24 08:31:49,470 INFO WorkflowNotificationXCommand:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000025-170823093043124-oozie-oozi-W 
2017-08-24 08:31:49,470 INFO WorkflowNotificationXCommand:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]:start:] No Notification URL is defined. Therefore nothing to notify for job [email protected]:start: 
2017-08-24 08:31:49,483 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Start action [[email protected]] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 
2017-08-24 08:31:49,516 INFO SparkActionExecutor:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Added into spark action configuration mapred.child.env=SPARK_HOME=.,HDP_VERSION=2.6.3.0-57 
2017-08-24 08:31:50,421 INFO SparkActionExecutor:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Trying to get job [job_1503473462790_0052], attempt [1] 
2017-08-24 08:31:50,437 INFO SparkActionExecutor:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] checking action, hadoop job ID [job_1503473462790_0052] status [RUNNING] 
2017-08-24 08:31:50,439 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] [***[email protected]***]Action status=RUNNING 
2017-08-24 08:31:50,439 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] [***[email protected]***]Action updated in DB! 
2017-08-24 08:31:50,451 INFO WorkflowNotificationXCommand:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] No Notification URL is defined. Therefore nothing to notify for job [email protected] 
2017-08-24 08:32:00,098 INFO CallbackServlet:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] callback for action [[email protected]] 
2017-08-24 08:32:00,154 INFO SparkActionExecutor:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Trying to get job [job_1503473462790_0052], attempt [1] 
2017-08-24 08:32:00,244 INFO SparkActionExecutor:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] action completed, external ID [job_1503473462790_0052] 
2017-08-24 08:32:00,247 WARN SparkActionExecutor:523 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org/apache/spark/deploy/SparkSubmit 
2017-08-24 08:32:00,247 WARN SparkActionExecutor:523 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Launcher exception: org/apache/spark/deploy/SparkSubmit 
java.lang.NoClassDefFoundError: org/apache/spark/deploy/SparkSubmit 
    at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:222) 
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58) 
    at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:62) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:239) 
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:422) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) 
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.SparkSubmit 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
    ... 16 more 

2017-08-24 08:32:00,293 INFO ActionEndXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] ERROR is considered as FAILED for SLA 
2017-08-24 08:32:00,358 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] Start action [[email protected]] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 
2017-08-24 08:32:00,358 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] [***[email protected]***]Action status=DONE 
2017-08-24 08:32:00,358 INFO ActionStartXCommand:520 - SERVER[BigData] USER[energia] GROUP[-] TOKEN[] APP[kafka-scala] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] [***[email protected]***]Action updated in DB! 
2017-08-24 08:32:00,435 INFO WorkflowNotificationXCommand:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] No Notification URL is defined. Therefore nothing to notify for job [email protected] 
2017-08-24 08:32:00,435 INFO WorkflowNotificationXCommand:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000025-170823093043124-oozie-oozi-W 
2017-08-24 08:32:00,435 INFO WorkflowNotificationXCommand:520 - SERVER[BigData] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000025-170823093043124-oozie-oozi-W] ACTION[[email protected]] No Notification URL is defined. Therefore nothing to notify for job [email protected]

aide, s'il vous plaît?

Source

2017-08-24 sartions

Est-ce que SPARK_HOME a été défini sur votre environnement? – FaigB

@FaigB oui. c'est – sartions

Cela pourrait être une solution rapide. Dans la fenêtre du flux de travail de teinte, accédez au dossier de l'espace de travail en cliquant sur l'icône du fichier en haut à droite. Créez un dossier 'lib' s'il n'y en a pas déjà un.

Accédez au dossier SPARK_HOME dans le système local (/usr/hdp/2.5../spark/lib) et vérifiez tous les fichiers jar disponibles. Téléchargez ces fichiers jars dans le dossier lib de votre espace de travail. Si vous souhaitez exécuter des travaux pyspark, téléchargez les fichiers jars dans le dossier 'python' de la mémoire de démarrage.

Réexécutez le flux de travail. P.S: Je suppose que ces jars peuvent être ajoutés au dossier sharelib dans les bibliothèques partagées oozie et mettre à jour la bibliothèque oozie. (Vérifiez pour la commande en ligne). Cela peut résoudre le problème de l'ajout de ces fichiers dans tous les dossiers de workflow.

Source

2017-09-19 14:43:27 Adiga

java.lang.NoClassDefFoundError: org/apache/spark/deploy/SparkSubmit

Répondre

Questions connexes