spark-app saveAsText erreur locale

Question

Impossible de créer une erreur d'application Spark due à Mkdirs. J'utilise étincelle 1.6.3 incapable de sauver la sortie sur mon dir locale

java.io.IOException: Mkdirs failed to create file:/home/zooms/output/sample1/sample1.txt/_temporary/0/_temporary/attempt_201709251225_0005_m_000000_10 
(exists=false, cwd=file:/grid/1/hadoop/yarn/local/usercache/zooms/appcache/application_1504506749061_0086/container_e01_1504506749061_0086_01_000003) 

journaux mis à jour

17/09/25 13:39:02 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 10, worker3.hdp.example.com): java.io.IOException: Mkdirs failed to create file:/home/zooms/output/sample1/sample1.txt/_temporary/0/_temporary/attempt_201709251339_0005_m_000000_10 (exists=false, cwd=file:/grid/1/hadoop/yarn/local/usercache/zooms/appcache/application_1504506749061_0099/container_e01_1504506749061_0099_01_000003) 
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456) 
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:442) 
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:930) 
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:823) 
    at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123) 
    at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1191) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) 
    at org.apache.spark.scheduler.Task.run(Task.scala:89) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
    at java.lang.Thread.run(Thread.java:748) 

code:

val output = "file:///home/zooms/output/sample1/sample1.txt" 
result.coalesce(1).saveAsTextFile(output) 

Answer 1

Voici la réponse à ma question. Étant donné que je suis en mode cluster ou en mode client, les utilisateurs ne pourront pas créer le répertoire sur chaque nœud à moins que vous ne le définissiez. Utilisez spark-submit -v --master local ...

Références: Writing files to local system with Spark in Cluster mode

Why does Spark job fails to write output?