Des travaux sont perdus dans ParallelPython?

Question

Je soumets environ 234 travaux (mais mon exemple n'en contient que 50 à des fins de démonstration) à mon cluster de 20 nœuds en utilisant ParallelPython. Je m'attendais à ce qu'il fasse la queue et les exécute mais il semble "perdre" des travaux et je ne comprends pas où les choses vont mal. Quand le script se termine, je ne peux pas voir 50 fichiers, c'est-à-dire info_1, info_2 .... info_50 mais je vois plutôt un comportement aléatoire. Aucune suggestion?

def readChecklist(): 
    f = open('/home/username/twisted/pp-1.6.0/checklist', 'r') 
    checklist = [line.strip() for line in f] 
    return checklist 

def processFile(num): 
    bl = readChecklist() 

    # pick a filename to write to 
    outfile = "info_" + str(num) 
    FILE = open(outfile, "a") 

    for i in range(num): 
     FILE.write(str(i)+"\n") 
     FILE.flush() 
    FILE.close() 
    return num 

ppservers=("*",) 

job_server = pp.Server(ppservers=ppservers) 

inputs = range(50) 
jobs = [(input, job_server.submit(processFile,(input,), (readCheckList,), ("os","math","time","sys","subprocess",))) for input in inputs] 
for input, job in jobs: 
    print "Job: ", input, " is", job() 

job_server.print_stats() 

Sortie:

Job: 0 is True 
Job: 1 is True 
Job: 2 is True 
Job: 3 is True 
Job: 4 is True 
Job: 5 is True 
Job: 6 is True 
Job: 7 is True 
Job: 8 is True 
Job: 9 is True 
Job: 10 is True 
Job: 11 is True 
Job: 12 is True 
Job: 13 is True 
Job: 14 is True 
Job: 15 is True 
Job: 16 is True 
Job: 17 is True 
Job: 18 is True 
Job: 19 is True 
Job: 20 is True 
Job: 21 is True 
Job: 22 is True 
Job: 23 is True 
Job: 24 is True 
Job: 25 is True 
Job: 26 is True 
Job: 27 is True 
Job: 28 is True 
Job: 29 is True 
Job: 30 is True 
Job: 31 is True 
Job: 32 is True 
Job: 33 is True 
Job: 34 is True 
Job: 35 is True 
Job: 36 is True 
Job: 37 is True 
Job: 38 is True 
Job: 39 is True 
Job: 40 is True 
Job: 41 is True 
Job: 42 is True 
Job: 43 is True 
Job: 44 is True 
Job: 45 is True 
Job: 46 is True 
Job: 47 is True 
Job: 48 is True 
Job: 49 is True 
Time elapsed: 0.592607975006 s 
Job execution statistics: 
job count | % of all jobs | job time sum | time per job | job server 
     3 |   6.00 |  0.3226 |  0.107546 | x.x.x.x:abcd 
     3 |   6.00 |  0.2849 |  0.094970 | x.x.x.x:abcd 
     2 |   4.00 |  0.2420 |  0.121004 | x.x.x.x:abcd 
     3 |   6.00 |  0.3328 |  0.110927 | x.x.x.x:abcd 
     2 |   4.00 |  0.2314 |  0.115687 | x.x.x.x:abcd 
     2 |   4.00 |  0.2634 |  0.131683 | x.x.x.x:abcd 
     3 |   6.00 |  0.2827 |  0.094223 | x.x.x.x:abcd 
     2 |   4.00 |  0.2496 |  0.124812 | x.x.x.x:abcd 
     1 |   2.00 |  0.1701 |  0.170140 | x.x.x.x:abcd 
     3 |   6.00 |  0.3053 |  0.101758 | x.x.x.x:abcd 
     1 |   2.00 |  0.1334 |  0.133415 | x.x.x.x:abcd 
     3 |   6.00 |  0.2777 |  0.092561 | x.x.x.x:abcd 
     1 |   2.00 |  0.1152 |  0.115169 | x.x.x.x:abcd 
     1 |   2.00 |  0.1273 |  0.127294 | x.x.x.x:abcd 
     3 |   6.00 |  0.3345 |  0.111503 | x.x.x.x:abcd 
     1 |   2.00 |  0.1128 |  0.112782 | x.x.x.x:abcd 
     2 |   4.00 |  0.2636 |  0.131819 | x.x.x.x:abcd 
     8 |   16.00 |  0.4413 |  0.055163 | local 
     1 |   2.00 |  0.1905 |  0.190510 | x.x.x.x:abcd 
     3 |   6.00 |  0.2774 |  0.092473 | x.x.x.x:abcd 
     2 |   4.00 |  0.2197 |  0.109835 | x.x.x.x:abcd 
Time elapsed since server creation 0.592818021774 

List of files created: (One per job) 
0 
1 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
2 
20 
21 
22 
3 
4 
5 
6 
7 
8 
9 

Answer 1

Ok mon erreur! Juste au cas où quelqu'un d'autre fait face à ce problème, assurez-vous que vos chemins de répertoire sont absolus si vous lisez un fichier ou écrivez dans un fichier ... 5 heures de débogage :(mais j'ai appris ma leçon :)