Tensorflow distribué: CreateSession n'attend toujours que les différents noeuds

Question

J'essaie d'obtenir mnist_replica.py exemple de travail. Selon la suggestion sur this question, je spécifie le filtre de dispositif.

Mon code fonctionne lorsque les tâches ps et de travail sont sur le même noeud. Lorsque j'essaie de placer la tâche ps sur le noeud 1 et la tâche de travail sur le noeud 2, je reçois "CreateSession toujours en attente".

Par exemple:

Pseudo version distribuée (travaux!)

Dump Terminal de Node1 (instance 1)

node1 $ python mnist_replica.py --worker_hosts=node1:2223 --job_name=ps --task_index=0 
Extracting /tmp/mnist-data/train-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/train-labels-idx1-ubyte.gz 
Extracting /tmp/mnist-data/t10k-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/t10k-labels-idx1-ubyte.gz 
job name = ps 
task index = 0 
2017-10-10 11:09:16.637006: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:2222} 
2017-10-10 11:09:16.637075: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> node1:2223} 
2017-10-10 11:09:16.640114: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2222 
... 

Dump Terminal de Node1 (exemple 2)

node1 $ python mnist_replica.py --worker_hosts=node1:2223 --job_name=worker --task_index=0 
Extracting /tmp/mnist-data/train-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/train-labels-idx1-ubyte.gz 
Extracting /tmp/mnist-data/t10k-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/t10k-labels-idx1-ubyte.gz 
job name = worker 
task index = 0 
2017-10-10 11:11:12.784982: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:2222} 
2017-10-10 11:11:12.785046: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:2223} 
2017-10-10 11:11:12.787685: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2223 
Worker 0: Initializing session... 
2017-10-10 11:11:12.991784: I tensorflow/core/distributed_runtime/master_session.cc:998] Start master session 418af3aa5ce103a3 with config: device_filters: "/job:ps" device_filters: "/job:worker/task:0" allow_soft_placement: true 
Worker 0: Session initialization complete. 
Training begins @ 1507648273.272837 
1507648273.443305: Worker 0: training step 1 done (global step: 0) 
1507648273.454537: Worker 0: training step 2 done (global step: 1) 
... 

2 noeuds distribués (ne fonctionnent pas)

Dump Terminal de Node1

node1 $ python mnist_replica.py --worker_hosts=node2:2222 --job_name=ps --task_index=0 
Extracting /tmp/mnist-data/train-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/train-labels-idx1-ubyte.gz 
Extracting /tmp/mnist-data/t10k-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/t10k-labels-idx1-ubyte.gz 
job name = ps 
task index = 0 
2017-10-10 10:54:27.419949: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:2222} 
2017-10-10 10:54:27.420064: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> node2:2222} 
2017-10-10 10:54:27.426168: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2222 
... 

Dump Terminal de Node2

node2 $ python mnist_replica.py --ps_hosts=node1:2222 --worker_hosts=node2:2222 --job_name=worker --task_index=0 
Extracting /tmp/mnist-data/train-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/train-labels-idx1-ubyte.gz 
Extracting /tmp/mnist-data/t10k-images-idx3-ubyte.gz 
Extracting /tmp/mnist-data/t10k-labels-idx1-ubyte.gz 
job name = worker 
task index = 0 
2017-10-10 10:51:13.303021: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> node1:2222} 
2017-10-10 10:51:13.303081: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:2222} 
2017-10-10 10:51:13.308288: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2222 
Worker 0: Initializing session... 
2017-10-10 10:51:23.508040: I tensorflow/core/distributed_runtime/master.cc:209] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0 
2017-10-10 10:51:33.508247: I tensorflow/core/distributed_runtime/master.cc:209] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0 
... 

Les deux nœuds exécuter CentOS7, tensorflow R1.3, Python 2.7. Les nœuds peuvent communiquer entre eux via ssh, les noms d'hôte sont corrects, le pare-feu est désactivé. Quelque chose manque?

Y a-t-il d'autres étapes que je dois prendre pour m'assurer que les nœuds peuvent communiquer entre eux en utilisant GRPC? Merci.

Answer 1

Je pense que vous feriez mieux de vérifier ClusterSpec et la partie serveur. Par exemple, vous devez vérifier l'adresse IP pour node1 et node2, vérifier le port et l'index des tâches, etc. Je veux donner une suggestion spécifique mais il est difficile de vous donner des suggestions sans le code. Merci.

Answer 2

Le problème était que le pare-feu bloquait les ports. J'ai désactivé le pare-feu sur tous les noeuds dans les questions et le problème résolu lui-même!