Affecter de la mémoire pour chaque travailleur en tempête

Question

Je dois attribuer différentes valeurs de mémoire pour chaque nouveau travailleur. J'ai donc essayé de changer la mémoire pour chaque boulon et chaque bec. J'utilise actuellement un planificateur personnalisé aussi. Voici mon approche du problème.

MON CODE:

TopologyBuilder builder = new TopologyBuilder(); 
builder.setSpout("spout", new EmailSpout(), 1).addConfiguration("node", "zoo1").setMemoryLoad(512.0); 
builder.setBolt("increment1", new IncrementBolt(), PARALLELISM).shuffleGrouping("spout").addConfiguration("node", "zoo2").setMemoryLoad(2048.0); 
builder.setBolt("increment2", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment1").addConfiguration("node", "zoo3").setMemoryLoad(2048.0); 
builder.setBolt("increment3", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment2").addConfiguration("node", "zoo4").setMemoryLoad(2048.0); 
builder.setBolt("output", new OutputBolt(), 1).globalGrouping("increment2").addConfiguration("node", "zoo1").setMemoryLoad(512.0); 
Config conf = new Config(); 
conf.setDebug(false); 
conf.setNumWorkers(4); 
StormSubmitter.submitTopologyWithProgressBar("Microbenchmark", conf, builder.createTopology()); 

MY STORM.YAML:

storm.zookeeper.servers: 
    - "zoo1" 
storm.zookeeper.port: 2181 
nimbus.seeds: ["zoo1"] 
storm.local.dir: "/home/ubuntu/eranga/storm-data" 
supervisor.slots.ports: 
    - 6700 
    - 6701 
    - 6702 
    - 6703 
    - 6704 
storm.scheduler: "org.apache.storm.scheduler.NodeBasedCustomScheduler" 
supervisor.scheduler.meta: 
    node: "zoo4" 
worker.profiler.enabled: true 
worker.profiler.childopts: "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder" 
worker.profiler.command: "flight.bash" 
worker.heartbeat.frequency.secs: 1 
worker.childopts: "-Xmx2048m -Xms2048m -Djava.net.preferIPv4Stack=true -Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl" 

Quand je soumets la topologie je reçois l'erreur suivante.

ERREUR:

Exception in thread "main" java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=768.0 < 2048.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount 
    at org.apache.storm.StormSubmitter.validateTopologyWorkerMaxHeapSizeMBConfigs(StormSubmitter.java:496) 

Toutes les suggestions?

Answer 1

Avez-vous essayé de suivre les conseils du message d'erreur?

Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount

Essayez d'ajouter ceci à storm.yaml:

topology.worker.max.heap.size.mb=2048.0 

Answer 2

Essayez d'utiliser cela.

import org.apache.storm.Config; 

public class TopologyExecuter{ 
    for(List<StormTopology> StormTopologyObject : StormTopologyObjects){ 
     Config topologyConf = new Config(); 
     topologyConf.put(Config.TOPOLOGY_WORKER_CHILDOPTS,"-Xmx512m -Xms256m"); 
     StormSubmitter.submitTopology("topology name", topologyConf, StormTopologyObject); 
    } 

}