ClojureのREPLで動きを解析
バイナリでznodeに格納されているデータを読み出してみた場合の例。
$ cd storm $ lein repl user=> (use '(backtype.storm config)) nil user=> (use '(backtype.storm zookeeper)) nil user=> (import '(backtype.storm.utils Utils)) backtype.storm.utils.Utils user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/storms/mytopology-4-1361812014", false))) 1496860 [Thread-1] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting #backtype.storm.daemon.common.StormBase{:storm-name "mytopology", :launch-time-secs 1361812014, :status {:type :active}, :num-workers 2, :component->executors {"split" 1, "__acker" 1, "count" 1, "sentence-parametrized" 1, "__system" 0, "sentence" 1}} 1496870 [Thread-1-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/assignments/mytopology-4-1361812014", false))) 1499720 [Thread-1] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting #backtype.storm.daemon.common.Assignment{:master-code-dir "storm-local/nimbus/stormdist/mytopology-4-1361812014", :node->host {"2de24c3a-9537-4522-a70b-117646d3bc44" "centos6"}, :executor->node+port {[3 3] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [5 5] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [4 4] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [2 2] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [1 1] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703]}, :executor->start-time-secs {[1 1] 1361812014, [5 5] 1361812014, [3 3] 1361812014, [2 2] 1361812014, [4 4] 1361812014}} 1499740 [Thread-1-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
rebalanceをコマンドを打ってから
$ bin/storm rebalance mytopology -w 60 -n 4 -e sentence=2 -e sentence-parametrized=2
反映されるまでの間と、
user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/storms/mytopology-4-1361812014", false))) 1570454 [Thread-1] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting #backtype.storm.daemon.common.StormBase{:storm-name "mytopology", :launch-time-secs 1361812014, :status {:type :rebalancing, :delay-secs 60, :old-status {:type :active}, :num-workers 4, :executor-overrides #<HashMap {sentence-parametrized=2, sentence=2}>}, :num-workers 2, :component->executors {"split" 1, "__acker" 1, "count" 1, "sentence-parametrized" 1, "__system" 0, "sentence" 1}} 1570480 [Thread-1-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/assignments/mytopology-4-1361812014", false))) 1576881 [Thread-1] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting #backtype.storm.daemon.common.Assignment{:master-code-dir "storm-local/nimbus/stormdist/mytopology-4-1361812014", :node->host {"2de24c3a-9537-4522-a70b-117646d3bc44" "centos6"}, :executor->node+port {[3 3] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [5 5] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [4 4] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [2 2] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [1 1] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703]}, :executor->start-time-secs {[1 1] 1361812014, [5 5] 1361812014, [3 3] 1361812014, [2 2] 1361812014, [4 4] 1361812014}} 1576888 [Thread-1-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
反映された後。
user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/storms/mytopology-4-1361812014", false))) 1660965 [Thread-1] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting #backtype.storm.daemon.common.StormBase{:storm-name "mytopology", :launch-time-secs 1361812014, :status {:type :active}, :num-workers 4, :component->executors {"split" 1, "__acker" 1, "count" 1, "sentence-parametrized" 2, "__system" 0, "sentence" 2}} 1660991 [Thread-1-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/assignments/mytopology-4-1361812014", false))) 1664021 [Thread-1] INFO com.netflix.curator.framework.imps.CuratorFrameworkImpl - Starting #backtype.storm.daemon.common.Assignment{:master-code-dir "storm-local/nimbus/stormdist/mytopology-4-1361812014", :node->host {"2de24c3a-9537-4522-a70b-117646d3bc44" "centos6"}, :executor->node+port {[3 3] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6701], [5 5] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [4 4] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6700], [2 2] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [1 1] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703]}, :executor->start-time-secs {[3 3] 1361812175, [1 1] 1361812175, [5 5] 1361812175, [4 4] 1361812175, [2 2] 1361812175}} 1664050 [Thread-1-EventThread] INFO backtype.storm.zookeeper - Zookeeper state update: :connected:none
rebalanceコマンドにはexecutorの数についてのオプションも用意されているが、指定しても数は変わらなかった。というのは、topology起動時に決まるタスク数の数は変えられないからだった。
https://github.com/nathanmarz/storm/issues/54
起動時に1 taskごとに1 executorの割りあてがある状態では、それ以上はexecutorスレッドは増やしようがない。