ClojureのREPLで動きを解析

バイナリでznodeに格納されているデータを読み出してみた場合の例。

$ cd storm

$ lein repl

user=> (use '(backtype.storm config))
nil

user=> (use '(backtype.storm zookeeper))
nil

user=> (import '(backtype.storm.utils Utils))
backtype.storm.utils.Utils

user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/storms/mytopology-4-1361812014", false)))
1496860 [Thread-1] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
#backtype.storm.daemon.common.StormBase{:storm-name "mytopology", :launch-time-secs 1361812014, :status {:type :active}, :num-workers 2, :component->executors {"split" 1, "__acker" 1, "count" 1, "sentence-parametrized" 1, "__system" 0, "sentence" 1}}
1496870 [Thread-1-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none

user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/assignments/mytopology-4-1361812014", false)))
1499720 [Thread-1] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
#backtype.storm.daemon.common.Assignment{:master-code-dir "storm-local/nimbus/stormdist/mytopology-4-1361812014", :node->host {"2de24c3a-9537-4522-a70b-117646d3bc44" "centos6"}, :executor->node+port {[3 3] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [5 5] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [4 4] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [2 2] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [1 1] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703]}, :executor->start-time-secs {[1 1] 1361812014, [5 5] 1361812014, [3 3] 1361812014, [2 2] 1361812014, [4 4] 1361812014}}
1499740 [Thread-1-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none

rebalanceをコマンドを打ってから

$ bin/storm rebalance mytopology -w 60 -n 4 -e sentence=2 -e sentence-parametrized=2

反映されるまでの間と、

user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/storms/mytopology-4-1361812014", false)))
1570454 [Thread-1] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
#backtype.storm.daemon.common.StormBase{:storm-name "mytopology", :launch-time-secs 1361812014, :status {:type :rebalancing, :delay-secs 60, :old-status {:type :active}, :num-workers 4, :executor-overrides #<HashMap {sentence-parametrized=2, sentence=2}>}, :num-workers 2, :component->executors {"split" 1, "__acker" 1, "count" 1, "sentence-parametrized" 1, "__system" 0, "sentence" 1}}
1570480 [Thread-1-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none

user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/assignments/mytopology-4-1361812014", false)))
1576881 [Thread-1] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
#backtype.storm.daemon.common.Assignment{:master-code-dir "storm-local/nimbus/stormdist/mytopology-4-1361812014", :node->host {"2de24c3a-9537-4522-a70b-117646d3bc44" "centos6"}, :executor->node+port {[3 3] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [5 5] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [4 4] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [2 2] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [1 1] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703]}, :executor->start-time-secs {[1 1] 1361812014, [5 5] 1361812014, [3 3] 1361812014, [2 2] 1361812014, [4 4] 1361812014}}
1576888 [Thread-1-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none

反映された後。

user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/storms/mytopology-4-1361812014", false)))
1660965 [Thread-1] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
#backtype.storm.daemon.common.StormBase{:storm-name "mytopology", :launch-time-secs 1361812014, :status {:type :active}, :num-workers 4, :component->executors {"split" 1, "__acker" 1, "count" 1, "sentence-parametrized" 2, "__system" 0, "sentence" 2}}
1660991 [Thread-1-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none

user=> (let [zk (let [conf (read-storm-config)] (mk-client conf ["localhost"] 2181))] (Utils/deserialize (get-data zk "/storm/assignments/mytopology-4-1361812014", false)))
1664021 [Thread-1] INFO  com.netflix.curator.framework.imps.CuratorFrameworkImpl  - Starting
#backtype.storm.daemon.common.Assignment{:master-code-dir "storm-local/nimbus/stormdist/mytopology-4-1361812014", :node->host {"2de24c3a-9537-4522-a70b-117646d3bc44" "centos6"}, :executor->node+port {[3 3] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6701], [5 5] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703], [4 4] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6700], [2 2] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6702], [1 1] ["2de24c3a-9537-4522-a70b-117646d3bc44" 6703]}, :executor->start-time-secs {[3 3] 1361812175, [1 1] 1361812175, [5 5] 1361812175, [4 4] 1361812175, [2 2] 1361812175}}
1664050 [Thread-1-EventThread] INFO  backtype.storm.zookeeper  - Zookeeper state update: :connected:none

rebalanceコマンドにはexecutorの数についてのオプションも用意されているが、指定しても数は変わらなかった。というのは、topology起動時に決まるタスク数の数は変えられないからだった。
https://github.com/nathanmarz/storm/issues/54
起動時に1 taskごとに1 executorの割りあてがある状態では、それ以上はexecutorスレッドは増やしようがない。