brain split
Intro
使用 Elasticsearch時，會先設定 Cluster Name。cluster name 就是用來判斷 cluster 裡面的 node 怎麼做 discover 與 怎麼 auto-join 這些 nodes。這裡必須再說明一下，關於 elasticsearch 的 discovery modules。 discovery module 負責來發現集群內節點，還有負責選出哪個節點是 master。
Elasticsearch 是的 Peer to Peer p2p baseed system，意味著每個nodes間，直接與另外的 node 做溝通。主要的API (index,delete,search ) 並不會與 master node做連線。
Master node 的主要工作，就是
- 維護 cluster 的 state
- 決定nodes 是否可以加入cluster 內
- 分配 shard
所以，在 production 的 cluster ，setting 內的 " cluster.name" 是強烈建議要修改的。預設值是 "elasticsearch"。
Split Brain Problem
split brian 的根本問題在 nodes 不可用(node unavailable)，一般而也可能是 out of memorey (constant GC)， 過高的 threads或者是網路的問題。所以，特定節點連不到，或者是該節點當好是master node，就會發生 split 成兩個 cluster的問題。
Elasticsearch 官方目前已經在改進 consensus/election algorithm 來避免發生 split briain 的情況。
"discovery.zen.publish_timeout" 預設值是 30 秒，當每個節點接受到 master的請求時，他會回復並且更新自己的狀態到 master節點上。當 timeout了，會繼續去執行queue裡面的 node list。
目前比較典型的解決方案，可以設置 "discovery.zen.minimum_master_nodes" 。設定 master 的最小數目，建議在超過兩個節點時，可以設置較高的值，推薦的數字為 n/2+1。( n 是 nodes 數)。 另外一方面，也可以設置 indices.recovery.max_bytes_per_sec default 20mb/s 去調降 cluster 內的網路流量。
一般來說，還是要找出，為什麼 loading 集中的問題點，才能徹底解決這個問題。(例如，requested node 在同一個時間內，要處理過多的requests。除了可以調整 thread pool 來觀察，也可以觀察 每個nodes 的memory 與 cpu 用量。)
以下附上幾個 mail
Ivan Brusic ivan@brusic.com 透過 googlegroups.com 5月14日 2014
寄給 elasticsearch The current approach used to fix split brain problems by the Elasticsearch is not to correct the symptoms, but address the cause. Split brains occur when one of more nodes becomes unavailable, usually due to out of memory (constant GC), overwhelmed threads or network issues. The last one is hard to avoid, but there have been improvements to help with memory and threading. Do any of these cases apply to your situation?
Elasticsearch is improving the consensus/election algorithm in order to not get into split brain situations, but in the mean time you should look into avoid situations that caused it.
Cheers,
Ivan
Mark Walkom markw@campaignmonitor.com 6月11日
寄給 elasticsearch For that size just make them all data+master nodes. You really want n/2+1 masters, where n = number of nodes, so that you prevent split-brain from occurring, and having all 3 as master will provide that.
Just set ES_HEAP_SIZE in the defaults, you don't need to worry about anything else.
Regards, Mark Walkom
Michael Hart hart.mike@gmail.com 6月25日 (13 天前)
寄給 elasticsearch Try setting "indices.recovery.max_bytes_per_sec" much higher for faster recovery. The default is 20mb/s, and there's a bug in versions prior to 1.2 that rate limit to even lower than that. You didn't specify how big your indices are, but I can fairly accurately predict how long it'll take for the cluster to go green with that parameter.
mike
Mark Walkom markw@campaignmonitor.com 5月15日
寄給 elasticsearch With 2 mater nodes and discovery.zen.minimum_master_nodes: 1 you still risk a split brain. If they master lose connectivity for whatever reason, they will each form their own cluster, and if the data nodes happen to be talking to only one of the master eligible nodes at that time then you will have two clusters, each with a master and a data node.
If you want HA as you are putting it then you want 3 master capable nodes with discovery.zen.minimum_master_nodes: 2.
Regards, Mark Walkom
cf.
- Configuration http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html
- Discovery http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery.html#modules-discovery
- Zen Discovery http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-zen.html
http://peichengnote.blogspot.tw/search/label/elasticsearch
peicheng note: [elasticsearch] 再談 _all field
http://peichengnote.blogspot.tw/2014/06/elasticsearch-all-field.html
peicheng note: [elasticsearch]range query depends on the field type
http://peichengnote.blogspot.tw/2014/06/elasticsearchrange-query-depends-on.htmlpeicheng-note: [elasticsearch] document id _id field uuid
http://peichengnote.blogspot.tw/2014/05/elasticsearch-document-id-id-field-uuid.html
peicheng-note: [elasticsearch/logstash] logstash id 自動產生 document id "_id" automatic id generation
http://peichengnote.blogspot.tw/2014/04/elasticsearchlogstash-logstash-id.html
沒有留言:
張貼留言