多個Cassandra Node在不同Data Center

本實作是採用Cassandra 1.0.12來實施

Topology

Token分配
由於Cassandra 1.0.12並沒有token generator, 建議下載https://raw.github.com/riptano/ComboAMI/2.2/tokentoolv2.py, 產生Token分配表, 如下所示:

Node	hostname	IP Address	Token	Data Center	Rack
node0	clouddb1.gc.net	172.16.70.32	0	DC1	RAC1
node1	clouddb2.gc.net	172.16.70.41	56713727820156410577229101238628035242	DC1	RAC1
node2	clouddb3.gc.net	172.16.70.42	113427455640312821154458202477256070485	DC1	RAC1
node3	clouddb4.gc.net	172.16.70.43	28356863910078205288614550619314017621	DC2	RAC1
node4	clouddb5.gc.net	172.16.70.44	85070591730234615865843651857942052863	DC2	RAC1
node5	clouddb6.gc.net	172.16.70.45	141784319550391026443072753096570088106	DC2	RAC1

修改cassandra.yaml

依token分配表將token和hostname填入initial_token和listen_address，例如

In node 0
initial_token: 0
listen_address: clouddb1.gc.net

In node 1
initial_token: 56713727820156410577229101238628035242
listen_address: clouddb2.gc.net

依此類推

Snitch用來設定Topology環境, 目的避免單一node failure.
其中環境可分為DataCenter和Rack, 在本文的測試環境分為DC1和DC2, 統一都是使用第一組機台

原文如下:
Set this to a class that implements
# IEndpointSnitch.  The snitch has two functions:
# - it teaches Cassandra enough about your network topology to route
#   requests efficiently
# - it allows Cassandra to spread replicas around your cluster to avoid
#   correlated failures. It does this by grouping machines into
#   "datacenters" and "racks."  Cassandra will do its best not to have
#   more than one replica on the same "rack" (which may not actually
#   be a physical location)

Cassandra提供幾種Snitch的方式

• SimpleSnitch
 Treats Strategy order as proximity. This improves cache locality  when disabling read repair, which can further improve throughput.  Only appropriate for single-datacenter deployments.

• PropertyFileSnitch
 Proximity is determined by rack and data center, which are  explicitly configured in cassandra-topology.properties.

• RackInferringSnitch
 Proximity is determined by rack and data center, which are  assumed to correspond to the 3rd and 2nd octet of each node's  IP address, respectively.  Unless this happens to match your  deployment conventions (as it did Facebook's), this is best used  as an example of writing a custom Snitch class.

• Ec2Snitch
 Appropriate for EC2 deployments in a single Region.  Loads Region  and Availability Zone information from the EC2 API. The Region is  treated as the Datacenter, and the Availability Zone as the rack.  Only private IPs are used, so this will not work across multiple  Regions.

• Ec2MultiRegionSnitch
 Uses public IPs as broadcast_address to allow cross-region  connectivity.  (Thus, you should set seed addresses to the public  IP as well.) You will need to open the storage_port or  ssl_storage_port on the public IP firewall.  (For intra-Region  traffic, Cassandra will switch to the private IP after  establishing a connection.)

本文使用的endpoint snitch為PropertyFileSnitch, 其支援Run-time更新已異動的Property值
在node0~node5的conf/cassandra.yaml

endpoint_snitch: PropertyFileSnitch

且修改conf/cassandra-topology.properties, 指定data center和rack

172.16.70.32=DC1:RAC1
172.16.70.41=DC1:RAC1
172.16.70.42=DC1:RAC1

172.16.70.43=DC2:RAC1
172.16.70.44=DC2:RAC1
172.16.70.45=DC2:RAC1

default=DC1:r1

Seed Node

指定node0, node1和node2為seed node
node0~node5的conf/cassandra.yaml

seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "clouddb1.gc.ubicloud.net,clouddb4.gc.ubicloud.net"

開起服務

依序開啟node0~node5的Cassandra process

結果如下

開起服務

1. Initializing Multiple Data Center Clusters on Cassandra
2. Generating Tokens
3. About Client Requests in Cassandra

================================================
後記

新增一個sample keyspace
$ ./cqlsh localhost
> CREATE KEYSPACE sample WITH strategy_class = 'NetworkTopologyStrategy' AND strategy_options:DC1 = '3' and strategy_options:DC2 = '3';

Ring的結果:
$ ./nodetool -h self ring
Address         DC          Rack        Status State   Load            Owns    Token
                                                                               169417178424467235000914166253263322299
node0  172.16.70.32    DC1         RAC1        Up     Normal  93.18 KB        0.43%   0
node4  172.16.70.44    DC2         RAC1        Up     Normal  74.67 KB        32.91%  55989722784154413846455963776007251813
node1  172.16.70.41    DC1         RAC1        Up     Normal  97.89 KB        0.43%   56713727820156410577229101238628035242
node5  172.16.70.45    DC2         RAC1        Up     Normal  81.01 KB        32.91%  112703450604310824423685065014635287055
node2  172.16.70.42    DC1         RAC1        Up     Normal  97.66 KB        0.43%   113427455640312821154458202477256070484
node3  172.16.70.43    DC2         RAC1        Up     Normal  81.01 KB        32.91%  169417178424467235000914166253263322299

$ ./nodetool -h self describering sample
TokenRange:
  TokenRange(start_token:55989722784154413846455963776007251813, end_token:56713727820156410577229101238628035242, endpoints:[172.16.70.45, 172.16.70.43, 172.16.70.44, 172.16.70.41, 172.16.70.42, 172.16.70.32], rpc_endpoints:[0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0], endpoint_details:[EndpointDetails(host:172.16.70.45, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.43, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.44, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.41, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.42, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.32, datacenter:DC1, rack:RAC1)])
  TokenRange(start_token:113427455640312821154458202477256070484, end_token:169417178424467235000914166253263322299, endpoints:[172.16.70.43, 172.16.70.44, 172.16.70.45, 172.16.70.32, 172.16.70.41, 172.16.70.42], rpc_endpoints:[0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0], endpoint_details:[EndpointDetails(host:172.16.70.43, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.44, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.45, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.32, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.41, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.42, datacenter:DC1, rack:RAC1)])
  TokenRange(start_token:169417178424467235000914166253263322299, end_token:0, endpoints:[172.16.70.44, 172.16.70.45, 172.16.70.43, 172.16.70.32, 172.16.70.41, 172.16.70.42], rpc_endpoints:[0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0], endpoint_details:[EndpointDetails(host:172.16.70.44, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.45, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.43, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.32, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.41, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.42, datacenter:DC1, rack:RAC1)])
  TokenRange(start_token:56713727820156410577229101238628035242, end_token:112703450604310824423685065014635287055, endpoints:[172.16.70.45, 172.16.70.43, 172.16.70.44, 172.16.70.42, 172.16.70.32, 172.16.70.41], rpc_endpoints:[0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0], endpoint_details:[EndpointDetails(host:172.16.70.45, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.43, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.44, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.42, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.32, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.41, datacenter:DC1, rack:RAC1)])
  TokenRange(start_token:112703450604310824423685065014635287055, end_token:113427455640312821154458202477256070484, endpoints:[172.16.70.43, 172.16.70.44, 172.16.70.45, 172.16.70.42, 172.16.70.32, 172.16.70.41], rpc_endpoints:[0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0], endpoint_details:[EndpointDetails(host:172.16.70.43, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.44, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.45, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.42, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.32, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.41, datacenter:DC1, rack:RAC1)])
  TokenRange(start_token:0, end_token:55989722784154413846455963776007251813, endpoints:[172.16.70.44, 172.16.70.45, 172.16.70.43, 172.16.70.41, 172.16.70.42, 172.16.70.32], rpc_endpoints:[0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0, 0.0.0.0], endpoint_details:[EndpointDetails(host:172.16.70.44, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.45, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.43, datacenter:DC2, rack:RAC1), EndpointDetails(host:172.16.70.41, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.42, datacenter:DC1, rack:RAC1), EndpointDetails(host:172.16.70.32, datacenter:DC1, rack:RAC1)])

從describering的結果顯示Ring的排列方式如下:
4 -> 1 -> 5 -> 2 -> 3 -> 0 -> 4

關於Cassandra的compaction

以下為最近研究Compaction時的一些心得，有錯歡迎指正

Compaction做什麼?

Compaction指的是將數個 SSTable合併成一個SSTable的動作，並且在合併的同時會merges keys, combines columns, discards tombstones and creates a new index in the merged SSTable.

Compaction的種類

一般將Compaction分成 minor compaction與major compaction兩種

Major compaction

當一個column family在做Major compaction時，會將這個column family的所有的SSTable合併成一個

Minor compaction

Minor compaction是指 Cassandra 在 runtime 時，經由compaction strategy判斷而做的compaction，這類的compaction通常一次只會處理部份的SSTable

Major compaction 與 Minor compaction的定義真的是如此嗎?

cassandra 官方似乎沒有很明確的定義 minor compaction。但在 Datastax的這份nodetool compact 文件中指出："You can specify a keyspace for compaction. If you do not specify a keyspace, the nodetool command uses the current keyspace. You can specify one or more tables for compaction. If you do not specify a table(s), compaction of all tables in the keyspace occurs. This is called a major compaction. If you do specify a table(s), compaction of the specified table(s) occurs. This is called a minor compaction. " 這說法其實是有問題的，實際去追source code (0.8~2.1版)會發現，不管有沒有指定 column family，nodetool compact最後都是用ColumnFamilyStore.forceMajorCompaction() 這個method，而實際操作確實不管有沒有指定column family，nodetool compact完之後，column family的SSTable都會被合併成一個。因此本文件以行為來區分Major與Minor，至少到2.1版為止的行為都可以解釋得通(不過更之後的版本可能未必還是如此)

Cassandra一定要Major compaction才會清除tombstone嗎?

早期cassandra (<0.6版) 確實是如此，因此會需要定期執行 nodetool compact，但是後來minor compact也可以清除 tombstone了 ( CASSANDRA-1074 )

Compaction是怎麼發生的?

Compaction會自動發生(稱為 autoCompaction)，也可以手動強制執行。

AutoCompaction

根據column family的 compaction options 來決定是否觸發，基本上 autoCompaction只會觸發 minor compaction。關於compaction options 請自行參考適合你的cassandra column family設定文件

Force compaction

透過 nodetool compact 的指令或是JMX可以觸發強制 compaction。強制compaction會觸發 major compaction

做過手動 compaction 之後，autoCompaction就不會再做了嗎?

錯，只要沒特地把 compaction關掉，不然auto compaction還是會持續被執行。這個問題在網路上有不少人在詢問，主要應該是因為Datastax的幾篇舊的tuning文件( 0.8, 1.0 )有以下這樣的敘述："once you run a major compaction, automatic minor compactions are no longer triggered frequently forcing you to manually run major compactions on a routine basis. So while read performance will be good immediately following a major compaction, it will continually degrade until the next major compaction is manually invoked. "，這段文字其實是指當做過手動 major compaction之後，column family的SSTable會被合併成一個，這個新的SSTable會是一個相對很大的檔案，而SizeTieredCompactionStrategy的觸發條件是當有幾個接近大小的SSTable存在時，才會做minor compaction將這幾個同大小的SSTable做compaction，因此這個相對較大的SSTable在SizeTieredCompactionStrategy的運作下，可能很難再發生minor compaction，導致管理者未來只能透過major compaction才有機會對這個SSTable做compaction。(可能是因為這段話造成太多誤會，Datastax在 1.1版文件 裡也修改了描述方式) (在cassandra 1.2之後提供了 sstablesplit 的指令可以將這類的大 SSTable檔案切成數個小檔)

Major compaction只會做用在 SizeTieredCompactionStrategy 或 DateTieredCompactionStrategy 這兩種strategy嗎?

是的，Major compaction只會作用在這兩種strategy的column family，對LeveledCompactionStrategy執行major compaction不會有任何反應

要怎麼關掉autoCompaction

• ColumnFamily的JMX method disableAutoCompaction() 可用來在runtime時暫時關掉autoCompaction，但是並沒有enable的method，所以只有重啟node才能回復。
• 1.x版的Cassandra，將SizeTieredCompactionStrategy的 min_compaction_threshold跟max_compaction_threshold 都設成0可以讓該column family不再做autoCompaction
• 2.x版的column family的Compaction options有個 enable的選項可以控制是否啟用autoCompaction