背景
- 避免CREATE/DROP/RENAME TABLE鎖
- 實現RENAME數據庫操作
- 正在執行的query發現表不存在異常
- rename errors(rename table A to B, C to A is not atomic)
- Atomic數據庫支持non-blocking DROP/RENAME表，支持原子交換表名EXCHANGE TABLES t1 AND t2
準備
- 啟動ClickHouse server
docker run -d --name test-clickhouse-server-20-12 --ulimit nofile=262144:262144 yandex/clickhouse-server:20.12.4.5
- 啟動ClickHouse client
docker run -it --rm --link test-clickhouse-server-20-12:clickhouse-server yandex/clickhouse-client:20.12.4.5 --host clickhouse-server
- 查看ClickHouse server文件系統
docker exec -ti test-clickhouse-server-20-12 /bin/bash
- 創建數據庫
CREATE DATABASE atom ENGINE = Atomic
select * from system.databases where name = 'atom' format Vertical;
Row 1:
──────
name: atom
engine: Atomic
data_path: /var/lib/clickhouse/store/
metadata_path: /var/lib/clickhouse/store/5db/5db78374-217a-42bf-97d2-9ceaddc176af/
uuid: 5db78374-217a-42bf-97d2-9ceaddc176af
- 創建表
create table atom.test(`id` Int64, `name` String, `birthday` DateTime DEFAULT toDateTime('0000-00-00 00:00:00')) ENGINE = MergeTree() PRIMARY KEY(id) ORDER BY(id);
insert into atom.test (id, name, birthday) VALUES (1, 'test1', '2019-10-02 01:00:00'), (2, 'test2', '2019-10-02 02:00:00'),(3, 'test3', '2019-10-02 03:00:00'),(4, 'test4', '2019-10-02 04:00:00');
Atomic數據庫的結構
數據庫有唯一的uuid,數據庫中的每個表有唯一的uuid
metadata存儲
root@0cd2cde6e19a:/var/lib/clickhouse/metadata# tree
.
├── atom -> /var/lib/clickhouse/store/5db/5db78374-217a-42bf-97d2-9ceaddc176af
├── atom.sql
├── default -> /var/lib/clickhouse/store/0c9/0c9063bd-26d8-438b-b3e9-5d8cb5745e3d
├── default.sql
├── system -> /var/lib/clickhouse/store/d39/d39d0db0-08e8-4752-9661-5034256821e9
└── system.sql
根目錄和老版本相同，仍然是/var/lib/clickhouse/metadata,其下目錄結構也相同，只不過每個數據庫名目錄實際上是一個超鏈接，指向實際的存儲目錄/var/lib/clickhouse/store/(prefix of database_uuid)/(database_uuid)
數據存儲
root@0cd2cde6e19a:/var/lib/clickhouse/data# tree
.
├── atom
│ ├── test -> /var/lib/clickhouse/store/6bb/6bb77284-2a07-4fff-aa84-932baba39fc0
│ └── test2 -> /var/lib/clickhouse/store/8c2/8c284554-1125-4bcf-8af0-5c697a735de4
├── default
└── system
├── asynchronous_metric_log -> /var/lib/clickhouse/store/7ed/7ed107ce-00c0-4080-96ac-8264d82a7e7d
├── metric_log -> /var/lib/clickhouse/store/b7f/b7ff5e00-b567-4e14-a89c-177d80808583
├── query_log -> /var/lib/clickhouse/store/50b/50b0ed02-df82-4109-a550-13d559cc28e3
├── query_thread_log -> /var/lib/clickhouse/store/6b5/6b58386c-da69-4136-baad-b3911a49fcc8
└── trace_log -> /var/lib/clickhouse/store/91a/91ad6abd-1e9f-44b6-a8e9-012d31dbb165
根目錄和老版本相同，仍然是/var/lib/clickhouse/data,其下目錄結構也相同，只不過每個表名目錄實際上是一個超鏈接，指向實際的存儲目錄/var/lib/clickhouse/store/(prefix of table_uuid)/(table_uuid)
以test表為例
以test表為例，其基本信息如下：
select * from system.tables where database = 'atom' and name = 'test' format Vertical;
Row 1:
──────
database: atom
name: test
uuid: 6bb77284-2a07-4fff-aa84-932baba39fc0
engine: MergeTree
is_temporary: 0
data_paths: ['/var/lib/clickhouse/store/6bb/6bb77284-2a07-4fff-aa84-932baba39fc0/']
metadata_path: /var/lib/clickhouse/store/5db/5db78374-217a-42bf-97d2-9ceaddc176af/test.sql
metadata_modification_time: 2021-03-16 12:23:11
dependencies_database: []
dependencies_table: []
create_table_query: CREATE TABLE atom.test (`id` Int64, `name` String, `birthday` DateTime DEFAULT toDateTime('0000-00-00 00:00:00')) ENGINE = MergeTree() PRIMARY KEY id ORDER BY id SETTINGS index_granularity = 8192
engine_full: MergeTree() PRIMARY KEY id ORDER BY id SETTINGS index_granularity = 8192
partition_key:
sorting_key: id
primary_key: id
sampling_key:
storage_policy: default
total_rows: 4
total_bytes: 266
lifetime_rows: ????
lifetime_bytes: ????
設計的好處
ClickHouse將邏輯目錄和數據存儲目錄分開，只是使用超鏈接地方式進行鏈接，這樣使得表邏輯結構地更改獨立與數據目錄，比如說重新命名數據庫，重新命名表等變得非常簡單。
基礎操作
重命名數據庫
0cd2cde6e19a :) rename database atom to atom2;
RENAME DATABASE atom TO atom2
Query id: 15f6dba4-1485-487a-91c7-447fa1f13155
Ok.
0 rows in set. Elapsed: 0.002 sec.
重命名數據庫，操作非常簡單，并且其底層uuid并不發生變化。
0cd2cde6e19a :) select * from system.databases where name = 'atom2' format Vertical;
SELECT *
FROM system.databases
WHERE name = 'atom2'
FORMAT Vertical
Query id: de4f0867-61ab-4fe4-9e1b-7394ec04da3c
Row 1:
──────
name: atom2
engine: Atomic
data_path: /var/lib/clickhouse/store/
metadata_path: /var/lib/clickhouse/store/5db/5db78374-217a-42bf-97d2-9ceaddc176af/
uuid: 5db78374-217a-42bf-97d2-9ceaddc176af
相當于重新創建了atom2的超鏈接，同時將文件atom.sql重命名為atom2。
root@0cd2cde6e19a:/var/lib/clickhouse/metadata# tree
.
├── atom2 -> /var/lib/clickhouse/store/5db/5db78374-217a-42bf-97d2-9ceaddc176af
├── atom2.sql
├── default -> /var/lib/clickhouse/store/0c9/0c9063bd-26d8-438b-b3e9-5d8cb5745e3d
├── default.sql
├── system -> /var/lib/clickhouse/store/d39/d39d0db0-08e8-4752-9661-5034256821e9
└── system.sql
重命名表
0cd2cde6e19a :) rename table test to new;
RENAME TABLE test TO new
Query id: f2faf21a-5190-4cae-9127-30941a1bf35d
Ok.
0 rows in set. Elapsed: 0.002 sec.
uuid不會改變，不會發生數據移動。
0cd2cde6e19a :) select * from system.tables where database = 'atom2' and name = 'new' format Vertical;
SELECT *
FROM system.tables
WHERE (database = 'atom2') AND (name = 'new')
FORMAT Vertical
Query id: 21ce2574-009a-409b-81a1-b80a036cf722
Row 1:
──────
database: atom2
name: new
uuid: 6bb77284-2a07-4fff-aa84-932baba39fc0
engine: MergeTree
is_temporary: 0
data_paths: ['/var/lib/clickhouse/store/6bb/6bb77284-2a07-4fff-aa84-932baba39fc0/']
metadata_path: /var/lib/clickhouse/store/5db/5db78374-217a-42bf-97d2-9ceaddc176af/new.sql
metadata_modification_time: 2021-03-16 12:23:11
dependencies_database: []
dependencies_table: []
create_table_query: CREATE TABLE atom2.new (`id` Int64, `name` String, `birthday` DateTime DEFAULT toDateTime('0000-00-00 00:00:00')) ENGINE = MergeTree() PRIMARY KEY id ORDER BY id SETTINGS index_granularity = 8192
engine_full: MergeTree() PRIMARY KEY id ORDER BY id SETTINGS index_granularity = 8192
partition_key:
sorting_key: id
primary_key: id
sampling_key:
storage_policy: default
total_rows: 4
total_bytes: 266
lifetime_rows: ????
lifetime_bytes: ????
1 rows in set. Elapsed: 0.005 sec.
相當于修改了表的超鏈接，同時修改表的metedata信息
root@0cd2cde6e19a:/var/lib/clickhouse/data# tree
.
├── atom2
│ ├── new -> /var/lib/clickhouse/store/6bb/6bb77284-2a07-4fff-aa84-932baba39fc0
│ └── test2 -> /var/lib/clickhouse/store/8c2/8c284554-1125-4bcf-8af0-5c697a735de4
├── default
└── system
├── asynchronous_metric_log -> /var/lib/clickhouse/store/7ed/7ed107ce-00c0-4080-96ac-8264d82a7e7d
├── metric_log -> /var/lib/clickhouse/store/b7f/b7ff5e00-b567-4e14-a89c-177d80808583
├── query_log -> /var/lib/clickhouse/store/50b/50b0ed02-df82-4109-a550-13d559cc28e3
├── query_thread_log -> /var/lib/clickhouse/store/6b5/6b58386c-da69-4136-baad-b3911a49fcc8
└── trace_log -> /var/lib/clickhouse/store/91a/91ad6abd-1e9f-44b6-a8e9-012d31dbb165
刪除表
- 刪除表不會發生數據移動，通過把metadata信息移動到
/var/lib/clickhouse/metadata_dropped標記刪除 - 真正運行的query可以繼續使用被刪除表，指導表不被使用，才會被真正刪除
- 刪除表的超鏈接
- 刪除表的metadata信息
0cd2cde6e19a :) drop table test2;
DROP TABLE test2
Query id: ec8aa65f-b4de-41ff-8e51-6b0ad4a59c76
Ok.
0 rows in set. Elapsed: 0.002 sec.
root@0cd2cde6e19a:/var/lib/clickhouse/metadata_dropped# ll
total 12
drwxr-x--- 2 clickhouse clickhouse 4096 Mar 16 13:35 ./
drwx------ 13 clickhouse clickhouse 4096 Mar 16 11:58 ../
-rw-r----- 1 clickhouse clickhouse 246 Mar 16 12:39 atom2.test2.8c284554-1125-4bcf-8af0-5c697a735de4.sql
