ClickHouse

ClickHouse是一個用于聯(lián)機分析(OLAP)的列式數(shù)據(jù)庫管理系統(tǒng)(DBMS)。

OLAP場景的關(guān)鍵特征

絕大多數(shù)是讀請求

數(shù)據(jù)以相當(dāng)大的批次(> 1000行)更新，而不是單行更新;或者根本沒有更新。

已添加到數(shù)據(jù)庫的數(shù)據(jù)不能修改。

對于讀取，從數(shù)據(jù)庫中提取相當(dāng)多的行，但只提取列的一小部分。

寬表，即每個表包含著大量的列

查詢相對較少(通常每臺服務(wù)器每秒查詢數(shù)百次或更少)

對于簡單查詢，允許延遲大約50毫秒

列中的數(shù)據(jù)相對較小：數(shù)字和短字符串(例如，每個URL 60個字節(jié))

處理單個查詢時需要高吞吐量(每臺服務(wù)器每秒可達數(shù)十億行)

事務(wù)不是必須的

對數(shù)據(jù)一致性要求低

每個查詢有一個大表。除了他意以外，其他的都很小。

查詢結(jié)果明顯小于源數(shù)據(jù)。換句話說，數(shù)據(jù)經(jīng)過過濾或聚合，因此結(jié)果適合于單個服務(wù)器的RAM中

很容易可以看出，OLAP場景與其他通常業(yè)務(wù)場景(例如,OLTP或K/V)有很大的不同， 因此想要使用OLTP或Key-Value數(shù)據(jù)庫去高效的處理分析查詢場景，并不是非常完美的適用方案。例如，使用OLAP數(shù)據(jù)庫去處理分析請求通常要優(yōu)于使用MongoDB或Redis去處理分析請求。

ClickHouse安裝和啟動

sudo apt-get install apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E0C56BD4
echo "deb https://repo.clickhouse.tech/deb/stable/ main/" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update
sudo apt-get install -y clickhouse-server clickhouse-client
sudo service clickhouse-server start
clickhouse-client

sudo apt-get update如果更新不了可以修改下源，然后修改source.list，clickhouse.list

image

image

新建表導(dǎo)入所需要的數(shù)據(jù)(120038310條經(jīng)緯度)

image

image

create table pnts (Lon  Float64, Lat Float64) engine = MergeTree() order by (Lon, Lat);
time clickhouse-client --query="INSERT INTO pnts FORMAT CSVWithNames" < test_data.csv

ClickHouse空間分析運用

計算最大小經(jīng)緯度

select min(Lon), max(Lon),min(Lat), max(Lat)FROM pnts

image

select geohashesInBox(-150.0565255, 24.5449115001 , -66.950609997, 65.1341342731, 4)

image

面篩選

SELECT count(1) FROM pnts WHERE -122.1603396012797162<=Lon and -122.1414371044578786>=Lon and 37.7842593304459271  <=Lat and 37.7965938218938646  >=Lat  and pointInPolygon((Lon, Lat), [( -122.158001836655117, 37.796593821893865 ), ( -122.151099864906271, 37.793632653369357 ), ( -122.146446600082044, 37.792029614769625 ), ( -122.141437104457879, 37.789224297220095 ), ( -122.143485431557536, 37.788534100045212 ), ( -122.14290655650764, 37.786641623920524 ), ( -122.143797133507491, 37.786196335420598 ), ( -122.147648879031848, 37.784437445845896 ), ( -122.14862851373168, 37.784259330445927 ), ( -122.15312592758093, 37.788734479870179 ), ( -122.154728966180656, 37.789736378995009 ), ( -122.156443326905375, 37.788801273145168 ), ( -122.160094692604758, 37.789335619345074 ), ( -122.15915958675491, 37.792185465744602 ), ( -122.160339601279716, 37.793788504344327 ), ( -122.159293173304889, 37.795369278519068 ), ( -122.159048264629931, 37.795525129494038 ), ( -122.159048264629931, 37.795525129494038 ), ( -122.158001836655117, 37.796593821893865 )]) = 1

image

緩沖區(qū)計算

SELECT count(1) from pnts WHERE greatCircleDistance(Lon, Lat, -122.158001836655117, 37.796593821893865 )<=2000

image

select geoToH3(Lon, Lat, 3),count(1) FROM pnts group by geoToH3(Lon, Lat, 3)

image

geohash聚合

SELECT geohashEncode(Lon, Lat, 5),count(1) FROM pnts group by geohashEncode(Lon, Lat, 5)

image