數(shù)據(jù)湖自身會(huì)提供一些元數(shù)據(jù)的信息，我們可以利用這些信息進(jìn)行快速粗略的數(shù)據(jù)校驗(yàn)

分區(qū)數(shù)據(jù)數(shù)校驗(yàn)

以Iceberg為例，catalog.db.tb2.partitions可以得到分區(qū)的元數(shù)據(jù)，里面包括了分區(qū)下數(shù)據(jù)條數(shù)和文件大小等信息

select a.*, b.*
from (
    select partition.date, sum(record_count) record_count, 'old' flag
    from catalog.db.tb1.partitions
    group by partition.date
) as a
full join (
    select partition.date, sum(record_count) record_count, 'new' flag
    from catalog.db.tb2.partitions
    group by partition.date
) as b
on a.date = b.date
where a.record_count is null or b.record_count is null or a.record_count != b.record_count