个人中心
gpt4 book ai didi

Clickhouse kafka integration tables cleaning(Clickhouse Kafka集成表清理)

转载 作者：bug小助手 更新时间：2023-10-25 17:57:39 26 4
gpt4 key购买 nike



I use kafka integration mechanism to transfer data via chain (kafka engine)->(materialized view with TO)->(destination table)

我使用Kafka集成机制通过链(Kafka引擎)->(带TO的实例化视图)->(目标表)进行数据传输


materialized view reads any data block coming to kafka table , and use view source select to add data to destination table and also the data is materialized inside the mat view inself. Sometimes I need to remove data from materilized view and remove data from dest table. I plan to remove old version rows based on two version keys with specific order. I have specific delete statement that working well to remove old version rows based mention keys (using PARTITION BY). So overall plan is DETACH mat view from dest table to stop kafka message streaming into mat view -> destination table , do my delete statement , delete data from mat view , attach mat view back after all. I use standard mergetree engine for dest table and I do not want any other engine (aggmergetre, replacingmergeetre or others) because of standard mergetree logic is better for me. I know that by using CREATE MATERIALIZED VIEW with "TO" make mat view engine to be inherited from dest table.

物化视图读取任何进入Kafka表的数据块，并使用view源SELECT将数据添加到目标表中，并将数据物化到mat view自身内部。有时我需要从物化视图中删除数据，并从DEST表中删除数据。我计划删除基于具有特定顺序的两个版本键的旧版本行。我有特定的DELETE语句，可以很好地删除基于提及键的旧版本行(使用PARTITION BY)。所以总体计划是将MAT视图从DEST表分离，以阻止Kafka消息流入MAT视图->目标表，执行我的DELETE语句，从MAT视图中删除数据，最后将MAT视图附加回来。我对DEST表使用标准的mergetree引擎，我不想要任何其他引擎(aggmergetre、replacingmergeetre或其他引擎)，因为标准的mergetree逻辑更适合我。我知道，通过使用带有“to”的CREATE MATERIALIZED VIEW使mat view引擎从DEST表继承。


Question is it safely to remove data from mat view during DETACH active time ? I mean will I not hurt integration mechanism ? Probably some rows came to mat view but not dest table because of my DETACH statement during same time , and I would probably delete data that not transfered to dest table else.

问题：在分离活动期间从MAT视图中删除数据是否安全？我的意思是，我会不会损害融合机制？可能在同一时间，由于我的DETACH语句，一些行进入了MAT视图，但没有进入DEST表，并且我可能会删除没有转移到其他DEST表的数据。


更多回答

There is no such meaning as "MV engine". MV itself doesn't store any data. MV is used to transform data you get from a Kafka Engine table and place this data to a destination table. You run DETACH on a Kafka Engine table, and yes, you don't loose any data in a Kafka topic when you run ATTACH afterwards. So, back to your question - yes, it's safe to do what you mentioned.

没有“MV引擎”这样的意思。MV本身并不存储任何数据。MV用于转换您从Kafka引擎表获得的数据，并将该数据放置到目标表中。您在Kafka引擎表上运行DETACH，是的，当您随后运行ATTACH时，您不会丢失Kafka主题中的任何数据。所以，回到你的问题上--是的，你提到的事情是安全的。

Thank you , so this means when I select MV created with "TO" I just select table which tied with this view?

谢谢，所以这意味着当我选择MV创建时，我只是选择了与此视图绑定的表？

If you query MV, the corresponding destination table (internal or used in TO) is queried internally with the same statement.

如果您查询MV，则会使用相同的语句在内部查询对应的目标表(内部或在To中使用)。

优秀答案推荐
更多回答

26 4 0
文章推荐： Compose Multiplatform (Desktop) - How to restrict window dragging to not go out of display bounds without flickering/artifacts?(合成多平台(桌面)-如何限制窗口拖动不超出显示边界而不闪烁/不出现瑕疵？)
文章推荐： Deleting image related to a product from static folder when deleting product. Django(删除产品时，从静态文件夹中删除与产品相关的图像。姜戈)
文章推荐： .net core HTTPClient with NTLM authentication(支持NTLM身份验证的.NET核心HTTPClient)
文章推荐： Why I get RAM leakage while running Yolov5 tensorflow inference server (FASTAPI) with docker?(为什么在运行带有docker的Yolov5 TensorFlow推理服务器(FASTAPI)时出现内存泄漏？)
bug小助手
个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章
滴滴打车优惠券免费领取
滴滴打车优惠券
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com