Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce hummock version overhead on L0 log store large retention #19960

Open
hzxa21 opened this issue Dec 30, 2024 · 0 comments
Open

Reduce hummock version overhead on L0 log store large retention #19960

hzxa21 opened this issue Dec 30, 2024 · 0 comments

Comments

@hzxa21
Copy link
Collaborator

hzxa21 commented Dec 30, 2024

Currently metadata of L0 log store is stored as TableChangeLog in hummock version, which will be propagated via the notification service.

Recently we have seen that then the metadata size of TableChangeLog is larger (e.g. due to 30-day subscription retention), the hummock version size can become several hundred MBs, which can cause inefficiency in hummock event handler (see the clone here) and potentially in other places.

Some ideas to improve it:

  • Don't store TableChangeLog in hummock version and exclude it from the notification. iter_log retrieves the TableChangeLog from meta on demand.
  • Keep TableChangeLog in hummock version and notification. But we can use a separate structure to store TableChangeLog in CN to avoid cloning the full TableChangLog on every new version delta.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant