Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate whether we can use auto-commit in arroyo #414

Open
untitaker opened this issue Dec 18, 2024 · 0 comments
Open

Investigate whether we can use auto-commit in arroyo #414

untitaker opened this issue Dec 18, 2024 · 0 comments

Comments

@untitaker
Copy link
Member

rdkafka has two options related to auto-committing:

  • enable.auto.commit -- commit staged offsets automatically
  • enable.auto.commit.store -- auto-stage offsets that have been read

we could potentially get rid of the internal offset stage that arroyo keeps track of, by storing the committed offsets in rdkafka and using enable.auto.commit to write them out periodically. this way we would not have to explicitly debounce commits.

arroyo still has a lot of code around retrying commits, but theoretically we could do away with that and only explicitly commit (with retries and everything else) in the revocation callbacks.

i'm not confident this would make the code simpler as it feels like we'd still need to store offsets in arroyo for other purposes + now we're interacting with rdkafka committing in two ways (auto vs in revocation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant