You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
enable.auto.commit.store -- auto-stage offsets that have been read
we could potentially get rid of the internal offset stage that arroyo keeps track of, by storing the committed offsets in rdkafka and using enable.auto.commit to write them out periodically. this way we would not have to explicitly debounce commits.
arroyo still has a lot of code around retrying commits, but theoretically we could do away with that and only explicitly commit (with retries and everything else) in the revocation callbacks.
i'm not confident this would make the code simpler as it feels like we'd still need to store offsets in arroyo for other purposes + now we're interacting with rdkafka committing in two ways (auto vs in revocation)
The text was updated successfully, but these errors were encountered:
rdkafka has two options related to auto-committing:
enable.auto.commit
-- commit staged offsets automaticallyenable.auto.commit.store
-- auto-stage offsets that have been readwe could potentially get rid of the internal offset stage that arroyo keeps track of, by storing the committed offsets in rdkafka and using
enable.auto.commit
to write them out periodically. this way we would not have to explicitly debounce commits.arroyo still has a lot of code around retrying commits, but theoretically we could do away with that and only explicitly commit (with retries and everything else) in the revocation callbacks.
i'm not confident this would make the code simpler as it feels like we'd still need to store offsets in arroyo for other purposes + now we're interacting with rdkafka committing in two ways (auto vs in revocation)
The text was updated successfully, but these errors were encountered: