Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated Kafka role for 0.10.x #58

Closed
wants to merge 977 commits into from
Closed

Updated Kafka role for 0.10.x #58

wants to merge 977 commits into from

Conversation

andrewkcarter
Copy link
Contributor

This updates the Kafka role from Kafka 0.9.x to 0.10.x. There are a series of steps to follow to perform a rolling upgrade of a functioning cluster. They are explained in the documentation linked below. I've distilled it it down the following specific steps.

https://kafka.apache.org/documentation/#upgrade

  1. Add the two following ansible variables so they apply to all brokers (i.e. production group vars file):
#### KAFKA VARS ####
kafka_inter_broker_protocol_version: "0.9.0"
kafka_log_message_format_version: "0.9.0"

These pin the message formats between brokers and the clients to our current versions.

For each broker in the Kafka cluster execute steps 2 - 4:

  1. Tail the broker logs to monitor the upgrade, shutdown, and restart:
tail -f /var/log/kafka/server.log
  1. Run the Kafka 0.10 role:
ansible-playbook -i <inventory> --limit <broker-node> --tags deps-kafka,site-kafka site-infrastructure.yml
  1. If the role didn't restart the Kafka broker after the update, then manually restart it.
supervisorctl restart kafka

-- GOTO Step 2 ---

Now the all brokers are running Kafka 0.10, we can now change the protocol versions to the latest.

  1. Remove the two Ansible variables introduced in step 1.

  2. Repeat steps 2 - 4 again for each broker to apply the protocol version change.

This process was tested on a two VM cluster running a series of Storm topologies actively using Kafka during the upgrade.

@andrewkcarter andrewkcarter requested a review from deecav February 13, 2017 20:35
deecav and others added 21 commits April 27, 2017 16:42
The latest including stats collector stopping before deletion which solved that issue.
…e conflicts and can't edit the conflicts due to an ansible-vault odd-length string error message due to the merge
Adds dogwhistle configuration (off by default), adds stats collection disabled
* Adding ability to define Kibana plugins, and run a local docker based ES for testing.

* script fix

* ensuring dockerfile updates are built

* Added config to turn on/off sll for kibana

* Fixed error in kibana deployment
@andrewkcarter andrewkcarter deleted the PULSE-3459 branch May 15, 2017 16:20
@andrewkcarter andrewkcarter restored the PULSE-3459 branch May 15, 2017 16:22
@andrewkcarter andrewkcarter reopened this May 15, 2017
@andrewkcarter andrewkcarter deleted the PULSE-3459 branch May 15, 2017 16:30
@andrewkcarter
Copy link
Contributor Author

This PR/branch got trashed by my GIT carelessness.

@andrewkcarter
Copy link
Contributor Author

New one is here: #66

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants