Portainer.io
Docker Swarm Setup Issue: memberlist: Was able to connect to 002.prod.if-0a35b2d92350 but other probes failed, network may be misconfigured #12471
Replies: 2 comments 4 replies
-
|
Are you running these nodes in VMware? This seems like an overlay network issue as your agents can't communicate with each other. please check our FAQ around this - https://portal.portainer.io/knowledge/known-issues-with-vmware. Another test is to deploy a service like busy box on each node using that network and test they can reach each other, we see overlay issues a fair bit as this is a common issue with Swarm not Portainer itself. |
Beta Was this translation helpful? Give feedback.
-
|
thank @Nick-Portainer for responding. these are physical servers and are able to communicate with each other in general over the wired network. As per your suggestion to check the overlay, i did a similar test using netshoot and found that it was working. but i will test it with the busybox too and come back. my initial suspect was docker network only, but all of my other services are able to talk to each other over the same overlay network. my idea is that somehow, the agents are trying to communicate with each other before they are ready and have the information about the host from the docker socket volume. it seems a race condition, and none of the agents are able to respond to each other what the other one is asking. the message logged also suggests the same. "connected with other agent but other probs are failed." agent doesn't provide much information in the debug mode, so I was not able to pinpoint the issue. when I have only one agent running, then it boots properly, and the portainer server also registers it successfully. I believe that the agent registration event loop makes them stop the other tasks, and they don't come in the ready state at all. on my request, please have a look into the piece of code that is responsible for making the agents ready and the registration event listeners. It seems that there is race condition here. thanks again for responding. |
Beta Was this translation helpful? Give feedback.
-
|
my suspicion seems to be true. since we all use the docker stack deploy for launching all the agent containers on all the nodes together they start communicating to others and declare each other dead because none of them were ready to respond. if we launch the agents one by one, they are able to work properly. here is what i did to validate my suspicions. added a label to each node as $ docker node update --label-add nodename=001.prod.if 001.prod.if
$ docker node update --label-add nodename=002.prod.if 002.prod.if
$ docker node update --label-add nodename=003.prod.if 003.prod.if
$ docker node update --label-add nodename=004.prod.if 004.prod.if
$ docker node update --label-add nodename=005.prod.if 005.prod.ifmodified my compose file as below: networks:
default:
name: proxy-network
external: true
services:
agent:
image: portainer/agent:latest
environment:
LOG_LEVEL: DEBUG
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
- /:/host
deploy:
mode: replicated
replicas: 1
placement:
preferences:
- spread: node.labels.nodename
constraints:
- node.platform.os == linuxplease note that i have changed the deployment mode to replicated in place of global with only 1 replicas. once the service is deployed, I updated the service to increase the number of replicas one at a time: $ docker service update --replicas=2 if-portainer-agent_agent
overall progress: 2 out of 2 tasks
$ docker service update --replicas=3 if-portainer-agent_agent
overall progress: 3 out of 3 tasks
$ docker service update --replicas=4 if-portainer-agent_agent
overall progress: 4 out of 4 tasks
$ docker service update --replicas=5 if-portainer-agent_agent
overall progress: 5 out of 5 tasksby this each agent got time to be ready before other agent starts probing it, for example agent the log for the agents are showing no issues and were successfully able to talk and register with each other. at this point when I deployed portainer (below compose file): it started working immediately. I believe that the agent code need optimisation. hope it is helpful in any sense to you or anyone else facing the similar issues and save some debugging efforts on overlay network. |
Beta Was this translation helpful? Give feedback.
-
|
Hello! I tried to replicate with the exact same setup as you (4 nodes instead of 5), but wasn't able to reproduce your issue.
docker swarm init --advertise-addr 192.168.99.252
docker swarm join --token SWMTKN-1-2n[...]fq 192.168.99.252:2377
docker network create --driver overlay --attachable --scope swarm proxy-network
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @xAt0mZ for looking into the issue. I am not sure what is different in what you have done and what I have here with me. I tried again to create the environment this time with VMs and i match the environment like OS version and docker version. I got same issues. if I start one by one they all work properly, but if deploy globally (simultaneously) then not. :( I think it is some sort of race condition, where on some environments, the nodes are ready to communicate immediately and others may not be. Can I request you to check the below information if it is matching. I remember this all started happening when my system kernel and docker version got updated. here is the system information. Operating System: Kernel Version: Docker Version: other docker relevant packages: Also, I observed that I have more than 1 master (3nos), may be that also can be a parameter as on masters agent behave differently as compared to workers and may take time to initialization in multi master environment. also your docker engine is quite old here is mine: Thanks again for looking into the issue! Regards, |
Beta Was this translation helpful? Give feedback.
-
|
there may be some compatibility issues too which might have been missed in the lts for docker version 27+ I was digging as saw some comment on compatibility relate discussion. |
Beta Was this translation helpful? Give feedback.
-
Ask a Question!
Reposting this issue here as i had already opened under portainer/agent portainer/agent#729
Hi i am not able to setup the agents to communicate with each other in s multi node Swarm. i have spent almost 3 days in trying to troubleshoot the issue but not able to figure out the reason and gone through almost all the forums and blogs and issue tracker.
Here the setup detail:
Servers:
Operating System:
Kernel Version:
Nodes:
Docker Version:
other docker relevant packages:
Agent Version:
created an overlay network as
output of
docker inspect proxy-network:[ { "Name": "proxy-network", "Id": "f32ckmy6wqgbo5iitd4yy69wv", "Created": "2025-01-06T10:43:47.998633153Z", "Scope": "swarm", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "10.0.1.0/24", "Gateway": "10.0.1.1" } ] }, "Internal": false, "Attachable": true, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": null, "Options": { "com.docker.network.driver.overlay.vxlanid_list": "4101" }, "Labels": null } ]compose file for the agent deployment:
output of
docker stack ps if-portainer-agent:output of
docker pson 001.prod.if:output of
docker logs d84eeec985ac:here is the output of
ip afrom 2 of the nodes just for the sake of MTU related diagnostics:from
001.prod.if:from
002.prod.if:please help me ... this my production environment and it is down from 3 days because of portainer. the only way forward, I believe is to mange the stacks using the CLI and go without portainer
Beta Was this translation helpful? Give feedback.
All reactions