xds: FileWatcherCertificateProvider is leaked

[ejona86]

In investigating #11678 (comment) , it was discovered that XdsSecurityClientServerTest looks to create two FileWatcherCertificateProviders each test, but only shuts down one. Since it is using the Channel/Server APIs, this means one of the client or server is highly likely leaking the certificate provider.

CC @kannanjgithub

[vinodhabib]

@ejona86 @shivaspeaks Request you to provide the repro steps for the same so that it will be helpful for us to debug and fix the root cause/leakage issue.
in the above linked comments it was not clear on how to repro the same issue which you found earlier as most of the comments are related to flaky test.
Thanks in Advance.

[ejona86]

The flakiness was unrelated. Run XdsSecurityClientServerTest, and see that FileWatcherCertificateProvider.start() is called more than close().

It appears to apply to most tests, so you can choose your favorite and run just the one test. For example, in the xds/ directory, run ../gradlew test --tests 'XdsSecurityClientServerTest.tlsClientServer_noClientAuthentication[enableSpiffe=false]'. If you use println debugging, you can find the stdout/stderr in build/reports/tests/test/index.html.

I see SecurityProtocolNegotiators$ClientSecurityHandler in the stack of the non-shutdown instance, so it appears the leak is on client-side. Unfortunately, the client-side is overly complex in the number of classes involved.

[vinodhabib]

build/reports/tests/test/index.html

@ejona86 @shivaspeaks As per your above suggestion I can see the below println (added in start() and close() methods ) statement for 1 of the test (as attached in below snippet) as below with 2 consecutive starts call execution followed by 1 close call.

Here the expected behaviour for this UT should have only 1 start call followed by 1 close call as below? please confirm ?

Executed start()
Executed close()
Executed start()
Executed close()

[ejona86]

@vinodhabib The number of start() calls should match the number of close() calls. The number of start() calls per test is likely to be 2, one for server and one for client. But this issue is that some are started but not closed.