Code coverage data for molecule tests

[shreekanthts]

Our team recently started using molecule for the test automation of ansible roles. I was looking for a way in which we could get the code coverage data/report after the molecule test runs.

I read molecule documentation, searched online regarding the code coverage, but I couldn't find any information regarding this. I really appreciate if someone can provide some details/doc/reference.

[cindyteng]

Hello. I would also like to know about this. A quantitative data would be a good representation.

[alexppg]

Any idea or roadmap for implementing this?

[ssbarnea]

DevTools team is aware about this and we do want to do something about it but the reality is that is bit unlikely to see anything implemented this year. Maybe if someone from the community would have some time to write a custom callback plugin for Ansible that gathers data that can be converted into coverage, we might be able to get some progress sooner.

CC @cidrblock @ganeshrn

[alexppg]

So your approach would be an ansible plugin rather that on molecule with verifiers? I ask because I wouldn't even know where to start, so maybe if the DevTools team could share some of the ideas on how to approach this, it would be kind of easier for somebody to try implement it.
Thanks for your response!

[drym3r]

I've been investigating a little bit and I've seen there's actually a junit callback plugin. I was ready to see if I could make it work with Coverage.py, which I figured it would be easier to make it work with this ecosystem. But when I read the code, I found that the xml wasn't particularly oriented to junit, it's actually quite generic.

i've been trying to make Coverage.py to generate a report, but it seems to need the database file, which I can't generate, since this is not really python code.

Anyway, the important bit is that there's actually an existing and working plugin that provides coverage information, which is what you @ssbarnea commented it would be useful.

I'll probably investigate how this could be used in molecule, but right now I don't know what would be the right approach.

[ssbarnea]

How about using https://github.com/ansible-collections/community.molecule to host this callback. We should make it work even without molecule and later we can add support for it molecule.

DevTools team will be more than happy to support anyone doing work in this area because that was a feature we already wanted to implement but had to delay to due having other more pressing fixes to implement.

Something that can be consumed by codecov would be great as this would allow us to use its github action and browsing abilities. Look at what it can use and see if you an even manually produce a report for it, at least to get an idea if that is doable.

[alexppg]

Codecov does have junit support. I checked with the allure framework, so there should be no problem in using it in codecov.

An allure capture:

The example playbook:

---
- hosts: localhost
  tasks:
    - ansible.builtin.shell: ls /
    - ansible.builtin.shell: hostname
    - ansible.builtin.shell: hostname
      when: 1 == 0

I can try if it really works in codecov, but before I'd like some clarification. Is this the wanted solution? I don't know well codecov, but what I've read is that is a SaSS solution, so it needs access to the website and it's not really a libre software option.

Is the intention to have something that works as quickly as possible and then work on alternatives or that would be it?

[alexanderbazhenoff]

At least I would like to have an opportunity to save a number of executed actions inside the role, the number of skipped to parse them manually. Is there any options to do that? Or manually molecule ... | tee -a executing.log then regex and wc -l? :)

[felipe4334]

Bump :)