Best practices for bundling tests with a contributed module?

[dcnorris]

When contributing a module via PR, what are best practices for bundling tests with the contribution?

• Will [suitably formatted] usage examples in predicate documentation headers get tangled, and automatically run at some point?
• Should modules self-test (say, via initialization/1) when they are loaded?
• What is going on in /tests/scryer/cli/? Is this a suitable place to lodge tests for contributed modules?
• How do contributors arrange to receive alerts in case their modules fail automatic tests?
• What additional questions did I neglect to ask?

[hurufu]

1. It would be a nice feature but it isn't supported, maybe (preliminary support for a subset of quads introduced by @UWN #2746) will evolve into it.
2. Probably only if module relies on runtime environment (for example if certain OS feature is supported or a certain dynamically loaded shared library exists).
3. It is the place for unit tests.
4. You will receive an email from github if CI pipline fails (it runs unit tests)

[hurufu]

Also, I've added a short note on how to run tests #2550

[dcnorris]

3. It is **the** place for unit tests.

Ah, I'm beginning to see how this works:

• To any src/lib/conmod.pl defining :- module(conmod, [...])., there should be a corresponding src/tests/conmod.pl that defines :- module(conmod_tests, []).
• This latter module should be filled with a bunch of test/2 clauses, each with a string identifier in arg 1 and goal in arg 2.
• Then in tests/scryer/cli/src_tests/ we need:
  • conmod_tests.toml giving command-line arguments to load src/tests/conmod.pl and run the tests in library(conmod_tests).
  • conmod.stderr and conmod.stdout for comparison against unit testing output.

[bakaq]

Kind of weird how my duct tape testing library that I made for library(dif) was de-facto adopted for this project. I guess it strikes a good balance between simplicity and expressivity. PlUnit seems to be the common way to do testing in Prolog, but it's extremely complicated and needs a lot of non-portable stuff to implement from what I saw.

I've been thinking of promoting this testing library I made to a builtin Scryer Prolog library after some adjustments. It's not good for everything, but it will make it so that users don't have to do the same improvised library themselves to do testing.

[dcnorris]

I can imagine that some libraries might need regression testing of a highly technical nature that users would rather not see. But in the case of the library(numerics/special_functions)) I'm preparing for PR, all tests I can conceive of at the moment warrant full transparency:

So I have included test code at the bottom of the module itself, and exported a self-test predicate:

?- use_module(library(numerics/special_functions)).
   true.
?- test_special_functions.
Looking for counterexamples ..
% erf is odd 
% pos root of erf(x)-x 
% erfc ≈ 1 - erf 
% inverf ≈ erf⁻¹ 
% ('false' is good) 
   false.

Given that I do however use the test/2 rubric established by your test_framework @bakaq, am I right in thinking tests/scryer/cli/src_tests/all_modules.toml serves as a catch-all, rendering a separate tests/special_functions.pl file unnecessary?

[bakaq]

all_modules.toml seems to test if you can load all modules at the same time. I think it would be beneficial to not overload the intention of each test case so that if they break is clearer what is broken. Notice that you don't actually need a tests/special_functions.pl for this library. Since you provide a test predicate, just put use_module(library(numerics/special_functions)). test_special_functions. in the .stdin file for your tests.