Skip to content

[UnitTest] Make unit tests deterministic with fixed seed

Context

Make tests deterministic when the --rng-seed <seed> option is passed, as of now this option did not actually fixed the randomness seed.

For instance without specifying a seed, running the tests twice:

> build/unit_tests/tests_aidge_core
Randomness seeded to: 470397667
...
All tests passed (33485 assertions in 76 test cases)
> build/unit_tests/tests_aidge_core
Randomness seeded to: 2478385801
...
All tests passed (29563 assertions in 76 test cases)

The actual number of unit tests changes because the tests cases are randomly generated.

With this change one can also force a seed to execute the sames tests when debugging/developping: for instance --rng-seed 0:

> build/unit_tests/tests_aidge_core --rng-seed 0
Randomness seeded to: 0
...
All tests passed (63597 assertions in 76 test cases)
> build/unit_tests/tests_aidge_core --rng-seed 0
Randomness seeded to: 0
...
All tests passed (63597 assertions in 76 test cases)

Or to reproduce a previous test reported in an issue or other source (in this case the reporter must provided the randomness seed used as output by the tests, see above outputs).

For instance to reproduce the first test above, the random seed was 470397667:

> build/unit_tests/tests_aidge_core --rng-seed 470397667
Randomness seeded to: `470397667`
...
All tests passed (33485 assertions in 76 test cases)

Modified files

I just modified the seed generator to use the Catch::Generators::Detail::getSeed() function instead of the std::random_device non-pseudo random generator.

I.e. the seed generator is now:

#include <catch2/generators/catch_generators_random.hpp>
...
auto rd = Catch::Generators::Detail::getSeed;

Then as before the random number generator is initialized with:

std::mt19937 gen(rd());

This was done in every test were randomness is used.

I did not want to change much things and add a new file/include for exposing directly a random number generator, which would avoid each time to create the seed generator as above and then the random number generator for it with: std::mt19937 gen(rd());. This would simplify the 2-lines seed-generator+random generator init into a one line generator init, it's probably not worth it though it's an option.

Detailed major modifications

No major modification.

TODO

No TODO forecasted

Note though that I still experienced some no-determinism while running the unit tests, but there I suppose it is due to the non-determinism of some implementations such as the walk order of the graphviews nodes sets or possibly sources of non-determinism in the code.

Non-determinism in the tests harness should be fixed by this MR.

Edited by Christophe Guillon

Merge request reports