Skip to content

PLCgen: testing plan and test set for S7-300

NOTE: New versions of the PLCgen self-certification test set are now no longer delivered via this issue. Instead, they are now delivered via nightly releases of the ESCET IDE. See #798 (comment 2967758).

RWS wants to test the new PLC code generator in 2024-09. This means we should do our own tests before then, to test PLCgen, find issues, and fix them, beforehand. We want to reach a point that we have confidence in the correct functioning of the new PLCgen. I discussed with RWS (Lars, Koen) and we came to the following 'plan'.

Scoping:

  • Priority is S7-300 target. The hardware setup at the TU/e lab has these PLCs. It is currently also planned for the RWS projects, although that may still change next year. Thus we will not focus on testing of other targets.

Different types of tests:

  1. Start with small models, and grow towards real RWS objects.
  2. Make an ESCET test set with small models.
    • They should cover the CIF language concepts supported by PLCgen.
    • They should cover the different parts of the code generator.
    • This differs from the current integration tests in ESCET, as they are not only for regressions, and not only for checking the output (code). The goal is to compile them in TIA Portal and run them.
    • We thus need not only the test models, but also the expected output. For instance a variable that eventually gets a certain value or so. This could then be inspected in the debugger.

Even if the PLC target changes, the test set should still be usable for testing other targets, is our expectation.

We agreed that Lars and Koen will work on (1). Within ESCET we will make the test set with test expectations for (2). Lars and Koen will execute the tests for (2) on the lab setup at TU/e.

Lars and Koen will report back with any issues found. They will also use the PLCgen documentation, and see whether that is understandable, whether anything is incorrect, etc. And they'll provide back feedback on it. That way we can improve the documentation as well, and we'll end up with 'tested' documentation.

Tasks

  • Make first tests, and see if approach works.
  • v1: Put first set of tests in a branch.
  • v2: Make second set of tests, fixing the issues of v1:
    • Leave out runtime errors (fixes #845 (closed))
    • Document alternative OK values for test output (literals become integers for S7-300) (addresses #846 (closed))
    • Document that non-listed variables can be ignored (addresses #846 (closed))
  • v3: Extend the test set with new tests.
    • Improved explanation of different values as outcome.
    • Test without simplifying values and inlining constants (if supported).
    • Test nearly empty model, enums, timers and tuples.
  • v4: Extend the test set with new tests.
    • Extend timer test.
    • Test more expressions (extended tuples test, added 'other' expressions test).
    • Test more declarations (algebraic variables, constants, state variables).
  • v5: Fix new issues found in test models.
    • #845 (closed) Still division by zero in expr_binary model.
    • #861 (closed) PLCgen test model enums does not comply with expected results.
  • v6: Fix new issues and add another model.
  • v7: Removed 'cbrt' and 'pow' as they are no longer supported by PLCgen.
  • v8: Fix 'tuples' model for too large real values, see #863 (comment 2471245)
  • v9: Extended/added test models.
    • Extended edges and tuples test models.
    • Added channels, io and sync_events test models.
  • v10: Various improvements/fixes.
    • ToolDef script: generate rename warnings.
    • Adapted all test models to prevent renames, and list the real PLC variable names for the expected output.
    • Fixed io test model.
    • Improved expected output of timers test model.
    • Extended sync_events test model.
    • Added iter_limit1, iter_limit2, max_neg_int, names and negative_zero test models.
  • v11: Fixed output of a test model.
    • #889 (closed) PLCgen test model "channels" results in incorrect value
  • v12: Various added/extended tests.
    • Test long identifiers/names.
    • names test indentation improvements.
    • Extend 'names' test with more S7 keywords.
    • Add op_bind_and_prio_bool and op_bind_and_prio_nums test models.
    • Rewrap output comments about real values.
    • Improve/extend tuples test model.
    • Add arrays test, inspired by tuples test models.
    • Add array_tuple_combi test models.
    • Don't test models with arrays for S7-300.
  • v13: Fix and extend.
    • Fix ToolDef script exclude typo (comb instead of combi).
    • Fix sync_events test model's sync10 automaton expected output.
    • Extend sync_events test model with more monitor testing.
  • v14: Fix expected output of 'iter_limit1' test model. (#920 (closed))
  • v15
    • Extra comments in 'sync_events' test model.
    • Extend 'io' model with an algebraic variable as output.
  • v16: fix some test models
    • #961 (closed) PLCgen test model "negative_zero" fails due to maximum PLC cycle time
    • #962 (closed) PLCgen test model "op_bind_and_prio_num" does not download to device
  • v17: duplicated sync_events test for code generated in main/functions (per !1046 (merged)).
  • v18: Test adherence to CIF controller properties checker execution scheme (new ctrl_chk_exec_scheme.cif test).
  • v19
    • #981 (closed) Fix expected test output of 'sync_events_*' tests.
    • Add 'iter_limit3' test.
  • v20: Split iter_limit3 test into two tests (for main/funcs), to test the working of the isProgress variable also for event functions.
  • v21:
    • Fixed expected output of the names test.
    • Indicate alternative boolean/bit values for eliminated enums in all test models where it applies.
    • Indent fix in a arrays model.
  • v22
    • 'assignments' test: add empty line before expected output, for consistency with other tests.
    • 'iter_limit2' test: additional test instructions, for clarity.
    • 'io' test: provide more sane default I/O table, taking sizes into account, preventing overlapping addresses, and reordered boolean outputs.
    • 'tuples' test: document import/block-generation order, to prevent compilation issues (#863 (closed)).
    • 'tuples' test: one expected output line was missing hex numbers.
  • v23: loopsKilled -> loopsSaturated (per #1013 (closed))
  • v24
    • Add generation scripts for S7-400/1200/1500 and TwinCAT testing.
    • arrays test: remove testing of nested arrays, as PLCgen doesn't support it.
    • arrays test: extend with partial array assignments testing.
  • v25: Enabled 'enums' test for TwinCAT.
  • v26
    • Documented the excludes for different targets in the ToolDef scripts. The excludes themselves have not changed.
    • Improved the test instructions in all test models. The test models and their expected outputs have not changed.
  • v27
    • Improved the instruction comments in all test models.
    • Various changes to alternative expected values (-> ...):
      • Changed the syntax to make it easier to process / machine-processable.
      • Added a syntax for not allowed alternative values (-> NOT ...).
      • Provide alternatives for various real values (less precision, ...e+5 -> ...e+05, etc).
      • Provide alternatives for various large integers (with underscores in values).
      • Provide values per tuple field / array element.
    • Changes to specific tests:
      • channels/ctrl_chk_exec_scheme tests: fix variable names in expected output (dot -> underscore).
      • empty test: no longer has any expected results listed in the expected results section.
      • enums test: rename some variables, for consistency between the model and its expected output.
      • tuples: changes to avoid ambiguous S7 output for nested tuples.
      • io test is now explicitly indicated as requiring manual testing/checking.
      • *_funcs tests: split these tests into *_funcs_event and *_funcs_scope (see #1031 (closed)).
    • Added a script for automatic comparing of actual S7 output against the expected output.
  • v28
    • The tuples test no longer lists the struct program block creation order. The order did not match the implementation anymore. We already refer to the documentation, which explains what the order is, and that a .txt file is generated for S7 with that order.
    • The S7 compare script now creates the actual output files on the first run, allowing you to fill them in and then run the script again to do a comparison.
    • The S7 compare script now has easier instructions for copying the full table with actual output from TIA Portal.
  • v29: Fix expected values for enums and automata locations (affects TwinCAT, not S7). (#1036 (closed))
  • v30: Split names test into names_general and names_s7. TwinCAT code generation script comment style improvement.
  • v31
    • op_bind_and_prio_num*: more alternative expected output (fixes #1052 (closed)).
    • channels: double space -> single space, between alternatives (cosmetic change).
  • v32
    • channels test: fix expected outcome syntax.
    • expr_binary/expr_stdlib/tuples: add alternative real values as expected output. (fixes #1057 (closed))
  • v33: Split io test into 4 tests, one for each value of the --input-output-code-form option. (#1055 (closed))
  • v34
    • Split part of io* tests to new io_real test: (#1062 (closed))
      • The existing I/O tests only test bool/int now. The new io_real test tests real values.
      • New test is only in 1 variant, so not for different settings, to prevent many new tests, and to prevent duplicate testing of same option.
      • The io_real test is excluded for S7.
    • Make timers test easier to succeed: (#1076 (closed))
      • Improved test instructions.
      • Added variable t to more easily determine when to stop.
      • You now get 1 second instead of 0.5 seconds to stop the test at the right moment.
  • v35: Fixed expected output of channels test. (#1082 (closed))
  • v36: Fixed TwinCAT code generation script to keep the project file in which to generate the code when not running in debug mode.
  • v37: expr_stdlib: 6 additional expected output values, for S7-300.
  • v38: For all tests, the test instructions now refer to PLCgen as being the stable PLC code generator (no longer experimental). (#1102 (closed))

To see the progress, check out the branch 798-plcgen-s7-300-self-certification-tests here.

Or download the test sets here:

Addresses #679

Edited by Dennis Hendriks

Copyright © Eclipse Foundation, Inc. All Rights Reserved.     Privacy Policy | Terms of Use | Copyright Agent