CIF use experience & feedback by Perry from ICT
I met Perry van Wesel at the ESI Symposium 2024. He mentioned he was trying out CIF, and wanted to share some experiences. He then tried to post to the escet-dev list, but it got rejected as he was not registered for the list. I asked if he wanted to create an issue for it, but he did not want to create an Eclipse Foundation account to do so. He did agree with me creating an issue for it.
The email
The email from Perry:
From: Perry van Wesel [email address omitted]
[...]
Subject: CIF use experience & feedbackAttempt two after mail getting rejected by the eclipse mailing list because I’m not a member:
Hello ESCET team,
Some time ago I met Michel at a conference, and he mentioned ESCET as a tool similar to the ones we have been exploring (Dezyne, Coco, little bit of LSAT) at our internal research/innovation project here at ICT. It sounded interesting so we spent some time looking at how this tool fits into our software ecosystem as it is now. I promised Dennis yesterday I would send an email with our experience and feedback, so here it is. If you have further questions, remarks, or want to discuss this further, feel free to reply.
What we did:
We have a sorting line that can sort disks to one of three output lanes, based on their color. This system looks like this:
Conveyor line: ----| color sensor | --------| ejector1 | ------- | ejector 2 | -------- | ejector 3 | | | | | | | | lane 1 | | lane 2 | | lane 3 |
The disk comes in on the left, is scanned by the color sensor and based on the detected color is pushed into lane 1/2/3 by the corresponding ejector. The controller for this is an industrial-grade PLC controller from Beckhoff, with the ability to run C(++) code.
We modelled this system in CIF and then replaced our hand-written/coco-generated controller code with the C99 output from the CIF code generator. We have not had any training on this and just figured it out based on the docs and our knowledge of model-based engineering. It took us roughly 2 days with 2 people to make the model, integrate it onto our hardware and to debug to get it working as we wanted. Note that our feedback is coming from this experience of trying it out with no outside help, maybe this is different when trainings are provided, or maybe this can be a nice outside view on your documentation and examples.
Our experiences and feedback:
The paradigm:
Coming from tools like Coco, we are used to write the supervisory control ourselves and know how difficult it is to get something that is safe but also performant. Using the paradigm of always-act-when-possible you ‘hide’ a lot of logic about execution flow in the underlying assumptions. This is both a blessing and a curse: At first it can be hard to oversee the impact of such a fundamental change in thinking about software, and it makes it hard to understand what the software is actually doing, just by looking at the code (because you need the information about how the runtime works). Having the ability to couple any plant to any other plant from any file makes it even harder to oversee what the full system actually does, you can have requirements for a certain edge possibly spread out over 10 different files, so it becomes extremely hard to understand what a plant is doing just by looking at its plant definition. As we get used to it we can definitely see the benefits to this paradigm by not having to write interaction/synchronization logic, more time is needed to get used to it and experiment.
Docs & examples:
We were really happy with the amount and quality of the docs. The fact we were able to get our model working and integrated with no previous experience shows the documentation is quite complete and understandable. We do think however that (based on the mess we were able to make in a short time), it would be good to have a ‘best practices’ prominently available, on how to write and organize your models. A kind of way-of-working would also be nice, i.e. when do you run which tool, for what purpose etc.
Having a lot of examples available locally is very nice, kudos. Our code was mostly made by looking at the examples. The quality of these examples provided with CIF could be improved. The readability of these examples varies a lot. Usage of one-letter variables and acronyms, and no comment headers explaining the purpose of the model make it very hard to understand what the model is actually meant to do. The complicated hybrid/bridge example is practically unreadable due to all the abbreviated variable names. It also contains a lot of weird ways of modelling and copy paste code, i.e. plant S1, plant S2, plant Red, Green and plant Red2 in bridge/synthesis/plant.cif, instead of reinstantiating the same plant multiple times with different names.
We also found that some of the examples appear to be broken when running certain checks, i.e. hybrid/fluid/fluid.cif fails with a lot of red errors when you run the controller checks. Running controller checks on the bridge plant never even finishes with no kind of progress indication.
SVG visualization:
This idea of using SVGs like this is very cool. It is really easy to make some nice visualizations that are useful during development with little effort. Only point of improvement I would mention is that in my opinion it is quite messy to have all of the SVG binding code mixed together with your actual production models in the example. I think I saw some examples where the SVG bindings are (partially) in a separate visualization file. If I were writing production code, I would not pollute my business logic with my dev viz, and keep it strictly separate.
Different checks and scripts:
We mostly used the ‘Apply Controller Checks’ and cifcodegen while developing. We unified this by making a toolsdef script to run the scripts. We noticed that sometimes the codegen will notice issue that the controller checks did not find, and sometimes the controller checks finds issue that the codegen was totally fine with. It is a bit confusing when to run what and what checks matter and when. As a contract, in Coco we have two buttons: ‘Check’ which does ALL verifications and validations of your model, and ‘Generate’ which just writes generated code to output. Having different scripts validate different things which can be forgotten, is an unnecessarily confusing process if you ask me.
External stimuli to the system:
As mentioned before, we are experienced with Dezyne/Coco where all stimuli to and from the model are events. These events can come from any external source and sent into the model via some glue. We could not figure out how to make this work with the CIF generated code, the only way we found to affect the model was to change the inputs, ie when a user presses a button, the only thing we can do is change one of the input bools from false to true, we could not send the pressed event into the system. This seems a bit awkward compared to what we are used to, but we can understand this is the paradigm and this is how to interact with the system. However, it is then very confusing to see the examples like button_lamp bind a svg click to an event in the model. Why is it that we can trigger controllable events from the SVG visualization and the simulation, but we cannot trigger events from code? It feels a bit schizophrenic.
State diagram visualization:
Minor gripe, but getting a decent state visualization of the system is quite annoying. Why do we need to install an external non-open source tool like yEd to get a visualization of our model? Why not output a dot file for graphviz, or a plant uml or mermaid.js diagram. Could show the diagram directly in Eclipse instead?
In conclusion, we think CIF is very interesting, and think that potentially it could be a great replacement for manually writing supervisory controllers where you can model your subcomponents easily. For now it still seems a little immature in comparison with our other tools. Like mentioned before if you have questions/feedback/want to discuss more, feel free to reply and we can discuss further!
Kind regards,
Perry van WeselICT Netherlands
Summary
I would summarize the experiences, in my own words, as follows:
Positive experiences
- Experiment: They managed to model their system, perform controller checks, and generate C99 code and get it to work on their PLC on their own, with just the documentation and examples. The also used ToolDef scripts for automation.
- Paradigm: They see the benefit of not having to explicitly write interaction/synchronization logic, and to get assistance in developing a supervisor that is safe and performant. They find it very interesting.
- SVG visualization: They like the idea of SVG visualization a lot, found it really easy to use, and useful.
Improvement points
- General impression: It seems to them that CIF less mature in comparison to the other tools they are accustomed to.
- Paradigm: The concept of synchronizing events is fundamentally different from explicitly coding interaction/synchronization logic, and requires a fundamentally different way of thinking about software, that takes some time to get used to.
- Paradigm: Information about events (such as edges) can be spread out over many automata, in many files, making it difficult to oversee the behavior for an event, although this may be something you just have to get used to.
- SVG visualization: It is quite messy to have all SVG declarations mixed throughout the model, as is done in some of the CIF example models. Business logic and visualization should not be mixed.
- Controller checks vs code generation: The controller checker and code generator report different issues, which is confusing as to what checks to do and when, leading to a confusing process.
- External stimuli: Being accustomed to interaction through events from the environment, they did not like that generated C code can only interact through input variable value changes. In SVG visualization in the simulator, as shown in CIF examples, they could link SVG inputs to events, but in generated code they had to use input variables. They found this very confusing and schizophrenic.
- Model visualization: They were quite annoyed by having to install yEd to graphically view the model, and would rather see something be integrated into the Eclipse ESCET IDE.
My thoughts
Positive experiences
My thoughts on their positive experiences:
I like that they were able to get it all working, by themselves, in just two days. They seem to have picked up on the strong points of our approach/toolkit.
Improvement points
My thoughts on their improvement points:
- General impression: To a large degree, maturity is in the eye of the beholder, I think. Your first brief experience influences that a lot. I don't see direct actions for us to take on this general impression feedback.
- Paradigm: We know that synchronizing automata, and thinking in restrictions takes some time to get used to. It is the way it is, and I have no intention of changing this paradigm.
- Paradigm: That information about events/behavior can be spread out over automata/models depends greatly on how you model it. But, indeed, this can happen. Experience helps. The yEd model and relations diagrams may also help here.
- SVG visualization: I see pros and cons to having the SVG mappings close to and mixed with the parts of the model that they affect. Having them close together makes it easier to change, as it easier to find them. Having them separated can also have benefits, as then there is separation of concerns. Both are indeed possible. We have CIF examples that showcase both. As such, I'm not sure something should be changed here.
- Controller checks vs code generation: The controller checker checks whether a supervisor model satisfies some properties that make it a valid controller. The CIF code generator checks that input models satisfy the precondition that allow code to be generated from them, thus checking that the model is supported by the code generator, which only supports a subset of language concepts. Indeed they check different things, by design. I don't think the CIF code generator lists the controller checker in its documentation as the new PLC code generator does. Likely it would be good to add it there as well. Furthermore, we have some ideas to add the outcome of the controller checks to the model with annotations, such that code generators can pick these up. They can then warn if controller checks were not done. We could later also consider integrating the controller checker into the code generators, to have them automatically check these conditions if the annotations are not present, to have it more automated and integrated.
- External stimuli: Recently we made it possible to have SVG input mappings link not only to events, but also to input variables (although we need to support this still in tools like the simulator). Generated code can currently indeed only interact via input variables, not events. Automata can be modeled that upon changes to input variables enable certain events, such that generated code can, though input variable changes, indirectly lead to events being taken. We could consider supporting the environment enabling/disabling certain (uncontrollable) events also in generated C/PLC/etc code, thus making putting SVG input mapping on the same level as environment interaction in generated code.
- Model visualization: Indeed yEd is not open source. But, it can be downloaded and used for free, I believe. If I remember correctly, there are commercial/paid plugins, also for the Eclipse IDE, that allow to integrate yEd/yFiles. But, we can't include that in our open-source project. We could check again, to see if another non-commercial is now available. We could also consider, besides generated yEd models, also having other visualizations of CIF models, including those that can be integrated into the Eclipse ESCET IDE.
Other thoughts
Besides the positive experiences and improvement points, I had the following thoughts when I read the email:
- I'm not sure whether they actually understood the concept of synthesis, with controllable and uncontrollable events, etc, and whether they actually even performed synthesis at all.
- They linked controllable events to SVG inputs, which I think should never be done. You should only link uncontrollable events to be decided by the environment. Controllable events should be decided by the controller.