Consider moving documentation sources closer to related code
Here we want to discuss whether it would be good, and possible, to move documentation sources closer to their related code. Thus, can we put e.g. the rail diagram generator documentation sources in the rail diagram generator plugin, regardless of it being part of some documentation set.
There is some relevant discussion from #73 (closed) that applies to this issue.
First introduction of the issue
At !46 (comment 9818), the following was mentioned:
The documentation sources of this plugin got moved to the doc corner. I know we did that before, and for languages I think it makes sense, as the tool collection is large, and often you need combined tools to get useful results. You want to describe all tools in one document, rather then each tool separately.
I am not entirely sure it makes sense for the more stand-alone tools. A distribution of this plugin now has no docs at all (no sources of the docs and no generated doc files). The 'docs' plugin has doc sources for tools that don't exist there.
The 'this plugin' relates to the rail diagram generator (documentation).
This discussion relates to the documentation set discussion, as that can also help to bring documentation somewhat closer to the code
- The Eclipse ESCET documentation includes too much.
- Move the
commondocumentation to a separate 'common' documentation set (in the
commondirectory), similar to the Chi, CIF, etc documentation. That way it is closer to the plugins that it applies to.
This leads to a few more documentation sets. But it would be more structured like the directory structure that we have, and would be 'closer to the source'. It would also fit the intended audience better.
We could make separate documentation sets for the application framework, rail diagram generator, etc that are part of `common, but that would lead to a lot of documentation sets, some of which may be very small.
Here it was identified that indeed we're discussing two things (which are now two separate issues)
I like the general direction you're taking.
Your discussion seems to use "documentation" in 2 different forms imho. One form is the source code, the *.asciidoc files. The other form are the generated viewable *.html or *.pdf files.
I am not sure why you don't want to move the documentation sources to the plugin if it makes sense. (While definitely keeping plugins that generate viewable versions as a collection of files in one place too, as a larger document with all related material together is useful for a reader.)
That is, with a Git checkout, the documentation builder has access to everything, so why do we stick documentation sources of a single document in a single directory?
The trick here is likely to find a good balance between keeping code and its documentation together as much as possible, while not scattering document sources across all plugins just because it discusses a few things of a plugin.
Possible concern with versions/qualifiers vs content/hashes
I like your idea of just including the sources from the separate plugins into the larger documentation sets. The common documentation plugin could have the general documentation sources and include the sources of the application framework from the application framework plugin. I had actually not considered this before.
However, one of the concerns with taking sources from various plugins is that the plugin version qualifier (the last part of the version) is determined by the last Git commit date. If you take files from other plugins (say B and C) to build your own plugin (say A), I'm not sure whether that's taken into account for the plugin version of plugin A. If not, then the version may not be updated, while the built contents is, and thus the JAR will have different content for the same version of the plugin. That could cause all kinds of issues when updating plugins to newer versions, when combining update sites, etc etc. We've faced these issues in the past.
I hope that adding a dependency from A to B and C will help to take their changes into account as well. If that works, we could take sources from other plugins. If that doesn't work, I fear we have to prevent taking sources from other plugins to prevent difficult to diagnose/resolve issues.
I wanted to check this anyway, as the idea is for the other issue about the railroad diagram generator to take the config file from the plugin and put it in the documentation. We would face similar issues there. I'll test it to see how it behaves. Maybe there is configuration that affects it that we can configure as well.
I experimented to see whether we can include sources from other plugins to build documentation plugins. I did the following:
- A dependency of the ESCET documentation on the railroad diagram generator.
- Including the railroad diagram generator in the common feature (just to be sure it is in a feature, in case not including it in a feature is a problem).
- Committed a change in the ESCET documentation plugin, at time X.
- Committed a change in the railroad diagram generator plugin, at a later time Y > X.
- Performed a full build locally.
Ideally both the ESCET documentation plugin and the railroad diagram generator get the 'qualifier' part of their version numbers set to something derived from Y. In practice, the railroad diagram generator gets Y as that is the last commit in that plugin, and the ESCET documentation gets X as that is the last commit in that plugin. It seems dependencies have no effect on qualifiers for the 'jgit' timestamp provider. The source code at https://github.com/eclipse/tycho/blob/master/tycho-extras/tycho-buildtimestamp-jgit/src/main/java/org/eclipse/tycho/extras/buildtimestamp/jgit/JGitBuildTimestampProvider.java seems to confirm this, as I can't find anything in the code that seems to take dependencies into account. Only the last commit of the basedir of the project is taken into account.
To some degree this makes sense to me, as typically dependencies are what you use to execute the plugin at runtime. In such cases you wouldn't want any minor change in dependencies and dependencies of dependencies to cause a version increase. However, it would be practical to consider build-time dependencies, and do consider them in determining the version. For features, this functionality is provided by the build-qualifier-aggregator goal of the tycho-packaging plugin, see https://www.eclipse.org/tycho/sitedocs/tycho-packaging-plugin/build-qualifier-aggregator-mojo.html. However, as far as I can find, this is only available for features.
I'll explore further.
One way to resolve all this, is to use fixed qualifiers, rather than different ones for each plugin/feature/etc based on Git timestamps. If we determine for a release the Git timestamp of the release tag, we could base the qualifier for all plugins, features and products on that timestamp, such that they are all based on the same timestamp and thus qualifier. Essentially, we'd be using the 'maximum' of all individual qualifiers for all plugins/features/etc. It would lead to 'unnecessary' qualifier increases, but would solve all the issues we have. Given that we update the major/minor/micro version for a release anyway, they are already unique and the qualifier seems to have reduced value anyway. For non-release builds (dev builds), we could keep the Git-based timestamp as we have now, as IMHO there the 'same version implies same content and hash' rule is less important. But in principle there is no reason we can't use the same strategy as for releases there as well.
Sounds good to me. I agree that development builds are mostly for sanity checking of the code, and not really intended for use to derive other work on. Having not exactly stable qualifiers shouldn't be much of a problem.
I've created #92 (closed) for the version/qualifier changes. Let's discuss that further there. Then this issue can focus on the documentation set structure.
If we can include sources from other plugins, it makes sense to spread the documentation sources even more, over several SeText plugins. We can then also easily include it in the developer documentation set, rather than having a separate documentation set.