Decide how to name and organize tests

Related to #1 (closed), we need to decide how to name and organize tests (i.e., the test data files).

In terms of naming, there are two questions:

1. What file extensions do we use for input, output, and config files?
2. What conventions should we follow when choosing a basename for the test files?

In terms of organization, there are three questions:

1. Where do the tests live? (#1 (closed) proposes putting them in the tests folder at the root of the repository)
2. How should tests be grouped?
3. How deep should tests be nested? Should there be a limit?

For naming, I'd like to propose the following suffixes for the test data files:

• -input.adoc - the test input; an AsciiDoc document
• -output.json - the JSON data of the ASG
• -config.yml - configuration settings for the tests, such as whether or not to trim trailing spaces / newlines, as well as the description (when needed)

Instead of -input.adoc / -output.json, we could consider -src.adoc / -asg.json. However, I'm concerned that would be too obtuse. I don't like using double file extensions, which is why each proposed suffix starts with -.

Here's an example of a config file for a test:

name: document will trailing empty lines
ensure_trailing_newline: true

As for how to choose the basename for test files, I think it should be the simplest name that uniquely describes the test scenario within a given context. It does not need to be a complete sentence as the description in the config file can serve that purpose. Abbreviations should be used prudently so the filename is not difficult to decipher. When the scenario gets complex, fall back to itemizing the elements in document order, without articles and conjunctions.

Here are some examples for paragraph tests:

• single-line
• multiple-lines
• sibling-paragraphs
• paragraph-empty-lines-paragraph

Here are some examples for header tests:

• title-only
• attribute-entries-below-title
• attribute-entries-above-title
• attribute-entries-around-title

For organization, I think we should divide the tests first by type (block, inline), then by node name (paragraph, section). For now, I'm not sure we want to go deeper, but we may need to as the number of test grows. We could add a level for "variant" and/or "form".

tests/
  block/
    paragraph/
      single-line-input.adoc
      single-line-output.json
      single-line-config.yml
    section/
  inline/
    image/
      isolated-input.adoc
      isolated-output.json

We may also consider using concepts at the level of the node names, such as "no-markup".

When assembling the test suite, the TCK should transform the folders and test basenames to make them pretty. They should also be sorted. Here's a preview of that test report:

  block
    header
      ✔ title only
      ✔ attribute entries below title
      ✔ attribute entries above title
      ✔ attribute entries around title
    paragraph
      ✔ multiple lines
      ✔ sibling paragraphs
      ✔ single line
  inline
    image
      ✔ isolated

Each folder under tests (that contains test files) gets represented by a level in the test suite (and hence the test result).

Instead of -input.adoc / -output.json, we could consider -src.adoc / -asg.json. However, I'm concerned that would be too obtuse. I don't like using double file extensions, which is why each proposed suffix starts with -.

In my opinion -src.adoc / -asg.json is a little less clear. Using input, output and config will be immediately understood by anyone. Arguably, we could use expected instead of output. Having said that, I quite like the i/o reference

Apart from that

One thing we'll also want to think about is where to put tests that are template tests for repeated structures like inline spans. We don't want to rewrite the same tests for strong, emphasis, code, etc, so we probably want to generate those from a common set of tests. This concern may need to be split into a separate issue.

I think we should generate them at build time. So we should store template tests outside of tests and write the result in tests. Both can be versioned.

This concern may need to be split into a separate issue.

@mojavelinux Currently, we have two top-level directory "block" and "inline". I want to test attribute assignment precedence but I'm not sure where I should add this tests? block > document > attributes? attributes > assignment-precedence?

Here's an example:

input.adoc

= Document Title
:name: Guillaume

{name}

config.json

{
  "options": {
    "attributes": {
      "name": "Dan"
    }
  }
}

Found examples in the AsciiDoc parsing lab defined in block > header: https://github.com/opendevise/asciidoc-parsing-lab/blob/main/test/tests/block/header/reference-attribute-in-body-input.adoc?plain=1

You got it!

However, we're going to be rethinking the way that document attributes are tracked and stored in the ASG since, currently, there's not enough information to properly implement and validate attribute precedence. I discovered this conundrum while playing around in my parsing lab. Look for an issue later this week on that topic.

I'm actually thinking that this issue should have been marked resolved with the release of milestone 0.1.0. Naturally, there are improvements that need to be made, but we did at least settle on the first draft enough to put a sampling of tests in place.

I agree

closed