Replace mocha with Node.js' built-in test runner

As of Node.js 16.17.0, Node.js now provides a built-in test runner which is strikingly similar to Mocha and thus effectively replaces it (at least for what we need it for). The built-in test runner has been further improved in Node.js 18 and 19. We should explore using it in the harness in place of Mocha (and Chai).

By switching the TCK harness to use Node.js' built-in test runner (possibly removing the use of Chai as well), we can avoid having to bundle external libraries, simplify the operation of the TCK, and make it easier to maintain in the long run.

Refer to https://nodejs.org/api/test.html to find the documentation for the test runner.

I checked whether it's possible to invoke the test runner from the API, and it turns out it's actually even easier. The test runner provides a run method which accepts a list of files to run and a setup function for adding hooks. This fits right into the operation of the harness proposed in !6 (merged). It also makes it easier to plug in a custom reporter, which is likely going to be needed for tailoring the display of the results to be more TCK-oriented.

It looks something like this:

import { run } from 'node:test'
import TCKReporter from './tck-reporter.js'
import ospath from 'node:path'

run({ files: [ospath.resolve('path/to/suite.js')] })
  .compose(new TCKReporter())
  .pipe(process.stdout)

Currently, Node.js does not export the built-in reporters, like the spec reporter. However, it's not too difficult to recreate that output and make it our own.

It also makes it easier to plug in a custom reporter, which is likely going to be needed for tailoring the display of the results to be more TCK-oriented.

I agree.

mentioned in merge request !6 (merged)

Let's do it!

assigned to @mojavelinux

The only hangup here is that it's going to take a bit of time to put together a custom reporter that does a decent job of showing the test results. I'm working on building something taking ideas from https://github.com/nodejs/node/blob/main/lib/internal/test_runner/reporter/spec.js. Once that is in place, the rest should go quickly.

I can give it a try. I might have some spare time next week.

I seem to have misplaced the prototype I was working on, so feel free to go ahead. I'm basically just trying to get roughly the same output that Mocha produces since it looks reasonable. I think Node.js 20 has most of the code we need to get the bones in place, then it's just a matter of tweaking the styling after that.

Test cases and the parse function (which calls the adapter manager) are passed using Mocha global context. As far as I understand Node.js built-in test runner does not have a global context. In fact, each test file is executed in a separate child process:

Each matching test file is executed in a separate child process. If the child process finishes with an exit code of 0, the test is considered passing. Otherwise, the test is considered to be a failure. Test files must be executable by Node.js, but are not required to use the node:test module internally.

https://nodejs.org/docs/latest-v18.x/api/test.html#test-runner-execution-model

I didn't find a way to communicate with this child process and/or to pass data. The TestsStream operates on a test level (not on a suite level) and ,as far as I know, we cannot retrieve the SuiteContext from outside the test file.

What we could do is create a parent test suite that calls/generates the block test suite and the inline test suite. Is that what you had in mind?

Yes. In fact, I'm pretty sure that's exactly what Mocha does internally. There is always a top-level suite, which is where Mocha is storing globals.

The latest version of https://github.com/vercel/pkg/ only supports Node <= 18.5.0. As a result, run is undefined since it was added in: v18.9.0.

The next release should include Node 19 and Node 18.15.

(node:34250) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
(Use `asciidoc-tck --trace-warnings ...` to show where the warning was created)
pkg/prelude/bootstrap.js:1927
      return wrapper.apply(this.exports, args);
                     ^

TypeError: undefined is not a function
    at run (/snapshot/asciidoc-tck/harness/lib/index.cjs)
    at /snapshot/asciidoc-tck/harness/lib/index.cjs
    at Object.<anonymous> (/snapshot/asciidoc-tck/harness/lib/index.cjs)
    at Module._compile (pkg/prelude/bootstrap.js:1927:22)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Module.require (node:internal/modules/cjs/loader:1012:19)
    at Module.require (pkg/prelude/bootstrap.js:1851:31)
    at require (node:internal/modules/cjs/helpers:102:18)

Node.js v18.5.0

I'm starting to wonder at this point if we should just require Node.js 20. By the time the TCK actually gets used by implementations, Node.js 20 will have been out for many months, and the advances in the built-it test runner are just too important to pass up IMO.

I agree

However, pkg (or more precisely pkg-fetch) is lagging behind: https://github.com/vercel/pkg-fetch/issues/302.
We could make the switch from pkg to https://nodejs.org/api/single-executable-applications.html but currently it has quite a few limitations and the support is experimental.

I will experiment with Node.js 20 and see if I'm able to create a single executable application.

Unlike pkg, Node single executable application does not have a snapshot filesystem. Having said that, we could run loadTests at build time and JSON.stringify the result.

On second thought, it might be a bad idea to include the test cases in the binary. Might be better to provide a binary and a directory that contains all the test cases (i.e., -input.adoc, -output.json, -config.jsson).

Node built-in test runner expects a list of files and since single executable application only supports a single Node.js file we will need to find a workaround. I've opened https://github.com/nodejs/node/issues/49606 but to be honest I'm not really convinced by my proposal (so it probably won't land in Node).

One way would be to write the content of block-test.js and inline-test.js in temporary files before calling test.run(). Another idea would be to distribute this two files as part of the "bundle".

asciidoc-tck
tests/
block-test.js
inline-test.js

Would it make sense to run one or the other or both? If so we could support something like:

$ asciidoc-tck --test ./block-test.js

It might make sense to configure the path of the tests/ directory using an environment variable.

$ ASCIIDOC_TCK_TESTS_DIR=/path/to/tests asciidoc-tck \
  --test /path/to/block-test.js /path/to/inline-test.js

One way would be to write the content of block-test.js and inline-test.js in temporary files before calling test.run(). Another idea would be to distribute this two files as part of the "bundle".

Either approach works for me. I had initially separated these files just to avoid creating a monolithic suite. But since the test suites ended up being thin and pretty similar, I don't see too much of an issue of just combining them for now and figuring how to bundle them later if we decide we need to split them back up.

It might make sense to configure the path of the tests/ directory using an environment variable.

My only question here is whether we will always require the tests dir to be specified, or whether we can somehow bundle the tests into the asciidoc-tck. It seems odd to distribute the TCK, but not the tests. The dir override is definitely a good thing for development. But I wonder if for official runs, the tests should be downloaded from a URL and unzipped behind the scenes. I just want to be sure that running the TCK doesn't end up feeling like a loose assortment of bits.

One way would be to write the content of block-test.js and inline-test.js in temporary files before calling test.run()

After giving it some thought, I think it's (currently) the best solution.

My only question here is whether we will always require the tests dir to be specified, or whether we can somehow bundle the tests into the asciidoc-tck. It seems odd to distribute the TCK, but not the tests. The dir override is definitely a good thing for development. But I wonder if for official runs, the tests should be downloaded from a URL and unzipped behind the scenes. I just want to be sure that running the TCK doesn't end up feeling like a loose assortment of bits.

I think it's fine if we have an executable and a directory that contains all the tests. I would prefer to avoid downloading the tests from Internet (a recipe for trouble with corporate network/proxy).

We won't need to specify the tests directory.
When using the executable, we can use "./tests" by default (i.e., a directory named "tests" located in the same directory of the executable). And when using npx we can load tests from the Node package by default.

Perfect.

And just to clarify, when I was referring to downloading tests, I was thinking that the tests themselves will be published as an artifact that someone can download, unzip, and pass to the asciidoc-tck. In other words, the tests will probably be in some sort of Eclipse process release (as the preferred alternative to cloning the repo and switching to a specific tag for an AsciiDoc spec version).

I also realized that the binary does not work on the main branch. I will add a test to make sure that pkg produces a working version of the TCK (probably using the echo adapter).

That would be most excellent.

Since pkg does not currently produce a working single executable, I will open a merge request that focuses on replacing Mocha with Nodes.js' built-in test runner.

In a following merge request, I will replace pkg with https://nodejs.org/api/single-executable-applications.html

mentioned in merge request !19 (merged)

closed with commit 4b4bd03b

closed with merge request !19 (merged)