Support adapter that runs as a server (client-server mode)

In the first draft of the TCK harness, the code assumes that the adapter is a CLI that is invoked per test (CLI mode). The input is passed over stdin and the result is passed over stdout. Then the CLI exits. While this mode may work for some implementations, it isn't going to scale for others. Some implementations will be based on a language runtime that has a cold start time (such as Java), Eclipse Austen being an example. If there are over a 1,000 tests (a conservative estimate), and the start time of the CLI is 1 second, the tests will take over 15 minutes to run. That's not reasonable. This mode also limits parallelization since systems may restrict the number of processes that run at once. And the process thrashing is just largely unnecessary.

If we allow the adapter to run as a server instead, we can avoid these limitations. We can assume that this server is a web server, and that information is exchanged over HTTP. Almost every language runtime provides an HTTP server out of the box or has libraries that are readily available. And Node.js has been able to communicate over HTTP from its inception.

When the TCK starts, it will call start on the adapter. If that method returns a PID, it will keep that process running. For each test, the TCK will post the test to the server over HTTP and retrieve the response. When the TCK ends, it will call stop on the adapter so it can stop the server.

If the start method of the adapter returns falsy (or perhaps the string cli), the TCK will run the adapter command instead of posting to the server, just as it does today.

I think we shouldn't care about starting/stopping the server. We could add CLI options to the asciidoc-tck command to support both modes:

$ asciidoc-tck --impl-server=http://localhost:1234

$ asciidoc-tck --impl-cli="/path/to/impl"

If the implementation needs to do something special before processing the first test, we could pass a test index:

{
  "test": "single-word-paragraph",
  "context": "block/paragraph",
  "path": "/test/single-word-paragraph.adoc",
  "contents": "AsciiDoc",
  "current": 1,
  "total": 1024
}

I think we shouldn't care about starting/stopping the server.

I don't agree. I think the whole point of the adapter is to prepare the implementation to be tested. That is literally what the adapter is for. The TCK wouldn't be starting and stopping the server, it would asking the adapter to do it, which is already external code. This also gives the adapter an opportunity to do setup/teardown routines.

I don't agree. I think the whole point of the adapter is to prepare the implementation to be tested. That is literally what the adapter is for. The TCK wouldn't be starting and stopping the server, it would asking the adapter to do it, which is already external code. This also gives the adapter an opportunity to do setup/teardown routines.

I disagree with the previous statement. The whole purpose of the adapter is to provide a generic communication protocol that allows for testing an implementation with as few requirements as possible.

The TCK is agnostic to any implementation-specific requirements, such as running a script or starting a Kubernetes cluster, before accepting test cases. If an implementation needs to execute setup/teardown routines, it can execute commands or programs before and after running the asciidoc-tck command.

asciidoc-tck.sh

echo 'Setup...'

asciidoc-tck --impl-server=http://localhost:1234

echo 'Teardown'

As mentioned earlier, if an implementation requires any specific actions before the first test or after the last test, we can include a test index in the input or extend the communication protocol to send a setup/teardown message. But again, that's not really necessary since an implementation could do that before/after running the asciidoc-tck command.

If we don't want require the adapter to have to be written in JavaScript, I think the first scenario is the only option...and the CLI would have to support start and stop subcommands

I don't see the point of executing commands as part of the TCK.

The other way is for the adapter to also have a post method (or perhaps test?). In this scenario, the harness does not have any knowledge about how the adapter is used. It sends the test to the post method and expects it to return the ASG as a string.

In my opinion, the only requirement for an adapter is to comply with the communication protocol. As an implementation, you must either provide a ready-to-use CLI that reads JSON inputs and returns ASG or a server that accepts HTTP POST requests with a JSON payload and returns ASG.

If we don't want require the adapter to have to be written in JavaScript

Definitely not, it should be possible to write an adapter in any language.

Definitely not, it should be possible to write an adapter in any language.

I did some more thinking about this, and I've come to realize that you're right. Having to write an adapter in JavaScript would be distasteful. So let's scratch that idea.

From that thinking, I arrived at a very reasonable way for the adapter to work. First, we can assume the adapter is a CLI. Let's just make sure that part is clear for what follows.

The adapter must answer to the start and stop commands. And if one of those commands is not specified (or perhaps the command is test), then the CLI can assume it's a test.

The start command gives the adapter a chance to set up. It must print either a URL or a file path to stdout. (Any other lines it sends back will simply be captured). If the response is a URL, then that's the URL to which the harness will post tests. (In this scenario, the adapter is not used for tests). If it prints a file path (or empty, which implies the original path), then the harness assumes it must exec this path to run a test (which could be different from the adapter path, btw).

The stop command gives the adapter a chance to tear down. The harness will send the response from the start command to this command (which includes any extra lines it asked the harness to remember). That way, the stop command should have all the information it needs to clean up and to stop the server (if applicable).

I really think this is going to work nicely. It gives adapters the choice to work entirely as a CLI or as a server. It's not that much work for the harness to support. And it avoids the implementation providing the adapter from having to do anything outside of the TCK run.

If an implementation needs to execute setup/teardown routines, it can execute commands or programs before and after running the asciidoc-tck command.

I'd really like to avoid this. It makes the TCK look even more difficult to use. I like the original idea that the interaction is encapsulated in a CLI. I just don't think a single subcommand is enough. Nearly every test suite out there offers at least a set up and tear down routine, and it's very reasonable that we follow that model. Having to start a server outside of the TCK adapter is just burdensome / unfriendly.

if an implementation requires any specific actions before the first test or after the last test, we can include a test index in the input or extend the communication protocol to send a setup/teardown message.

To me, that's just a hack. I don't see how that is better than clear start/stop commands.

In my opinion, the only requirement for an adapter is to comply with the communication protocol.

I think we are still true to this. We are saying, look, if you want a server, there is a nice way to start and stop it. Just have the adapter listen to the subcommands when the harness sends them. It's still up to the adapter to figure out how to start and stop a server. But it's cleaner. ...and it's still all a CLI interaction.

That's a much more reasonable proposal but let's take a concret and plausible scenario.
Someone at Red Hat is working on an AsciiDoc implementation. Of course, they are using Quarkus with live-reload to continuously test their implementation. The Quarkus server has a route that accepts HTTP POST requests and returns the corresponding ASG.

They are using the TCK but they need to have a CLI that returns http://localhost:8081/asciidoc-tck when the command is start. They would rather just use: asciidoc-tck --impl-server=http://localhost:8081/asciidoc-tck.

And it avoids the implementation providing the adapter from having to do anything outside of the TCK run.

I can see the benefit but it also adds a requirement. You can't just have a running server.

To me, that's just a hack. I don't see how that is better than clear start/stop commands.

Yes and no, the test index might be useful and we are not advocating to use it as a setup/teardown mechanism but someone might have a valid use case. For instance, someone might want to print a bunch of blank lines before the TCK suite runs again or add a conditional break point in their IDE or send a message to Slack using a Webhook...

I can see the benefit but it also adds a requirement. You can't just have a running server.

Yes you can. The start command can just return the URL of the running server. It doesn't have to start a server.

Honestly, I don't see any downsides to what I'm proposing, other than it's a little extra work for us in the harness. But that work is not hard.

Yes and no, the test index might be useful and we are not advocating to use it as a setup/teardown mechanism but someone might have a valid use case.

I'm not advocating against a test index. I'm advocating against using it to hack in set up logic for the reason that there is no other way...when there is another way.

Yes you can. The start command can just return the URL of the running server. It doesn't have to start a server.

You mean something like:

ASCIIDOC_TCK_ADAPTER=/path/to/tck-adapter asciidoc-tck 

/path/to/tck-adapter

#!/usr/bin/env bash
echo 'http://localhost:8081/asciidoc-tck'

Honestly, I don't see any downsides to what I'm proposing

In some cases, it's a bit more convoluted (see above) than having two modes: CLI or server.

asciidoc-tck --cli=/usr/bin/mangata

asciidoc-tck --server=http://localhost:8081/asciidoc-tck

other than it's a little extra work for us in the harness.

I don't mind the extra work, I'm just worried about usability in different contexts and I don't want dictate a workflow.

Yes, that's essentially the idea.

I was about to suggest a compromise which is pretty close to what you suggested. If the adapter path is a URL, then the harness skips calling the start and stop command and just sends test to it.

But your proposal is still missing an important scenario, which is an adapter CLI that prepares and manages a server. So the different cases would be more like this:

asciidoc-tck /path/to/adapter-that-manages-server

asciidoc-tck /path/to/adapter-thus-runs-standalone

asciidoc-tck http://localhost:8081/asciidoc-tck-adapter

As you can see, there are actually three permutations.

In the first case, the adapter starts and stops a server, and the harness gets the URL from the start command (e.g., /path/to/adapter-that-manages-server start will return http://localhost:8081/asciidoc-tck-adapter).

In the second case, the adapter is what should accept the tests, so it will return itself (or nothing) when it receives the start command.

In the third case, the harness just starts passing tests to the server. There's no CLI to invoke (and thus no start and stop).

I don't mind the extra work, I'm just worried about usability in different contexts and I don't want dictate a workflow.

I think what I'm suggesting (with your revisions) is providing plenty of flexibility and is not dictating workflow. It's very adaptable.

I think what I'm suggesting (with your revisions) is providing plenty of flexibility and is not dictating workflow. It's very adaptable.

I like it

I prefer explicit CLI options over a single argument.

$ asciidoc-tck server --start-command=/path/to/start-server --stop-command=/path/to/stop-server --url=https://localhost:8081/asciidoc-tck-adapter

• --start-command or --url must be defined.
• --url is optional, if missing use the value returned by --start-command.
• --start-command is optional, if missing use the --url value.
• --stop-command is optional.

$ asciidoc-tck cli --command=/path/to/adapter-thus-runs-standalone

• --command is mandatory

Could be the default command and the default argument:

# shorthand
$ asciidoc-tck cli /path/to/adapter-thus-runs-standalone
$ asciidoc-tck /path/to/adapter-thus-runs-standalone

I see two possible ways the harness can support these two different modes (CLI vs client-server).

One way is for the adapter to only have a start and stop method (or start and stop subcommand), but for the harness to handle posting the test. In this scenario, the harness has to know and be responsible for whether it's invoking a command or sending an HTTP POST request. The start method/subcommand of the adapter would tell the harness which mode to use.

The other way is for the adapter to also have a post method (or perhaps test?). In this scenario, the harness does not have any knowledge about how the adapter is used. It sends the test to the post method and expects it to return the ASG as a string.

The second scenario puts more burden on the adapter as it has to handle invoking a CLI or sending an HTTP request. However, it makes the harness simpler. And we could provide base classes that adapters can extend to make it easier to implement. In this case, the adapter would have to be written in JavaScript.

If we don't want require the adapter to have to be written in JavaScript, I think the first scenario is the only option...and the CLI would have to support start and stop subcommands. If it accepts tests over the CLI, it would have to support the post (or test) subcommand.

I've revised this idea in the other thread. We don't want to require adapters to be written in JavaScript.

We seem to be in agreement. I can submit a MR.

mentioned in issue #13 (closed)

mentioned in merge request !17 (merged)

mentioned in merge request !18 (merged)

I really like the idea that the TCK can be run over a client/server protocol. That allows lots of tests to be performed at once with very little overhead, essential for a Java implementation.

I would still like to have the option for the adapter to forgo the use of a server and be able to process tests directly through a command. The way I see that working is that if the start command does not return a URL (or returns a file path), that command will be used for running each test (stdin being the request and stout being the response). If that's already implemented, then great. If not, I think it's fine to come back to it later. I just want to make sure we leave the door open so that mode can be added.

I would still like to have the option for the adapter to forgo the use of a server and be able to process tests directly through a command.

The following is not enough?

$ asciidoc-tck cli /path/to/adapter-thus-runs-standalone

If that's already implemented, then great. If not, I think it's fine to come back to it later. I just want to make sure we leave the door open so that mode can be added.

We do have two modes, cli (stin/stdout) and client/server (http).

Solid. Thanks for clarifying. I just wanted to check to be sure.

closed with commit 45fcc69a