Unverified Commit ad5312e1 authored by Joe Osborn's avatar Joe Osborn Committed by GitHub
Browse files

Merge pull request #445 from osbornjd/jay/MarkIII

Updates to Commands
parents cf1ea7fd d1b8b470
......@@ -131,7 +131,8 @@ public class SWTConnectionAuthorizationHandler extends ConnectionAuthorizationHa
*
* @param password
*/
public void setPassword(char[] password) {
this.password = password;
@Override
public void setOption(String option) {
this.password = option.toCharArray();
}
}
# Commands Package
This README serves as an overview of the commands package, which is a standalone maven package that can be used within or outside of ICE. The package provides the necessary API to set up and run jobs on either one's local computer or a remote host. Additionally, the API includes file transfer and file system browsing capabilities, with the option to move or copy files on the local host or remote host. It is suggested that users encode their file processing logic into a bash/python/powershell script to be run locally/remotely.
This README serves as an overview of the commands package, which is a standalone maven package that can be used within or outside of ICE. The package provides the necessary API to set up and run jobs on either one's local computer or a remote host. Additionally, the API includes file transfer and file system browsing capabilities, with the option to move or copy files on the local host or remote host. It is suggested that users encode their file processing logic into a bash/python/powershell script to be run locally/remotely. For example, a remote job on a remote host B could be run from commands on local host A which executes a remote job on remote host C, assuming the script contains the necessary logic to connect remote host B to remote host C.
Examples can be found in either the `src/test/java/org/eclipse/ice/tests/commands` directory or in the standalone package within ICE `org/eclipse/ice/demo/commands/`.
......@@ -34,7 +34,9 @@ hostname
Windows users need to put their ssh credentials into the file located at `C:\Users\Adminstrator\ice-remote-creds.txt` in order for the tests to properly function.
The automated tests will then grab the necessary credentials from this file to run the tests. Any valid ssh connection will work. If you still find that the tests fail, ensure that the ssh connection you are using has been logged into before from your host computer such that there is a key fingerprint associated to that host in your `~/.ssh/known_hosts` file. The Commands package requires that this key exists in order for authentication to proceed, no matter what form of authentication you use. Alternatively, you can set `StrictHostKeyChecking` to false in the `ConnectionManager`, which is not advised as it is inherently unsecure. To do this for the static `ConnectionManager`, just write:
The automated tests will then grab the necessary credentials from this file to run. Any valid ssh connection will work. If you still find that the tests fail, ensure that the ssh connection you are using has been logged into before from your host computer such that there is a key fingerprint associated to that host in your `~/.ssh/known_hosts` file. The Commands package requires that this key exists in order for authentication to proceed, no matter what form of authentication you use. In the event that tests fail on a host that already exists in `known_hosts` (e.g. with the error message `server key did not validate`, try deleting your `known_hosts` file (or the entries in your `known_hosts` that correspond to the host you are trying to run the tests on), logging in again to re-establish a fingerprint, and running the tests again.
Alternatively, you can set `StrictHostKeyChecking` to false in the `ConnectionManager`, which is in general not advised as it is inherently unsecure. To do this for the static `ConnectionManager`, just write:
```java
ConnectionManagerFactory.getConnectionManager().setRequireStrictHostKeyChecking(false);
......@@ -43,16 +45,28 @@ ConnectionManagerFactory.getConnectionManager().setRequireStrictHostKeyChecking(
Note that this is also a way through which ssh validation can be performed in the package for running actual remote commands/file transfers.
#### EmailHandler test
To test the `EmailUpdateHandler` class, a similar file to the `/tmp/ice-remote-creds.txt` file must be created. Instead, a file in the location `/tmp/ice-email-creds.txt` must exist which contains the following information:
```
email@address
password
SmtpHost
```
The EmailHandler will send an email from your own address to the same address with updates on when the job finishes. In order for this to happen, the email address must be authenticated. In the case of the tests, and for CI purposes, these authentications are placed in the above text file. For developer use, one could simply enter this information as it is entered in EmailHandlerTest, or you could implement another method (e.g. through use of the text file).
#### KeyGen Tests and Connections
Connections may be established via a public/private key pair that is generated between the local and remote host. The JSch API only works with RSA keys - Commands can also function with ECDSA, but it is advised to use RSA. You should be sure to generate a key similarly to the following snip of shell code:
Connections may be established via a public/private key pair that is generated between the local and remote host. Commands can function with ECDSA or RSA type keys. To generate an RSA key, for example, use:
```bash
$ ssh-keygen -t rsa -m PEM
$ ssh-copy-id -i ~/.ssh/keyname.pub username@hostname
```
Then you should be able to remotely login via `ssh -i /path/to/key username@hostname` without a password requirement.
For the keygen connection tests to pass, you should also create a key to a remote host that the tests expect to find. This can be done with any arbitrary remote server that you have credential access to; however, the key must be named dummyhostkey and must exist in your home `.ssh` directory. In other words, the key must be here:
```
......
......@@ -25,10 +25,25 @@
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-core</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-sftp</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>com.sun.mail</groupId>
<artifactId>javax.mail</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>0.1.55</version>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.10</version>
</dependency>
</dependencies>
</dependencies>
</project>
\ No newline at end of file
......@@ -70,6 +70,12 @@ public abstract class Command {
*/
protected ConnectionManager manager = ConnectionManagerFactory.getConnectionManager();
/**
* An optional update handler that can provide status updates for the command
*/
ICommandUpdateHandler updater = null;
/**
* Default constructor
*/
......@@ -104,6 +110,18 @@ public abstract class Command {
// Confirm the job finished with some status
logger.info("The job finished with status: " + status);
if(updater != null) {
String message = "Job number " + commandConfig.getCommandId()
+ " finished with status " + status +"\n "
+ "Check your log/out/error files for more details";
updater.setMessage(message);
try {
updater.postUpdate();
} catch (IOException e) {
// Just warn since it is not indicative of a job failing
logger.warn("Couldn't post update. Check stack trace" , e);
}
}
return status;
}
......@@ -161,7 +179,7 @@ public abstract class Command {
* This function is intended to clean up any (possible) remaining loose ends
* after the job is finished processing.
*
* @return
* @return CommandStatus - indicating that the configuration completed clean up
*/
protected CommandStatus cleanUp() {
......@@ -182,7 +200,7 @@ public abstract class Command {
* log/error information.
*
* @return - CommandStatus indicating that configuration completed and job can
* start running
* start running
*/
protected CommandStatus setConfiguration() {
......@@ -201,34 +219,37 @@ public abstract class Command {
String separator = "/";
if (commandConfig.getOS().toLowerCase().contains("win"))
separator = "\\";
// Check if the working directory exists
String workingDir = commandConfig.getWorkingDirectory();
boolean exists = false, execExists = false;
if(workingDir != null) {
if (!workingDir.endsWith(separator))
workingDir += separator;
// Check that the directory exists
// Get the file handler factory
FileHandlerFactory factory = new FileHandlerFactory();
boolean exists = false, execExists = false;
try {
// Get the handler for this particular connection, whether local or remote
FileHandler handler = factory.getFileHandler(connectionConfig);
// If the working directory was set, check that it exists. If it wasn't set,
// then the paths should have been explicitly identified
if (workingDir != null)
exists = handler.exists(workingDir);
// Check if the executable exists in the working directory
execExists = handler.exists(workingDir + exec);
// Check if the executable exists in the working directory
execExists = handler.exists(workingDir + exec);
} catch (IOException e) {
// If we can't get the file handler, then there was an error in the connection
// configuration
logger.error("Unable to connect to filehandler and check file existence. Exiting.", e);
return CommandStatus.INFOERROR;
}
}
// If the working directory doesn't exist, we won't be able to continue the job
// processing unless the full paths were specified. Warn the user
if (!exists) {
......@@ -330,7 +351,7 @@ public abstract class Command {
* This function is a simple helper function to check and make sure that the
* command status is not set to a flagged error, e.g. failed.
*
* @param current_status
* @param current_status to be checked
* @return boolean indicating whether or not status is good to continue (true)
* or whether or not job has failed (returns false)
*/
......@@ -340,8 +361,12 @@ public abstract class Command {
logger.info("The current status is: " + current_status);
return true;
} else {
logger.error("The job failed with status: " + current_status);
logger.error("Check your error logfile for more details! Exiting now!");
String statusString = "The job failed with status: " + current_status;
String checkString = "Check your error logfile for more details! Exiting now!";
logger.error(statusString);
logger.error(checkString);
return false;
}
......@@ -371,7 +396,7 @@ public abstract class Command {
* This function returns to the user the configuration that was used to create a
* particular command.
*
* @return - the particular configuration for this command
* @return - the particular CommandConfiguration for this command
*/
public CommandConfiguration getCommandConfiguration() {
return commandConfig;
......@@ -380,7 +405,7 @@ public abstract class Command {
/**
* This function sets the command configuration for a particular command
*
* @param config
* @param commandConfig - CommandConfiguration to be set
*/
public void setCommandConfiguration(CommandConfiguration commandConfig) {
this.commandConfig = commandConfig;
......@@ -400,7 +425,7 @@ public abstract class Command {
* This function sets the configuration that is to be used to set up a
* particular connection.
*
* @param connect
* @param connectionConfig - ConnectionConfiguration to be set
*/
public void setConnectionConfiguration(ConnectionConfiguration connectionConfig) {
this.connectionConfig = connectionConfig;
......@@ -427,4 +452,12 @@ public abstract class Command {
public ConnectionManager getConnectionManager() {
return manager;
}
/**
* Setter for update handler, see {@link org.eclipse.ice.commands.Command#updater}
* @param updater
*/
public void setUpdateHandler(ICommandUpdateHandler updater) {
this.updater = updater;
}
}
......@@ -73,10 +73,10 @@ public class CommandConfiguration {
private HashMap<String, String> inputFiles = new HashMap<String, String>();
/**
* This is a list of arguments that the user might want to append to the executable
* name that are _not_ input files. These will not be explicitly checked by the
* file handler for whether or not they exist, as they are presumed to be
* flags/arguments for the job to run.
* This is a list of arguments that the user might want to append to the
* executable name that are _not_ input files. These will not be explicitly
* checked by the file handler for whether or not they exist, as they are
* presumed to be flags/arguments for the job to run.
*/
private ArrayList<String> argumentList = new ArrayList<String>();
/**
......@@ -308,16 +308,16 @@ public class CommandConfiguration {
String separator = "/";
// Add the arguments to the executable name
for(String arg : argumentList) {
for (String arg : argumentList) {
fixedExecutableName += " " + arg;
}
// If the input files should be appended, append it
if (appendInput)
fixedExecutableName += " " + getInputFiles();
fullCommand = fixedExecutableName;
// Determine the proper separator
if (installDirectory != null && installDirectory.contains(":\\"))
separator = "\\";
......@@ -326,8 +326,6 @@ public class CommandConfiguration {
if (installDirectory != null && !installDirectory.endsWith(separator))
installDirectory = installDirectory + separator;
// Search for and replace the ${inputFile} to properly configure the input file
for (Map.Entry<String, String> entry : inputFiles.entrySet()) {
if (fixedExecutableName.contains("${" + entry.getKey() + "}") && !appendInput)
......@@ -374,7 +372,7 @@ public class CommandConfiguration {
* Setter for CommandId, see
* {@link org.eclipse.ice.commands.CommandConfiguration#commandId}
*
* @param _commandId
* @param commandId - integer of command ID to be run
*/
public void setCommandId(int commandId) {
this.commandId = commandId;
......@@ -385,7 +383,7 @@ public class CommandConfiguration {
* Getter for CommandId, see
* {@link org.eclipse.ice.commands.CommandConfiguration#commandId}
*
* @return commandId
* @return commandId - CommandConfiguration ID
*/
public int getCommandId() {
return commandId;
......@@ -395,7 +393,7 @@ public class CommandConfiguration {
* Setter for executable, see
* {@link org.eclipse.ice.commands.CommandConfiguration#executable}
*
* @param exec
* @param executable - executable to be run
*/
public void setExecutable(String executable) {
this.executable = executable;
......@@ -406,17 +404,18 @@ public class CommandConfiguration {
* Getter for executable, see
* {@link org.eclipse.ice.commands.CommandConfiguration#executable}
*
* @return executable
* @return executable - executable to be run
*/
public String getExecutable() {
return executable;
}
/**
* Setter for inputFile, see
* Adder for inputFile, see
* {@link org.eclipse.ice.commands.CommandConfiguration#inputFile}
*
* @param input
* @param name - name of inputFile to add
* @param path - path to inputFile relative to workingDirectory
*/
public void addInputFile(String name, String path) {
inputFiles.put(name, path);
......@@ -427,7 +426,7 @@ public class CommandConfiguration {
* Getter for a concatenated string of inputFiles, see
* {@link org.eclipse.ice.commands.CommandConfiguration#inputFiles}
*
* @return inputFile
* @return String - a string of all the inputFiles concatenated
*/
public String getInputFiles() {
String files = "";
......@@ -441,7 +440,7 @@ public class CommandConfiguration {
* Getter for the inputFile hashmap itself, see
* {@link org.eclipse.ice.commands.CommandConfiguration#inputFiles}
*
* @return inputFile
* @return inputFile - Hashmap with input file list
*/
public HashMap<String, String> getInputFileList() {
return inputFiles;
......@@ -451,7 +450,7 @@ public class CommandConfiguration {
* Setter for stdErrFileName, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdErrFileName}
*
* @param errFile
* @param stdErrFileName - name for StdErr file
*/
public void setErrFileName(String stdErrFileName) {
this.stdErrFileName = stdErrFileName;
......@@ -462,7 +461,7 @@ public class CommandConfiguration {
* Getter for stdErrFileName, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdErrFileName}
*
* @return stdErrFileName
* @return stdErrFileName - name of StdErr file
*/
public String getErrFileName() {
return stdErrFileName;
......@@ -472,7 +471,7 @@ public class CommandConfiguration {
* Setter for stdOutFileName, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOutFileName}
*
* @param outFile
* @param outFile - name of StdOut file
*/
public void setOutFileName(String stdOutFileName) {
this.stdOutFileName = stdOutFileName;
......@@ -483,7 +482,7 @@ public class CommandConfiguration {
* Getter for stdOutFileName, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOutFileName}
*
* @return stdOutFileName
* @return stdOutFileName - name of StdOut file
*/
public String getOutFileName() {
return stdOutFileName;
......@@ -493,7 +492,7 @@ public class CommandConfiguration {
* Setter for numProcs, see
* {@link org.eclipse.ice.commands.CommandConfiguration#numProcs}
*
* @param procs
* @param numProcs - number of processes
*/
public void setNumProcs(String numProcs) {
this.numProcs = numProcs;
......@@ -504,7 +503,7 @@ public class CommandConfiguration {
* Getter for numProcs, see
* {@link org.eclipse.ice.commands.CommandConfiguration#numProcs}
*
* @return numProcs
* @return numProcs - number of processes
*/
public String getNumProcs() {
return numProcs;
......@@ -514,7 +513,7 @@ public class CommandConfiguration {
* Setter for installDirectory, see
* {@link org.eclipse.ice.commands.CommandConfiguration#installDirectory}
*
* @param installDir
* @param installDir - String corresponding to path of install directory
*/
public void setInstallDirectory(String installDirectory) {
this.installDirectory = installDirectory;
......@@ -525,7 +524,7 @@ public class CommandConfiguration {
* Getter for installDirectory, see
* {@link org.eclipse.ice.commands.CommandConfiguration#installDirectory}
*
* @return installDirectory
* @return installDirectory - path of install directory
*/
public String getInstallDirectory() {
return installDirectory;
......@@ -535,7 +534,7 @@ public class CommandConfiguration {
* Getter for os, see {@link org.eclipse.ice.commands.CommandConfiguration#os}
* Note that this is set to the default of the local OS
*
* @return os
* @return os - operating system that Command is running on
*/
public String getOS() {
return os;
......@@ -545,7 +544,7 @@ public class CommandConfiguration {
* Setter for operating system, see
* {@link org.eclipse.ice.commands.CommandConfiguration#os}
*
* @param OS
* @param os - operating system that Command is running on
*/
public void setOS(String os) {
this.os = os;
......@@ -555,11 +554,18 @@ public class CommandConfiguration {
* Setter for workingDirectory, see
* {@link org.eclipse.ice.commands.CommandConfiguration#workingDirectory}
*
* @param workingDir
* @param workingDir - working directory containing input files/scripts
*/
public void setWorkingDirectory(String workingDirectory) {
// Check to see if the directory ends with a separator
String separator = FileSystems.getDefault().getSeparator();
// Just get the one that corresponds to the string. In the case of
// commands which are running on *nix but starting on windows,
// or vice versa, this can get things mixed up
String separator = "/";
// Make a special case for windows
if (workingDirectory.contains("\\"))
separator = "\\";
if (!workingDirectory.endsWith(separator))
workingDirectory += separator;
this.workingDirectory = workingDirectory;
......@@ -570,7 +576,7 @@ public class CommandConfiguration {
* Getter for workingDirectory, see
* {@link org.eclipse.ice.commands.CommandConfiguration#workingDirectory}
*
* @return workingDirectory
* @return workingDirectory - working directory containing files/scripts
*/
public String getWorkingDirectory() {
return workingDirectory;
......@@ -580,7 +586,8 @@ public class CommandConfiguration {
* Setter for appendInput, see
* {@link org.eclipse.ice.commands.CommandConfiguration#appendInput}
*
* @param _appendInput
* @param appendInput - boolean of whether or not to append input file names to
* executable
*/
public void setAppendInput(boolean appendInput) {
this.appendInput = appendInput;
......@@ -591,7 +598,8 @@ public class CommandConfiguration {
* Getter for appendInput, see
* {@link org.eclipse.ice.commands.CommandConfiguration#appendInput}
*
* @return appendInput
* @return appendInput - boolean of whether or not to append input file names to
* executable
*/
public boolean getAppendInput() {
return appendInput;
......@@ -603,7 +611,7 @@ public class CommandConfiguration {
* setter protected with the intent that only ConnectionConfiguration can modify
* this member variable.
*
* @param host
* @param hostname - hostname for command to run on
*/
protected void setHostname(String hostname) {
this.hostname = hostname;
......@@ -614,7 +622,7 @@ public class CommandConfiguration {
* Getter for hostname, see
* {@link org.eclipse.ice.commands.CommandConfiguration#hostname}.
*
* @return
* @return hostname - hostname for command to run on
*/
public String getHostname() {
return hostname;
......@@ -624,7 +632,7 @@ public class CommandConfiguration {
* Setter for stdErr, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdErr}
*
* @param writer
* @param stdErr - BufferedWriter to write error output
*/
public void setStdErr(BufferedWriter stdErr) {
this.stdErr = stdErr;
......@@ -635,7 +643,7 @@ public class CommandConfiguration {
* Getter for stdErr, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdErr}
*
* @return stdErr
* @return stdErr - Buffered writer to write error output
*/
public BufferedWriter getStdErr() {
return stdErr;
......@@ -645,7 +653,7 @@ public class CommandConfiguration {
* Setter for stdOut, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOut}
*
* @param writer
* @param stdOut - BufferedWriter to write log output
*/
public void setStdOut(BufferedWriter stdOut) {
this.stdOut = stdOut;
......@@ -656,7 +664,7 @@ public class CommandConfiguration {
* Getter for stdOut, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOut}
*
* @return stdOut
* @return stdOut - BufferedWriter to write log output
*/
public BufferedWriter getStdOut() {
return stdOut;
......@@ -666,7 +674,7 @@ public class CommandConfiguration {
* Setter for stdOutput, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOutput}
*
* @return stdOutput
* @return stdOutput - String for StdOutput name
*/
public void setStdOutputString(String stdOutput) {
this.stdOutput = stdOutput;
......@@ -676,7 +684,7 @@ public class CommandConfiguration {
* This function adds the String string to the String
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOutput}
*
* @param string
* @param string - String to concatenate to the stdOutput string
*/
public void addToStdOutputString(String string) {
stdOutput += "\n" + string;
......@@ -686,7 +694,7 @@ public class CommandConfiguration {
* Getter for stdOutput, see
* {@link org.eclipse.ice.commands.CommandConfiguration#stdOutput}
*
* @return stdOutput
* @return stdOutput - String of all stdOutput, separated by '\n'
*/
public String getStdOutputString() {
return stdOutput;
......@@ -694,9 +702,11 @@ public class CommandConfiguration {
/**
* Getter for splitCommand, see
* {@link org.eclipse.ice.commands.CommandConfiguration#splitCommand} No setter
* {@link org.eclipse.ice.commands.CommandConfiguration#splitCommand}. No setter
* since splitCommand is determined by
* {@link org.eclipse.ice.commands.CommandConfiguration#getExecutableName()}
*
* @return splitCommand - ArrayList<String> of split commands
*/
public ArrayList<String> getSplitCommand() {
return splitCommand;
......@@ -708,9 +718,8 @@ public class CommandConfiguration {
* the setter for full command protected so that it can only be accessed within
* the package and not by (e.g.) the user
*
* @param command
* @param fullComand - String of the full command to be executed
*
*
*/
protected void setFullCommand(String fullCommand) {
this.fullCommand = fullCommand;
......@@ -721,7 +730,7 @@ public class CommandConfiguration {
* Getter for fullCommand, see
* {@link org.eclipse.ice.commands.CommandConfiguration#fullCommand}
*
* @return fullCommand
* @return fullCommand - String of the full command to be executed
*/
public String getFullCommand() {
return fullCommand;
......@@ -731,7 +740,7 @@ public class CommandConfiguration {
* Add to error message string, see
* {@link org.eclipse.ice.commands.CommandConfiguration#errMsg}
*
* @param
* @param errMsg - String to add to the error message string
*/
public void addToErrString(String errMsg) {
this.errMsg += errMsg;
......@@ -741,7 +750,7 @@ public class CommandConfiguration {