Thrift module: compute_resource_model

Module	Services	Data types	Constants
compute_resource_model		BatchQueue CloudJobSubmission ComputeResourceDescription DMType FileSystems GlobusJobSubmission JobManagerCommand JobSubmissionInterface JobSubmissionProtocol LOCALSubmission MonitorMode ProviderName ResourceJobManager ResourceJobManagerType SSHJobSubmission UnicoreJobSubmission

Enumerations

Enumeration: ResourceJobManagerType

* Enumeration of local resource job manager types supported by Airavata
*
* FORK:
*  Forking of commands without any job manager
*
* PBS:
*  Job manager supporting the Portal Batch System (PBS) protocol. Some examples include TORQUE, PBSPro, Grid Engine.
*
* SLURM:
*  The Simple Linux Utility for Resource Management is a open source workload manager.
 *
 * UGE:
 *  Univa Grid Engine, a variation of PBS implementation.
 *
 * LSF:
 *  IBM Platform Load Sharing Facility is dominantly installed on IBM clusters.
*

`FORK`	`0`
`PBS`	`1`
`SLURM`	`2`
`LSF`	`3`
`UGE`	`4`
`CLOUD`	`5`
`AIRAVATA_CUSTOM`	`6`

Enumeration: JobManagerCommand

Enumeration of resource job manager commands

SUBMISSION:
 Ex: qsub, sbatch

JOBMONITORING:
 Ex: qstat, squeue

DELETION:
 Ex: qdel, scancel

CHECK_JOB:
 Detailed Status about the Job. Ex: checkjob

SHOW_QUEUE:
 List of Queued Job by the schedular. Ex: showq

SHOW_RESERVATION:
 List all reservations. Ex:showres, show_res

SHOW_START:
 Display the start time of the specified job. Ex: showstart

`SUBMISSION`	`0`
`JOB_MONITORING`	`1`
`DELETION`	`2`
`CHECK_JOB`	`3`
`SHOW_QUEUE`	`4`
`SHOW_RESERVATION`	`5`
`SHOW_START`	`6`

Enumeration: FileSystems

Enumeration of File Systems on the resource

FORK:
 Forking of commands without any job manager

PBS:
 Job manager supporting the Portal Batch System (PBS) protocol. Some examples include TORQUE, PBSPro, Grid Engine.

UGE:
 Univa Grid Engine, a variation of PBS implementation.

SLURM:
 The Simple Linux Utility for Resource Management is a open source workload manager.

`HOME`	`0`
`WORK`	`1`
`LOCALTMP`	`2`
`SCRATCH`	`3`
`ARCHIVE`	`4`

Enumeration: JobSubmissionProtocol

Enumeration of Airavata supported Job Submission Mechanisms for High Performance Computing Clusters.

SSH:
 Execute remote job submission commands using via secure shell protocol.

GRAM:
 Execute remote jobs via Globus GRAM service.

UNICORE:
 Execute remote jobs via Unicore services

`LOCAL`	`0`
`SSH`	`1`
`GLOBUS`	`2`
`UNICORE`	`3`
`CLOUD`	`4`
`SSH_FORK`	`5`
`LOCAL_FORK`	`6`

Enumeration: MonitorMode

Monitoring modes

POLL_JOB_MANAGER:
GFac need to pull job status changes.

XSEDE_AMQP_SUBSCRIBE:
Server will publish job status changes to amqp servert.

`POLL_JOB_MANAGER`	`0`
`CLOUD_JOB_MONITOR`	`1`
`JOB_EMAIL_NOTIFICATION_MONITOR`	`2`
`XSEDE_AMQP_SUBSCRIBE`	`3`
`FORK`	`4`
`LOCAL`	`5`

Enumeration: DMType

`COMPUTE_RESOURCE`	`0`
`STORAGE_RESOURCE`	`1`

Enumeration: ProviderName

Provider name

`EC2`	`0`
`AWSEC2`	`1`
`RACKSPACE`	`2`

Data structures

Struct: ResourceJobManager

Key	Field	Type	Requiredness	Default value
1	resourceJobManagerId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	resourceJobManagerType	`ResourceJobManagerType`	required
3	pushMonitoringEndpoint	`string`	optional
4	jobManagerBinPath	`string`	optional
5	jobManagerCommands	`map<JobManagerCommand, string>`	optional
6	parallelismPrefix	`map<parallelism_model.ApplicationParallelismType, string>`	optional

Resource Job Manager Information

resourceJobManagerType:
 A typical HPC cluster has a single Job Manager to manage the resources.

pushMonitoringEndpoint:
 If the job manager pushes out state changes to a database or bus, specify the service endpoint.
  Ex: Moab Web Service, Moab MongoDB URL, AMQP (GLUE2) Broker

jobManagerBinPath:
 Path to the Job Manager Installation Binary directory.

jobManagerCommands:
 An enumeration of commonly used manager commands.

Struct: BatchQueue

Key	Field	Type	Requiredness
1	queueName	`string`	required
2	queueDescription	`string`	optional
3	maxRunTime	`i32`	optional
4	maxNodes	`i32`	optional
5	maxProcessors	`i32`	optional
6	maxJobsInQueue	`i32`	optional
7	maxMemory	`i32`	optional
8	cpuPerNode	`i32`	optional
9	defaultNodeCount	`i32`	optional
10	defaultCPUCount	`i32`	optional
11	defaultWalltime	`i32`	optional
12	queueSpecificMacros	`string`	optional
13	isDefaultQueue	`bool`	optional

Batch Queue Information on SuperComputers

maxRunTime:
 Maximum allowed run time in hours.

Struct: LOCALSubmission

Key	Field	Type	Requiredness	Default value
1	jobSubmissionInterfaceId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	resourceJobManager	`ResourceJobManager`	required
3	securityProtocol	`data_movement_models.SecurityProtocol`	required

Locally Fork Jobs as OS processes

alternativeSSHHostName:
 If the login to ssh is different than the hostname itself, specify it here

sshPort:
 If a non-default port needs to used, specify it.

Struct: SSHJobSubmission

Key	Field	Type	Requiredness	Default value
1	jobSubmissionInterfaceId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	securityProtocol	`data_movement_models.SecurityProtocol`	required
3	resourceJobManager	`ResourceJobManager`	required
4	alternativeSSHHostName	`string`	optional
5	sshPort	`i32`	optional	`22`
6	monitorMode	`MonitorMode`	optional
7	batchQueueEmailSenders	`list<string>`	optional

Authenticate using Secured Shell

alternativeSSHHostName:
 If the login to ssh is different than the hostname itself, specify it here

sshPort:
 If a non-default port needs to used, specify it.

batchQueueEmailSenders:
 If a resource always sends the monitoring from a specific address, specify the
  full email address. If a resource sends emails from multiple addresses (
   example: based on the submitted login node) then use the wildchar * to indicate
   the same. Example: *@*.example.com or *@example.com

Struct: GlobusJobSubmission

Key	Field	Type	Requiredness	Default value
1	jobSubmissionInterfaceId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	securityProtocol	`data_movement_models.SecurityProtocol`	required
3	globusGateKeeperEndPoint	`list<string>`	optional

Struct: UnicoreJobSubmission

Key	Field	Type	Requiredness	Default value
1	jobSubmissionInterfaceId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	securityProtocol	`data_movement_models.SecurityProtocol`	required
3	unicoreEndPointURL	`string`	required

Unicore Job Submission

unicoreEndPointURL:
 unicoreGateway End Point. The provider will query this service to fetch required service end points.
authenticationMode
 The authenticationMode defines the way certificate is fetched.

Struct: CloudJobSubmission

Key	Field	Type	Requiredness	Default value
1	jobSubmissionInterfaceId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	securityProtocol	`data_movement_models.SecurityProtocol`	required
3	nodeId	`string`	required
4	executableType	`string`	required
5	providerName	`ProviderName`	required
6	userAccountName	`string`	required

Cloud Job Submission

Struct: JobSubmissionInterface

Key	Field	Type	Requiredness	Default value
1	jobSubmissionInterfaceId	`string`	required
2	jobSubmissionProtocol	`JobSubmissionProtocol`	required
3	priorityOrder	`i32`	required	`0`

Job Submission Interfaces

jobSubmissionInterfaceId: The Job Submission Interface has to be previously registered and referenced here.

priorityOrder:
 For resources with multiple interfaces, the priority order should be selected.
  Lower the numerical number, higher the priority

Struct: ComputeResourceDescription

Key	Field	Type	Requiredness	Default value
1	computeResourceId	`string`	required	`"DO_NOT_SET_AT_CLIENTS"`
2	hostName	`string`	required
3	hostAliases	`list<string>`	optional
4	ipAddresses	`list<string>`	optional
5	resourceDescription	`string`	optional
6	enabled	`bool`	optional
7	batchQueues	`list<BatchQueue>`	optional
8	fileSystems	`map<FileSystems, string>`	optional
9	jobSubmissionInterfaces	`list<JobSubmissionInterface>`	optional
10	dataMovementInterfaces	`list<data_movement_models.DataMovementInterface>`	optional
11	maxMemoryPerNode	`i32`	optional
12	gatewayUsageReporting	`bool`	optional
13	gatewayUsageModuleLoadCommand	`string`	optional
14	gatewayUsageExecutable	`string`	optional
15	cpusPerNode	`i32`	optional
16	defaultNodeCount	`i32`	optional
17	defaultCPUCount	`i32`	optional
18	defaultWalltime	`i32`	optional

Computational Resource Description

computeResourceId: Airavata Internal Unique Identifier to distinguish Compute Resource.

hostName:
Fully Qualified Host Name.

hostAliases:
Aliases if any.

ipAddress:
IP Addresses of the Resource.

resourceDescription:
A user friendly description of the resource.

JobSubmissionProtocols:
A computational resources may have one or more ways of submitting Jobs. This structure
will hold all available mechanisms to interact with the resource.
The key is the priority

DataMovementProtocol:
Option to specify a prefered data movement mechanism of the available options.

fileSystems:
Map of file systems type and the path.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

source/api-docs/master/compute_resource_model.html [1:366]:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Thrift module: compute_resource_model

Thrift module: compute_resource_model
Module Services Data types Constants

compute_resource_model
BatchQueue

CloudJobSubmission

ComputeResourceDescription

DMType

FileSystems

GlobusJobSubmission

JobManagerCommand

JobSubmissionInterface

JobSubmissionProtocol

LOCALSubmission

MonitorMode

ProviderName

ResourceJobManager

ResourceJobManagerType

SSHJobSubmission

UnicoreJobSubmission

Enumerations
Enumeration: ResourceJobManagerType
* Enumeration of local resource job manager types supported by Airavata
*
* FORK:
* Forking of commands without any job manager
*
* PBS:
* Job manager supporting the Portal Batch System (PBS) protocol. Some examples include TORQUE, PBSPro, Grid Engine.
*
* SLURM:
* The Simple Linux Utility for Resource Management is a open source workload manager.
*
* UGE:
* Univa Grid Engine, a variation of PBS implementation.
*
* LSF:
* IBM Platform Load Sharing Facility is dominantly installed on IBM clusters.
*

FORK 0

PBS 1

SLURM 2

LSF 3

UGE 4

CLOUD 5

AIRAVATA_CUSTOM 6

Enumeration: JobManagerCommand
Enumeration of resource job manager commands

SUBMISSION:
Ex: qsub, sbatch

JOBMONITORING:
Ex: qstat, squeue

DELETION:
Ex: qdel, scancel

CHECK_JOB:
Detailed Status about the Job. Ex: checkjob

SHOW_QUEUE:
List of Queued Job by the schedular. Ex: showq

SHOW_RESERVATION:
List all reservations. Ex:showres, show_res

SHOW_START:
Display the start time of the specified job. Ex: showstart

SUBMISSION 0

JOB_MONITORING 1

DELETION 2

CHECK_JOB 3

SHOW_QUEUE 4

SHOW_RESERVATION 5

SHOW_START 6

Enumeration: FileSystems
Enumeration of File Systems on the resource

FORK:
Forking of commands without any job manager

PBS:
Job manager supporting the Portal Batch System (PBS) protocol. Some examples include TORQUE, PBSPro, Grid Engine.

UGE:
Univa Grid Engine, a variation of PBS implementation.

SLURM:
The Simple Linux Utility for Resource Management is a open source workload manager.

HOME 0

WORK 1

LOCALTMP 2

SCRATCH 3

ARCHIVE 4

Enumeration: JobSubmissionProtocol
Enumeration of Airavata supported Job Submission Mechanisms for High Performance Computing Clusters.

SSH:
Execute remote job submission commands using via secure shell protocol.

GRAM:
Execute remote jobs via Globus GRAM service.

UNICORE:
Execute remote jobs via Unicore services

LOCAL 0

SSH 1

GLOBUS 2

UNICORE 3

CLOUD 4

SSH_FORK 5

LOCAL_FORK 6

Enumeration: MonitorMode
Monitoring modes

POLL_JOB_MANAGER:
GFac need to pull job status changes.

XSEDE_AMQP_SUBSCRIBE:
Server will publish job status changes to amqp servert.

POLL_JOB_MANAGER 0

CLOUD_JOB_MONITOR 1

JOB_EMAIL_NOTIFICATION_MONITOR 2

XSEDE_AMQP_SUBSCRIBE 3

FORK 4

LOCAL 5

Enumeration: DMType

COMPUTE_RESOURCE 0

STORAGE_RESOURCE 1

Enumeration: ProviderName
Provider name

EC2 0

AWSEC2 1

RACKSPACE 2

Data structures
Struct: ResourceJobManager
Key Field Type Description Requiredness Default value

1 resourceJobManagerId string required "DO_NOT_SET_AT_CLIENTS"
2 resourceJobManagerType ResourceJobManagerType required
3 pushMonitoringEndpoint string optional
4 jobManagerBinPath string optional
5 jobManagerCommands map<JobManagerCommand, string> optional
6 parallelismPrefix map<parallelism_model.ApplicationParallelismType, string> optional

Resource Job Manager Information

resourceJobManagerType:
A typical HPC cluster has a single Job Manager to manage the resources.

pushMonitoringEndpoint:
If the job manager pushes out state changes to a database or bus, specify the service endpoint.
Ex: Moab Web Service, Moab MongoDB URL, AMQP (GLUE2) Broker

jobManagerBinPath:
Path to the Job Manager Installation Binary directory.

jobManagerCommands:
An enumeration of commonly used manager commands.

Struct: BatchQueue
Key Field Type Description Requiredness Default value

1 queueName string required
2 queueDescription string optional
3 maxRunTime i32 optional
4 maxNodes i32 optional
5 maxProcessors i32 optional
6 maxJobsInQueue i32 optional
7 maxMemory i32 optional
8 cpuPerNode i32 optional
9 defaultNodeCount i32 optional
10 defaultCPUCount i32 optional
11 defaultWalltime i32 optional
12 queueSpecificMacros string optional
13 isDefaultQueue bool optional

Batch Queue Information on SuperComputers

maxRunTime:
Maximum allowed run time in hours.

Struct: LOCALSubmission
Key Field Type Description Requiredness Default value

1 jobSubmissionInterfaceId string required "DO_NOT_SET_AT_CLIENTS"
2 resourceJobManager ResourceJobManager required
3 securityProtocol data_movement_models.SecurityProtocol required

Locally Fork Jobs as OS processes

alternativeSSHHostName:
If the login to ssh is different than the hostname itself, specify it here

sshPort:
If a non-default port needs to used, specify it.

Struct: SSHJobSubmission
Key Field Type Description Requiredness Default value

1 jobSubmissionInterfaceId string required "DO_NOT_SET_AT_CLIENTS"
2 securityProtocol data_movement_models.SecurityProtocol required
3 resourceJobManager ResourceJobManager required
4 alternativeSSHHostName string optional
5 sshPort i32 optional 22
6 monitorMode MonitorMode optional
7 batchQueueEmailSenders list<string> optional

Authenticate using Secured Shell

alternativeSSHHostName:
If the login to ssh is different than the hostname itself, specify it here

sshPort:
If a non-default port needs to used, specify it.

batchQueueEmailSenders:
If a resource always sends the monitoring from a specific address, specify the
full email address. If a resource sends emails from multiple addresses (
example: based on the submitted login node) then use the wildchar * to indicate
the same. Example: *@*.example.com or *@example.com

Struct: GlobusJobSubmission
Key Field Type Description Requiredness Default value

1 jobSubmissionInterfaceId string required "DO_NOT_SET_AT_CLIENTS"
2 securityProtocol data_movement_models.SecurityProtocol required
3 globusGateKeeperEndPoint list<string> optional

Struct: UnicoreJobSubmission
Key Field Type Description Requiredness Default value

1 jobSubmissionInterfaceId string required "DO_NOT_SET_AT_CLIENTS"
2 securityProtocol data_movement_models.SecurityProtocol required
3 unicoreEndPointURL string required

Unicore Job Submission

unicoreEndPointURL:
unicoreGateway End Point. The provider will query this service to fetch required service end points.
authenticationMode
The authenticationMode defines the way certificate is fetched.

Struct: CloudJobSubmission
Key Field Type Description Requiredness Default value

1 jobSubmissionInterfaceId string required "DO_NOT_SET_AT_CLIENTS"
2 securityProtocol data_movement_models.SecurityProtocol required
3 nodeId string required
4 executableType string required
5 providerName ProviderName required
6 userAccountName string required

Cloud Job Submission

Struct: JobSubmissionInterface
Key Field Type Description Requiredness Default value

1 jobSubmissionInterfaceId string required
2 jobSubmissionProtocol JobSubmissionProtocol required
3 priorityOrder i32 required 0

Job Submission Interfaces

jobSubmissionInterfaceId: The Job Submission Interface has to be previously registered and referenced here.

priorityOrder:
For resources with multiple interfaces, the priority order should be selected.
Lower the numerical number, higher the priority

Struct: ComputeResourceDescription
Key Field Type Description Requiredness Default value

1 computeResourceId string required "DO_NOT_SET_AT_CLIENTS"
2 hostName string required
3 hostAliases list<string> optional
4 ipAddresses list<string> optional
5 resourceDescription string optional
6 enabled bool optional
7 batchQueues list<BatchQueue> optional
8 fileSystems map<FileSystems, string> optional
9 jobSubmissionInterfaces list<JobSubmissionInterface> optional
10 dataMovementInterfaces list<data_movement_models.DataMovementInterface> optional
11 maxMemoryPerNode i32 optional
12 gatewayUsageReporting bool optional
13 gatewayUsageModuleLoadCommand string optional
14 gatewayUsageExecutable string optional
15 cpusPerNode i32 optional
16 defaultNodeCount i32 optional
17 defaultCPUCount i32 optional
18 defaultWalltime i32 optional