JAWS API
JAWS API Usage Guide
The JAWS API provides a programmatic interface to interact with the JAWS system. This guide provides an overview of the available commands, along with examples to help users interact with JAWS programmatically using Python.
Getting Started
Installation
To use the JAWS API, you need to install the JAWS client package. You can install it using pip:
pip install jaws-client==2.3.0 --index-url https://code.jgi.doe.gov/api/v4/projects/312/packages/pypi/simple
Initialization
Before using any commands, you need to initialize the JAWS API by configuring your environment.
from jaws_client import api
from jaws_client.config import Configuration
jaws_config_file_path = '/clusterfs/jgi/groups/dsi/homes/svc-jaws/dori-staging/jaws-staging.conf' # Please replace the JAWS Config path based on the site you are using
user_token_file_path = '/clusterfs/jgi/groups/dsi/homes/dcassol/jaws.conf' # Please replace the user token path with your own pat
# Initialize the Configuration object
config = Configuration(jaws_config_file_path, user_token_file_path)
# Initialize the JAWS API
jaws = api.JawsApi(config)
Command Reference
Below is a list of available commands you can use in your scripts.
Check System Health
Retrieve the status of the JAWS services:
jaws.health()
{
"Central": "UP",
"RabbitMQ": "UP",
"defiant-Site": "UP",
"dori-Site": "UP",
"jgi-Site": "UP",
"nmdc-Site": "UP",
"nmdc_tahoma-Site": "UP",
"perlmutter-Site": "UP",
"tahoma-Site": "UP"
}
Submit a Workflow
Submit a workflow to be executed at a specific site.
- Parameters:
wdl_file: Path to the WDL file.
site: Compute site where the workflow will run.
inputs: Path to the JSON file containing input parameters.
- Optional Parameters:
team: The team under which the workflow run is being submitted.
tag: A tag to identify the run.
no_cache: Disable Cromwell call-caching for this run.
quiet: Suppress non-essential output.
sub: Path to a ZIP file containing subworkflows.
overwrite_inputs: Overwrite inputs in the staging area.
jaws.submit(
wdl_file="workflow.wdl", # Path to the WDL file or to the URL WDL file
site="dori", # Site where the workflow will run
inputs="inputs.json", # (Optional) Path to the input JSON file
team="dsi-aa", # (Optional) Team under which the run is submitted
tag="experiment1", # (Optional) Tag to identify the run
no_cache=True, # (Optional) Disable caching
quiet=False, # (Optional) Suppress non-essential output
sub="subworkflows.zip", # (Optional) Subworkflow file
overwrite_inputs=False # (Optional) Overwrite inputs in staging
)
jaws.submit(wdl_file="align_final.wdl", site="dori", inputs="inputs.json", tag="experiment1", no_cache=True)
Using cached copy of sample.fastq.bz2
Using cached copy of sample.fasta
{'run_id': 21811}
List Active Runs
Get the list of active workflow runs for the current user.
jaws.queue(site="dori", all=False)
[
{
"compute_site_id": "dori",
"id": 21811,
"result": null,
"status": "running",
"tag": "experiment1",
"team_id": "dsi-aa",
"updated": "2024-09-08 16:41:08"
}
]
List Past Runs
Get the list of past workflow runs submitted by the user.
jaws.history(days=1, site="dori", result="any", verbose=False, all=False)
[
{
"compute_site_id": "dori",
"cpu_hours": null,
"cromwell_run_id": "d33be53a-1905-4214-99f9-9b9f3d2a42a6",
"id": 21811,
"input_site_id": "dori",
"json_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/inputs.json",
"output_dir": null,
"result": null,
"status": "running",
"status_detail": "The run is being executed; you can check `tasks` for more detail",
"submitted": "2024-09-08 16:38:54",
"tag": "experiment1",
"team_id": "dsi-aa",
"updated": "2024-09-08 16:41:08",
"user_id": "dcassol",
"wdl_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/align_final.wdl",
"workflow_name": "bbtools",
"workflow_root": "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/cromwell-executions/bbtools/d33be53a-1905-4214-99f9-9b9f3d2a42a6"
}
]
Get Run Status
Retrieve the current status of a specific workflow run using its run_id.
jaws.status("21811", verbose=False)
{
"compute_site_id": "dori",
"cpu_hours": 0.0,
"cromwell_run_id": "d33be53a-1905-4214-99f9-9b9f3d2a42a6",
"id": 21811,
"input_site_id": "dori",
"json_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/inputs.json",
"output_dir": "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/dsi-aa/dcassol/21811/d33be53a-1905-4214-99f9-9b9f3d2a42a6",
"result": "succeeded",
"status": "done",
"status_detail": "The run is complete.",
"submitted": "2024-09-08 16:38:54",
"tag": "experiment1",
"team_id": "dsi-aa",
"updated": "2024-09-08 16:44:20",
"user_id": "dcassol",
"wdl_file": "/clusterfs/jgi/groups/dsi/homes/dcassol/jaws/jaws-tutorial-examples/5min_example/align_final.wdl",
"workflow_name": "bbtools",
"workflow_root": "/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/cromwell-executions/bbtools/d33be53a-1905-4214-99f9-9b9f3d2a42a6"
}
View Run Logs
Get the log of state transitions for a specific workflow run.
jaws.log("21811")
{'data': [['created', 'upload queued', '2024-09-08 16:38:57', ''], ['upload queued', 'upload complete', '2024-09-08 16:38:57', ''], ['upload complete', 'ready', '2024-09-08 16:39:08', ''], ['ready', 'submitted', '2024-09-08 16:39:16', ''], ['submitted', 'queued', '2024-09-08 16:39:27', ''], ['queued', 'running', '2024-09-08 16:41:08', ''], ['running', 'succeeded', '2024-09-08 16:42:59', ''], ['succeeded', 'complete', '2024-09-08 16:43:09', ''], ['complete', 'finished', '2024-09-08 16:43:19', ''], ['finished', 'download queued', '2024-09-08 16:43:29', ''], ['download queued', 'download complete', '2024-09-08 16:43:39', ''], ['download complete', 'fix perms complete', '2024-09-08 16:43:49', ''], ['fix perms complete', 'sync complete', '2024-09-08 16:44:00', ''], ['sync complete', 'slack succeeded', '2024-09-08 16:44:10', ''], ['slack succeeded', 'done', '2024-09-08 16:44:20', '']], 'header': ['STATUS_FROM', 'STATUS_TO', 'TIMESTAMP', 'COMMENT']}
View Tasks Details
Retrieve detailed information about individual tasks within a workflow run.
jaws.tasks("21811")
{'data': [['call-alignment', '24135', 'succeeded', '2024-09-08 16:39:21', '2024-09-08 16:41:02', '2024-09-08 16:41:07', 2, 0, False, 'bbtools.alignment', 1, 8, 10, 0.0, 0], ['call-samtools', '24136', 'succeeded', '2024-09-08 16:42:09', '2024-09-08 16:42:21', '2024-09-08 16:42:23', 0, 0, False, 'bbtools.samtools', 1, 4, 20, 0.0, 0]], 'header': ['TASK_DIR', 'JOB_ID', 'STATUS', 'QUEUE_START', 'RUN_START', 'RUN_END', 'QUEUE_MIN', 'RUN_MIN', 'CACHED', 'TASK_NAME', 'REQ_CPU', 'REQ_GB', 'REQ_MIN', 'CPU_HRS', 'RETURN_CODE']}
Cancel Workflow Run
Cancel a specific workflow run or multiple runs.
jaws.cancel(run_ids=("21811",))
{'21811': True}
Cancel All Active Runs
Cancel all currently active workflow runs.
jaws.cancel_all()
{'21647': True, '21661': True, '21738': True, '21747': True, '21801': True, '21812': True}
Resubmit a Workflow
Resubmit a failed workflow run. Please investigate the cause of the failure before resubmitting. If the issue is related to the WDL or inputs, please fix them and use the jaws.submit command. To resubmit a failed workflow run caused by system issues, you can use the jaws.resubmit command. JAWS will reuse the same WDL and inputs in the JAWS staging area.
Example:
jaws.resubmit(run_id="21811")
{'21811': 'resubmitted'}
Download Run Results
Get detailed task information for a specific workflow run.
jaws.download(run_id="21811")
{'download_id': 12297, 'id': 21811, 'status': 'download queued'}
List Available Compute Sites
List all available compute sites for submitting workflows.
jaws.list_sites()
{
"defiant": {
"max_cpu": "128",
"max_ram_gb": "256",
"max_time": "1425",
"status": true
},
"dori": {
"max_cpu": "64",
"max_ram_gb": "1480",
"max_time": "4320",
"status": true
},
"jgi": {
"max_cpu": "32",
"max_ram_gb": "492",
"max_time": "4320",
"status": true
},
"nmdc": {
"max_cpu": "128",
"max_ram_gb": "492",
"max_time": "2865",
"status": true
},
"nmdc_tahoma": {
"max_cpu": "36",
"max_ram_gb": "1480",
"max_time": "2865",
"status": true
},
"perlmutter": {
"max_cpu": "128",
"max_ram_gb": "492",
"max_time": "1440",
"status": true
},
"tahoma": {
"max_cpu": "36",
"max_ram_gb": "1480",
"max_time": "2865",
"status": true
}
}
Generate Inputs Template
Generate an input template (JSON) from a WDL file.
jaws.inputs(wdl_file="align.wdl")
'{\n "bbtools.ref": "File",\n "bbtools.reads1": "File"\n}'
Validate WDL and Inputs
Validate a WDL file and its corresponding inputs.
jaws.validate(shell_check=True, wdl_file="align.wdl", inputs_file="inputs.json")
{'result': 'succeeded', 'message': 'align_final.wdl\n workflow bbtools\n call alignment\n call samtools\n task alignment\n task samtools\n', 'error': ''}
Get User Information
Retrieve the current user’s information.
jaws.get_user()
{'email': 'dcassol@lbl.gov', 'name': 'Daniela Cassol', 'slack_id': 'XXXXXXX', 'teams': ['dsi-aa'], 'uid': 'dcassol'}
Update User Information
Update the current user’s information.
jaws.update_user(data=UserUpdate(slack_id="<Member_ID>", slack_webhook="<WebHook_URL>"))
List All Teams
List all teams, including those the current user does not belong to.
jaws.list_teams()
[
"gt-ga",
"nmdc",
"dsi-ii",
"bionano",
"sc-mcr",
"ornl",
"workshop",
"plant_ha",
"plantcompgen",
"gaa",
"gt-seqtech",
"sc-mds",
"gt-usa",
"micro-scale",
"rqc",
"dsi-aa",
"gt-syn",
"phytzm"
]
List User Teams
List all teams the current user belongs to.
jaws.my_teams()
['dsi-aa']
List Team Members
List all members of a specific team.
jaws.team_get_users(team_id="dsi-aa")
['dcassol', 'ekirton', 'elais', 'jfroula', 'mmelara', 'ramanik', 'ssul']
Get JAWS Teams Path
Get the path to the JAWS Team directory in a specific site.
jaws.team_get_site("dsi-aa", "dori")
'/clusterfs/jgi/scratch/dsi/aa/jaws/dori-staging/dsi-aa'
JAWS CENTRAL REST API Documentation
Discover the detailed definition and specifications of the CENTRAL REST API for the JAWS by clicking here.