Handling Heavy Workloads and Automated Workflows on NCIP Hub

By Michael McLennan

Purdue University

Download (PPTX)

Licensed according to this deed.

Published on

Abstract

In this June 2014 NCIP Hub Webinar, we see how to use "workspaces" within the hub to access significant computational power.  The "submit" command is a gateway to many different execution venues that can be integrated into each hub.  This seminar shows how to use "submit" to send blastx jobs to DiaGrid machines, and how to upload input data files and download results.  It also provides a brief introduction to the Pegasus Workflow Management System, which can be used to define complex workflows with dozens or hundreds of programs working together in an analysis flow.  Once a workflow is defined and compiled to a DAX representation, the workflow can be executed from a workspace via "submit".  All of these capabilities can be used by researchers directly at the command line, or built into tools published on the hub so that many other users can access the same power with the press of a button.

Cite this work

Researchers should cite this work as follows:

  • Michael McLennan (2014), "Handling Heavy Workloads and Automated Workflows on NCIP Hub," https://nciphub.org/resources/424.

    BibTex | EndNote

Tags