DAGMan Workflows

DAGMan is a HTCondor tool that allows multiple jobs to be organized in workflows, represented as a directed acyclic graph (DAG). A DAGMan workflow automatically submits jobs in a particular order, such that certain jobs need to complete before others start running. This allows the outputs of some jobs to be used as inputs for others, and makes it easy to replicate a workflow multiple times in the future.

Example DAG Visualized

        flowchart TD
 A --> B
    

A simple example is a workflow that requires output from node A to become input for node B. This can be described as a DAGMan workflow as follows:

Example DAG description file

# Example DAGMan Workflow
JOB A produce_data.sub
JOB B process_data.sub

PARENT A CHILD B

Note

A video introducing the DAGman tool for beginners is available at https://www.youtube.com/watch?v=1MvVHxRs7iU and another video, for intermediate users, is available at https://www.youtube.com/watch?v=C2RkdxE_ph0 . A link to the slides is available in the videos’ description.

• DAGMan Introduction
  • Describing Workflows with DAGMan
  • JOB
  • PARENT/CHILD Relationships
  • Scripts
  • Node Submit Descriptions
• Running and Managing DAGMan
  • Basic DAG Controls
  • File Paths in DAGs
  • Managing Large Numbers of Jobs
• DAGMan Completion
  • Node Success/Failure
  • Resubmitting a Failed DAG
  • DAG Save Point Files
• Composing Workflows from DAGs
  • Terminology
  • A DAG Within a DAG Is a SUBDAG
  • DAG Splicing
• Advance DAGMan Functionality
  • Custom Job Macros for Nodes
  • Special Node Types
  • Node Priorities
  • DAGMan and Accounting Groups
  • ALL_NODES Option
  • INCLUDE
  • DAG Manager Job Specifications
  • DAG Specific Configuration
  • Visualizing DAGs
• Informational Files
  • Workflow Metrics
  • Current Node Status File
  • Machine-Readable Event History
• Quick Reference
  • DAG Commands
  • Produced Files
  • Referable DAG Information
  • DAG Submission and Management