Machinery for reading, editing, and writing Condor DAG files.
When running DAGs on Condor compute clusters, very often one will wish to
re-run a portion of a DAG. This can be done by marking all jobs except the
ones to be re-run as "DONE". Unfortunately the Condor software suite lacks
an I/O library for reading and writing Condor DAG files, so there is no
easy way to edit DAG files except by playing games sed, awk, or once-off
Python or Perl scripts. That's where this module comes in. This module
will read a DAG file into an in-ram representation that is easily edited,
and allow the file to be written to disk again.
Example:
>>> from gstlal import dagfile
>>> dag = dagfile.DAG.parse(open("pipeline.dag"))
>>> dag.write(open("pipeline.dag", "w"))
Although it is possible to machine-generate an original DAG file using this
module and write it to disk, this module does not provide the tools
required to do any of the other tasks associated with pipeline
construction. For example there is no facility here to generate or manage
submit files, data files, or any other files that are associated with a
full pipeline. Only the DAG file itself is considered here. For general
pipeline construction see the pipeline module. The focus of this module is
on editing existing DAG files.
Developers should also consider doing any new pipeline development using
DAX files as the fundamental workflow description, instead of DAGs. See
http://pegasus.isi.edu for more information.
A DAG file is loaded using the .parse() class method of the DAG class.
This parses the file-like object passed to it and returns an instance of
the DAG class representing the file's contents. Once loaded, the nodes in
the DAG can all be found in the .nodes dictionary, whose keys are the node
names and whose values are the corresponding node objects. Among each node
object's attributes are sets .children and .parents containing references
to the child and parent nodes (not their names) for each node. Note that
every node must appear listed as a parent of each of its children, and vice
versa. The other attributes of a DAG instance contain information about
the DAG, for example the CONFIG file or the DOT file, and so on. All of
the data for each node in the DAG, for example the node's VARS value, its
initial working directory, and so on, can be found in the attributes of the
nodes themselves. A DAG is written to a file using the .write() method of
the DAG object.
1.17.0