|
| | __init__ (self, nodes={}, maxjobs={}, config=None, dot=None, dotupdate=False, dotoverwrite=True, dotinclude=None, node_status_file=None, node_status_file_updatetime=None, jobstate_log=None) |
| | reindex (self) |
| | parse (cls, f, progress=None) |
| | select_nodes_by_name (cls, dag, nodenames) |
| | get_all_parent_names (self, names) |
| | get_all_child_names (self, names) |
| | check_edges (self) |
| | load_rescue (self, f, progress=None) |
| | write (self, f, progress=None, rescue=None) |
| | dot_source (self, title="DAG", rename=False, colour="black", bgcolour="#a3a3a3", statecolours={ 'wait':'yellow', 'idle':'yellow', 'run':'lightblue', 'abort':'red', 'stop':'red', 'success':'green', 'fail':'red'}) |
|
| | dotpat = re.compile(r'^DOT\s+(?P<filename>\S+)(\s+(?P<options>.+))?', re.IGNORECASE) |
| | jobpat = re.compile(r'^JOB\s+(?P<name>\S+)\s+(?P<filename>\S+)(\s+DIR\s+(?P<directory>\S+))?(\s+(?P<noop>NOOP))?(\s+(?P<done>DONE))?', re.IGNORECASE) |
| | datapat = re.compile(r'^DATA\s+(?P<name>\S+)\s+(?P<filename>\S+)(\s+DIR\s+(?P<directory>\S+))?(\s+(?P<noop>NOOP))?(\s+(?P<done>DONE))?', re.IGNORECASE) |
| | subdagpat = re.compile(r'^SUBDAG\s+EXTERNAL\s+(?P<name>\S+)\s+(?P<filename>\S+)(\s+DIR\s+(?P<directory>\S+))?(\s+(?P<noop>NOOP))?(\s+(?P<done>DONE))?', re.IGNORECASE) |
| | splicepat = re.compile(r'^SPLICE\s+(?P<name>\S+)\s+(?P<filename>\S+)(\s+DIR\s+(?P<directory>\S+))?', re.IGNORECASE) |
| | prioritypat = re.compile(r'^PRIORITY\s+(?P<name>\S+)\s+(?P<value>\S+)', re.IGNORECASE) |
| | categorypat = re.compile(r'^CATEGORY\s+(?P<name>\S+)\s+(?P<category>\S+)', re.IGNORECASE) |
| | retrypat = re.compile(r'^RETRY\s+(?P<name>\S+)\s+(?P<retries>\S+)(\s+UNLESS-EXIT\s+(?P<retry_unless_exit_value>\S+))?', re.IGNORECASE) |
| | varspat = re.compile(r'^VARS\s+(?P<name>\S+)\s+(?P<vars>.+)', re.IGNORECASE) |
| | varsvaluepat = re.compile(r'(?P<name>\S+)\s*=\s*"(?P<value>.*?)(?<!\\)"', re.IGNORECASE) |
| | scriptpat = re.compile(r'^SCRIPT\s+(?P<type>(PRE)|(POST))\s(?P<name>\S+)\s+(?P<executable>\S+)(\s+(?P<arguments>.+))?', re.IGNORECASE) |
| | abortdagonpat = re.compile(r'^ABORT-DAG-ON\s+(?P<name>\S+)\s+(?P<exitvalue>\S+)(\s+RETURN\s+(?P<returnvalue>\S+))?', re.IGNORECASE) |
| | arcpat = re.compile(r'^PARENT\s+(?P<parents>.+?)\s+CHILD\s+(?P<children>.+)', re.IGNORECASE) |
| | maxjobspat = re.compile(r'^MAXJOBS\s+(?P<category>\S+)\s+(?P<value>\S+)', re.IGNORECASE) |
| | configpat = re.compile(r'^CONFIG\s+(?P<filename>\S+)', re.IGNORECASE) |
| | nodestatuspat = re.compile(r'^NODE_STATUS_FILE\s+(?P<filename>\S+)(\s+(?P<updatetime>\S+))?', re.IGNORECASE) |
| | jobstatepat = re.compile(r'^JOBSTATE_LOG\s+(?P<filename>\S+)', re.IGNORECASE) |
| | donepat = re.compile(r'^DONE\s+(?P<name>\S+)', re.IGNORECASE) |
Representation of the contents of a Condor DAG file.
BUGS: the semantics of the "+" special character in category names
is not understood. For now, it is an error for a node's category
to not be found verbatim in a MAXJOBS line. The "+" character is a
wildcard-like character used in the assignment of MAXJOBS values to
job categories in splices; see the Condor documentation for more
information.
Definition at line 346 of file dagfile.py.
| python.dagfile.DAG.dot_source |
( |
| self, |
|
|
| title = "DAG", |
|
|
| rename = False, |
|
|
| colour = "black", |
|
|
| bgcolour = "#a3a3a3", |
|
|
| statecolours = {'wait': 'yellow', 'idle': 'yellow', 'run': 'lightblue', 'abort': 'red', 'stop': 'red', 'success': 'green', 'fail': 'red'} ) |
Generator yielding a sequence of strings containing DOT
code to generate a visualization of the DAG graph. See
http://www.graphviz.org for more information.
title provides a title for the graph. If rename is True,
instead of using the names of the nodes for the node names
in the graph, numbers will be used instead. The numbers
are assigned to the nodes in alphabetical order by node
name. This might be required if the nodes have names that
are incompatible with the DOT syntax.
colour and bgcolour set the outline colour of the graph
nodes and the background colour for the graph respectively.
statecolours is a dictionary mapping node state (see the
.state attribute of the JOB class and its derivatives) to a
colour. Set statecolours to None to disable state-based
colouring of graph nodes.
Example:
>>> import sys
>>> sys.stdout.writelines(dag.dot_source(statecolours = None))
BUGS: the JOB class does not implement the ability to
retrieve the job state at this time, therefore it is always
necessary to set statecolours to None. This might change
in the future.
Definition at line 950 of file dagfile.py.
| python.dagfile.DAG.load_rescue |
( |
| self, |
|
|
| f, |
|
|
| progress = None ) |
Parse the file-like object f as a rescue DAG, using the
DONE lines therein to set the job states of this DAG.
In the past, rescue DAGs were full copies of the original
DAG with the word DONE added to the JOB lines of completed
jobs. In version 7.7.2 of Condor, the default format of
rescue DAGs was changed to a condensed format consisting of
only the names of completed jobs and the number of retries
remaining for incomplete jobs. Currently Condor still
supports the original rescue DAG format, but the user must
set the DAGMAN_WRITE_PARTIAL_RESCUE config variable to
false to obtain one. This module does not directly support
the new format, however this method allows a new-style
rescue DAG to be parsed to set the states of the jobs in a
DAG. This, in effect, converts a new-style rescue DAG to
an old-style rescue DAG, allowing the result to be
manipulated as before.
If the progress argument is not None, it should be a
callable object. This object will be called periodically
and passed the f argument, the current line number, and a
boolean indicating if parsing is complete. The boolean is
always False until parsing is complete, then the callable
will be invoked one last time with the final line count and
the boolean set to True.
Definition at line 773 of file dagfile.py.
| python.dagfile.DAG.parse |
( |
| cls, |
|
|
| f, |
|
|
| progress = None ) |
Parse the file-like object f as a Condor DAG file. Return
a DAG object. The file object must be iterable, yielding
one line of text of the DAG file in each iteration.
If the progress argument is not None, it should be a
callable object. This object will be called periodically
and passed the f argument, the current line number, and a
boolean indicating if parsing is complete. The boolean is
always False until parsing is complete, then the callable
will be invoked one last time with the final line count and
the boolean set to True.
Example:
>>> def progress(f, n, done):
... print("reading %s: %d lines\\r" % (f.name, n)),
... if done:
... print
...
>>> dag = DAG.parse(open("pipeline.dag"), progress = progress)
Definition at line 476 of file dagfile.py.
| python.dagfile.DAG.select_nodes_by_name |
( |
| cls, |
|
|
| dag, |
|
|
| nodenames ) |
Construct a new DAG object containing only the nodes whose
names are in nodenames.
Example:
>>> names_to_rerun = set(["triggergen"])
>>> dag = DAG.select_nodes_by_name(dag, names_to_rerun | dag.get_all_parent_names(names_to_rerun))
NOTE: the new DAG object is given references to the node
(JOB, DATA, etc.) objects in the original DAG, not copies
of them. Therefore, editing the node objects, for example
modifying their parent or child sets, will affect both
DAGs. To obtain an independent DAG with its own node
objects, make a deepcopy of the object that is returned
(see the copy module in the Python standard library for
more information).
Example:
>>> import copy
>>> dag = copy.deepcopy(DAG.select_nodes_by_name(dag, names_to_rerun | dag.get_all_parent_names(names_to_rerun)))
Definition at line 677 of file dagfile.py.
| python.dagfile.DAG.write |
( |
| self, |
|
|
| f, |
|
|
| progress = None, |
|
|
| rescue = None ) |
Write the DAG to the file-like object f. The object must
provide a .write() method. In the special case that the
optional rescue argument is not None (see below) then f can
be set to None and no DAG file will be written (just the
rescue DAG will be written).
If the progress argument is not None, it should be a
callable object. This object will be called periodically
and passed the f argument, the current line number, and a
boolean indicating if writing is complete. The boolean is
always False until writing is complete, then the callable
will be invoked one last time with the final line count and
the boolean set to True.
Example:
>>> def progress(f, n, done):
... print "writing %s: %d lines\\r" % (f.name, n),
... if done:
... print
...
>>> dag.write(open("pipeline.dag", "w"), progress = progress)
NOTE: when writing PARENT/CHILD graph edges, this method
will silently skip any node names that are not in this
DAG's graph. This is a convenience to simplify writing
DAGs constructed by the .select_nodes_by_name() class
method. If one wishes to check for broken parent/child
links before writing the DAG use the .check_edges() method.
If the optional rescue argument is not None, it must be a
file-like object providing a .write() method and the DONE
state of jobs will be written to this file instead of the
.dag (in the .dag all jobs will be marked not done).
Example:
>>> dag.write(open("pipeline.dag", "w"), rescue = open("pipeline.dag.rescue001", "w"))
NOTE: it is left as an exercise for the calling code to
ensure the name chosen for the rescue file is consistent
with the naming convention assumed by condor_dagman when it
starts up.
Definition at line 832 of file dagfile.py.
1.17.0