comparison env/lib/python3.9/site-packages/cwltool/schemas/v1.1.0-dev1/Workflow.yml @ 0:4f3585e2f14b draft default tip

"planemo upload commit 60cee0fc7c0cda8592644e1aad72851dec82c959"
author shellac
date Mon, 22 Mar 2021 18:12:50 +0000
parents
children
comparison
equal deleted inserted replaced
-1:000000000000 0:4f3585e2f14b
1 saladVersion: v1.1
2 $base: "https://w3id.org/cwl/cwl#"
3
4 $namespaces:
5 cwl: "https://w3id.org/cwl/cwl#"
6 rdfs: "http://www.w3.org/2000/01/rdf-schema#"
7
8 $graph:
9
10 - name: "WorkflowDoc"
11 type: documentation
12 doc:
13 - |
14 # Common Workflow Language (CWL) Workflow Description, v1.1.0-dev1
15
16 This version:
17 * https://w3id.org/cwl/v1.1.0-dev1/
18
19 Current version:
20 * https://w3id.org/cwl/
21 - "\n\n"
22 - {$include: contrib.md}
23 - "\n\n"
24 - |
25 # Abstract
26
27 One way to define a workflow is: an analysis task represented by a
28 directed graph describing a sequence of operations that transform an
29 input data set to output. This specification defines the Common Workflow
30 Language (CWL) Workflow description, a vendor-neutral standard for
31 representing workflows intended to be portable across a variety of
32 computing platforms.
33
34 - {$include: intro.md}
35
36 - |
37
38 ## Introduction to CWL Workflow standard v1.1.0-dev1
39
40 This specification represents the latest stable release from the
41 CWL group. Since the v1.0 release, v1.1.0-dev1 introduces the
42 following updates to the CWL Workflow standard.
43 Documents should to use `cwlVersion: v1.1.0-dev1` to make use of new
44 syntax and features introduced in v1.1.0-dev1. Existing v1.0 documents
45 should be trivially updatable by changing `cwlVersion`, however
46 CWL documents that relied on previously undefined or
47 underspecified behavior may have slightly different behavior in
48 v1.1.0-dev1.
49
50 ## Changelog
51
52 * Clarify intent of ExpressionTool and clarify that they can be run standalone.
53 * Clarify how Requirements are inherited by step processes.
54 * Clarify behavior around step inputs not consumed by the target
55 run process.
56 * Document explicitly that CWL allows pass-through of workflow inputs to outputs.
57 * Added optional name and doc fields to `OutputRecordSchema`.
58 * Default values for some fields are now expressed in the schema.
59 * When defining record types, fields of type `File` may now include
60 `format`, `secondaryFiles`, and `streamable`.
61 * The `Input` schemas (`Record`, `Array`, `Enum`), now permits a `doc` field.
62 * Any `doc` field may be an array of strings in addition to the
63 previously allowed single string.
64 * The `run` field now introduces a subscope so that embedded
65 definitions do not have identifier conflicts with the
66 outer workflow.
67 * `Workflow.inputs` and `ExpressionTool.inputs` now refer to the type
68 `WorkflowInputParameter` instead of `InputParameter`; the only syntax
69 difference from v1.0 is that `inputBinding` is now deprecated (as
70 there is no command line to bind input values to) and `inputBinding`
71 in a `Workflow` context will be removed in v2.0. Use the new
72 `WorkflowInputParameter.loadContents` instead.
73 * The presence of `OutputRecordField.outputBinding` in v1.0 was due to
74 a technical error and as it makes no semantic sense it was removed.
75 * `WorkflowStepInput` now has a `label` field.
76 * [Addition](#Requirements_and_hints) of `cwl:requirements` field to
77 input object documents.
78 * `secondaryFiles` can now be explicitly marked as `required` or not.
79 * Add note that undefined details related to execution are left open to
80 the platform.
81 * [Added](#WorkReuse) `WorkReuse` feature, allowing to enable or disable the reuse
82 behavior for a particular tool or step for implementations that
83 support reusing output from past work.
84 * [Added](#NetworkAccess) `NetworkAccess` feature, allowing to indicate whether a
85 process requires outgoing network access.
86 * [Added](#InplaceUpdateRequirement) `InplaceUpdateRequirement` feature, allowing tools to directly
87 update files with `writable: true` in `InitialWorkDirRequirement`.
88 * [Added](#LoadListingRequirement) `LoadListingRequirement`
89 and [loadListing](#LoadContents) to control whether and how
90 `Directory` listings should be loaded for use in expressions.
91 * [Better explain](#map) the `map<…>` notation that has existed since v1.0.
92 * Fixed schema error where the `type` field inside the `inputs` and
93 `outputs` fields for both `Workflow` and `ExpressionTool` was
94 incorrectly listed as optional.
95 * For multi-Process CWL documents, if no particular process is named then
96 the process with the `id` of `#main` is chosen.
97
98 See also the [CWL Command Line Tool Description, v1.1.0-dev1 changelog](CommandLineTool.html#Changelog).
99
100 ## Purpose
101
102 The Common Workflow Language Command Line Tool Description express
103 workflows for data-intensive science, such as Bioinformatics, Chemistry,
104 Physics, and Astronomy. This specification is intended to define a data
105 and execution model for Workflows that can be implemented on top of a
106 variety of computing platforms, ranging from an individual workstation to
107 cluster, grid, cloud, and high performance computing systems. Details related
108 to execution of these workflow not laid out in this specification are open to
109 interpretation by the computing platform implementing this specification.
110
111 - {$include: concepts.md}
112
113 - name: ExpressionToolOutputParameter
114 type: record
115 extends: OutputParameter
116 fields:
117 - name: type
118 type:
119 - CWLType
120 - OutputRecordSchema
121 - OutputEnumSchema
122 - OutputArraySchema
123 - string
124 - type: array
125 items:
126 - CWLType
127 - OutputRecordSchema
128 - OutputEnumSchema
129 - OutputArraySchema
130 - string
131 jsonldPredicate:
132 "_id": "sld:type"
133 "_type": "@vocab"
134 refScope: 2
135 typeDSL: True
136 doc: |
137 Specify valid types of data that may be assigned to this parameter.
138
139
140 - name: WorkflowInputParameter
141 type: record
142 extends: InputParameter
143 docParent: "#Workflow"
144 fields:
145 - name: type
146 type:
147 - CWLType
148 - InputRecordSchema
149 - InputEnumSchema
150 - InputArraySchema
151 - string
152 - type: array
153 items:
154 - CWLType
155 - InputRecordSchema
156 - InputEnumSchema
157 - InputArraySchema
158 - string
159 jsonldPredicate:
160 "_id": "sld:type"
161 "_type": "@vocab"
162 refScope: 2
163 typeDSL: True
164 doc: |
165 Specify valid types of data that may be assigned to this parameter.
166 - name: inputBinding
167 type: InputBinding?
168 doc: |
169 Deprecated. Preserved for v1.0 backwards compatability. Will be removed in
170 CWL v2.0. Use `WorkflowInputParameter.loadContents` instead.
171 jsonldPredicate: "cwl:inputBinding"
172
173
174 - type: record
175 name: ExpressionTool
176 extends: Process
177 specialize:
178 - specializeFrom: InputParameter
179 specializeTo: WorkflowInputParameter
180 - specializeFrom: OutputParameter
181 specializeTo: ExpressionToolOutputParameter
182 documentRoot: true
183 doc: |
184 An ExpressionTool is a type of Process object that can be run by itself
185 or as a Workflow step. It executes a pure Javascript expression that has
186 access to the same input parameters as a workflow. It is meant to be used
187 sparingly as a way to isolate complex Javascript expressions that need to
188 operate on input data and produce some result; perhaps just a
189 rearrangement of the inputs. No Docker software container is required
190 or allowed.
191 fields:
192 - name: class
193 jsonldPredicate:
194 "_id": "@type"
195 "_type": "@vocab"
196 type: string
197 - name: expression
198 type: [string, Expression]
199 doc: |
200 The expression to execute. The expression must return a JSON object which
201 matches the output parameters of the ExpressionTool.
202
203 - name: LinkMergeMethod
204 type: enum
205 docParent: "#WorkflowStepInput"
206 doc: The input link merge method, described in [WorkflowStepInput](#WorkflowStepInput).
207 symbols:
208 - merge_nested
209 - merge_flattened
210
211
212 - name: WorkflowOutputParameter
213 type: record
214 extends: OutputParameter
215 docParent: "#Workflow"
216 doc: |
217 Describe an output parameter of a workflow. The parameter must be
218 connected to one or more parameters defined in the workflow that
219 will provide the value of the output parameter. It is legal to
220 connect a WorkflowInputParameter to a WorkflowOutputParameter.
221 fields:
222 - name: outputSource
223 doc: |
224 Specifies one or more workflow parameters that supply the value of to
225 the output parameter.
226 jsonldPredicate:
227 "_id": "cwl:outputSource"
228 "_type": "@id"
229 refScope: 0
230 type:
231 - string?
232 - string[]?
233 - name: linkMerge
234 type: ["null", LinkMergeMethod]
235 jsonldPredicate: "cwl:linkMerge"
236 default: merge_nested
237 doc: |
238 The method to use to merge multiple sources into a single array.
239 If not specified, the default method is "merge_nested".
240 - name: type
241 type:
242 - CWLType
243 - OutputRecordSchema
244 - OutputEnumSchema
245 - OutputArraySchema
246 - string
247 - type: array
248 items:
249 - CWLType
250 - OutputRecordSchema
251 - OutputEnumSchema
252 - OutputArraySchema
253 - string
254 jsonldPredicate:
255 "_id": "sld:type"
256 "_type": "@vocab"
257 refScope: 2
258 typeDSL: True
259 doc: |
260 Specify valid types of data that may be assigned to this parameter.
261
262
263 - name: Sink
264 type: record
265 abstract: true
266 fields:
267 - name: source
268 doc: |
269 Specifies one or more workflow parameters that will provide input to
270 the underlying step parameter.
271 jsonldPredicate:
272 "_id": "cwl:source"
273 "_type": "@id"
274 refScope: 2
275 type:
276 - string?
277 - string[]?
278 - name: linkMerge
279 type: LinkMergeMethod?
280 jsonldPredicate: "cwl:linkMerge"
281 default: merge_nested
282 doc: |
283 The method to use to merge multiple inbound links into a single array.
284 If not specified, the default method is "merge_nested".
285
286
287 - type: record
288 name: WorkflowStepInput
289 extends: [Identified, Sink, LoadContents, Labeled]
290 docParent: "#WorkflowStep"
291 doc: |
292 The input of a workflow step connects an upstream parameter (from the
293 workflow inputs, or the outputs of other workflows steps) with the input
294 parameters of the process specified by the `run` field. Only input parameters
295 declared by the target process will be passed through at runtime to the process
296 though additonal parameters may be specified (for use within `valueFrom`
297 expressions for instance) - unconnected or unused parameters do not represent an
298 error condition.
299
300 ## Input object
301
302 A WorkflowStepInput object must contain an `id` field in the form
303 `#fieldname` or `#prefix/fieldname`. When the `id` field contains a slash
304 `/` the field name consists of the characters following the final slash
305 (the prefix portion may contain one or more slashes to indicate scope).
306 This defines a field of the workflow step input object with the value of
307 the `source` parameter(s).
308
309 ## Merging
310
311 To merge multiple inbound data links,
312 [MultipleInputFeatureRequirement](#MultipleInputFeatureRequirement) must be specified
313 in the workflow or workflow step requirements.
314
315 If the sink parameter is an array, or named in a [workflow
316 scatter](#WorkflowStep) operation, there may be multiple inbound data links
317 listed in the `source` field. The values from the input links are merged
318 depending on the method specified in the `linkMerge` field. If not
319 specified, the default method is "merge_nested".
320
321 * **merge_nested**
322
323 The input must be an array consisting of exactly one entry for each
324 input link. If "merge_nested" is specified with a single link, the value
325 from the link must be wrapped in a single-item list.
326
327 * **merge_flattened**
328
329 1. The source and sink parameters must be compatible types, or the source
330 type must be compatible with single element from the "items" type of
331 the destination array parameter.
332 2. Source parameters which are arrays are concatenated.
333 Source parameters which are single element types are appended as
334 single elements.
335
336 fields:
337 - name: default
338 type: ["null", Any]
339 doc: |
340 The default value for this parameter to use if either there is no
341 `source` field, or the value produced by the `source` is `null`. The
342 default must be applied prior to scattering or evaluating `valueFrom`.
343 jsonldPredicate:
344 _id: "sld:default"
345 noLinkCheck: true
346 - name: valueFrom
347 type:
348 - "null"
349 - string
350 - Expression
351 jsonldPredicate: "cwl:valueFrom"
352 doc: |
353 To use valueFrom, [StepInputExpressionRequirement](#StepInputExpressionRequirement) must
354 be specified in the workflow or workflow step requirements.
355
356 If `valueFrom` is a constant string value, use this as the value for
357 this input parameter.
358
359 If `valueFrom` is a parameter reference or expression, it must be
360 evaluated to yield the actual value to be assiged to the input field.
361
362 The `self` value in the parameter reference or expression must be
363 1. `null` if there is no `source` field
364 2. the value of the parameter(s) specified in the `source` field when this
365 workflow input parameter **is not** specified in this workflow step's `scatter` field.
366 3. an element of the parameter specified in the `source` field when this workflow input
367 parameter **is** specified in this workflow step's `scatter` field.
368
369 The value of `inputs` in the parameter reference or expression must be
370 the input object to the workflow step after assigning the `source`
371 values, applying `default`, and then scattering. The order of
372 evaluating `valueFrom` among step input parameters is undefined and the
373 result of evaluating `valueFrom` on a parameter must not be visible to
374 evaluation of `valueFrom` on other parameters.
375
376
377 - type: record
378 name: WorkflowStepOutput
379 docParent: "#WorkflowStep"
380 extends: Identified
381 doc: |
382 Associate an output parameter of the underlying process with a workflow
383 parameter. The workflow parameter (given in the `id` field) be may be used
384 as a `source` to connect with input parameters of other workflow steps, or
385 with an output parameter of the process.
386
387 A unique identifier for this workflow output parameter. This is
388 the identifier to use in the `source` field of `WorkflowStepInput`
389 to connect the output value to downstream parameters.
390
391
392 - name: ScatterMethod
393 type: enum
394 docParent: "#WorkflowStep"
395 doc: The scatter method, as described in [workflow step scatter](#WorkflowStep).
396 symbols:
397 - dotproduct
398 - nested_crossproduct
399 - flat_crossproduct
400
401
402 - name: WorkflowStep
403 type: record
404 extends: [Identified, Labeled, sld:Documented]
405 docParent: "#Workflow"
406 doc: |
407 A workflow step is an executable element of a workflow. It specifies the
408 underlying process implementation (such as `CommandLineTool` or another
409 `Workflow`) in the `run` field and connects the input and output parameters
410 of the underlying process to workflow parameters.
411
412 # Scatter/gather
413
414 To use scatter/gather,
415 [ScatterFeatureRequirement](#ScatterFeatureRequirement) must be specified
416 in the workflow or workflow step requirements.
417
418 A "scatter" operation specifies that the associated workflow step or
419 subworkflow should execute separately over a list of input elements. Each
420 job making up a scatter operation is independent and may be executed
421 concurrently.
422
423 The `scatter` field specifies one or more input parameters which will be
424 scattered. An input parameter may be listed more than once. The declared
425 type of each input parameter is implicitly becomes an array of items of the
426 input parameter type. If a parameter is listed more than once, it becomes
427 a nested array. As a result, upstream parameters which are connected to
428 scattered parameters must be arrays.
429
430 All output parameter types are also implicitly wrapped in arrays. Each job
431 in the scatter results in an entry in the output array.
432
433 If any scattered parameter runtime value is an empty array, all outputs are
434 set to empty arrays and no work is done for the step, according to
435 applicable scattering rules.
436
437 If `scatter` declares more than one input parameter, `scatterMethod`
438 describes how to decompose the input into a discrete set of jobs.
439
440 * **dotproduct** specifies that each of the input arrays are aligned and one
441 element taken from each array to construct each job. It is an error
442 if all input arrays are not the same length.
443
444 * **nested_crossproduct** specifies the Cartesian product of the inputs,
445 producing a job for every combination of the scattered inputs. The
446 output must be nested arrays for each level of scattering, in the
447 order that the input arrays are listed in the `scatter` field.
448
449 * **flat_crossproduct** specifies the Cartesian product of the inputs,
450 producing a job for every combination of the scattered inputs. The
451 output arrays must be flattened to a single level, but otherwise listed in the
452 order that the input arrays are listed in the `scatter` field.
453
454 # Subworkflows
455
456 To specify a nested workflow as part of a workflow step,
457 [SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) must be
458 specified in the workflow or workflow step requirements.
459
460 It is a fatal error if a workflow directly or indirectly invokes itself as
461 a subworkflow (recursive workflows are not allowed).
462
463 fields:
464 - name: in
465 type: WorkflowStepInput[]
466 jsonldPredicate:
467 _id: "cwl:in"
468 mapSubject: id
469 mapPredicate: source
470 doc: |
471 Defines the input parameters of the workflow step. The process is ready to
472 run when all required input parameters are associated with concrete
473 values. Input parameters include a schema for each parameter which is
474 used to validate the input object. It may also be used build a user
475 interface for constructing the input object.
476 - name: out
477 type:
478 - type: array
479 items: [string, WorkflowStepOutput]
480 jsonldPredicate:
481 _id: "cwl:out"
482 _type: "@id"
483 identity: true
484 doc: |
485 Defines the parameters representing the output of the process. May be
486 used to generate and/or validate the output object.
487 - name: requirements
488 type: ProcessRequirement[]?
489 jsonldPredicate:
490 _id: "cwl:requirements"
491 mapSubject: class
492 doc: |
493 Declares requirements that apply to either the runtime environment or the
494 workflow engine that must be met in order to execute this workflow step. If
495 an implementation cannot satisfy all requirements, or a requirement is
496 listed which is not recognized by the implementation, it is a fatal
497 error and the implementation must not attempt to run the process,
498 unless overridden at user option.
499 - name: hints
500 type: Any[]?
501 jsonldPredicate:
502 _id: "cwl:hints"
503 noLinkCheck: true
504 mapSubject: class
505 doc: |
506 Declares hints applying to either the runtime environment or the
507 workflow engine that may be helpful in executing this workflow step. It is
508 not an error if an implementation cannot satisfy all hints, however
509 the implementation may report a warning.
510 - name: run
511 type: [string, Process]
512 jsonldPredicate:
513 _id: "cwl:run"
514 _type: "@id"
515 subscope: run
516 doc: |
517 Specifies the process to run.
518 - name: scatter
519 type:
520 - string?
521 - string[]?
522 jsonldPredicate:
523 "_id": "cwl:scatter"
524 "_type": "@id"
525 "_container": "@list"
526 refScope: 0
527 - name: scatterMethod
528 doc: |
529 Required if `scatter` is an array of more than one element.
530 type: ScatterMethod?
531 jsonldPredicate:
532 "_id": "cwl:scatterMethod"
533 "_type": "@vocab"
534
535
536 - name: Workflow
537 type: record
538 extends: "#Process"
539 documentRoot: true
540 specialize:
541 - specializeFrom: InputParameter
542 specializeTo: WorkflowInputParameter
543 - specializeFrom: OutputParameter
544 specializeTo: WorkflowOutputParameter
545 doc: |
546 A workflow describes a set of **steps** and the **dependencies** between
547 those steps. When a step produces output that will be consumed by a
548 second step, the first step is a dependency of the second step.
549
550 When there is a dependency, the workflow engine must execute the preceeding
551 step and wait for it to successfully produce output before executing the
552 dependent step. If two steps are defined in the workflow graph that
553 are not directly or indirectly dependent, these steps are **independent**,
554 and may execute in any order or execute concurrently. A workflow is
555 complete when all steps have been executed.
556
557 Dependencies between parameters are expressed using the `source` field on
558 [workflow step input parameters](#WorkflowStepInput) and [workflow output
559 parameters](#WorkflowOutputParameter).
560
561 The `source` field expresses the dependency of one parameter on another
562 such that when a value is associated with the parameter specified by
563 `source`, that value is propagated to the destination parameter. When all
564 data links inbound to a given step are fufilled, the step is ready to
565 execute.
566
567 ## Workflow success and failure
568
569 A completed step must result in one of `success`, `temporaryFailure` or
570 `permanentFailure` states. An implementation may choose to retry a step
571 execution which resulted in `temporaryFailure`. An implementation may
572 choose to either continue running other steps of a workflow, or terminate
573 immediately upon `permanentFailure`.
574
575 * If any step of a workflow execution results in `permanentFailure`, then
576 the workflow status is `permanentFailure`.
577
578 * If one or more steps result in `temporaryFailure` and all other steps
579 complete `success` or are not executed, then the workflow status is
580 `temporaryFailure`.
581
582 * If all workflow steps are executed and complete with `success`, then the
583 workflow status is `success`.
584
585 # Extensions
586
587 [ScatterFeatureRequirement](#ScatterFeatureRequirement) and
588 [SubworkflowFeatureRequirement](#SubworkflowFeatureRequirement) are
589 available as standard [extensions](#Extensions_and_Metadata) to core
590 workflow semantics.
591
592 fields:
593 - name: "class"
594 jsonldPredicate:
595 "_id": "@type"
596 "_type": "@vocab"
597 type: string
598 - name: steps
599 doc: |
600 The individual steps that make up the workflow. Each step is executed when all of its
601 input data links are fufilled. An implementation may choose to execute
602 the steps in a different order than listed and/or execute steps
603 concurrently, provided that dependencies between steps are met.
604 type:
605 - type: array
606 items: "#WorkflowStep"
607 jsonldPredicate:
608 mapSubject: id
609
610
611 - type: record
612 name: SubworkflowFeatureRequirement
613 extends: ProcessRequirement
614 doc: |
615 Indicates that the workflow platform must support nested workflows in
616 the `run` field of [WorkflowStep](#WorkflowStep).
617 fields:
618 - name: "class"
619 type: "string"
620 doc: "Always 'SubworkflowFeatureRequirement'"
621 jsonldPredicate:
622 "_id": "@type"
623 "_type": "@vocab"
624
625 - name: ScatterFeatureRequirement
626 type: record
627 extends: ProcessRequirement
628 doc: |
629 Indicates that the workflow platform must support the `scatter` and
630 `scatterMethod` fields of [WorkflowStep](#WorkflowStep).
631 fields:
632 - name: "class"
633 type: "string"
634 doc: "Always 'ScatterFeatureRequirement'"
635 jsonldPredicate:
636 "_id": "@type"
637 "_type": "@vocab"
638
639 - name: MultipleInputFeatureRequirement
640 type: record
641 extends: ProcessRequirement
642 doc: |
643 Indicates that the workflow platform must support multiple inbound data links
644 listed in the `source` field of [WorkflowStepInput](#WorkflowStepInput).
645 fields:
646 - name: "class"
647 type: "string"
648 doc: "Always 'MultipleInputFeatureRequirement'"
649 jsonldPredicate:
650 "_id": "@type"
651 "_type": "@vocab"
652
653 - type: record
654 name: StepInputExpressionRequirement
655 extends: ProcessRequirement
656 doc: |
657 Indicate that the workflow platform must support the `valueFrom` field
658 of [WorkflowStepInput](#WorkflowStepInput).
659 fields:
660 - name: "class"
661 type: "string"
662 doc: "Always 'StepInputExpressionRequirement'"
663 jsonldPredicate:
664 "_id": "@type"
665 "_type": "@vocab"