Mercurial > repos > shellac > sam_consensus_v3
diff env/lib/python3.9/site-packages/cwltool/schemas/v1.2.0-dev4/concepts.md @ 0:4f3585e2f14b draft default tip
"planemo upload commit 60cee0fc7c0cda8592644e1aad72851dec82c959"
| author | shellac |
|---|---|
| date | Mon, 22 Mar 2021 18:12:50 +0000 |
| parents | |
| children |
line wrap: on
line diff
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/env/lib/python3.9/site-packages/cwltool/schemas/v1.2.0-dev4/concepts.md Mon Mar 22 18:12:50 2021 +0000 @@ -0,0 +1,621 @@ +## References to other specifications + +**Javascript Object Notation (JSON)**: http://json.org + +**JSON Linked Data (JSON-LD)**: http://json-ld.org + +**YAML**: http://yaml.org + +**Avro**: https://avro.apache.org/docs/1.8.1/spec.html + +**Uniform Resource Identifier (URI) Generic Syntax**: https://tools.ietf.org/html/rfc3986) + +**Internationalized Resource Identifiers (IRIs)**: +https://tools.ietf.org/html/rfc3987 + +**Portable Operating System Interface (POSIX.1-2008)**: http://pubs.opengroup.org/onlinepubs/9699919799/ + +**Resource Description Framework (RDF)**: http://www.w3.org/RDF/ + +**XDG Base Directory Specification**: https://specifications.freedesktop.org/basedir-spec/basedir-spec-0.6.html + + +## Scope + +This document describes CWL syntax, execution, and object model. It +is not intended to document a CWL specific implementation, however it may +serve as a reference for the behavior of conforming implementations. + +## Terminology + +The terminology used to describe CWL documents is defined in the +Concepts section of the specification. The terms defined in the +following list are used in building those definitions and in describing the +actions of a CWL implementation: + +**may**: Conforming CWL documents and CWL implementations are permitted but +not required to behave as described. + +**must**: Conforming CWL documents and CWL implementations are required to behave +as described; otherwise they are in error. + +**error**: A violation of the rules of this specification; results are +undefined. Conforming implementations may detect and report an error and may +recover from it. + +**fatal error**: A violation of the rules of this specification; results are +undefined. Conforming implementations must not continue to execute the current +process and may report an error. + +**at user option**: Conforming software may or must (depending on the modal verb in +the sentence) behave as described; if it does, it must provide users a means to +enable or disable the behavior described. + +**deprecated**: Conforming software may implement a behavior for backwards +compatibility. Portable CWL documents should not rely on deprecated behavior. +Behavior marked as deprecated may be removed entirely from future revisions of +the CWL specification. + +# Data model + +## Data concepts + +An **object** is a data structure equivalent to the "object" type in JSON, +consisting of a unordered set of name/value pairs (referred to here as +**fields**) and where the name is a string and the value is a string, number, +boolean, array, or object. + +A **document** is a file containing a serialized object, or an array of objects. + +A **process** is a basic unit of computation which accepts input data, +performs some computation, and produces output data. Examples include +CommandLineTools, Workflows, and ExpressionTools. + +An **input object** is an object describing the inputs to an invocation of +a process. + +An **output object** is an object describing the output resulting from an +invocation of a process. + +An **input schema** describes the valid format (required fields, data types) +for an input object. + +An **output schema** describes the valid format for an output object. + +**Metadata** is information about workflows, tools, or input items. + +## Syntax + +CWL documents must consist of an object or array of objects represented using +JSON or YAML syntax. Upon loading, a CWL implementation must apply the +preprocessing steps described in the +[Semantic Annotations for Linked Avro Data (SALAD) Specification](SchemaSalad.html). +An implementation may formally validate the structure of a CWL document using +SALAD schemas located at https://github.com/common-workflow-language/cwl-v1.1/ + +### map + +Note: This section is non-normative. +> type: array<ComplexType> | +> map<`key_field`, ComplexType> + +The above syntax in the CWL specifications means there are two or more ways to write the given value. + +Option one is a array and is the most verbose option. + +Option one generic example: +``` +some_cwl_field: + - key_field: a_complex_type1 + field2: foo + field3: bar + - key_field: a_complex_type2 + field2: foo2 + field3: bar2 + - key_field: a_complex_type3 +``` + +Option one specific example using [Workflow](Workflow.html#Workflow).[inputs](Workflow.html#WorkflowInputParameter): +> array<InputParameter> | +> map<`id`, `type` | InputParameter> + + +``` +inputs: + - id: workflow_input01 + type: string + - id: workflow_input02 + type: File + format: http://edamontology.org/format_2572 +``` + +Option two is enabled by the `map<…>` syntax. Instead of an array of entries we +use a mapping, where one field of the `ComplexType` (here named `key_field`) +becomes the key in the map, and its value is the rest of the `ComplexType` +without the key field. If all of the other fields of the `ComplexType` are +optional and unneeded, then we can indicate this with an empty mapping as the +value: `a_complex_type3: {}` + +Option two generic example: +``` +some_cwl_field: + a_complex_type1: # this was the "key_field" from above + field2: foo + field3: bar + a_complex_type2: + field2: foo2 + field3: bar2 + a_complex_type3: {} # we accept the defualt values for "field2" and "field3" +``` + +Option two specific example using [Workflow](Workflow.html#Workflow).[inputs](Workflow.html#WorkflowInputParameter): +> array<InputParameter> | +> map<`id`, `type` | InputParameter> + + +``` +inputs: + workflow_input01: + type: string + workflow_input02: + type: File + format: http://edamontology.org/format_2572 +``` + +Option two specific example using [SoftwareRequirement](#SoftwareRequirement).[packages](#SoftwarePackage): +> array<SoftwarePackage> | +> map<`package`, `specs` | SoftwarePackage> + + +``` +hints: + SoftwareRequirement: + packages: + sourmash: + specs: [ https://doi.org/10.21105/joss.00027 ] + screed: + version: [ "1.0" ] + python: {} +``` +` +Sometimes we have a third and even more compact option denoted like this: +> type: array<ComplexType> | +> map<`key_field`, `field2` | ComplexType> + +For this example, if we only need the `key_field` and `field2` when specifying +our `ComplexType`s (because the other fields are optional and we are fine with +their default values) then we can abbreviate. + +Option three generic example: +``` +some_cwl_field: + a_complex_type1: foo # we accept the default value for field3 + a_complex_type2: foo2 # we accept the default value for field3 + a_complex_type3: {} # we accept the default values for "field2" and "field3" +``` + +Option three specific example using [Workflow](Workflow.html#Workflow).[inputs](Workflow.html#WorkflowInputParameter): +> array<InputParameter> | +> map<`id`, `type` | InputParameter> + + +``` +inputs: + workflow_input01: string + workflow_input02: File # we accept the default of no File format +``` + +Option three specific example using [SoftwareRequirement](#SoftwareRequirement).[packages](#SoftwarePackage): +> array<SoftwarePackage> | +> map<`package`, `specs` | SoftwarePackage> + + +``` +hints: + SoftwareRequirement: + packages: + sourmash: [ https://doi.org/10.21105/joss.00027 ] + python: {} +``` + + +What if some entries we want to mix the option 2 and 3? You can! + +Mixed option 2 and 3 generic example: +``` +some_cwl_field: + my_complex_type1: foo # we accept the default value for field3 + my_complex_type2: + field2: foo2 + field3: bar2 # we did not accept the default value for field3 + # so we had to use the slightly expanded syntax + my_complex_type3: {} # as before, we accept the default values for both + # "field2" and "field3" +``` + +Mixed option 2 and 3 specific example using [Workflow](Workflow.html#Workflow).[inputs](Workflow.html#WorkflowInputParameter): +> array<InputParameter> | +> map<`id`, `type` | InputParameter> + + +``` +inputs: + workflow_input01: string + workflow_input02: # we use the longer way + type: File # because we want to specify the "format" too + format: http://edamontology.org/format_2572 +``` + +Mixed option 2 and 3 specific example using [SoftwareRequirement](#SoftwareRequirement).[packages](#SoftwarePackage): +> array<SoftwarePackage> | +> map<`package`, `specs` | SoftwarePackage> + + +``` +hints: + SoftwareRequirement: + packages: + sourmash: [ https://doi.org/10.21105/joss.00027 ] + screed: + specs: [ https://github.com/dib-lab/screed ] + version: [ "1.0" ] + python: {} +``` + +Note: The `map<…>` (compact) versions are optional for users, the verbose option #1 is +always allowed, but for presentation reasons option 3 and 2 may be preferred +by human readers. Consumers of CWL must support all three options. + +The normative explanation for these variations, aimed at implementors, is in the +[Schema Salad specification](SchemaSalad.html#Identifier_maps). + +## Identifiers + +If an object contains an `id` field, that is used to uniquely identify the +object in that document. The value of the `id` field must be unique over the +entire document. Identifiers may be resolved relative to either the document +base and/or other identifiers following the rules are described in the +[Schema Salad specification](SchemaSalad.html#Identifier_resolution). + +An implementation may choose to only honor references to object types for +which the `id` field is explicitly listed in this specification. + +## Document preprocessing + +An implementation must resolve [$import](SchemaSalad.html#Import) and +[$include](SchemaSalad.html#Import) directives as described in the +[Schema Salad specification](SchemaSalad.html). + +Another transformation defined in Schema salad is simplification of data type definitions. +Type `<T>` ending with `?` should be transformed to `[<T>, "null"]`. +Type `<T>` ending with `[]` should be transformed to `{"type": "array", "items": <T>}` + +## Extensions and metadata + +Input metadata (for example, a sample identifier) may be represented within +a tool or workflow using input parameters which are explicitly propagated to +output. Future versions of this specification may define additional facilities +for working with input/output metadata. + +Implementation extensions not required for correct execution (for example, +fields related to GUI presentation) and metadata about the tool or workflow +itself (for example, authorship for use in citations) may be provided as +additional fields on any object. Such extensions fields must use a namespace +prefix listed in the `$namespaces` section of the document as described in the +[Schema Salad specification](SchemaSalad.html#Explicit_context). + +It is recommended that concepts from schema.org are used whenever possible. +For the `$schema` field we recommend their RDF encoding: http://schema.org/version/latest/schema.rdf + +Implementation extensions which modify execution semantics must be [listed in +the `requirements` field](#Requirements_and_hints). + +# Execution model + +## Execution concepts + +A **parameter** is a named symbolic input or output of process, with an +associated datatype or schema. During execution, values are assigned to +parameters to make the input object or output object used for concrete +process invocation. + +A **CommandLineTool** is a process characterized by the execution of a +standalone, non-interactive program which is invoked on some input, +produces output, and then terminates. + +A **workflow** is a process characterized by multiple subprocess steps, +where step outputs are connected to the inputs of downstream steps to +form a directed acylic graph, and independent steps may run concurrently. + +A **runtime environment** is the actual hardware and software environment when +executing a command line tool. It includes, but is not limited to, the +hardware architecture, hardware resources, operating system, software runtime +(if applicable, such as the specific Python interpreter or the specific Java +virtual machine), libraries, modules, packages, utilities, and data files +required to run the tool. + +A **workflow platform** is a specific hardware and software implementation +capable of interpreting CWL documents and executing the processes specified by +the document. The responsibilities of the workflow platform may include +scheduling process invocation, setting up the necessary runtime environment, +making input data available, invoking the tool process, and collecting output. + +A workflow platform may choose to only implement the Command Line Tool +Description part of the CWL specification. + +It is intended that the workflow platform has broad leeway outside of this +specification to optimize use of computing resources and enforce policies +not covered by this specification. Some areas that are currently out of +scope for CWL specification but may be handled by a specific workflow +platform include: + +* Data security and permissions +* Scheduling tool invocations on remote cluster or cloud compute nodes. +* Using virtual machines or operating system containers to manage the runtime +(except as described in [DockerRequirement](CommandLineTool.html#DockerRequirement)). +* Using remote or distributed file systems to manage input and output files. +* Transforming file paths. +* Pausing, resuming or checkpointing processes or workflows. + +Conforming CWL processes must not assume anything about the runtime +environment or workflow platform unless explicitly declared though the use +of [process requirements](#Requirements_and_hints). + +## Generic execution process + +The generic execution sequence of a CWL process (including workflows and +command line line tools) is as follows. + +1. Load input object. +1. Load, process and validate a CWL document, yielding one or more process objects. +The [`$namespaces`](SchemaSalad.html#Explicit_context) present in the CWL document +are also used when validating and processing the input object. +1. If there are multiple process objects (due to [`$graph`](SchemaSalad.html#Document_graph)) +and which process object to start with is not specified in the input object (via +a [`cwl:tool`](#Executing_CWL_documents_as_scripts) entry) or by any other means +(like a URL fragment) then choose the process with the `id` of "#main" or "main". +1. Validate the input object against the `inputs` schema for the process. +1. Validate process requirements are met. +1. Perform any further setup required by the specific process type. +1. Execute the process. +1. Capture results of process execution into the output object. +1. Validate the output object against the `outputs` schema for the process. +1. Report the output object to the process caller. + +## Requirements and hints + +A **process requirement** modifies the semantics or runtime +environment of a process. If an implementation cannot satisfy all +requirements, or a requirement is listed which is not recognized by the +implementation, it is a fatal error and the implementation must not attempt +to run the process, unless overridden at user option. + +A **hint** is similar to a requirement; however, it is not an error if an +implementation cannot satisfy all hints. The implementation may report a +warning if a hint cannot be satisfied. + +Optionally, implementations may allow requirements to be specified in the input +object document as an array of requirements under the field name +`cwl:requirements`. If implementations allow this, then such requirements +should be combined with any requirements present in the corresponding Process +as if they were specified there. + +Requirements specified in a parent Workflow are inherited by step processes +if they are valid for that step. If the substep is a CommandLineTool +only the `InlineJavascriptRequirement`, `SchemaDefRequirement`, `DockerRequirement`, +`SoftwareRequirement`, `InitialWorkDirRequirement`, `EnvVarRequirement`, +`ShellCommandRequirement`, `ResourceRequirement` are valid. + +*As good practice, it is best to have process requirements be self-contained, +such that each process can run successfully by itself.* + +If the same process requirement appears at different levels of the +workflow, the most specific instance of the requirement is used, that is, +an entry in `requirements` on a process implementation such as +CommandLineTool will take precedence over an entry in `requirements` +specified in a workflow step, and an entry in `requirements` on a workflow +step takes precedence over the workflow. Entries in `hints` are resolved +the same way. + +Requirements override hints. If a process implementation provides a +process requirement in `hints` which is also provided in `requirements` by +an enclosing workflow or workflow step, the enclosing `requirements` takes +precedence. + +## Parameter references + +Parameter references are denoted by the syntax `$(...)` and may be used in any +field permitting the pseudo-type `Expression`, as specified by this document. +Conforming implementations must support parameter references. Parameter +references use the following subset of +[Javascript/ECMAScript 5.1](http://www.ecma-international.org/ecma-262/5.1/) +syntax, but they are designed to not require a Javascript engine for evaluation. + +In the following [BNF grammar](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form), +character classes and grammar rules are denoted in '{}', '-' denotes +exclusion from a character class, '(())' denotes grouping, '|' denotes +alternates, trailing '*' denotes zero or more repeats, '+' denote one +or more repeats, and all other characters are literal values. + +<p> +<table class="table"> +<tr><td>symbol:: </td><td>{Unicode alphanumeric}+</td></tr> +<tr><td>singleq:: </td><td>[' (( {character - { | \ ' \} } ))* ']</td></tr> +<tr><td>doubleq:: </td><td>[" (( {character - { | \ " \} } ))* "]</td></tr> +<tr><td>index:: </td><td>[ {decimal digit}+ ]</td></tr> +<tr><td>segment:: </td><td>. {symbol} | {singleq} | {doubleq} | {index}</td></tr> +<tr><td>parameter reference::</td><td>$( {symbol} {segment}*)</td></tr> +</table> +</p> + +Use the following algorithm to resolve a parameter reference: + + 1. Match the leading symbol as the key + 2. Look up the key in the parameter context (described below) to get the current value. + It is an error if the key is not found in the parameter context. + 3. If there are no subsequent segments, terminate and return current value + 4. Else, match the next segment + 5. Extract the symbol, string, or index from the segment as the key + 6. Look up the key in current value and assign as new current value. If + the key is a symbol or string, the current value must be an object. + If the key is an index, the current value must be an array or string. + It is an error if the key does not match the required type, or the key is not found or out + of range. + 7. Repeat steps 3-6 + +The root namespace is the parameter context. The following parameters must +be provided: + + * `inputs`: The input object to the current Process. + * `self`: A context-specific value. The contextual values for 'self' are + documented for specific fields elsewhere in this specification. If + a contextual value of 'self' is not documented for a field, it + must be 'null'. + * `runtime`: An object containing configuration details. Specific to the + process type. An implementation may provide + opaque strings for any or all fields of `runtime`. These must be + filled in by the platform after processing the Tool but before actual + execution. Parameter references and expressions may only use the + literal string value of the field and must not perform computation on + the contents, except where noted otherwise. + +If the value of a field has no leading or trailing non-whitespace +characters around a parameter reference, the effective value of the field +becomes the value of the referenced parameter, preserving the return type. + +### String interpolation + +If the value of a field has non-whitespace leading or trailing characters +around a parameter reference, it is subject to string interpolation. The +effective value of the field is a string containing the leading characters, +followed by the string value of the parameter reference, followed by the +trailing characters. The string value of the parameter reference is its +textual JSON representation with the following rules: + + * Strings are replaced the literal text of the string, any escaped + characters replaced by the literal characters they represent, and + there are no leading or trailing quotes. + * Objects entries are sorted by key + +Multiple parameter references may appear in a single field. This case +must be treated as a string interpolation. After interpolating the first +parameter reference, interpolation must be recursively applied to the +trailing characters to yield the final string value. + +When text embedded in a CWL file represents code for another +programming language, the use of `$(...)` (and `${...}` in the case of +expressions) may conflict with the syntax of that language. For +example, when writing shell scripts, `$(...)` is used to execute a +command in a subshell and replace a portion of the command line with +the standard output of that command. + +The following escaping rules apply. The scanner makes a single pass +from start to end with 3-character lookahead. After performing a +replacement scanning resumes at the next character following the +replaced substring. + + 1. The substrings `\$(` and `\${` are replaced by `$(` and `${` + respectively. No parameter or expression evaluation + interpolation occurs. + 2. A double backslash `\\` is replaced by a single backslash `\`. + 3. A substring starting with a backslash that does not match one of + the previous rules is left unchanged. + +## Expressions (Optional) + +An expression is a fragment of [Javascript/ECMAScript +5.1](http://www.ecma-international.org/ecma-262/5.1/) code evaluated by the +workflow platform to affect the inputs, outputs, or +behavior of a process. In the generic execution sequence, expressions may +be evaluated during step 5 (process setup), step 6 (execute process), +and/or step 7 (capture output). Expressions are distinct from regular +processes in that they are intended to modify the behavior of the workflow +itself rather than perform the primary work of the workflow. + +Expressions in CWL are an optional feature and are not required to be +implemented by all consumers of CWL documents. They should be used sparingly, +when there is no other way to achieve the desired outcome. Excessive use of +expressions may be a signal that other refactoring of the tools or workflows +would benefit the author, runtime, and users of the CWL document in question. + +To declare the use of expressions, the document must include the process +requirement `InlineJavascriptRequirement`. Expressions may be used in any +field permitting the pseudo-type `Expression`, as specified by this +document. + +Expressions are denoted by the syntax `$(...)` or `${...}`. A code +fragment wrapped in the `$(...)` syntax must be evaluated as a +[ECMAScript expression](http://www.ecma-international.org/ecma-262/5.1/#sec-11). A +code fragment wrapped in the `${...}` syntax must be evaluated as a +[ECMAScript function body](http://www.ecma-international.org/ecma-262/5.1/#sec-13) +for an anonymous, zero-argument function. Expressions must return a valid JSON +data type: one of null, string, number, boolean, array, object. Other return +values must result in a `permanentFailure`. Implementations must permit any +syntactically valid Javascript and account for nesting of parenthesis or braces +and that strings that may contain parenthesis or braces when scanning for +expressions. + +The runtime must include any code defined in the ["expressionLib" field of +InlineJavascriptRequirement](#InlineJavascriptRequirement) prior to +executing the actual expression. + +Before executing the expression, the runtime must initialize as global +variables the fields of the parameter context described above. + +The effective value of the field after expression evaluation follows the +same rules as parameter references discussed above. Multiple expressions +may appear in a single field. + +Expressions must be evaluated in an isolated context (a "sandbox") which +permits no side effects to leak outside the context. Expressions also must +be evaluated in [Javascript strict mode](http://www.ecma-international.org/ecma-262/5.1/#sec-4.2.2). + +The order in which expressions are evaluated is undefined except where +otherwise noted in this document. + +An implementation may choose to implement parameter references by +evaluating as a Javascript expression. The results of evaluating +parameter references must be identical whether implemented by Javascript +evaluation or some other means. + +Implementations may apply other limits, such as process isolation, timeouts, +and operating system containers/jails to minimize the security risks associated +with running untrusted code embedded in a CWL document. + +Javascript exceptions thrown from a CWL expression must result in a +`permanentFailure` of the CWL process. + +## Executing CWL documents as scripts + +By convention, a CWL document may begin with `#!/usr/bin/env cwl-runner` +and be marked as executable (the POSIX "+x" permission bits) to enable it +to be executed directly. A workflow platform may support this mode of +operation; if so, it must provide `cwl-runner` as an alias for the +platform's CWL implementation. + +A CWL input object document may similarly begin with `#!/usr/bin/env +cwl-runner` and be marked as executable. In this case, the input object +must include the field `cwl:tool` supplying an IRI to the default CWL +document that should be executed using the fields of the input object as +input parameters. + +The `cwl-runner` interface is required for conformance testing and is +documented in [cwl-runner.cwl](cwl-runner.cwl). + +## Discovering CWL documents on a local filesystem + +To discover CWL documents look in the following locations: + +For each value in the `XDG_DATA_DIRS` environment variable (which is a `:` colon +separated list), check the `./commonwl` subdirectory. If `XDG_DATA_DIRS` is +unset or empty, then check using the default value for `XDG_DATA_DIRS`: +`/usr/local/share/:/usr/share/` (That is to say, check `/usr/share/commonwl/` +and `/usr/local/share/commonwl/`) + +Then check `$XDG_DATA_HOME/commonwl/`. + +If the `XDG_DATA_HOME` environment variable is unset, its default value is +`$HOME/.local/share` (That is to say, check `$HOME/.local/share/commonwl`) + +`$XDG_DATA_HOME` and `$XDG_DATA_DIRS` are from the [XDG Base Directory +Specification](http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html)