Mercurial > repos > fubar > toolfactory_gtn
annotate toolfactory/README.md @ 4:2a46da701dde draft
Uploaded
author | fubar |
---|---|
date | Mon, 26 Apr 2021 05:25:26 +0000 |
parents | 43edf22e8cbc |
children | e2c8c2fa192d |
rev | line source |
---|---|
4 | 1 ## Breaking news! Completely refactored |
0
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
2 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
3 ### New demonstration of planemo tool_factory command ![Planemo ToolFactory demonstration](images/lintplanemo-2021-01-08_18.02.45.mkv?raw=false "Demonstration inside Planemo") |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
4 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
5 # WARNING |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
6 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
7 Install this tool to a throw-away private Galaxy or Docker container ONLY! |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
8 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
9 Please NEVER on a public or production instance where a hostile user may |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
10 be able to gain access if they can acquire an administrative account login. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
11 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
12 It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
13 it can install new tools to the Galaxy server it executes on! This is not something you should allow other than |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
14 on a throw away instance that is protected from potentially hostile users. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
15 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
16 ## Short Story |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
17 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
18 Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
19 a tool to Galaxy requires an XML document describing how the application interacts with Galaxy. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
20 This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
21 as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
22 and easy to use interface once installed in the local Galaxy server. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
23 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
24 Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
25 automates much of the boilerplate and makes the process much easier. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
26 The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
27 line complexities. The user will still need appropriate skills in terms of describing the interface between |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
28 Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
29 settings, together with automated testing and uploading to a toolshed with optional local installation. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
30 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
31 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
32 ## ToolFactory generated tools are ordinary Galaxy tools |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
33 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
34 A TF generated tool that passes the Planemo test is ready to publish in any Galaxy Toolshed and ready to install in any running Galaxy instance. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
35 They are fully workflow compatible and work exactly like any hand-written tool. The user can select input files of the specified type(s) from their |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
36 history and edit each of the specified parameters. The tool form will show all the labels and help text supplied when the tool was built. When the tool |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
37 is executed, the dependent binary or script will be passed all the i/o files and parameters as specified, and will write outputs to the specified new |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
38 history datasets - just like any other Galaxy tool. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
39 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
40 ## Models for tool command line construction |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
41 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
42 The key to turning any software package into a Galaxy tool is the automated construction of a suitable command line. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
43 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
44 The TF can build a new tool that will allow the tool user to select input files from their history, set any parameters and when run will send the |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
45 new output files to the history as specified when the tool builder completed the form and built the new tool. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
46 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
47 That tool can contain instructions to run any Conda dependency or a system executable like bash. Whether a bash script you have written or |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
48 a Conda package like bwa, the executable will expect to find settings for input, output and parameters on a command line. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
49 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
50 These are often passed as "--name value" (argparse style) or in a fixed order (positional style). |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
51 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
52 The ToolFactory allows either, or for "filter" applications that process input from STDIN and write processed output to STDOUT. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
53 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
54 The simplest tool model wraps a simple script or Conda dependency package requiring only input and output files, with no user supplied settings illustrated by |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
55 the Tacrev demonstration tool found in the Galaxy running in the ToolFactory docker container. It passes a user selected input file from the current history on STDIN |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
56 to a bash script. The bash script runs the unix tac utility (reverse cat) piped to the unix rev (reverse lines in a text file) utility. It's a one liner: |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
57 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
58 `tac | rev` |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
59 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
60 The tool building form allows zero or more Conda package name(s) and version(s) and an optional script to be executed by either a system |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
61 executable like ``bash`` or the first of any named Conda dependency package/version. Tacrev uses a tiny bash script shown above and uses the system |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
62 bash. Conda bash can be specified if it is important to use the same version consistently for the tool. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
63 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
64 On the tool form, the repeat section allowing zero or more input files was set to be a text file to be selected by the tool user and |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
65 in the repeat section allowing one or more outputs, a new output file with special value `STDOUT` as the positional parameter, causes the TF to |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
66 generate a command to capture STDOUT and send it to the new history file containing the reversed input text. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
67 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
68 By reversed, we mean really, truly reversed. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
69 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
70 That simple model can be made much more complicated, and can pass inputs and outputs as named or positional parameters, |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
71 to allow more complicated scripts or dependent binaries that require: |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
72 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
73 1. Any number of input data files selected by the user from existing history data |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
74 2. Any number of output data files written to the user's history |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
75 3. Any number of user supplied parameters. These can be passed as command line arguments to the script or the dependency package. Either |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
76 positional or named (argparse) style command line parameter passing can be used. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
77 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
78 More complex models can be seen in the Sedtest, Pyrevpos and Pyrevargparse tools illustrating positional and argparse parameter passing. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
79 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
80 The most complex demonstration is the Planemo advanced tool tutorial BWA tool. There is one version using a command-override to implement |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
81 exactly the same command structure in the Planemo tutorial. A second version uses a bash script and positional parameters to achieve the same |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
82 result. Some tool builders may find the bash version more familiar and cleaner but the choice is yours. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
83 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
84 ## Overview |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
85 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
86 ![IHello example ToolFactory tool form](files/hello_toolfactory_form.png?raw=true "Part of the Hello world example ToolFactory tool form") |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
87 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
88 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
89 Steps in building a new Galaxy tool are all conducted through Galaxy running in the docker container: |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
90 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
91 1. Login to the Galaxy running in the container at http://localhost:8080 using an admin account. They are specified in config/galaxy.yml and |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
92 in the documentation at |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
93 and the ToolFactory will error out and refuse to run for non-administrative tool builders as a minimal protection from opportunistic hostile use. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
94 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
95 2. Start the TF and fill in the form, providing sample inputs and parameter values to suit the Conda package being wrapped. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
96 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
97 3. Execute the tool to create a new XML tool wrapper using the sample inputs and parameter settings for the inbuilt tool test. Planemo runs twice. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
98 firstly to generate the test outputs and then to perform a proper test. The completed toolshed archive is written to the history |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
99 together with the planemo test report. Optionally the new tool archive can be uploaded |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
100 to the toolshed running in the same container (http://localhost:9009) and then installed inside the Galaxy in the container for further testing. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
101 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
102 4. If the test fails, rerun the failed history job and correct errors on the tool form before rerunning until everything works correctly. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
103 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
104 ![How it works](files/TFasIDE.png?raw=true "Overview of the ToolFactory as an Integrated Development Environment") |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
105 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
106 ## Planning and building new Galaxy tool wrappers. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
107 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
108 It is best to have all the required planning done to wrap any new script or binary before firing up the TF. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
109 Conda is the only current dependency manager supported. Before starting, at the very least, the tool builder will need |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
110 to know the required software package name in Conda and the version to use, how the command line for |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
111 the package must be constructed, and there must be sample inputs in the working history for each of the required data inputs |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
112 for the package, together with values for every parameter to suit these sample inputs. These are required on the TF form |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
113 for preparing the inbuilt tool test. That test is run using Planemo, as part of the tool generation process. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
114 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
115 A new tool is specified by filling in the usual Galaxy tool form. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
116 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
117 The form starts with a new tool name. Most tools will need dependency packages and versions |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
118 for the executable. Only Conda is currently supported. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
119 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
120 If a script is needed, it can be pasted into a text box and the interpreter named. Available system executables |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
121 can be used such as bash, or an interpreter such as python, perl or R can be nominated as conda dependencies |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
122 to ensure reproducible analyses. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
123 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
124 The tool form will be generated from the input data and the tool builder supplied parameters. The command line for the |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
125 executable is built using positional or argparse (named e.g. --input_file /foo/baz) style |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
126 parameters and is completely dependent on the executable. These can include: |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
127 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
128 1. Any number of input data sets needed by the executable. Each appears to the tool user on the run form and is included |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
129 on the command line for the executable. The tool builder must supply a small representative sample for each one as |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
130 an input for the automated tool test. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
131 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
132 2. Any number of output data sets generated by the package can be added to the command line and will appear in |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
133 the user's history at the end of the job |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
134 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
135 3. Any number of text or numeric parameters. Each will appear to the tool user on the run form and are included |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
136 on the command line to the executable. The tool builder must supply a suitable representative value for each one as |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
137 the value to be used for the automated tool test. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
138 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
139 Once the form is completed, executing the TF will build a new XML tool wrapper |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
140 including a functional test based on the sample settings and data. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
141 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
142 If the Planemo test passes, the tool can be optionally uploaded to the local Galaxy used in the image for more testing. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
143 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
144 A local toolshed runs inside the container to allow an automated installation, although any toolshed and any accessible |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
145 Galaxy can be specified for this process by editing the default URL and API keys to provide appropriate credentials. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
146 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
147 ## Generated Tool Dependency management |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
148 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
149 Conda is used for all dependency management although tools that use system utilities like sed, bash or awk |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
150 may be available on job execution nodes. Sed and friends are available as Conda (conda-forge) dependencies if necessary. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
151 Versioned Conda dependencies are always baked-in to the tool and will be used for reproducible calculation. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
152 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
153 ## Requirements |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
154 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
155 These are all managed automagically. The TF relies on galaxyxml to generate tool xml and uses ephemeris and |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
156 bioblend to load tools to the toolshed and to Galaxy. Planemo is used for testing and runs in a biocontainer currently at |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
157 https://quay.io/fubar2/planemo-biocontainer |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
158 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
159 ## Caveats |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
160 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
161 This docker image requires privileged mode so exposes potential security risks if hostile tool builders gain access. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
162 Please, do not run it in any situation where that is a problem - never, ever on a public facing Galaxy server. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
163 On a laptop or workstation should be fine in a non-hostile environment. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
164 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
165 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
166 ## Example generated XML |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
167 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
168 For the bwa-mem example, a supplied bash script is included as a configfile and so has escaped characters. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
169 ``` |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
170 <tool name="bwatest" id="bwatest" version="0.01"> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
171 <!--Cite: Creating re-usable tools from scripts doi:10.1093/bioinformatics/bts573--> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
172 <!--Source in git at: https://github.com/fubar2/toolfactory--> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
173 <!--Created by admin@galaxy.org at 30/11/2020 07:12:10 using the Galaxy Tool Factory.--> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
174 <description>Planemo advanced tool building sample bwa mem mapper as a ToolFactory demo</description> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
175 <requirements> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
176 <requirement version="0.7.15" type="package">bwa</requirement> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
177 <requirement version="1.3" type="package">samtools</requirement> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
178 </requirements> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
179 <configfiles> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
180 <configfile name="runme"><![CDATA[ |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
181 REFFILE=\$1 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
182 FASTQ=\$2 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
183 BAMOUT=\$3 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
184 rm -f "refalias" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
185 ln -s "\$REFFILE" "refalias" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
186 bwa index -a is "refalias" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
187 bwa mem -t "2" -v 1 "refalias" "\$FASTQ" > tempsam |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
188 samtools view -Sb tempsam > temporary_bam_file.bam |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
189 samtools sort -o "\$BAMOUT" temporary_bam_file.bam |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
190 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
191 ]]></configfile> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
192 </configfiles> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
193 <version_command/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
194 <command><![CDATA[bash |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
195 $runme |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
196 $input1 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
197 $input2 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
198 $bam_output]]></command> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
199 <inputs> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
200 <param optional="false" label="Reference sequence for bwa to map the fastq reads against" help="" format="fasta" multiple="false" type="data" name="input1" argument="input1"/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
201 <param optional="false" label="Reads as fastqsanger to align to the reference sequence" help="" format="fastqsanger" multiple="false" type="data" name="input2" argument="input2"/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
202 </inputs> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
203 <outputs> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
204 <data name="bam_output" format="bam" label="bam_output" hidden="false"/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
205 </outputs> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
206 <tests> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
207 <test> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
208 <output name="bam_output" value="bam_output_sample" compare="sim_size" format="bam" delta_frac="0.1"/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
209 <param name="input1" value="input1_sample"/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
210 <param name="input2" value="input2_sample"/> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
211 </test> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
212 </tests> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
213 <help><![CDATA[ |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
214 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
215 **What it Does** |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
216 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
217 Planemo advanced tool building sample bwa mem mapper |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
218 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
219 Reimagined as a bash script for a ToolFactory demonstration |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
220 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
221 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
222 ------ |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
223 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
224 Script:: |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
225 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
226 REFFILE=$1 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
227 FASTQ=$2 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
228 BAMOUT=$3 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
229 rm -f "refalias" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
230 ln -s "$REFFILE" "refalias" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
231 bwa index -a is "refalias" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
232 bwa mem -t "2" -v 1 "refalias" "$FASTQ" > tempsam |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
233 samtools view -Sb tempsam > temporary_bam_file.bam |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
234 samtools sort -o "$BAMOUT" temporary_bam_file.bam |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
235 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
236 ]]></help> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
237 </tool> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
238 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
239 ``` |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
240 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
241 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
242 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
243 ## More Explanation |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
244 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
245 The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
246 It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
247 using instructions provided by the user and the results of Planemo lint and tool testing using |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
248 small sample inputs provided by the TF user. The small samples become tests built in to the new tool. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
249 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
250 It offers a familiar Galaxy form driven way to define how the user of the new tool will |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
251 choose input data from their history, and what parameters the new tool user will be able to adjust. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
252 The TF user must know, or be able to read, enough about the tool to be able to define the details of |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
253 the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
254 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
255 Tools always depend on other things. Most tools in Galaxy depend on third party |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
256 scientific packages, so TF tools usually have one or more dependencies. These can be |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
257 scientific packages such as BWA or scripting languages such as Python and are |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
258 managed by Conda. If the new tool relies on a system utility such as bash or awk |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
259 where the importance of version control on reproducibility is low, these can be used without |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
260 Conda management - but remember the potential risks of unmanaged dependencies on computational |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
261 reproducibility. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
262 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
263 The TF user can optionally supply a working script where scripting is |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
264 required and the chosen dependency is a scripting language such as Python or a system |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
265 scripting executable such as bash. Whatever the language, the script must correctly parse the command line |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
266 arguments it receives at tool execution, as they are defined by the TF user. The |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
267 text of that script is "baked in" to the new tool and will be executed each time |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
268 the new tool is run. It is highly recommended that scripts and their command lines be developed |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
269 and tested until proven to work before the TF is invoked. Galaxy as a software development |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
270 environment is actually possible, but not recommended being somewhat clumsy and inefficient. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
271 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
272 Tools nearly always take one or more data sets from the user's history as input. TF tools |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
273 allow the TF user to define what Galaxy datatypes the tool end user will be able to choose and what |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
274 names or positions will be used to pass them on a command line to the package or script. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
275 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
276 Tools often have various parameter settings. The TF allows the TF user to define how each |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
277 parameter will appear on the tool form to the end user, and what names or positions will be |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
278 used to pass them on the command line to the package. At present, parameters are limited to |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
279 simple text and number fields. Pull requests for other kinds of parameters that galaxyxml |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
280 can handle are welcomed. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
281 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
282 Best practice Galaxy tools have one or more automated tests. These should use small sample data sets and |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
283 specific parameter settings so when the tool is tested, the outputs can be compared with their expected |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
284 values. The TF will automatically create a test for the new tool. It will use the sample data sets |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
285 chosen by the TF user when they built the new tool. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
286 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
287 The TF works by exposing *unrestricted* and therefore extremely dangerous scripting |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
288 to all designated administrators of the host Galaxy server, allowing them to |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
289 run scripts in R, python, sh and perl. For this reason, a Docker container is |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
290 available to help manage the associated risks. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
291 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
292 ## Scripting uses |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
293 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
294 To use a scripting language to create a new tool, you must first prepared and properly test a script. Use small sample |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
295 data sets for testing. When the script is working correctly, upload the small sample datasets |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
296 into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
297 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
298 ### Outputs |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
299 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
300 The TF will generate the new tool described on the TF form, and test it |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
301 using planemo. Optionally if a local toolshed is running, it can be used to |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
302 install the new tool back into the generating Galaxy. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
303 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
304 A toolshed is built in to the Docker container and configured |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
305 so a tool can be tested, sent to that toolshed, then installed in the Galaxy |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
306 where the TF is running using the default toolshed and Galaxy URL and API keys. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
307 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
308 Once it's in a ToolShed, it can be installed into any local Galaxy server |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
309 from the server administrative interface. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
310 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
311 Once the new tool is installed, local users can run it - each time, the |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
312 package and/or script that was supplied when it was built will be executed with the input chosen |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
313 from the user's history, together with user supplied parameters. In other words, the tools you generate with the |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
314 TF run just like any other Galaxy tool. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
315 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
316 TF generated tools work as normal workflow components. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
317 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
318 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
319 ## Limitations |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
320 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
321 The TF is flexible enough to generate wrappers for many common scientific packages |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
322 but the inbuilt automation will not cope with all possible situations. Users can |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
323 supply overrides for two tool XML segments - tests and command and the BWA |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
324 example in the supplied samples workflow illustrates their use. It does not deal with |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
325 repeated elements or conditional parameters such as allowing a user to choose to see "simple" |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
326 or "advanced" parameters (yet) and there will be plenty of packages it just |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
327 won't cover - but it's a quick and efficient tool for the other 90% of cases. Perfect for |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
328 that bash one liner you need to get that workflow functioning correctly for this |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
329 afternoon's demonstration! |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
330 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
331 ## Installation |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
332 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
333 The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
334 is the best way to use the TF because it is preconfigured |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
335 to automate new tool testing and has a built in local toolshed where each new tool |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
336 is uploaded. If you grab the docker container, it should just work after a restart and you |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
337 can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
338 jobs that generated them allows you to add fields and experiment to see how things work. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
339 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
340 It can be installed like any other tool from the Toolshed, but you will need to make some |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
341 configuration changes (TODO write a configuration). You can install it most conveniently using the |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
342 administrative "Search and browse tool sheds" link. Find the Galaxy Main |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
343 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
344 repository in the Tool Maker section. Open it and review the code and select the option to install it. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
345 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
346 If not already there please add: |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
347 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
348 ``` |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
349 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" /> |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
350 ``` |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
351 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
352 to your local config/data_types_conf.xml. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
353 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
354 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
355 ## Restricted execution |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
356 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
357 The tool factory tool itself will ONLY run for admin users - |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
358 people with IDs in config/galaxy.yml "admin_users". |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
359 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
360 *ONLY admin_users can run this tool* |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
361 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
362 That doesn't mean it's safe to install on a shared or exposed instance - please don't. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
363 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
364 ## Generated tool Security |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
365 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
366 Once you install a generated tool, it's just |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
367 another tool - assuming the script is safe. They just run normally and their |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
368 user cannot do anything unusually insecure but please, practice safe toolshed. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
369 Read the code before you install any tool. Especially this one - it is really scary. |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
370 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
371 ## Attribution |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
372 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
373 Creating re-usable tools from scripts: The Galaxy Tool Factory |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
374 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
375 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
376 |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
377 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref |
43edf22e8cbc
Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff
changeset
|
378 |