annotate toolfactory_docker/README.md @ 6:482386d6cc43 draft default tip

Uploaded
author fubar
date Sun, 24 Jan 2021 03:54:01 +0000
parents a5c5652823a6
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
1 # Breaking news! Docker container at https://github.com/fubar2/toolfactory-galaxy-docker recommended as at December 2020
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
2
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
3 ## This ToolFactory is for docker use and is used in the new recommended Docker ToolFactory
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
4
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
5 ## For non-Docker situations, use the ordinary ToolFactory https://github.com/fubar2/toolfactory
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
6
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
7
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
8 # WARNING
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
9
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
10 Install this tool to a throw-away private Galaxy or Docker container ONLY!
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
11
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
12 Please NEVER on a public or production instance where a hostile user may
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
13 be able to gain access if they can acquire an administrative account login.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
14
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
15 It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
16 it can install new tools to the Galaxy server it executes on! This is not something you should allow other than
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
17 on a throw away instance that is protected from potentially hostile users.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
18
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
19 ## Short Story
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
20
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
21 Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
22 a tool to Galaxy requires an XML document describing how the application interacts with Galaxy.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
23 This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
24 as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
25 and easy to use interface once installed in the local Galaxy server.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
26
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
27 Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
28 automates much of the boilerplate and makes the process much easier.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
29 The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
30 line complexities. The user will still need appropriate skills in terms of describing the interface between
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
31 Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
32 settings, together with automated testing and uploading to a toolshed with optional local installation.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
33
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
34 ## More Explanation
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
35
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
36 The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
37 It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
38 using instructions provided by the user and the results of Planemo lint and tool testing using
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
39 small sample inputs provided by the TF user. The small samples become tests built in to the new tool.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
40
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
41 It offers a familiar Galaxy form driven way to define how the user of the new tool will
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
42 choose input data from their history, and what parameters the new tool user will be able to adjust.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
43 The TF user must know, or be able to read, enough about the tool to be able to define the details of
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
44 the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
45
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
46 Tools always depend on other things. Most tools in Galaxy depend on third party
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
47 scientific packages, so TF tools usually have one or more dependencies. These can be
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
48 scientific packages such as BWA or scripting languages such as Python and are
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
49 managed by Conda. If the new tool relies on a system utility such as bash or awk
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
50 where the importance of version control on reproducibility is low, these can be used without
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
51 Conda management - but remember the potential risks of unmanaged dependencies on computational
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
52 reproducibility.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
53
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
54 The TF user can optionally supply a working script where scripting is
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
55 required and the chosen dependency is a scripting language such as Python or a system
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
56 scripting executable such as bash. Whatever the language, the script must correctly parse the command line
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
57 arguments it receives at tool execution, as they are defined by the TF user. The
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
58 text of that script is "baked in" to the new tool and will be executed each time
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
59 the new tool is run. It is highly recommended that scripts and their command lines be developed
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
60 and tested until proven to work before the TF is invoked. Galaxy as a software development
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
61 environment is actually possible, but not recommended being somewhat clumsy and inefficient.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
62
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
63 Tools nearly always take one or more data sets from the user's history as input. TF tools
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
64 allow the TF user to define what Galaxy datatypes the tool end user will be able to choose and what
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
65 names or positions will be used to pass them on a command line to the package or script.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
66
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
67 Tools often have various parameter settings. The TF allows the TF user to define how each
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
68 parameter will appear on the tool form to the end user, and what names or positions will be
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
69 used to pass them on the command line to the package. At present, parameters are limited to
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
70 simple text and number fields. Pull requests for other kinds of parameters that galaxyxml
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
71 can handle are welcomed.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
72
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
73 Best practice Galaxy tools have one or more automated tests. These should use small sample data sets and
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
74 specific parameter settings so when the tool is tested, the outputs can be compared with their expected
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
75 values. The TF will automatically create a test for the new tool. It will use the sample data sets
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
76 chosen by the TF user when they built the new tool.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
77
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
78 The TF works by exposing *unrestricted* and therefore extremely dangerous scripting
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
79 to all designated administrators of the host Galaxy server, allowing them to
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
80 run scripts in R, python, sh and perl. For this reason, a Docker container is
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
81 available to help manage the associated risks.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
82
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
83 ## Scripting uses
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
84
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
85 To use a scripting language to create a new tool, you must first prepared and properly test a script. Use small sample
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
86 data sets for testing. When the script is working correctly, upload the small sample datasets
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
87 into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
88
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
89 ### Outputs
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
90
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
91 The TF will generate the new tool described on the TF form, and test it
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
92 using planemo. Optionally if a local toolshed is running, it can be used to
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
93 install the new tool back into the generating Galaxy.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
94
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
95 A toolshed is built in to the Docker container and configured
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
96 so a tool can be tested, sent to that toolshed, then installed in the Galaxy
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
97 where the TF is running using the default toolshed and Galaxy URL and API keys.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
98
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
99 Once it's in a ToolShed, it can be installed into any local Galaxy server
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
100 from the server administrative interface.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
101
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
102 Once the new tool is installed, local users can run it - each time, the
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
103 package and/or script that was supplied when it was built will be executed with the input chosen
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
104 from the user's history, together with user supplied parameters. In other words, the tools you generate with the
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
105 TF run just like any other Galaxy tool.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
106
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
107 TF generated tools work as normal workflow components.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
108
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
109
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
110 ## Limitations
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
111
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
112 The TF is flexible enough to generate wrappers for many common scientific packages
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
113 but the inbuilt automation will not cope with all possible situations. Users can
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
114 supply overrides for two tool XML segments - tests and command and the BWA
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
115 example in the supplied samples workflow illustrates their use. It does not deal with
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
116 repeated elements or conditional parameters such as allowing a user to choose to see "simple"
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
117 or "advanced" parameters (yet) and there will be plenty of packages it just
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
118 won't cover - but it's a quick and efficient tool for the other 90% of cases. Perfect for
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
119 that bash one liner you need to get that workflow functioning correctly for this
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
120 afternoon's demonstration!
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
121
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
122 ## Installation
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
123
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
124 The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
125 is the best way to use the TF because it is preconfigured
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
126 to automate new tool testing and has a built in local toolshed where each new tool
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
127 is uploaded. If you grab the docker container, it should just work after a restart and you
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
128 can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
129 jobs that generated them allows you to add fields and experiment to see how things work.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
130
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
131 It can be installed like any other tool from the Toolshed, but you will need to make some
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
132 configuration changes (TODO write a configuration). You can install it most conveniently using the
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
133 administrative "Search and browse tool sheds" link. Find the Galaxy Main
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
134 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
135 repository in the Tool Maker section. Open it and review the code and select the option to install it.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
136
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
137 If not already there please add:
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
138
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
139 ```
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
140 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
141 ```
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
142
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
143 to your local config/data_types_conf.xml.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
144
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
145
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
146 ## Restricted execution
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
147
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
148 The tool factory tool itself will ONLY run for admin users -
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
149 people with IDs in config/galaxy.yml "admin_users".
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
150
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
151 *ONLY admin_users can run this tool*
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
152
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
153 That doesn't mean it's safe to install on a shared or exposed instance - please don't.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
154
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
155 ## Generated tool Security
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
156
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
157 Once you install a generated tool, it's just
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
158 another tool - assuming the script is safe. They just run normally and their
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
159 user cannot do anything unusually insecure but please, practice safe toolshed.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
160 Read the code before you install any tool. Especially this one - it is really scary.
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
161
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
162 ## Attribution
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
163
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
164 Creating re-usable tools from scripts: The Galaxy Tool Factory
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
165 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
166 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
167
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
168 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
a5c5652823a6 Uploaded
fubar
parents:
diff changeset
169