annotate toolfactory/README.md @ 40:51fa77152988 draft

Uploaded
author fubar
date Thu, 13 Aug 2020 02:55:26 -0400
parents c5290ea7bae0
children f8c1694190f0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
33
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
1 ´╗┐Note as at August 8 2020
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
2
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
3 Until a PR is accepted, galaxyxml requires a small tweak - you will need to install
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
4 ```
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
5 https://github.com/fubar2/galaxyxml
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
6 ```
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
7 instead into the virtualenv used to run galaxy and when using planemo, the
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
8
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
9 ```
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
10 --skip_venv
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
11 ```
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
12
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
13 flag must be used.
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
14
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
15
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
16 *WARNING before you start*
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
17
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
18 Install this tool on a private Galaxy ONLY
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
19 Please NEVER on a public or production instance
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
20
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
21 Updated august 2014 by John Chilton adding citation support
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
22
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
23 Updated august 8 2014 to fix bugs reported by Marius van den Beek
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
24
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
25 Please cite the resource at
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
26 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
27 if you use this tool in your published work.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
28
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
29 **Short Story**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
31 This is an unusual Galaxy tool capable of generating new Galaxy tools.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
32 It works by exposing *unrestricted* and therefore extremely dangerous scripting
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
33 to all designated administrators of the host Galaxy server, allowing them to
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
34 run scripts in R, python, sh and perl over multiple selected input data sets,
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
35 writing a single new data set as output.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
36
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
37 *You have a working r/python/perl/bash script or any executable with positional or argparse style parameters*
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
38
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
39 It can be turned into an ordinary Galaxy tool in minutes, using a Galaxy tool.
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
40
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
41
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
42 **Automated generation of new Galaxy tools for installation into any Galaxy**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
43
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
44 A test is generated using small sample test data inputs and parameter settings you supply.
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
45 Once the test case outputs have been produced, they can be used to build a
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
46 new Galaxy tool. The supplied script or executable is baked as a requirement
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
47 into a new, ordinary Galaxy tool, fully workflow compatible out of the box.
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
48 Generated tools are installed via a tool shed by an administrator
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
49 and work exactly like all other Galaxy tools for your users.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
50
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
51 **More Detail**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
52
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
53 To use the ToolFactory, you should have prepared a script to paste into a
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
54 text box, or have a package in mind and a small test input example ready to select from your history
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
55 to test your new script.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
56
33
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
57 ```planemo test --no_cleanup --no_dependency_resolution --skip_venv --galaxy_root ~/galaxy ~/rossgit/toolfactory``` works for me
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
58
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
59 There is an example in each scripting language on the Tool Factory form. You
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
60 can just cut and paste these to try it out - remember to select the right
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
61 interpreter please. You'll also need to create a small test data set using
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
62 the Galaxy history add new data tool.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
63
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
64 If the script fails somehow, use the "redo" button on the tool output in
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
65 your history to recreate the form complete with broken script. Fix the bug
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
66 and execute again. Rinse, wash, repeat.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
67
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
68 Once the script runs sucessfully, a new Galaxy tool that runs your script
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
69 can be generated. Select the "generate" option and supply some help text and
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
70 names. The new tool will be generated in the form of a new Galaxy datatype
33
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
71 *tgz* - as the name suggests, it's an archive ready to upload to a
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
72 Galaxy ToolShed as a new tool repository.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
73
33
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
74 Until tgz is added to the galaxy distribution datatypes you will need to add the following:
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
75 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" subclass="true" mimetype="multipart/x-gzip" display_in_upload="true" />
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
76
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
77 to::
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
78 [galaxyroot]/config/datatypes_conf.xml
c5290ea7bae0 Uploaded
fubar
parents: 32
diff changeset
79
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
80 Once it's in a ToolShed, it can be installed into any local Galaxy server
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
81 from the server administrative interface.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
82
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
83 Once the new tool is installed, local users can run it - each time, the script
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
84 that was supplied when it was built will be executed with the input chosen
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
85 from the user's history. In other words, the tools you generate with the
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
86 ToolFactory run just like any other Galaxy tool,but run your script every time.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
87
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
88 Tool factory tools are perfect for workflow components. One input, one output,
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
89 no variables.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
90
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
91 *To fully and safely exploit the awesome power* of this tool,
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
92 Galaxy and the ToolShed, you should be a developer installing this
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
93 tool on a private/personal/scratch local instance where you are an
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
94 admin_user. Then, if you break it, you get to keep all the pieces see
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
95 https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
96
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
97 **Installation**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
98 This is a Galaxy tool. You can install it most conveniently using the
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
99 administrative "Search and browse tool sheds" link. Find the Galaxy Main
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
100 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
101 repository. Open it and review the code and select the option to install it.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
102
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
103 If you can't get the tool that way, the xml and py files here need to be
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
104 copied into a new tools
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
105 subdirectory such as tools/toolfactory Your tool_conf.xml needs a new entry
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
106 pointing to the xml
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
107 file - something like::
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
108
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
109 <section name="Tool building tools" id="toolbuilders">
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
110 <tool file="toolfactory/rgToolFactory.xml"/>
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
111 </section>
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
112
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
113 If not already there,
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
114 please add:
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
115 <datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary"
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
116 mimetype="multipart/x-gzip" subclass="True" />
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
117 to your local data_types_conf.xml.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
118
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
119
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
120 **Restricted execution**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
121
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
122 The tool factory tool itself will then be usable ONLY by admin users -
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
123 people with IDs in admin_users in universe_wsgi.ini **Yes, that's right. ONLY
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
124 admin_users can run this tool** Think about it for a moment. If allowed to
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
125 run any arbitrary script on your Galaxy server, the only thing that would
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
126 impede a miscreant bent on destroying all your Galaxy data would probably
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
127 be lack of appropriate technical skills.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
128
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
129 **What it does**
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
130
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
131 This is a tool factory for simple scripts in python, R and
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
132 perl currently. Functional tests are automatically generated. How cool is that.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
133
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
134 LIMITED to simple scripts that read one input from the history. Optionally can
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
135 write one new history dataset, and optionally collect any number of outputs
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
136 into links on an autogenerated HTML index page for the user to navigate -
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
137 useful if the script writes images and output files - pdf outputs are shown
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
138 as thumbnails and R's bloated pdf's are shrunk with ghostscript so that and
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
139 imagemagik need to be available.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
140
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
141 Generated tools can be edited and enhanced like any Galaxy tool, so start
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
142 small and build up since a generated script gets you a serious leg up to a
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
143 more complex one.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
144
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
145 **What you do**
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
146
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
147 You paste and run your script, you fix the syntax errors and
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
148 eventually it runs. You can use the redo button and edit the script before
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
149 trying to rerun it as you debug - it works pretty well.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
150
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
151 Once the script works on some test data, you can generate a toolshed compatible
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
152 gzip file containing your script ready to run as an ordinary Galaxy tool in
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
153 a repository on your local toolshed. That means safe and largely automated
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
154 installation in any production Galaxy configured to use your toolshed.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
155
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
156 **Generated tool Security**
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
157
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
158 Once you install a generated tool, it's just
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
159 another tool - assuming the script is safe. They just run normally and their
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
160 user cannot do anything unusually insecure but please, practice safe toolshed.
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
161 Read the code before you install any tool. Especially this one - it is really scary.
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
162
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
163 **Send Code**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
164
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
165 Patches and suggestions welcome as bitbucket issues please?
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
166
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
167 **Attribution**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
168
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
169 Creating re-usable tools from scripts: The Galaxy Tool Factory
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
170 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
171 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
172
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
173 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
174
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
175 **Licensing**
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
176
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
177 Copyright Ross Lazarus 2010
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
178 ross lazarus at g mail period com
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
179
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
180 All rights reserved.
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
181
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
182 Licensed under the LGPL
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
183
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
184 **Obligatory screenshot**
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
185
32
4d578c8c1613 passes planemo test
fubar
parents: 30
diff changeset
186 http://bitbucket.org/fubar/galaxytoolmaker/src/fda8032fe989/images/dynamicScriptTool.png
30
6f48315c32c1 Uploaded
fubar
parents:
diff changeset
187