annotate toolfactory/README.md @ 44:23ad2924bf78 draft

Deleted selected files
author fubar
date Sun, 16 Aug 2020 08:51:49 -0400
parents e7e9732ebed6
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
43
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
1 Note as at August 8 2020
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
2
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
3
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
4 *WARNING before you start*
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
5
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
6 Install this tool on a private Galaxy ONLY
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
7 Please NEVER on a public or production instance
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
8
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
9 Please cite the resource at
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
10 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
11 if you use this tool in your published work.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
12
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
13 **Short Story**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
14
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
15 This is an unusual Galaxy tool capable of generating new Galaxy tools.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
16 It works by exposing *unrestricted* and therefore extremely dangerous scripting
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
17 to all designated administrators of the host Galaxy server, allowing them to
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
18 run scripts in R, python, sh and perl over multiple selected input data sets,
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
19 writing a single new data set as output.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
20
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
21 *You have a working r/python/perl/bash script or any executable with positional or argparse style parameters*
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
22
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
23 It can be turned into an ordinary Galaxy tool in minutes, using a Galaxy tool.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
24
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
25 **Automated generation of new Galaxy tools for installation into any Galaxy**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
26
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
27 A test is generated using small sample test data inputs and parameter settings you supply.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
28 Once the test case outputs have been produced, they can be used to build a
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
29 new Galaxy tool. The supplied script or executable is baked as a requirement
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
30 into a new, ordinary Galaxy tool, fully workflow compatible out of the box.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
31 Generated tools are installed via a tool shed by an administrator
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
32 and work exactly like all other Galaxy tools for your users.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
33
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
34 **More Detail**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
35
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
36 To use the ToolFactory, you should have prepared a script to paste into a
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
37 text box, or have a package in mind and a small test input example ready to select from your history
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
38 to test your new script.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
39
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
40 ```planemo test --no_cleanup --no_dependency_resolution --skip_venv --galaxy_root ~/galaxy ~/rossgit/toolfactory``` works for me
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
41
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
42 There is an example in each scripting language on the Tool Factory form. You
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
43 can just cut and paste these to try it out - remember to select the right
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
44 interpreter please. You'll also need to create a small test data set using
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
45 the Galaxy history add new data tool.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
46
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
47 If the script fails somehow, use the "redo" button on the tool output in
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
48 your history to recreate the form complete with broken script. Fix the bug
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
49 and execute again. Rinse, wash, repeat.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
50
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
51 Once the script runs sucessfully, a new Galaxy tool that runs your script
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
52 can be generated. Select the "generate" option and supply some help text and
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
53 names. The new tool will be generated in the form of a new Galaxy datatype
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
54 *tgz* - as the name suggests, it's an archive ready to upload to a
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
55 Galaxy ToolShed as a new tool repository.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
56
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
57
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
58 Once it's in a ToolShed, it can be installed into any local Galaxy server
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
59 from the server administrative interface.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
60
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
61 Once the new tool is installed, local users can run it - each time, the script
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
62 that was supplied when it was built will be executed with the input chosen
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
63 from the user's history. In other words, the tools you generate with the
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
64 ToolFactory run just like any other Galaxy tool,but run your script every time.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
65
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
66 Tool factory tools are perfect for workflow components. One input, one output,
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
67 no variables.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
68
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
69 *To fully and safely exploit the awesome power* of this tool,
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
70 Galaxy and the ToolShed, you should be a developer installing this
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
71 tool on a private/personal/scratch local instance where you are an
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
72 admin_user. Then, if you break it, you get to keep all the pieces see
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
73 https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
74
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
75 **Installation**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
76 This is a Galaxy tool. You can install it most conveniently using the
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
77 administrative "Search and browse tool sheds" link. Find the Galaxy Main
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
78 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
79 repository. Open it and review the code and select the option to install it.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
80
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
81 If you can't get the tool that way, the xml and py files here need to be
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
82 copied into a new tools
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
83 subdirectory such as tools/toolfactory Your tool_conf.xml needs a new entry
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
84 pointing to the xml
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
85 file - something like::
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
86
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
87 <section name="Tool building tools" id="toolbuilders">
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
88 <tool file="toolfactory/rgToolFactory.xml"/>
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
89 </section>
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
90
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
91 If not already there,
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
92 please add:
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
93 <datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary"
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
94 mimetype="multipart/x-gzip" subclass="True" />
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
95 to your local data_types_conf.xml.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
96
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
97
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
98 **Restricted execution**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
99
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
100 The tool factory tool itself will then be usable ONLY by admin users -
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
101 people with IDs in admin_users in universe_wsgi.ini **Yes, that's right. ONLY
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
102 admin_users can run this tool** Think about it for a moment. If allowed to
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
103 run any arbitrary script on your Galaxy server, the only thing that would
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
104 impede a miscreant bent on destroying all your Galaxy data would probably
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
105 be lack of appropriate technical skills.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
106
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
107 **What it does**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
108
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
109 This is a tool factory for simple scripts in python, R and
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
110 perl currently. Functional tests are automatically generated.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
111
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
112 LIMITED to simple scripts that read one input from the history. Optionally can
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
113 write one new history dataset, and optionally collect any number of outputs
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
114 into links on an autogenerated HTML index page for the user to navigate -
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
115 useful if the script writes images and output files - pdf outputs are shown
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
116 as thumbnails and R's bloated pdf's are shrunk with ghostscript so that and
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
117 imagemagik need to be available.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
118
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
119 Generated tools can be edited and enhanced like any Galaxy tool, so start
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
120 small and build up since a generated script gets you a serious leg up to a
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
121 more complex one.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
122
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
123 **What you do**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
124
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
125 You paste and run your script, you fix the syntax errors and
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
126 eventually it runs. You can use the redo button and edit the script before
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
127 trying to rerun it as you debug - it works pretty well.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
128
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
129 Once the script works on some test data, you can generate a toolshed compatible
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
130 gzip file containing your script ready to run as an ordinary Galaxy tool in
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
131 a repository on your local toolshed. That means safe and largely automated
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
132 installation in any production Galaxy configured to use your toolshed.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
133
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
134 **Generated tool Security**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
135
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
136 Once you install a generated tool, it's just
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
137 another tool - assuming the script is safe. They just run normally and their
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
138 user cannot do anything unusually insecure but please, practice safe toolshed.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
139 Read the code before you install any tool. Especially this one - it is really scary.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
140
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
141 **Send Code**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
142
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
143 Patches and suggestions welcome as bitbucket issues please?
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
144
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
145 **Attribution**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
146
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
147 Creating re-usable tools from scripts: The Galaxy Tool Factory
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
148 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
149 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
150
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
151 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
152
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
153 **Licensing**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
154
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
155 Copyright Ross Lazarus 2010
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
156 ross lazarus at g mail period com
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
157
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
158 All rights reserved.
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
159
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
160 Licensed under the LGPL
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
161
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
162 **Obligatory screenshot**
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
163
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
164 http://bitbucket.org/fubar/galaxytoolmaker/src/fda8032fe989/images/dynamicScriptTool.png
e7e9732ebed6 Uploaded
fubar
parents:
diff changeset
165