| Previous changeset 48:5a7a5b06bce0 (2020-08-23) Next changeset 50:bf432f4486c7 (2020-08-30) |
|
Commit message:
Can now make the bwa example from planemo :) |
|
modified:
toolfactory/README.md toolfactory/rgToolFactory2.py toolfactory/rgToolFactory2.xml |
|
added:
toolfactory/testtf.sh |
| b |
| diff -r 5a7a5b06bce0 -r 35a912ce0c83 toolfactory/README.md --- a/toolfactory/README.md Sun Aug 23 21:03:48 2020 -0400 +++ b/toolfactory/README.md Thu Aug 27 23:11:01 2020 -0400 |
| [ |
| b'@@ -1,137 +1,164 @@\n-\xef\xbb\xbf*WARNING before you start*\n+\xef\xbb\xbf**Breaking news! Docker container is recommended as at August 2020**\n+\n+A Docker container can be built - see the docker directory.\n+It is highly recommended for isolation. It also has an integrated toolshed to allow installation of new tools back \n+into the Galaxy being used to generate them. \n+\n+Built from quay.io/bgruening/galaxy:20.05 but updates the\n+Galaxy code to the dev branch - it seems to work fine with updated bioblend>=0.14\n+with planemo and the right version of gxformat2 needed by the ToolFactory (TF).\n \n- Install this tool on a private Galaxy ONLY\n- Please NEVER on a public or production instance\n- \n-Updated august 2014 by John Chilton adding citation support\n+The runclean.sh script run from the docker subdirectory of your local clone of this repository\n+should create a container (eventually) and serve it at localhost:8080 with a toolshed at\n+localhost:9009.\n \n-Updated august 8 2014 to fix bugs reported by Marius van den Beek\n+Once it\'s up, please restart Galaxy in the container with \n+```docker exec [container name] supervisorctl restart galaxy: ```\n+Jobs just do not seem to run properly otherwise and the next steps won\'t work!\n \n-Please cite the resource at\n-http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref\n-if you use this tool in your published work.\n+The generated container includes a workflow and 2 sample data sets for the workflow\n \n-**Short Story**\n+Load the workflow. Adjust the inputs for each as labelled. The perl example counts GC in phiX.fasta. \n+The python scripts use the rgToolFactory.py as their input - any text file will work but I like the\n+recursion. The BWA example has some mitochondrial reads and reference. Run the workflow and watch.\n+This should fill the history with some sample tools you can rerun and play with.\n+Note that each new tool will have been tested using Planemo. In the workflow, in Galaxy.\n+Extremely cool to watch.\n+\n+*WARNING* \n+\n+ Install this tool on a throw-away private Galaxy or Docker container ONLY\n+ Please NEVER on a public or production instance\n \n-This is an unusual Galaxy tool capable of generating new Galaxy tools.\n-It works by exposing *unrestricted* and therefore extremely dangerous scripting\n-to all designated administrators of the host Galaxy server, allowing them to\n-run scripts in R, python, sh and perl over multiple selected input data sets,\n-writing a single new data set as output.\n+*Short Story*\n+\n+Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as\n+a tool to Galaxy requires some special instructions to be written. This is sometimes termed "wrapping" the package\n+because the instructions tell Galaxy how to run the package as a new Galaxy tool. Any tool in a Galaxy is \n+readily available to all the users through a consistent and easy to use interface.\n \n-*You have a working r/python/perl/bash script or any executable with positional or argparse style parameters*\n+Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it \n+automates much of the basic boilerplate and makes the process much easier. The ToolFactory (TF) \n+uses Planemo under the hood for many functions, but hides the command\n+line complexities from the TF user. \n \n-It can be turned into an ordinary Galaxy tool in minutes, using a Galaxy tool.\n-\n+*More Explanation*\n \n-**Automated generation of new Galaxy tools for installation into any Galaxy**\n+The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools. \n+It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated\n+using instructions provided by the user and the results of Planemo lint and tool testing using\n+small sample inputs provided by the TF user. The small samples become tests built in to the new tool.\n+\n+It offers a familiar Galaxy form driven way to define how '..b'flexible enough to generate wrappers for many common scientific packages\n+but the inbuilt automation will not cope with all possible situations. Users can\n+supply overrides for two tool XML segments - tests and command and the BWA\n+example in the supplied samples workflow illustrates their use. \n \n-**Installation**\n-This is a Galaxy tool. You can install it most conveniently using the\n+*Installation*\n+\n+The Docker container is the best way to use the TF because it is preconfigured\n+to automate new tool testing and has a built in local toolshed where each new tool\n+is uploaded. It is easy to install without Docker, but you will need to make some \n+configuration changes (TODO write a configuration). You can install it most conveniently using the\n administrative "Search and browse tool sheds" link. Find the Galaxy Main\n toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory\n-repository. Open it and review the code and select the option to install it.\n-\n-If you can\'t get the tool that way, the xml and py files here need to be\n-copied into a new tools\n-subdirectory such as tools/toolfactory Your tool_conf.xml needs a new entry\n-pointing to the xml\n-file - something like::\n+repository in the Tool Maker section. Open it and review the code and select the option to install it.\n \n- <section name="Tool building tools" id="toolbuilders">\n- <tool file="toolfactory/rgToolFactory.xml"/>\n- </section>\n-\n-If not already there,\n+Otherwise, if not already there pending an accepted PR,\n please add:\n-<datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary"\n+<datatype extension="tgz" type="galaxy.datatypes.binary:Binary"\n mimetype="multipart/x-gzip" subclass="True" />\n to your local data_types_conf.xml.\n \n \n-**Restricted execution**\n+*Restricted execution*\n \n The tool factory tool itself will then be usable ONLY by admin users -\n-people with IDs in admin_users in universe_wsgi.ini **Yes, that\'s right. ONLY\n+people with IDs in admin_users. **Yes, that\'s right. ONLY\n admin_users can run this tool** Think about it for a moment. If allowed to\n run any arbitrary script on your Galaxy server, the only thing that would\n impede a miscreant bent on destroying all your Galaxy data would probably\n be lack of appropriate technical skills.\n \n-**What it does** \n-\n-This is a tool factory for simple scripts in python, R and\n-perl currently. Functional tests are automatically generated. How cool is that.\n-\n-LIMITED to simple scripts that read one input from the history. Optionally can\n-write one new history dataset, and optionally collect any number of outputs\n-into links on an autogenerated HTML index page for the user to navigate -\n-useful if the script writes images and output files - pdf outputs are shown\n-as thumbnails and R\'s bloated pdf\'s are shrunk with ghostscript so that and\n-imagemagik need to be available.\n-\n-Generated tools can be edited and enhanced like any Galaxy tool, so start\n-small and build up since a generated script gets you a serious leg up to a\n-more complex one.\n-\n-**What you do**\n-\n-You paste and run your script, you fix the syntax errors and\n-eventually it runs. You can use the redo button and edit the script before\n-trying to rerun it as you debug - it works pretty well.\n-\n-Once the script works on some test data, you can generate a toolshed compatible\n-gzip file containing your script ready to run as an ordinary Galaxy tool in\n-a repository on your local toolshed. That means safe and largely automated\n-installation in any production Galaxy configured to use your toolshed.\n-\n **Generated tool Security**\n \n Once you install a generated tool, it\'s just\n@@ -141,7 +168,7 @@\n \n **Send Code**\n \n-Patches and suggestions welcome as bitbucket issues please?\n+Pull requests and suggestions welcome as git issues please?\n \n **Attribution**\n \n@@ -160,7 +187,3 @@\n \n Licensed under the LGPL\n \n-**Obligatory screenshot**\n-\n-http://bitbucket.org/fubar/galaxytoolmaker/src/fda8032fe989/images/dynamicScriptTool.png\n-\n' |
| b |
| diff -r 5a7a5b06bce0 -r 35a912ce0c83 toolfactory/rgToolFactory2.py --- a/toolfactory/rgToolFactory2.py Sun Aug 23 21:03:48 2020 -0400 +++ b/toolfactory/rgToolFactory2.py Thu Aug 27 23:11:01 2020 -0400 |
| [ |
| b'@@ -6,7 +6,8 @@\n #\n # all rights reserved\n # Licensed under the LGPL\n-# suggestions for improvement and bug fixes welcome at https://github.com/fubar2/toolfactory\n+# suggestions for improvement and bug fixes welcome at\n+# https://github.com/fubar2/toolfactory\n #\n # July 2020: BCC was fun and I feel like rip van winkle after 5 years.\n # Decided to\n@@ -46,6 +47,10 @@\n debug = True\n toolFactoryURL = "https://github.com/fubar2/toolfactory"\n ourdelim = "~~~"\n+ALOT = 10000000 # srsly. command or test overrides use read() so just in case\n+STDIOXML = """<stdio>\n+<exit_code range="100:" level="debug" description="shite happens" />\n+</stdio>"""\n \n # --input_files="$input_files~~~$CL~~~$input_formats~~~$input_label\n # ~~~$input_help"\n@@ -56,11 +61,13 @@\n IHELPOS = 4\n IOCLPOS = 5\n \n-# --output_files "$otab.history_name~~~$otab.history_format~~~$otab.CL\n+# --output_files "$otab.history_name~~~$otab.history_format~~~$otab.CL~~~otab.history_test\n ONAMEPOS = 0\n OFMTPOS = 1\n OCLPOS = 2\n-OOCLPOS = 3\n+OTESTPOS = 3\n+OOCLPOS = 4\n+\n \n # --additional_parameters="$i.param_name~~~$i.param_value~~~\n # $i.param_label~~~$i.param_help~~~$i.param_type~~~$i.CL~~~i$.param_CLoverride"\n@@ -76,6 +83,10 @@\n \n foo = len(lxml.__version__)\n # fug you, flake8. Say my name!\n+FAKEEXE = "~~~REMOVE~~~ME~~~"\n+# need this until a PR/version bump to fix galaxyxml prepending the exe even\n+# with override.\n+\n \n def timenow():\n """return current time as a string\n@@ -120,9 +131,7 @@\n if citation.startswith("doi"):\n citation_tuples.append(("doi", citation[len("doi") :].strip()))\n else:\n- citation_tuples.append(\n- ("bibtex", citation[len("bibtex") :].strip())\n- )\n+ citation_tuples.append(("bibtex", citation[len("bibtex") :].strip()))\n return citation_tuples\n \n \n@@ -137,7 +146,6 @@\n prepare command line cl for running the tool here\n and prepare elements needed for galaxyxml tool generation\n """\n-\n self.infiles = [x.split(ourdelim) for x in args.input_files]\n self.outfiles = [x.split(ourdelim) for x in args.output_files]\n self.addpar = [x.split(ourdelim) for x in args.additional_parameters]\n@@ -148,28 +156,31 @@\n self.cl = []\n self.xmlcl = []\n self.is_positional = self.args.parampass == "positional"\n+ if self.args.packages:\n+ self.executeme = self.args.packages.split(",")[0].split(":")[0]\n+ else:\n+ self.executeme = self.args.sysexe\n+ assert (\n+ self.executeme is not None\n+ ), "No system or managed executable passed in. Cannot build"\n aCL = self.cl.append\n+ aXCL = self.xmlcl.append\n assert args.parampass in [\n "0",\n "argparse",\n "positional",\n- ], \'Parameter passing in args.parampass must be "0","positional" or "argparse"\'\n+ ], \'args.parampass must be "0","positional" or "argparse"\'\n self.tool_name = re.sub("[^a-zA-Z0-9_]+", "", args.tool_name)\n self.tool_id = self.tool_name\n- if self.args.interpreter_name:\n- exe = "$runMe"\n- else:\n- exe = self.args.exe_package\n- assert (\n- exe is not None\n- ), "No interpeter or executable passed in - nothing to run so cannot build"\n self.tool = gxt.Tool(\n self.args.tool_name,\n self.tool_id,\n self.args.tool_version,\n self.args.tool_desc,\n- exe,\n+ FAKEEXE, \n )\n+ if self.args.script_path:\n+ self.tool.interpreter = self.executeme\n self.tooloutdir = "tfout"\n self.repdir = "TF_run_report_tempdir"\n self.testdir = os.path.join(self.tooloutdir, "test-data")\n@@ -182,21 +193,42 @@\n self.tinputs = gxtp.Inputs()\n self.toutputs = gxtp.Outputs()\n self.testparam = []\n- if (\n- self.args.runmode == "Executable" or self.args.runmode == "s'..b'nly")\n a("--help_text", default=None)\n a("--tool_desc", default=None)\n a("--tool_version", default=None)\n a("--citations", default=None)\n+ a("--command_override", default=None)\n+ a("--test_override", default=None)\n a("--additional_parameters", action="append", default=[])\n a("--edit_additional_parameters", action="store_true", default=False)\n a("--parampass", default="positional")\n a("--tfout", default="./tfout")\n a("--new_tool", default="new_tool")\n a("--runmode", default=None)\n- a("--galaxy_url", default="http://localhost")\n- a("--galaxy_api_key", default=\'fakekey\')\n+ a("--galaxy_url", default="http://localhost:8080")\n+ a("--galaxy_api_key", default="fbdd3c2eecd191e88939fffc02eeeaf8")\n a("--toolshed_url", default="http://localhost:9009")\n- a("--toolshed_api_key", default=None)\n- a("--planemo_test", default="yes")\n+ a("--toolshed_api_key", default="d46e5ed0e242ed52c6e1f506b5d7f9f7")\n+ a("--galaxy_root", default="/home/ross/galaxy")\n \n args = parser.parse_args()\n assert not args.bad_user, (\n \'UNAUTHORISED: %s is NOT authorized to use this tool until Galaxy admin adds %s to "admin_users" in the Galaxy configuration file\'\n % (args.bad_user, args.bad_user)\n )\n+ assert args.tool_name, "## Tool Factory expects a tool name - eg --tool_name=DESeq"\n assert (\n- args.tool_name\n- ), "## Tool Factory expects a tool name - eg --tool_name=DESeq"\n- assert (\n- args.interpreter_name or args.exe_package\n+ args.sysexe or args.packages\n ), "## Tool Factory wrapper expects an interpreter or an executable package"\n- assert args.exe_package or (\n- len(args.script_path) > 0 and os.path.isfile(args.script_path)\n- ), "## Tool Factory wrapper expects a script path - eg --script_path=foo.R if no executable"\n- args.input_files = [\n- x.replace(\'"\', "").replace("\'", "") for x in args.input_files\n- ]\n+ args.input_files = [x.replace(\'"\', "").replace("\'", "") for x in args.input_files]\n # remove quotes we need to deal with spaces in CL params\n for i, x in enumerate(args.additional_parameters):\n- args.additional_parameters[i] = args.additional_parameters[i].replace(\n- \'"\', ""\n- )\n+ args.additional_parameters[i] = args.additional_parameters[i].replace(\'"\', "")\n r = ScriptRunner(args)\n- if args.make_Tool:\n- r.writeShedyml()\n- retcode = r.makeTooltar()\n- if retcode == 0:\n- if args.planemo_test == "yes":\n- r.install_load()\n+ r.writeShedyml()\n+ r.makeTool()\n+ if args.make_Tool == "runonly":\n+ retcode = r.run()\n+ if retcode:\n+ sys.stderr.write(\n+ "## Run failed with return code %d. Cannot build yet. Please fix and retry"\n+ % retcode\n+ )\n+ sys.exit(1)\n+ else:\n+ r.moveRunOutputs()\n+ elif args.make_Tool in ["gentestinstall", "generate", "gentest"]:\n+ retcode = r.run()\n+ if retcode:\n+ sys.stderr.write(\n+ "## Run failed with return code %d. Cannot build yet. Please fix and retry"\n+ % retcode\n+ )\n+ sys.exit(1)\n+ r.moveRunOutputs()\n+ r.makeToolTar()\n+ if args.make_Tool in ["gentestinstall","gentest"]:\n+ r.planemo_test(genoutputs=False)\n+ r.moveRunOutputs()\n+ r.planemo_shedload()\n+ r.eph_galaxy_load()\n else:\n- retcode = r.run()\n- # if retcode:\n+ retcode = r.planemo_test(genoutputs=True) # this fails :(\n+ r.moveRunOutputs()\n+ r.makeToolTar()\n+ retcode = r.planemo_test(genoutputs=False)\n+ r.moveRunOutputs()\n+ if args.make_Tool == "planemotestinstall":\n+ r.planemo_shedload() \n+ r.eph_galaxy_load()\n+ # if retcode:\n # sys.exit(retcode) # indicate failure to job runner\n \n \n' |
| b |
| diff -r 5a7a5b06bce0 -r 35a912ce0c83 toolfactory/rgToolFactory2.xml --- a/toolfactory/rgToolFactory2.xml Sun Aug 23 21:03:48 2020 -0400 +++ b/toolfactory/rgToolFactory2.xml Thu Aug 27 23:11:01 2020 -0400 |
| [ |
| b'@@ -1,9 +1,46 @@\n <tool id="rgTF2" name="toolfactory" version="2.00" profile="16.04" >\n <description>Scripts into tools</description>\n <macros>\n+ <xml name="planemo_steps">\n+ <param name="tool_version" label="Tool Version - bump this to warn users trying to redo old analyses" type="text" value="0.01"\n+ help="If you change your script and regenerate the \'same\' tool, you should inform Galaxy (and users) by changing (bumping is traditional) this number"/>\n+ <param name="tool_desc" label="Tool Description" type="text" value=""\n+ help="Supply a brief tool description for the Galaxy tool menu entry (optional - appears after the tool name)" />\n+ <param name="help_text" label="Tool form documentation and help text for users" type="text" area="true"\n+ value="**What it Does**"\n+ help="Supply user documentation to appear on the new tool form as reStructured text - http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html" >\n+ <sanitizer>\n+ <valid initial="string.printable">\n+ </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param>\n+ <repeat name="citations" title="Citation">\n+ <conditional name="citation_type">\n+ <param name="type" type="select" display="radio" label="Citation Type">\n+ <option value="doi">DOI</option>\n+ <option value="bibtex">BibTeX</option>\n+ </param>\n+ <when value="doi">\n+ <param name="doi" label="DOI" type="text" value=""\n+ help="Supply a DOI (e.g. doi: 10.1111/j.1740-9713.2007.00258.x) that should be cited when this tool is used in published research." />\n+ </when>\n+ <when value="bibtex">\n+ <param name="bibtex" label="BibTex" type="text" area="true"\n+ help="Supply a BibTex entry that should be cited when this tool is used in published research." value="" >\n+ <sanitizer>\n+ <valid initial="string.printable">\n+ </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param>\n+ </when>\n+ </conditional>\n+ </repeat>\n+ </xml>\n <xml name="io">\n <repeat name="history_inputs" title="Add a data file from your history to pass in to the script. Use the \'+\' button as needed"\n- help="USE SMALL SAMPLES because these will be used for the new tool\'s test. The names will become a history item selector as input for users of the new tool you are making">\n+ help="USE SMALL SAMPLES for the new tool\'s test. Prompts will form a history item selector as input for users of this new tool">\n <param name="input_files" type="data" format="data" label="Select an input file from your history" optional="true" multiple="false"\n help=""/>\n <param name="input_formats" type="select" multiple="true" label="Select the datatype(s) that your tool/script accepts as input"\n@@ -41,6 +78,8 @@\n </param>\n <param name="history_CL" type="text" label="Positional: ordinal integer. Argparse: argument name expected for this output parameter. Use STDOUT if \'>\' required."\n help="If positional parameters, enter the integer ordinal expected for this parameter. If Argparse style, \'--\' will be prepended or \'-\' if single character" value=""/>\n+ <param name="history_test" type="text" label="Test pass decision criterion for this output compared to test generation"\n+ help="Available options are diff:lines, sim_size" value="diff:2"/>\n </repeat>\n </xml>\n <xml n'..b' <option value="yes">DIY XML needed to replace autogenerated command and/or test segments will be pasted below</option>\n+ </param> \n+ <when value="no">\n+ <param name="command_override" type="hidden" value="" />\n+ <param name="test_override" type="hidden" value="" />\n </when>\n- <when value = "">\n+ <when value="yes">\n+ <param name="command_override" type="text" area="True" value="" label="Optional. Human wrought command element override XML/template - e.g. for bwa"\n+ help="For arbitrary and artfull command lines. All i/o and parameters must be passed. Leave blank unless needed. Not for the faint of heart">\n+ <sanitizer>\n+ <valid initial="string.printable">\n+ </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param>\n+ <param name="test_override" type="text" area="True" value="" label="Optional. Human wrought test element override XML/template - e.g. for bwa"\n+ help="For arbitrary and artfull scripts. Leave blank unless needed. Not for the faint of heart">\n+ <sanitizer>\n+ <valid initial="string.printable">\n+ </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param> \n </when>\n </conditional>\n+\n+ <conditional name="makeMode">\n+ <param name="make_Tool" type="select" display="radio" label="Choose the steps you want to run. Docker is recommended if you want local installation"\n+ help="Testing and installation in this Galaxy is optional. Planemo run must be used if you are providing XML overrides for the command or test sections" >\n+ <option value="runonly">Run this configuration. NOT suitable for Executable command/test overriden tools</option>\n+ <option value="generate" selected="true">Run and generate toolshed archive containing the new tool. NOT suitable for command/test overriden tools</option>\n+ <option value="gentest">Run, generate and test the new tool with planemo. NOT suitable for command/test overriden tools</option>\n+ <option value="gentestinstall">Run, generate, planemo test and install (via the local toolshed) the new tool. NOT suitable for command/test overriden tools</option>\n+ <option value="planemogenerate">Planemo test twice - once to generate test outputs. Suitable for tools with or without command/test overrides but takes time</option>\n+ <option value="planemogeninstall">Planemo test twice and install. Suitable for tools with or without command/test overrides.</option>\n+ </param>\n+ <when value="runonly"/>\n+ <when value="generate">\n+ <expand macro="planemo_steps" />\n+ </when>\n+ <when value="gentest">\n+ <expand macro="planemo_steps" />\n+ </when>\n+ <when value="gentestinstall">\n+ <expand macro="planemo_steps" />\n+ </when>\n+ <when value="planemogenerate">\n+ <expand macro="planemo_steps" />\n+ </when>\n+ <when value="planemogeninstall">\n+ <expand macro="planemo_steps" />\n+ </when>\n+ </conditional>\n+\n </inputs>\n <outputs>\n \n <data format="tgz" name="new_tool" label="${tool_name}_toolshed.tgz" >\n- <filter>makeMode[\'make_Tool\'] == "yes"</filter>\n+ <filter>makeMode[\'make_Tool\'] != "runonly"</filter>\n </data>\n \n <collection name="TF_run_report" type="list" label="${tool_name} outputs">\n@@ -351,7 +373,7 @@\n <param name="input_help" value="help" />\n <param name="tool_name" value="pyrevpos" />\n <param name="parampass" value="positional" />\n- <param name="make_Tool" value="yes" />\n+ <param name="make_Tool" value="generate" />\n <param name="tool_version" value="0.01" />\n <param name="tool_desc" value="positional reverse" />\n <param name="help_text" value="help text goes here" />\n' |
| b |
| diff -r 5a7a5b06bce0 -r 35a912ce0c83 toolfactory/testtf.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/toolfactory/testtf.sh Thu Aug 27 23:11:01 2020 -0400 |
| b |
| @@ -0,0 +1,2 @@ +planemo test --no_cleanup --no_dependency_resolution --skip_venv --galaxy_root ~/galaxy ~/galaxy/tools/tool_makers/toolfactory &>foo + |