| Previous changeset 119:8ea1133b9d9a (2021-01-05) Next changeset 121:2050b2475ae5 (2021-01-07) |
|
Commit message:
Uploaded |
|
added:
home/ross/galaxy/tools/tool_makers/toolfactory/LICENSE home/ross/galaxy/tools/tool_makers/toolfactory/README.md home/ross/galaxy/tools/tool_makers/toolfactory/galaxy-tool-test home/ross/galaxy/tools/tool_makers/toolfactory/rgToolFactory2.py home/ross/galaxy/tools/tool_makers/toolfactory/rgToolFactory2.xml home/ross/galaxy/tools/tool_makers/toolfactory/test-data/input1_sample home/ross/galaxy/tools/tool_makers/toolfactory/test-data/output2_sample home/ross/galaxy/tools/tool_makers/toolfactory/test-data/pyrevpos.python home/ross/galaxy/tools/tool_makers/toolfactory/test-data/test1_log.txt home/ross/galaxy/tools/tool_makers/toolfactory/test-data/toolfactory_pyrevpos_tgz_sample home/ross/galaxy/tools/tool_makers/toolfactory/testtf.sh home/ross/galaxy/tools/tool_makers/toolfactory/whoosh.sh |
|
removed:
toolfactory/.planemo.yml toolfactory/.shed.yml toolfactory/LICENSE toolfactory/README.md toolfactory/images/TFasIDE.png toolfactory/images/dynamicScriptTool.png toolfactory/images/hello_toolfactory_form.png toolfactory/install_tf_demos.py toolfactory/install_tf_demos_toolshed.tgz toolfactory/planemo_install_tfdemo.tar.gz toolfactory/rgToolFactory2.py toolfactory/rgToolFactory2.xml toolfactory/test-data/input1_sample toolfactory/test-data/output2_sample toolfactory/test-data/pyrevpos.python toolfactory/test-data/test1_log.txt toolfactory/test-data/toolfactory_pyrevpos_tgz_sample toolfactory/whoosh.sh |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/LICENSE --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/LICENSE Thu Jan 07 09:22:51 2021 +0000 |
| b |
| b"@@ -0,0 +1,504 @@\n+GNU LESSER GENERAL PUBLIC LICENSE\n+ Version 2.1, February 1999\n+\n+ Copyright (C) 1991, 1999 Free Software Foundation, Inc.\n+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA\n+ Everyone is permitted to copy and distribute verbatim copies\n+ of this license document, but changing it is not allowed.\n+\n+(This is the first released version of the Lesser GPL. It also counts\n+ as the successor of the GNU Library Public License, version 2, hence\n+ the version number 2.1.)\n+\n+ Preamble\n+\n+ The licenses for most software are designed to take away your\n+freedom to share and change it. By contrast, the GNU General Public\n+Licenses are intended to guarantee your freedom to share and change\n+free software--to make sure the software is free for all its users.\n+\n+ This license, the Lesser General Public License, applies to some\n+specially designated software packages--typically libraries--of the\n+Free Software Foundation and other authors who decide to use it. You\n+can use it too, but we suggest you first think carefully about whether\n+this license or the ordinary General Public License is the better\n+strategy to use in any particular case, based on the explanations below.\n+\n+ When we speak of free software, we are referring to freedom of use,\n+not price. Our General Public Licenses are designed to make sure that\n+you have the freedom to distribute copies of free software (and charge\n+for this service if you wish); that you receive source code or can get\n+it if you want it; that you can change the software and use pieces of\n+it in new free programs; and that you are informed that you can do\n+these things.\n+\n+ To protect your rights, we need to make restrictions that forbid\n+distributors to deny you these rights or to ask you to surrender these\n+rights. These restrictions translate to certain responsibilities for\n+you if you distribute copies of the library or if you modify it.\n+\n+ For example, if you distribute copies of the library, whether gratis\n+or for a fee, you must give the recipients all the rights that we gave\n+you. You must make sure that they, too, receive or can get the source\n+code. If you link other code with the library, you must provide\n+complete object files to the recipients, so that they can relink them\n+with the library after making changes to the library and recompiling\n+it. And you must show them these terms so they know their rights.\n+\n+ We protect your rights with a two-step method: (1) we copyright the\n+library, and (2) we offer you this license, which gives you legal\n+permission to copy, distribute and/or modify the library.\n+\n+ To protect each distributor, we want to make it very clear that\n+there is no warranty for the free library. Also, if the library is\n+modified by someone else and passed on, the recipients should know\n+that what they have is not the original version, so that the original\n+author's reputation will not be affected by problems that might be\n+introduced by others.\n+\n+ Finally, software patents pose a constant threat to the existence of\n+any free program. We wish to make sure that a company cannot\n+effectively restrict the users of a free program by obtaining a\n+restrictive license from a patent holder. Therefore, we insist that\n+any patent license obtained for a version of the library must be\n+consistent with the full freedom of use specified in this license.\n+\n+ Most GNU software, including some libraries, is covered by the\n+ordinary GNU General Public License. This license, the GNU Lesser\n+General Public License, applies to certain designated libraries, and\n+is quite different from the ordinary General Public License. We use\n+this license for certain libraries in order to permit linking those\n+libraries into non-free programs.\n+\n+ When a program is linked with a library, whether statically or using\n+a shared library, the combination of the two is legally speaking a\n+combined work, a derivative "..b'oes not specify a\n+license version number, you may choose any version ever published by\n+the Free Software Foundation.\n+\n+ 14. If you wish to incorporate parts of the Library into other free\n+programs whose distribution conditions are incompatible with these,\n+write to the author to ask for permission. For software which is\n+copyrighted by the Free Software Foundation, write to the Free\n+Software Foundation; we sometimes make exceptions for this. Our\n+decision will be guided by the two goals of preserving the free status\n+of all derivatives of our free software and of promoting the sharing\n+and reuse of software generally.\n+\n+ NO WARRANTY\n+\n+ 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO\n+WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.\n+EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR\n+OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY\n+KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE\n+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE\n+LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME\n+THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n+\n+ 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN\n+WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY\n+AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU\n+FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR\n+CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE\n+LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING\n+RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A\n+FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF\n+SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH\n+DAMAGES.\n+\n+ END OF TERMS AND CONDITIONS\n+\n+ How to Apply These Terms to Your New Libraries\n+\n+ If you develop a new library, and you want it to be of the greatest\n+possible use to the public, we recommend making it free software that\n+everyone can redistribute and change. You can do so by permitting\n+redistribution under these terms (or, alternatively, under the terms of the\n+ordinary General Public License).\n+\n+ To apply these terms, attach the following notices to the library. It is\n+safest to attach them to the start of each source file to most effectively\n+convey the exclusion of warranty; and each file should have at least the\n+"copyright" line and a pointer to where the full notice is found.\n+\n+ {description}\n+ Copyright (C) {year} {fullname}\n+\n+ This library is free software; you can redistribute it and/or\n+ modify it under the terms of the GNU Lesser General Public\n+ License as published by the Free Software Foundation; either\n+ version 2.1 of the License, or (at your option) any later version.\n+\n+ This library is distributed in the hope that it will be useful,\n+ but WITHOUT ANY WARRANTY; without even the implied warranty of\n+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n+ Lesser General Public License for more details.\n+\n+ You should have received a copy of the GNU Lesser General Public\n+ License along with this library; if not, write to the Free Software\n+ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301\n+ USA\n+\n+Also add information on how to contact you by electronic and paper mail.\n+\n+You should also get your employer (if you work as a programmer) or your\n+school, if any, to sign a "copyright disclaimer" for the library, if\n+necessary. Here is a sample; alter the names:\n+\n+ Yoyodyne, Inc., hereby disclaims all copyright interest in the\n+ library `Frob\' (a library for tweaking knobs) written by James Random\n+ Hacker.\n+\n+ {signature of Ty Coon}, 1 April 1990\n+ Ty Coon, President of Vice\n+\n+That\'s all there is to it!\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/README.md --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/README.md Thu Jan 07 09:22:51 2021 +0000 |
| [ |
| b'@@ -0,0 +1,211 @@\n+\xef\xbb\xbf**Breaking news! Docker container is recommended as at August 2020**\n+\n+A Docker container can be built - see the docker directory.\n+It is highly recommended for isolation. It also has an integrated toolshed to allow installation of new tools back \n+into the Galaxy being used to generate them. \n+\n+Built from quay.io/bgruening/galaxy:20.05 but updates the\n+Galaxy code to the dev branch - it seems to work fine with updated bioblend>=0.14\n+with planemo and the right version of gxformat2 needed by the ToolFactory (TF).\n+\n+The runclean.sh script run from the docker subdirectory of your local clone of this repository\n+should create a container (eventually) and serve it at localhost:8080 with a toolshed at\n+localhost:9009.\n+\n+Once it\'s up, please restart Galaxy in the container with \n+```docker exec [container name] supervisorctl restart galaxy: ```\n+Jobs just do not seem to run properly otherwise and the next steps won\'t work!\n+\n+The generated container includes a workflow and 2 sample data sets for the workflow\n+\n+Load the workflow. Adjust the inputs for each as labelled. The perl example counts GC in phiX.fasta. \n+The python scripts use the rgToolFactory.py as their input - any text file will work but I like the\n+recursion. The BWA example has some mitochondrial reads and reference. Run the workflow and watch.\n+This should fill the history with some sample tools you can rerun and play with.\n+Note that each new tool will have been tested using Planemo. In the workflow, in Galaxy.\n+Extremely cool to watch.\n+\n+*WARNING* \n+\n+ Install this tool on a throw-away private Galaxy or Docker container ONLY\n+ Please NEVER on a public or production instance\n+\n+*Short Story*\n+\n+Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as\n+a tool to Galaxy requires some special instructions to be written. This is sometimes termed "wrapping" the package\n+because the instructions tell Galaxy how to run the package as a new Galaxy tool. Any tool in a Galaxy is \n+readily available to all the users through a consistent and easy to use interface.\n+\n+Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it \n+automates much of the basic boilerplate and makes the process much easier. The ToolFactory (TF) \n+uses Planemo under the hood for many functions, but hides the command\n+line complexities from the TF user. \n+\n+*More Explanation*\n+\n+The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools. \n+It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated\n+using instructions provided by the user and the results of Planemo lint and tool testing using\n+small sample inputs provided by the TF user. The small samples become tests built in to the new tool.\n+\n+It offers a familiar Galaxy form driven way to define how the user of the new tool will \n+choose input data from their history, and what parameters the new tool user will be able to adjust.\n+The TF user must know, or be able to read, enough about the tool to be able to define the details of\n+the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples.\n+\n+Tools always depend on other things. Most tools in Galaxy depend on third party\n+scientific packages, so TF tools usually have one or more dependencies. These can be\n+scientific packages such as BWA or scripting languages such as Python and are\n+usually managed by Conda. If the new tool relies on a system utility such as bash or awk \n+where the importance of version control on reproducibility is low, these can be used without \n+Conda management - but remember the potential risks of unmanaged dependencies on computational\n+reproducibility.\n+\n+The TF user can optionally supply a working script where scripting is\n+required and the chosen dependency is a scripting language such as Python or a system\n+scripting executable suc'..b'rs can run it - each time, the \n+package and/or script that was supplied when it was built will be executed with the input chosen\n+from the user\'s history, together with user supplied parameters. In other words, the tools you generate with the\n+ToolFactory run just like any other Galaxy tool.\n+\n+TF generated tools work as normal workflow components.\n+\n+\n+*Limitations*\n+\n+The TF is flexible enough to generate wrappers for many common scientific packages\n+but the inbuilt automation will not cope with all possible situations. Users can\n+supply overrides for two tool XML segments - tests and command and the BWA\n+example in the supplied samples workflow illustrates their use. \n+\n+*Installation*\n+\n+The Docker container is the best way to use the TF because it is preconfigured\n+to automate new tool testing and has a built in local toolshed where each new tool\n+is uploaded. If you grab the docker container, it should just work. \n+\n+If you build the container, there are some things to watch out for. Let it run for 10 minutes\n+or so once you build it - check with top until conda has finished fussing. Once everything quietens\n+down, find the container with\n+```docker ps```\n+and use\n+```docker exec [containername] supervisorctl restart galaxy:```\n+That colon is not a typographical mistake.\n+Not restarting after first boot seems to leave the job/worflow system confused and the workflow\n+just will not run properly until Galaxy has restarted.\n+\n+Login as admin@galaxy.org with password "password". Feel free to change it once you are logged in. \n+There should be a companion toolshed at localhost:9090. The history should have some sample data for\n+the workflow.\n+\n+Run the workflow and make sure the right dataset is selected for each of the input files. Most of the \n+examples use text files so should run, but the bwa example needs the right ones to work properly.\n+\n+When the workflow is finished, you will have half a dozen examples to rerun and play with. They have also \n+all been tested and installed so you should find them in your tool menu under "Generated Tools"\n+\n+It is easy to install without Docker, but you will need to make some \n+configuration changes (TODO write a configuration). You can install it most conveniently using the\n+administrative "Search and browse tool sheds" link. Find the Galaxy Main\n+toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory\n+repository in the Tool Maker section. Open it and review the code and select the option to install it.\n+\n+Otherwise, if not already there pending an accepted PR,\n+please add:\n+<datatype extension="tgz" type="galaxy.datatypes.binary:Binary"\n+mimetype="multipart/x-gzip" subclass="True" />\n+to your local data_types_conf.xml.\n+\n+\n+*Restricted execution*\n+\n+The tool factory tool itself will then be usable ONLY by admin users -\n+people with IDs in admin_users. **Yes, that\'s right. ONLY\n+admin_users can run this tool** Think about it for a moment. If allowed to\n+run any arbitrary script on your Galaxy server, the only thing that would\n+impede a miscreant bent on destroying all your Galaxy data would probably\n+be lack of appropriate technical skills.\n+\n+**Generated tool Security**\n+\n+Once you install a generated tool, it\'s just\n+another tool - assuming the script is safe. They just run normally and their\n+user cannot do anything unusually insecure but please, practice safe toolshed.\n+Read the code before you install any tool. Especially this one - it is really scary.\n+\n+**Send Code**\n+\n+Pull requests and suggestions welcome as git issues please?\n+\n+**Attribution**\n+\n+Creating re-usable tools from scripts: The Galaxy Tool Factory\n+Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team\n+Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573\n+\n+http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref\n+\n+**Licensing**\n+\n+Copyright Ross Lazarus 2010\n+ross lazarus at g mail period com\n+\n+All rights reserved.\n+\n+Licensed under the LGPL\n+\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/galaxy-tool-test --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/galaxy-tool-test Thu Jan 07 09:22:51 2021 +0000 |
| [ |
| b'@@ -0,0 +1,457 @@\n+#!/usr/bin/env python\n+\n+import argparse\n+import datetime as dt\n+import json\n+import logging\n+import os\n+import sys\n+import tempfile\n+from collections import namedtuple\n+from concurrent.futures import thread, ThreadPoolExecutor\n+\n+import yaml\n+\n+from galaxy.tool_util.verify.interactor import (\n+ DictClientTestConfig,\n+ GalaxyInteractorApi,\n+ verify_tool,\n+)\n+\n+DESCRIPTION = """Script to quickly run a tool test against a running Galaxy instance."""\n+DEFAULT_SUITE_NAME = "Galaxy Tool Tests"\n+ALL_TESTS = -1\n+ALL_TOOLS = "*"\n+ALL_VERSION = "*"\n+LATEST_VERSION = None\n+\n+\n+TestReference = namedtuple("TestReference", ["tool_id", "tool_version", "test_index"])\n+TestException = namedtuple("TestException", ["tool_id", "exception", "was_recorded"])\n+\n+\n+class Results:\n+\n+ def __init__(self, default_suitename, test_json, append=False):\n+ self.test_json = test_json or "-"\n+ test_results = []\n+ test_exceptions = []\n+ suitename = default_suitename\n+ if append:\n+ assert test_json != "-"\n+ with open(test_json) as f:\n+ previous_results = json.load(f)\n+ test_results = previous_results["tests"]\n+ if "suitename" in previous_results:\n+ suitename = previous_results["suitename"]\n+ self.test_results = test_results\n+ self.test_exceptions = test_exceptions\n+ self.suitename = suitename\n+\n+ def register_result(self, result):\n+ self.test_results.append(result)\n+\n+ def register_exception(self, test_exception):\n+ self.test_exceptions.append(test_exception)\n+\n+ def already_successful(self, test_reference):\n+ test_id = _test_id_for_reference(test_reference)\n+ for test_result in self.test_results:\n+ if test_result.get(\'id\') != test_id:\n+ continue\n+\n+ has_data = test_result.get(\'has_data\', False)\n+ if has_data:\n+ test_data = test_result.get("data", {})\n+ if \'status\' in test_data and test_data[\'status\'] == \'success\':\n+ return True\n+\n+ return False\n+\n+ def write(self):\n+ tests = sorted(self.test_results, key=lambda el: el[\'id\'])\n+ n_passed, n_failures, n_skips = 0, 0, 0\n+ n_errors = len([e for e in self.test_exceptions if not e.was_recorded])\n+ for test in tests:\n+ has_data = test.get(\'has_data\', False)\n+ if has_data:\n+ test_data = test.get("data", {})\n+ if \'status\' not in test_data:\n+ raise Exception(f"Test result data {test_data} doesn\'t contain a status key.")\n+ status = test_data[\'status\']\n+ if status == "success":\n+ n_passed += 1\n+ elif status == "error":\n+ n_errors += 1\n+ elif status == "skip":\n+ n_skips += 1\n+ elif status == "failure":\n+ n_failures += 1\n+ report_obj = {\n+ \'version\': \'0.1\',\n+ \'suitename\': self.suitename,\n+ \'results\': {\n+ \'total\': n_passed + n_failures + n_skips + n_errors,\n+ \'errors\': n_errors,\n+ \'failures\': n_failures,\n+ \'skips\': n_skips,\n+ },\n+ \'tests\': tests,\n+ }\n+ if self.test_json == "-":\n+ print(json.dumps(report_obj))\n+ else:\n+ with open(self.test_json, "w") as f:\n+ json.dump(report_obj, f)\n+\n+ def info_message(self):\n+ messages = []\n+ passed_tests = self._tests_with_status(\'success\')\n+ messages.append("Passed tool tests ({}): {}".format(\n+ len(passed_tests),\n+ [t["id"] for t in passed_tests]\n+ ))\n+ failed_tests = self._tests_with_status(\'failure\')\n+ messages.append("Failed tool tests ({}): {}".format(\n+ len(failed_tests),\n+'..b' logger.setLevel(logging.DEBUG if verbose else logging.INFO)\n+ logger.addHandler(console)\n+\n+ if not log_file:\n+ # delete = false is chosen here because it is always nice to have a log file\n+ # ready if you need to debug. Not having the "if only I had set a log file"\n+ # moment after the fact.\n+ temp = tempfile.NamedTemporaryFile(prefix="ephemeris_", delete=False)\n+ log_file = temp.name\n+ file_handler = logging.FileHandler(log_file)\n+ logger.addHandler(file_handler)\n+ logger.info(f"Storing log file in: {log_file}")\n+ return logger\n+\n+\n+def _arg_parser():\n+ parser = argparse.ArgumentParser(description=DESCRIPTION)\n+ parser.add_argument(\'-u\', \'--galaxy-url\', default="http://localhost:8080", help=\'Galaxy URL\')\n+ parser.add_argument(\'-k\', \'--key\', default=None, help=\'Galaxy User API Key\')\n+ parser.add_argument(\'-a\', \'--admin-key\', default=None, help=\'Galaxy Admin API Key\')\n+ parser.add_argument(\'--force_path_paste\', default=False, action="store_true", help=\'This requires Galaxy-side config option "allow_path_paste" enabled. Allows for fetching test data locally. Only for admins.\')\n+ parser.add_argument(\'-t\', \'--tool-id\', default=ALL_TOOLS, help=\'Tool ID\')\n+ parser.add_argument(\'--tool-version\', default=None, help=\'Tool Version (if tool id supplied). Defaults to just latest version, use * to test all versions\')\n+ parser.add_argument(\'-i\', \'--test-index\', default=ALL_TESTS, type=int, help=\'Tool Test Index (starting at 0) - by default all tests will run.\')\n+ parser.add_argument(\'-o\', \'--output\', default=None, help=\'directory to dump outputs to\')\n+ parser.add_argument(\'--append\', default=False, action="store_true", help="Extend a test record json (created with --output-json) with additional tests.")\n+ parser.add_argument(\'--skip-successful\', default=False, action="store_true", help="When used with --append, skip previously run successful tests.")\n+ parser.add_argument(\'-j\', \'--output-json\', default=None, help=\'output metadata json\')\n+ parser.add_argument(\'--verbose\', default=False, action="store_true", help="Verbose logging.")\n+ parser.add_argument(\'-c\', \'--client-test-config\', default=None, help="Test config YAML to help with client testing")\n+ parser.add_argument(\'--suite-name\', default=DEFAULT_SUITE_NAME, help="Suite name for tool test output")\n+ parser.add_argument(\'--with-reference-data\', dest="with_reference_data", default=False, action="store_true")\n+ parser.add_argument(\'--skip-with-reference-data\', dest="with_reference_data", action="store_false", help="Skip tests the Galaxy server believes use data tables or loc files.")\n+ parser.add_argument(\'--history-per-suite\', dest="history_per_test_case", default=False, action="store_false", help="Create new history per test suite (all tests in same history).")\n+ parser.add_argument(\'--history-per-test-case\', dest="history_per_test_case", action="store_true", help="Create new history per test case.")\n+ parser.add_argument(\'--no-history-cleanup\', default=False, action="store_true", help="Perserve histories created for testing.")\n+ parser.add_argument(\'--parallel-tests\', default=1, type=int, help="Parallel tests.")\n+ parser.add_argument(\'--retries\', default=0, type=int, help="Retry failed tests.")\n+ parser.add_argument(\'--page-size\', default=0, type=int, help="If positive, use pagination and just run one \'page\' to tool tests.")\n+ parser.add_argument(\'--page-number\', default=0, type=int, help="If page size is used, run this \'page\' of tests - starts with 0.")\n+ parser.add_argument(\'--download-attempts\', default=1, type=int, help="Galaxy may return a transient 500 status code for download if test results are written but not yet accessible.")\n+ parser.add_argument(\'--download-sleep\', default=1, type=int, help="If download attempts is greater than 1, the amount to sleep between download attempts.")\n+ return parser\n+\n+\n+if __name__ == "__main__":\n+ main()\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/rgToolFactory2.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/rgToolFactory2.py Thu Jan 07 09:22:51 2021 +0000 |
| [ |
| b'@@ -0,0 +1,1037 @@\n+# replace with shebang for biocontainer\n+# see https://github.com/fubar2/toolfactory\n+#\n+# copyright ross lazarus (ross stop lazarus at gmail stop com) May 2012\n+#\n+# all rights reserved\n+# Licensed under the LGPL\n+# suggestions for improvement and bug fixes welcome at\n+# https://github.com/fubar2/toolfactory\n+#\n+# July 2020: BCC was fun and I feel like rip van winkle after 5 years.\n+# Decided to\n+# 1. Fix the toolfactory so it works - done for simplest case\n+# 2. Fix planemo so the toolfactory function works\n+# 3. Rewrite bits using galaxyxml functions where that makes sense - done\n+\n+import argparse\n+import copy\n+import logging\n+import os\n+import re\n+import shutil\n+import subprocess\n+import sys\n+import tarfile\n+import tempfile\n+import time\n+\n+from bioblend import ConnectionError\n+from bioblend import toolshed\n+\n+import galaxyxml.tool as gxt\n+import galaxyxml.tool.parameters as gxtp\n+\n+import lxml\n+\n+import yaml\n+\n+myversion = "V2.1 July 2020"\n+verbose = True\n+debug = True\n+toolFactoryURL = "https://github.com/fubar2/toolfactory"\n+ourdelim = "~~~"\n+\n+# --input_files="$intab.input_files~~~$intab.input_CL~~~\n+# $intab.input_formats# ~~~$intab.input_label\n+# ~~~$intab.input_help"\n+IPATHPOS = 0\n+ICLPOS = 1\n+IFMTPOS = 2\n+ILABPOS = 3\n+IHELPOS = 4\n+IOCLPOS = 5\n+\n+# --output_files "$otab.history_name~~~$otab.history_format~~~\n+# $otab.history_CL~~~$otab.history_test"\n+ONAMEPOS = 0\n+OFMTPOS = 1\n+OCLPOS = 2\n+OTESTPOS = 3\n+OOCLPOS = 4\n+\n+\n+# --additional_parameters="$i.param_name~~~$i.param_value~~~\n+# $i.param_label~~~$i.param_help~~~$i.param_type\n+# ~~~$i.CL~~~i$.param_CLoverride"\n+ANAMEPOS = 0\n+AVALPOS = 1\n+ALABPOS = 2\n+AHELPPOS = 3\n+ATYPEPOS = 4\n+ACLPOS = 5\n+AOVERPOS = 6\n+AOCLPOS = 7\n+\n+\n+foo = len(lxml.__version__)\n+# fug you, flake8. Say my name!\n+FAKEEXE = "~~~REMOVE~~~ME~~~"\n+# need this until a PR/version bump to fix galaxyxml prepending the exe even\n+# with override.\n+\n+\n+def timenow():\n+ """return current time as a string"""\n+ return time.strftime("%d/%m/%Y %H:%M:%S", time.localtime(time.time()))\n+\n+\n+def quote_non_numeric(s):\n+ """return a prequoted string for non-numerics\n+ useful for perl and Rscript parameter passing?\n+ """\n+ try:\n+ _ = float(s)\n+ return s\n+ except ValueError:\n+ return \'"%s"\' % s\n+\n+\n+html_escape_table = {\n+ "&": "&",\n+ ">": ">",\n+ "<": "<",\n+ "#": "#",\n+ "$": "$",\n+}\n+cheetah_escape_table = {"$": "\\\\$", "#": "\\\\#"}\n+\n+\n+def html_escape(text):\n+ """Produce entities within text."""\n+ return "".join([html_escape_table.get(c, c) for c in text])\n+\n+\n+def cheetah_escape(text):\n+ """Produce entities within text."""\n+ return "".join([cheetah_escape_table.get(c, c) for c in text])\n+\n+\n+def parse_citations(citations_text):\n+ """"""\n+ citations = [c for c in citations_text.split("**ENTRY**") if c.strip()]\n+ citation_tuples = []\n+ for citation in citations:\n+ if citation.startswith("doi"):\n+ citation_tuples.append(("doi", citation[len("doi") :].strip()))\n+ else:\n+ citation_tuples.append(("bibtex", citation[len("bibtex") :].strip()))\n+ return citation_tuples\n+\n+\n+class ScriptRunner:\n+ """Wrapper for an arbitrary script\n+ uses galaxyxml\n+\n+ """\n+\n+ def __init__(self, args=None):\n+ """\n+ prepare command line cl for running the tool here\n+ and prepare elements needed for galaxyxml tool generation\n+ """\n+ self.ourcwd = os.getcwd()\n+ self.ourenv = copy.deepcopy(os.environ)\n+ self.infiles = [x.split(ourdelim) for x in args.input_files]\n+ self.outfiles = [x.split(ourdelim) for x in args.output_files]\n+ self.addpar = [x.split(ourdelim) for x in args.additional_parameters]\n+ self.args = args\n+ self.cleanuppar()\n+ self.lastclredirect = None\n+ self.lastxclredirect = None\n+ self.cl = []\n+ self.xmlcl = []\n+ self.is_positional = self.args.parampass == "'..b'ath(xreal),\n+ ]\n+ p = subprocess.run(\n+ cll,\n+ env=self.ourenv,\n+ shell=False,\n+ cwd=self.tooloutdir,\n+ stderr=dummy,\n+ stdout=dummy,\n+ )\n+\n+ else:\n+ cll = [\n+ "planemo",\n+ "test",\n+ "--test_data",\n+ os.path.abspath(self.testdir),\n+ "--test_output",\n+ os.path.abspath(tool_test_path),\n+ "--skip_venv",\n+ "--galaxy_root",\n+ self.args.galaxy_root,\n+ os.path.abspath(xreal),\n+ ]\n+ p = subprocess.run(\n+ cll,\n+ shell=False,\n+ env=self.ourenv,\n+ cwd=self.tooloutdir,\n+ stderr=tout,\n+ stdout=tout,\n+ )\n+ tout.close()\n+ return p.returncode\n+\n+\n+def main():\n+ """\n+ This is a Galaxy wrapper.\n+ It expects to be called by a special purpose tool.xml\n+\n+ """\n+ parser = argparse.ArgumentParser()\n+ a = parser.add_argument\n+ a("--script_path", default=None)\n+ a("--history_test", default=None)\n+ a("--cl_prefix", default=None)\n+ a("--sysexe", default=None)\n+ a("--packages", default=None)\n+ a("--tool_name", default="newtool")\n+ a("--tool_dir", default=None)\n+ a("--input_files", default=[], action="append")\n+ a("--output_files", default=[], action="append")\n+ a("--user_email", default="Unknown")\n+ a("--bad_user", default=None)\n+ a("--make_Tool", default="runonly")\n+ a("--help_text", default=None)\n+ a("--tool_desc", default=None)\n+ a("--tool_version", default=None)\n+ a("--citations", default=None)\n+ a("--command_override", default=None)\n+ a("--test_override", default=None)\n+ a("--additional_parameters", action="append", default=[])\n+ a("--edit_additional_parameters", action="store_true", default=False)\n+ a("--parampass", default="positional")\n+ a("--tfout", default="./tfout")\n+ a("--new_tool", default="new_tool")\n+ a("--galaxy_url", default="http://localhost:8080")\n+ a("--toolshed_url", default="http://localhost:9009")\n+ # make sure this is identical to tool_sheds_conf.xml\n+ # localhost != 127.0.0.1 so validation fails\n+ a("--toolshed_api_key", default="fakekey")\n+ a("--galaxy_api_key", default="fakekey")\n+ a("--galaxy_root", default="/galaxy-central")\n+ a("--galaxy_venv", default="/galaxy_venv")\n+ args = parser.parse_args()\n+ assert not args.bad_user, (\n+ \'UNAUTHORISED: %s is NOT authorized to use this tool until Galaxy \\\n+admin adds %s to "admin_users" in the galaxy.yml Galaxy configuration file\'\n+ % (args.bad_user, args.bad_user)\n+ )\n+ assert args.tool_name, "## Tool Factory expects a tool name - eg --tool_name=DESeq"\n+ assert (\n+ args.sysexe or args.packages\n+ ), "## Tool Factory wrapper expects an interpreter \\\n+or an executable package in --sysexe or --packages"\n+ args.input_files = [x.replace(\'"\', "").replace("\'", "") for x in args.input_files]\n+ # remove quotes we need to deal with spaces in CL params\n+ for i, x in enumerate(args.additional_parameters):\n+ args.additional_parameters[i] = args.additional_parameters[i].replace(\'"\', "")\n+ r = ScriptRunner(args)\n+ r.writeShedyml()\n+ r.makeTool()\n+ if args.make_Tool == "generate":\n+ retcode = r.run()\n+ r.moveRunOutputs()\n+ r.makeToolTar()\n+ else:\n+ retcode = r.planemo_test(genoutputs=True) # this fails :( - see PR\n+ r.moveRunOutputs()\n+ r.makeToolTar()\n+ retcode = r.planemo_test(genoutputs=False)\n+ r.moveRunOutputs()\n+ r.makeToolTar()\n+ print(f"second planemo_test returned {retcode}")\n+ if args.make_Tool == "gentestinstall":\n+ r.shedLoad()\n+ r.eph_galaxy_load()\n+\n+\n+if __name__ == "__main__":\n+ main()\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/rgToolFactory2.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/rgToolFactory2.xml Thu Jan 07 09:22:51 2021 +0000 |
| [ |
| b'@@ -0,0 +1,536 @@\n+<tool id="rgtf2" name="toolfactory" version="2.00" profile="16.04" >\n+ <description>Scripts into tools v2.0</description>\n+ <macros>\n+ <xml name="tool_metadata">\n+ <param name="tool_version" label="Tool Version - bump this to warn users trying to redo old analyses" type="text" value="0.01"\n+ help="If you change your script and regenerate the \'same\' tool, you should inform Galaxy (and users) by changing (bumping is traditional) this number"/>\n+ <param name="tool_desc" label="Tool Synopsis" type="text" value=""\n+ help="Supply a brief tool description for the Galaxy tool menu entry" />\n+ <param name="help_text" label="Tool form documentation and help text for users" type="text" area="true"\n+ value="**What it Does**"\n+ help="Supply user documentation to appear on the new tool form as reStructured text - http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html" >\n+ <sanitizer>\n+ <valid initial="string.printable">\n+ </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param>\n+ <repeat name="citations" title="Citation">\n+ <conditional name="citation_type">\n+ <param name="type" type="select" display="radio" label="Citation Type">\n+ <option value="doi">DOI</option>\n+ <option value="bibtex">BibTeX</option>\n+ </param>\n+ <when value="doi">\n+ <param name="doi" label="DOI" type="text" value=""\n+ help="Supply a DOI (e.g. doi: 10.1111/j.1740-9713.2007.00258.x) to be cited when this tool is used in published research." />\n+ </when>\n+ <when value="bibtex">\n+ <param name="bibtex" label="BibTex" type="text" area="true"\n+ help="Supply a BibTex entry that should be cited when this tool is used in published research." value="" >\n+ <sanitizer>\n+ <valid initial="string.printable">\n+ </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param>\n+ </when>\n+ </conditional>\n+ </repeat>\n+ </xml>\n+ <xml name="io">\n+ <section name="io" title="Input and output files" expanded="true">\n+ <repeat name="history_inputs" title="zero or more input data files to pass as parameters to the executable."\n+ help="USE SMALL SAMPLES for the new tool\'s test. Prompts will form a history item selector as input for users of this new tool">\n+ <param name="input_files" type="data" format="data" label="Select an input file from your history" optional="true" multiple="false"\n+ help=""/>\n+ <param name="input_formats" type="select" multiple="true" label="Select the datatype(s) that your tool/script accepts as input"\n+ help="If your datatype is not listed here, it has to be added in galaxy\'s datatypes_conf.xml" value="tabular">\n+ <options from_parameter="tool.app.datatypes_registry.upload_file_formats">\n+ <column name="value" index="0"/>\n+ </options>\n+ </param>\n+ <param name="input_label" type="text" value="" label="This will become the user prompt for the form so please make it informative"\n+ help="Note that \'~~~\' is an internal delimiter so must not appear in this text field - please work around this technical limitation" >\n+ <sanitizer invalid_char="">\n+ <valid initial="string.printable"> <remove value=\'~~~\'/> </valid>\n+ <mapping initial="none"/>\n+ </sanitizer>\n+ </param>\n+ <param'..b'value="import sys; inp = sys.argv[1]; outp = sys.argv[2]; inlist = open(inp,\'r\').readlines(); o = open(outp,\'w\'); rs = [\'\'.join(list(reversed(x.rstrip()))) for x in inlist]; o.write(\'\\n\'.join(rs)); o.close()"/>\n+ <param name="choosescript" value="yes" />\n+ <param name="script_path" value="$runme"/>\n+ <output name="new_tool" file="toolfactory_pyrevpos_tgz_sample" compare="sim_size" delta="6000" />\n+</test>\n+</tests>\n+<help>\n+\n+.. class:: warningmark\n+\n+**Details and attribution**\n+(see GTF_)\n+\n+**Local Admins ONLY**\n+Only users whose IDs found in the local admin_user configuration setting in universe_wsgi.ini can run this tool.\n+\n+**If you find a bug**\n+Please raise an issue, or even better, submit a pull request fixing it, on the github repository GTF_\n+\n+**What it does**\n+This tool optionally generates normal workflow compatible first class Galaxy tools\n+\n+Generated tools can run existing binary packages that become requirements, existing scripts, or new scripts pasted into this tool form.\n+Pasted scripts are written so they are part of the new tool and cannot be adjusted by the downstream user.\n+Binary packages are managed by the dependency subsystem - conda usually, so anything in bioconda or conda_forge is available for example.\n+\n+Any number of parameters can be built into the new tool form for passing in to the script or executable at runtime.\n+These can be editable by the downstream user or baked in.\n+\n+When you run this tool, your executable or script and supplied parameter values will be run to produce a canonical\n+set of outputs - these are used to construct a test for the new tool.\n+\n+If tool generation is required, a new tarball compatible with any Galaxy toolshed is created.\n+It can be unpacked in your galaxy/tools directory and manually added to tool_conf.xml, or\n+installed into any toolshed from where it can be installed into your Galaxy.\n+\n+\n+.. class:: warningmark\n+\n+**Note to system administrators**\n+This tool offers *NO* built in protection against malicious scripts. It should only be installed on private/personnal Galaxy instances.\n+Admin_users will have the power to do anything they want as the Galaxy user if you install this tool.\n+\n+.. class:: warningmark\n+\n+**Use on public servers** is STRONGLY discouraged for obvious reasons\n+\n+The tools generated by this tool will run just as securely as any other normal installed Galaxy tool but like any other new tools, should always be checked carefully before installation.\n+We recommend that you follow the good code hygiene practices associated with safe toolshed practices.\n+\n+Here\'s a sample python script that can be cut and pasted into the tool form, suitable for positional parameter passing:\n+\n+::\n+\n+ # reverse order of text by row\n+ import sys\n+ inp = sys.argv[1]\n+ outp = sys.argv[2]\n+ i = open(inp,\'r\').readlines()\n+ o = open(outp,\'w\')\n+ for row in i:\n+ rs = row.rstrip()\n+ rs = list(rs)\n+ rs.reverse()\n+ o.write(\'\'.join(rs))\n+ o.write(\'\\n\')\n+ o.close()\n+\n+With argparse style parameters:\n+\n+::\n+\n+ # reverse order of text by row\n+ import argparse\n+ parser = argparse.ArgumentParser()\n+ a = parser.add_argument\n+ a(\'--infile\',default=\'\')\n+ a(\'--outfile\',default=None)\n+ args = parser.parse_args()\n+ inp = args.infile\n+ outp = args.outfile\n+ i = open(inp,\'r\').readlines()\n+ o = open(outp,\'w\')\n+ for row in i:\n+ rs = row.rstrip()\n+ rs = list(rs)\n+ rs.reverse()\n+ o.write(\'\'.join(rs))\n+ o.write(\'\\n\')\n+ o.close()\n+\n+\n+\n+Paper_\n+\n+*Licensing*\n+\n+Copyright Ross Lazarus (ross period lazarus at gmail period com) May 2012\n+All rights reserved.\n+Licensed under the LGPL_\n+\n+.. _LGPL: http://www.gnu.org/copyleft/lesser.html\n+.. _GTF: https://github.com/fubar2/toolfactory\n+.. _Paper: http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573\n+\n+\n+</help>\n+<citations>\n+ <citation type="doi">10.1093/bioinformatics/bts573</citation>\n+</citations>\n+</tool>\n+\n+\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/test-data/input1_sample --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/test-data/input1_sample Thu Jan 07 09:22:51 2021 +0000 |
| b |
| @@ -0,0 +1,166 @@ +*WARNING before you start* + + Install this tool on a private Galaxy ONLY + Please NEVER on a public or production instance + +Updated august 2014 by John Chilton adding citation support + +Updated august 8 2014 to fix bugs reported by Marius van den Beek + +Please cite the resource at +http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref +if you use this tool in your published work. + +**Short Story** + +This is an unusual Galaxy tool capable of generating new Galaxy tools. +It works by exposing *unrestricted* and therefore extremely dangerous scripting +to all designated administrators of the host Galaxy server, allowing them to +run scripts in R, python, sh and perl over multiple selected input data sets, +writing a single new data set as output. + +*You have a working r/python/perl/bash script or any executable with positional or argparse style parameters* + +It can be turned into an ordinary Galaxy tool in minutes, using a Galaxy tool. + + +**Automated generation of new Galaxy tools for installation into any Galaxy** + +A test is generated using small sample test data inputs and parameter settings you supply. +Once the test case outputs have been produced, they can be used to build a +new Galaxy tool. The supplied script or executable is baked as a requirement +into a new, ordinary Galaxy tool, fully workflow compatible out of the box. +Generated tools are installed via a tool shed by an administrator +and work exactly like all other Galaxy tools for your users. + +**More Detail** + +To use the ToolFactory, you should have prepared a script to paste into a +text box, or have a package in mind and a small test input example ready to select from your history +to test your new script. + +```planemo test rgToolFactory2.xml --galaxy_root ~/galaxy --test_data ~/galaxy/tools/tool_makers/toolfactory/test-data``` works for me + +There is an example in each scripting language on the Tool Factory form. You +can just cut and paste these to try it out - remember to select the right +interpreter please. You'll also need to create a small test data set using +the Galaxy history add new data tool. + +If the script fails somehow, use the "redo" button on the tool output in +your history to recreate the form complete with broken script. Fix the bug +and execute again. Rinse, wash, repeat. + +Once the script runs sucessfully, a new Galaxy tool that runs your script +can be generated. Select the "generate" option and supply some help text and +names. The new tool will be generated in the form of a new Galaxy datatype +*toolshed.gz* - as the name suggests, it's an archive ready to upload to a +Galaxy ToolShed as a new tool repository. + +Once it's in a ToolShed, it can be installed into any local Galaxy server +from the server administrative interface. + +Once the new tool is installed, local users can run it - each time, the script +that was supplied when it was built will be executed with the input chosen +from the user's history. In other words, the tools you generate with the +ToolFactory run just like any other Galaxy tool,but run your script every time. + +Tool factory tools are perfect for workflow components. One input, one output, +no variables. + +*To fully and safely exploit the awesome power* of this tool, +Galaxy and the ToolShed, you should be a developer installing this +tool on a private/personal/scratch local instance where you are an +admin_user. Then, if you break it, you get to keep all the pieces see +https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home + +**Installation** +This is a Galaxy tool. You can install it most conveniently using the +administrative "Search and browse tool sheds" link. Find the Galaxy Main +toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory +repository. Open it and review the code and select the option to install it. + +If you can't get the tool that way, the xml and py files here need to be +copied into a new tools +subdirectory such as tools/toolfactory Your tool_conf.xml needs a new entry +pointing to the xml +file - something like:: + + <section name="Tool building tools" id="toolbuilders"> + <tool file="toolfactory/rgToolFactory.xml"/> + </section> + +If not already there, +please add: +<datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary" +mimetype="multipart/x-gzip" subclass="True" /> +to your local data_types_conf.xml. + + +**Restricted execution** + +The tool factory tool itself will then be usable ONLY by admin users - +people with IDs in admin_users in universe_wsgi.ini **Yes, that's right. ONLY +admin_users can run this tool** Think about it for a moment. If allowed to +run any arbitrary script on your Galaxy server, the only thing that would +impede a miscreant bent on destroying all your Galaxy data would probably +be lack of appropriate technical skills. + +**What it does** + +This is a tool factory for simple scripts in python, R and +perl currently. Functional tests are automatically generated. How cool is that. + +LIMITED to simple scripts that read one input from the history. Optionally can +write one new history dataset, and optionally collect any number of outputs +into links on an autogenerated HTML index page for the user to navigate - +useful if the script writes images and output files - pdf outputs are shown +as thumbnails and R's bloated pdf's are shrunk with ghostscript so that and +imagemagik need to be available. + +Generated tools can be edited and enhanced like any Galaxy tool, so start +small and build up since a generated script gets you a serious leg up to a +more complex one. + +**What you do** + +You paste and run your script, you fix the syntax errors and +eventually it runs. You can use the redo button and edit the script before +trying to rerun it as you debug - it works pretty well. + +Once the script works on some test data, you can generate a toolshed compatible +gzip file containing your script ready to run as an ordinary Galaxy tool in +a repository on your local toolshed. That means safe and largely automated +installation in any production Galaxy configured to use your toolshed. + +**Generated tool Security** + +Once you install a generated tool, it's just +another tool - assuming the script is safe. They just run normally and their +user cannot do anything unusually insecure but please, practice safe toolshed. +Read the code before you install any tool. Especially this one - it is really scary. + +**Send Code** + +Patches and suggestions welcome as bitbucket issues please? + +**Attribution** + +Creating re-usable tools from scripts: The Galaxy Tool Factory +Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team +Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573 + +http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref + +**Licensing** + +Copyright Ross Lazarus 2010 +ross lazarus at g mail period com + +All rights reserved. + +Licensed under the LGPL + +**Obligatory screenshot** + +http://bitbucket.org/fubar/galaxytoolmaker/src/fda8032fe989/images/dynamicScriptTool.png + |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/test-data/output2_sample --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/test-data/output2_sample Thu Jan 07 09:22:51 2021 +0000 |
| b |
| @@ -0,0 +1,165 @@ +*trats uoy erofeb GNINRAW* + +YLNO yxalaG etavirp a no loot siht llatsnI +ecnatsni noitcudorp ro cilbup a no REVEN esaelP + +troppus noitatic gnidda notlihC nhoJ yb 4102 tsugua detadpU + +keeB ned nav suiraM yb detroper sgub xif ot 4102 8 tsugua detadpU + +ta ecruoser eht etic esaelP +fer=epytyek&JWYdwMrWs1hQzcl=yekji?375stb/tnirper/igc/gro.slanruojdrofxo.scitamrofnioib//:ptth +.krow dehsilbup ruoy ni loot siht esu uoy fi + +**yrotS trohS** + +.sloot yxalaG wen gnitareneg fo elbapac loot yxalaG lausunu na si sihT +gnitpircs suoregnad ylemertxe erofereht dna *detcirtsernu* gnisopxe yb skrow tI +ot meht gniwolla ,revres yxalaG tsoh eht fo srotartsinimda detangised lla ot +,stes atad tupni detceles elpitlum revo lrep dna hs ,nohtyp ,R ni stpircs nur +.tuptuo sa tes atad wen elgnis a gnitirw + +*sretemarap elyts esrapgra ro lanoitisop htiw elbatucexe yna ro tpircs hsab/lrep/nohtyp/r gnikrow a evah uoY* + +.loot yxalaG a gnisu ,setunim ni loot yxalaG yranidro na otni denrut eb nac tI + + +**yxalaG yna otni noitallatsni rof sloot yxalaG wen fo noitareneg detamotuA** + +.ylppus uoy sgnittes retemarap dna stupni atad tset elpmas llams gnisu detareneg si tset A +a dliub ot desu eb nac yeht ,decudorp neeb evah stuptuo esac tset eht ecnO +tnemeriuqer a sa dekab si elbatucexe ro tpircs deilppus ehT .loot yxalaG wen +.xob eht fo tuo elbitapmoc wolfkrow ylluf ,loot yxalaG yranidro ,wen a otni +rotartsinimda na yb dehs loot a aiv dellatsni era sloot detareneG +.sresu ruoy rof sloot yxalaG rehto lla ekil yltcaxe krow dna + +**liateD eroM** + +a otni etsap ot tpircs a deraperp evah dluohs uoy ,yrotcaFlooT eht esu oT +yrotsih ruoy morf tceles ot ydaer elpmaxe tupni tset llams a dna dnim ni egakcap a evah ro ,xob txet +.tpircs wen ruoy tset ot + +em rof skrow ```atad-tset/yrotcafloot/srekam_loot/sloot/yxalag/~ atad_tset-- yxalag/~ toor_yxalag-- lmx.2yrotcaFlooTgr tset omenalp``` + +uoY .mrof yrotcaF looT eht no egaugnal gnitpircs hcae ni elpmaxe na si erehT +thgir eht tceles ot rebmemer - tuo ti yrt ot eseht etsap dna tuc tsuj nac +gnisu tes atad tset llams a etaerc ot deen osla ll'uoY .esaelp reterpretni +.loot atad wen dda yrotsih yxalaG eht + +ni tuptuo loot eht no nottub "oder" eht esu ,wohemos sliaf tpircs eht fI +gub eht xiF .tpircs nekorb htiw etelpmoc mrof eht etaercer ot yrotsih ruoy +.taeper ,hsaw ,esniR .niaga etucexe dna + +tpircs ruoy snur taht loot yxalaG wen a ,yllufssecus snur tpircs eht ecnO +dna txet pleh emos ylppus dna noitpo "etareneg" eht tceleS .detareneg eb nac +epytatad yxalaG wen a fo mrof eht ni detareneg eb lliw loot wen ehT .seman +a ot daolpu ot ydaer evihcra na s'ti ,stseggus eman eht sa - *zg.dehsloot* +.yrotisoper loot wen a sa dehSlooT yxalaG + +revres yxalaG lacol yna otni dellatsni eb nac ti ,dehSlooT a ni s'ti ecnO +.ecafretni evitartsinimda revres eht morf + +tpircs eht ,emit hcae - ti nur nac sresu lacol ,dellatsni si loot wen eht ecnO +nesohc tupni eht htiw detucexe eb lliw tliub saw ti nehw deilppus saw taht +eht htiw etareneg uoy sloot eht ,sdrow rehto nI .yrotsih s'resu eht morf +.emit yreve tpircs ruoy nur tub,loot yxalaG rehto yna ekil tsuj nur yrotcaFlooT + +,tuptuo eno ,tupni enO .stnenopmoc wolfkrow rof tcefrep era sloot yrotcaf looT +.selbairav on + +,loot siht fo *rewop emosewa eht tiolpxe ylefas dna ylluf oT* +siht gnillatsni repoleved a eb dluohs uoy ,dehSlooT eht dna yxalaG +na era uoy erehw ecnatsni lacol hctarcs/lanosrep/etavirp a no loot +ees seceip eht lla peek ot teg uoy ,ti kaerb uoy fi ,nehT .resu_nimda +emoH/ikiw/yrotcaflootyxalag/rabuf/gro.tekcubtib//:sptth + +**noitallatsnI** +eht gnisu yltneinevnoc tsom ti llatsni nac uoY .loot yxalaG a si sihT +niaM yxalaG eht dniF .knil "sdehs loot esworb dna hcraeS" evitartsinimda +yrotcafloot eht rof hcraes dna /ude.usp.xb.2g.dehsloot//:sptth ta dehsloot +.ti llatsni ot noitpo eht tceles dna edoc eht weiver dna ti nepO .yrotisoper + +eb ot deen ereh selif yp dna lmx eht ,yaw taht loot eht teg t'nac uoy fI +sloot wen a otni deipoc +yrtne wen a sdeen lmx.fnoc_loot ruoY yrotcafloot/sloot sa hcus yrotceridbus +lmx eht ot gnitniop +::ekil gnihtemos - elif + +>"sredliubloot"=di "sloot gnidliub looT"=eman noitces< +>/"lmx.yrotcaFlooTgr/yrotcafloot"=elif loot< +>noitces/< + +,ereht ydaerla ton fI +:dda esaelp +"yraniB:yranib.sepytatad.yxalag"=epyt "zg.dehsloot"=noisnetxe epytatad< +>/ "eurT"=ssalcbus "pizg-x/trapitlum"=epytemim +.lmx.fnoc_sepyt_atad lacol ruoy ot + + +**noitucexe detcirtseR** + +- sresu nimda yb YLNO elbasu eb neht lliw flesti loot yrotcaf loot ehT +YLNO .thgir s'taht ,seY** ini.igsw_esrevinu ni sresu_nimda ni sDI htiw elpoep +ot dewolla fI .tnemom a rof ti tuoba knihT **loot siht nur nac sresu_nimda +dluow taht gniht ylno eht ,revres yxalaG ruoy no tpircs yrartibra yna nur +ylbaborp dluow atad yxalaG ruoy lla gniyortsed no tneb tnaercsim a edepmi +.slliks lacinhcet etairporppa fo kcal eb + +**seod ti tahW** + +dna R ,nohtyp ni stpircs elpmis rof yrotcaf loot a si sihT +.taht si looc woH .detareneg yllacitamotua era stset lanoitcnuF .yltnerruc lrep + +nac yllanoitpO .yrotsih eht morf tupni eno daer taht stpircs elpmis ot DETIMIL +stuptuo fo rebmun yna tcelloc yllanoitpo dna ,tesatad yrotsih wen eno etirw +- etagivan ot resu eht rof egap xedni LMTH detarenegotua na no sknil otni +nwohs era stuptuo fdp - selif tuptuo dna segami setirw tpircs eht fi lufesu +dna taht os tpircstsohg htiw knurhs era s'fdp detaolb s'R dna slianbmuht sa +.elbaliava eb ot deen kigamegami + +trats os ,loot yxalaG yna ekil decnahne dna detide eb nac sloot detareneG +a ot pu gel suoires a uoy steg tpircs detareneg a ecnis pu dliub dna llams +.eno xelpmoc erom + +**od uoy tahW** + +dna srorre xatnys eht xif uoy ,tpircs ruoy nur dna etsap uoY +erofeb tpircs eht tide dna nottub oder eht esu nac uoY .snur ti yllautneve +.llew ytterp skrow ti - gubed uoy sa ti nurer ot gniyrt + +elbitapmoc dehsloot a etareneg nac uoy ,atad tset emos no skrow tpircs eht ecnO +ni loot yxalaG yranidro na sa nur ot ydaer tpircs ruoy gniniatnoc elif pizg +detamotua ylegral dna efas snaem tahT .dehsloot lacol ruoy no yrotisoper a +.dehsloot ruoy esu ot derugifnoc yxalaG noitcudorp yna ni noitallatsni + +**ytiruceS loot detareneG** + +tsuj s'ti ,loot detareneg a llatsni uoy ecnO +rieht dna yllamron nur tsuj yehT .efas si tpircs eht gnimussa - loot rehtona +.dehsloot efas ecitcarp ,esaelp tub erucesni yllausunu gnihtyna od tonnac resu +.yracs yllaer si ti - eno siht yllaicepsE .loot yna llatsni uoy erofeb edoc eht daeR + +**edoC dneS** + +?esaelp seussi tekcubtib sa emoclew snoitseggus dna sehctaP + +**noitubirttA** + +yrotcaF looT yxalaG ehT :stpircs morf sloot elbasu-er gnitaerC +maeT yxalaG ehT ;nnameiZ kraM ;ipsaK ynotnA ;surazaL ssoR +375stb/scitamrofnioib/3901.01 :iod ;2102 scitamrofnioiB + +fer=epytyek&JWYdwMrWs1hQzcl=yekji?375stb/tnirper/igc/gro.slanruojdrofxo.scitamrofnioib//:ptth + +**gnisneciL** + +0102 surazaL ssoR thgirypoC +moc doirep liam g ta surazal ssor + +.devreser sthgir llA + +LPGL eht rednu desneciL + +**tohsneercs yrotagilbO** + +gnp.looTtpircScimanyd/segami/989ef2308adf/crs/rekamlootyxalag/rabuf/gro.tekcubtib//:ptth |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/test-data/pyrevpos.python --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/test-data/pyrevpos.python Thu Jan 07 09:22:51 2021 +0000 |
| [ |
| @@ -0,0 +1,13 @@ +# reverse order of text by row +import sys +inp = sys.argv[1] +outp = sys.argv[2] +i = open(inp,'r').readlines() +o = open(outp,'w') +for row in i: + rs = row.rstrip() + rs = list(rs) + rs.reverse() + o.write(''.join(rs)) +o.close() + |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/test-data/test1_log.txt --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/test-data/test1_log.txt Thu Jan 07 09:22:51 2021 +0000 |
| b |
| @@ -0,0 +1,1 @@ +## Executing Toolfactory generated command line = python /tmp/pyrevposq5dmcdy1.python /tmp/tmpqrksf8sd/files/5/b/9/dataset_5b952a86-87df-44ad-a415-ea549f3f0cee.dat output2 |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/test-data/toolfactory_pyrevpos_tgz_sample |
| b |
| Binary file home/ross/galaxy/tools/tool_makers/toolfactory/test-data/toolfactory_pyrevpos_tgz_sample has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/testtf.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/testtf.sh Thu Jan 07 09:22:51 2021 +0000 |
| b |
| @@ -0,0 +1,2 @@ +planemo test --no_cleanup --no_dependency_resolution --skip_venv --galaxy_root ~/galaxy ~/galaxy/tools/tool_makers/toolfactory &>foo + |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 home/ross/galaxy/tools/tool_makers/toolfactory/whoosh.sh --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/home/ross/galaxy/tools/tool_makers/toolfactory/whoosh.sh Thu Jan 07 09:22:51 2021 +0000 |
| [ |
| @@ -0,0 +1,16 @@ +# using the galaxy venv this seems to work +. /galaxy_venv/bin/activate +python scripts/tool_shed/build_ts_whoosh_index.py -c config/tool_shed.yml --config-section tool_shed +# eeesh. /etc/galaxy is where the actual galaxy.yml lives - despite other configs being where they might be expected +# fix tool_shed.yml to 0.0.0.0 and admin email +# add tgz to datatypes :( +# need to source a venv in /export/tool_deps/toolfactorydeps/0.01/fubar/toolfactorydeps/9e9428fe9134/env.sh +# as nothing was done by the setup_virtualenv step apparently. +# gcc and friends for planemo pip installation +# File "/galaxy-central/lib/galaxy/tool_util/verify/interactor.py", line 595, in <listcomp> +# test_user = [user for user in all_users if user["email"] == email][0] +# add local to tool_sheds_conf.xml +# <tool_sheds> +# <tool_shed name="local" url="http://localhost:9009"/> +# mercurial > 5.5!! + |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/.planemo.yml --- a/toolfactory/.planemo.yml Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,23 +0,0 @@ -## Planemo Global Configuration File. -## Everything in this file is completely optional - these values can all be -## configured via command line options for the corresponding commands. - -## Specify a default galaxy_root for test and server commands here. -galaxy_root: /galaxy-central -## Username used with toolshed(s). -shed_username: galaxy -sheds: - # For each tool shed you wish to target, uncomment key or both email and - # password. - toolshed: - #key: "<TODO>" - #email: "<TODO>" - #password: "<TODO>" - testtoolshed: - #key: "<TODO>" - #email: "<TODO>" - #password: "<TODO>" - local: - key: "fakekey" - email: "admin@galaxy.org" - password: "password" |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/.shed.yml --- a/toolfactory/.shed.yml Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,13 +0,0 @@ -name: toolfactory -owner: fubar -description: ToolFactory - tool to make Galaxy tools ready for the toolshed -homepage_url: https://github.com/fubar2/toolfactory -long_description: | - ToolFactory - turn executable packages and R/python/perl/bash scripts into ordinary Galaxy tools - - Creating re-usable tools from scripts: The Galaxy Tool Factory Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team - Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573 -remote_repository_url: https://github.com/fubar2/toolfactory -type: tool_dependency_definition -categories: -- Tool Generators |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/LICENSE --- a/toolfactory/LICENSE Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| b"@@ -1,504 +0,0 @@\n-GNU LESSER GENERAL PUBLIC LICENSE\n- Version 2.1, February 1999\n-\n- Copyright (C) 1991, 1999 Free Software Foundation, Inc.\n- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA\n- Everyone is permitted to copy and distribute verbatim copies\n- of this license document, but changing it is not allowed.\n-\n-(This is the first released version of the Lesser GPL. It also counts\n- as the successor of the GNU Library Public License, version 2, hence\n- the version number 2.1.)\n-\n- Preamble\n-\n- The licenses for most software are designed to take away your\n-freedom to share and change it. By contrast, the GNU General Public\n-Licenses are intended to guarantee your freedom to share and change\n-free software--to make sure the software is free for all its users.\n-\n- This license, the Lesser General Public License, applies to some\n-specially designated software packages--typically libraries--of the\n-Free Software Foundation and other authors who decide to use it. You\n-can use it too, but we suggest you first think carefully about whether\n-this license or the ordinary General Public License is the better\n-strategy to use in any particular case, based on the explanations below.\n-\n- When we speak of free software, we are referring to freedom of use,\n-not price. Our General Public Licenses are designed to make sure that\n-you have the freedom to distribute copies of free software (and charge\n-for this service if you wish); that you receive source code or can get\n-it if you want it; that you can change the software and use pieces of\n-it in new free programs; and that you are informed that you can do\n-these things.\n-\n- To protect your rights, we need to make restrictions that forbid\n-distributors to deny you these rights or to ask you to surrender these\n-rights. These restrictions translate to certain responsibilities for\n-you if you distribute copies of the library or if you modify it.\n-\n- For example, if you distribute copies of the library, whether gratis\n-or for a fee, you must give the recipients all the rights that we gave\n-you. You must make sure that they, too, receive or can get the source\n-code. If you link other code with the library, you must provide\n-complete object files to the recipients, so that they can relink them\n-with the library after making changes to the library and recompiling\n-it. And you must show them these terms so they know their rights.\n-\n- We protect your rights with a two-step method: (1) we copyright the\n-library, and (2) we offer you this license, which gives you legal\n-permission to copy, distribute and/or modify the library.\n-\n- To protect each distributor, we want to make it very clear that\n-there is no warranty for the free library. Also, if the library is\n-modified by someone else and passed on, the recipients should know\n-that what they have is not the original version, so that the original\n-author's reputation will not be affected by problems that might be\n-introduced by others.\n-\n- Finally, software patents pose a constant threat to the existence of\n-any free program. We wish to make sure that a company cannot\n-effectively restrict the users of a free program by obtaining a\n-restrictive license from a patent holder. Therefore, we insist that\n-any patent license obtained for a version of the library must be\n-consistent with the full freedom of use specified in this license.\n-\n- Most GNU software, including some libraries, is covered by the\n-ordinary GNU General Public License. This license, the GNU Lesser\n-General Public License, applies to certain designated libraries, and\n-is quite different from the ordinary General Public License. We use\n-this license for certain libraries in order to permit linking those\n-libraries into non-free programs.\n-\n- When a program is linked with a library, whether statically or using\n-a shared library, the combination of the two is legally speaking a\n-combined work, a derivative "..b'oes not specify a\n-license version number, you may choose any version ever published by\n-the Free Software Foundation.\n-\n- 14. If you wish to incorporate parts of the Library into other free\n-programs whose distribution conditions are incompatible with these,\n-write to the author to ask for permission. For software which is\n-copyrighted by the Free Software Foundation, write to the Free\n-Software Foundation; we sometimes make exceptions for this. Our\n-decision will be guided by the two goals of preserving the free status\n-of all derivatives of our free software and of promoting the sharing\n-and reuse of software generally.\n-\n- NO WARRANTY\n-\n- 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO\n-WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.\n-EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR\n-OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY\n-KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE\n-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n-PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE\n-LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME\n-THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.\n-\n- 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN\n-WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY\n-AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU\n-FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR\n-CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE\n-LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING\n-RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A\n-FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF\n-SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH\n-DAMAGES.\n-\n- END OF TERMS AND CONDITIONS\n-\n- How to Apply These Terms to Your New Libraries\n-\n- If you develop a new library, and you want it to be of the greatest\n-possible use to the public, we recommend making it free software that\n-everyone can redistribute and change. You can do so by permitting\n-redistribution under these terms (or, alternatively, under the terms of the\n-ordinary General Public License).\n-\n- To apply these terms, attach the following notices to the library. It is\n-safest to attach them to the start of each source file to most effectively\n-convey the exclusion of warranty; and each file should have at least the\n-"copyright" line and a pointer to where the full notice is found.\n-\n- {description}\n- Copyright (C) {year} {fullname}\n-\n- This library is free software; you can redistribute it and/or\n- modify it under the terms of the GNU Lesser General Public\n- License as published by the Free Software Foundation; either\n- version 2.1 of the License, or (at your option) any later version.\n-\n- This library is distributed in the hope that it will be useful,\n- but WITHOUT ANY WARRANTY; without even the implied warranty of\n- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU\n- Lesser General Public License for more details.\n-\n- You should have received a copy of the GNU Lesser General Public\n- License along with this library; if not, write to the Free Software\n- Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301\n- USA\n-\n-Also add information on how to contact you by electronic and paper mail.\n-\n-You should also get your employer (if you work as a programmer) or your\n-school, if any, to sign a "copyright disclaimer" for the library, if\n-necessary. Here is a sample; alter the names:\n-\n- Yoyodyne, Inc., hereby disclaims all copyright interest in the\n- library `Frob\' (a library for tweaking knobs) written by James Random\n- Hacker.\n-\n- {signature of Ty Coon}, 1 April 1990\n- Ty Coon, President of Vice\n-\n-That\'s all there is to it!\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/README.md --- a/toolfactory/README.md Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| b'@@ -1,380 +0,0 @@\n-\xef\xbb\xbf## Breaking news! Docker container at https://github.com/fubar2/toolfactory-galaxy-docker recommended as at December 2020\n-\n-## This is the original ToolFactory suitable for non-docker situations. Please use the docker container if you can because it\'s integrated with a Toolshed...\n-\n-# WARNING\n-\n-Install this tool to a throw-away private Galaxy or Docker container ONLY!\n-\n-Please NEVER on a public or production instance where a hostile user may\n-be able to gain access if they can acquire an administrative account login.\n-\n-It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since\n-it can install new tools to the Galaxy server it executes on! This is not something you should allow other than\n-on a throw away instance that is protected from potentially hostile users.\n-\n-## Short Story\n-\n-Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as\n-a tool to Galaxy requires an XML document describing how the application interacts with Galaxy.\n-This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package\n-as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent\n-and easy to use interface once installed in the local Galaxy server.\n-\n-Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it\n-automates much of the boilerplate and makes the process much easier.\n-The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command\n-line complexities. The user will still need appropriate skills in terms of describing the interface between\n-Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed\n-settings, together with automated testing and uploading to a toolshed with optional local installation.\n-\n-\n-## ToolFactory generated tools are ordinary Galaxy tools\n-\n-A TF generated tool that passes the Planemo test is ready to publish in any Galaxy Toolshed and ready to install in any running Galaxy instance.\n-They are fully workflow compatible and work exactly like any hand-written tool. The user can select input files of the specified type(s) from their\n-history and edit each of the specified parameters. The tool form will show all the labels and help text supplied when the tool was built. When the tool\n-is executed, the dependent binary or script will be passed all the i/o files and parameters as specified, and will write outputs to the specified new\n-history datasets - just like any other Galaxy tool.\n-\n-## Models for tool command line construction\n-\n-The key to turning any software package into a Galaxy tool is the automated construction of a suitable command line.\n-\n-The TF can build a new tool that will allow the tool user to select input files from their history, set any parameters and when run will send the\n-new output files to the history as specified when the tool builder completed the form and built the new tool.\n-\n-That tool can contain instructions to run any Conda dependency or a system executable like bash. Whether a bash script you have written or\n-a Conda package like bwa, the executable will expect to find settings for input, output and parameters on a command line.\n-\n-These are often passed as "--name value" (argparse style) or in a fixed order (positional style).\n-\n-The ToolFactory allows either, or for "filter" applications that process input from STDIN and write processed output to STDOUT.\n-\n-The simplest tool model wraps a simple script or Conda dependency package requiring only input and output files, with no user supplied settings illustrated by\n-the Tacrev demonstration tool found in the Galaxy running in the ToolFactory docker container. It passes a user selected input file from the current history on STDIN\n-to a bash script. The bash script runs the unix tac utility (r'..b'll sample\n-data sets for testing. When the script is working correctly, upload the small sample datasets\n-into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form.\n-\n-### Outputs\n-\n-The TF will generate the new tool described on the TF form, and test it\n-using planemo. Optionally if a local toolshed is running, it can be used to\n-install the new tool back into the generating Galaxy.\n-\n-A toolshed is built in to the Docker container and configured\n-so a tool can be tested, sent to that toolshed, then installed in the Galaxy\n-where the TF is running using the default toolshed and Galaxy URL and API keys.\n-\n-Once it\'s in a ToolShed, it can be installed into any local Galaxy server\n-from the server administrative interface.\n-\n-Once the new tool is installed, local users can run it - each time, the\n-package and/or script that was supplied when it was built will be executed with the input chosen\n-from the user\'s history, together with user supplied parameters. In other words, the tools you generate with the\n-TF run just like any other Galaxy tool.\n-\n-TF generated tools work as normal workflow components.\n-\n-\n-## Limitations\n-\n-The TF is flexible enough to generate wrappers for many common scientific packages\n-but the inbuilt automation will not cope with all possible situations. Users can\n-supply overrides for two tool XML segments - tests and command and the BWA\n-example in the supplied samples workflow illustrates their use. It does not deal with\n-repeated elements or conditional parameters such as allowing a user to choose to see "simple"\n-or "advanced" parameters (yet) and there will be plenty of packages it just\n-won\'t cover - but it\'s a quick and efficient tool for the other 90% of cases. Perfect for\n-that bash one liner you need to get that workflow functioning correctly for this\n-afternoon\'s demonstration!\n-\n-## Installation\n-\n-The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md\n-is the best way to use the TF because it is preconfigured\n-to automate new tool testing and has a built in local toolshed where each new tool\n-is uploaded. If you grab the docker container, it should just work after a restart and you\n-can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory\n-jobs that generated them allows you to add fields and experiment to see how things work.\n-\n-It can be installed like any other tool from the Toolshed, but you will need to make some\n-configuration changes (TODO write a configuration). You can install it most conveniently using the\n-administrative "Search and browse tool sheds" link. Find the Galaxy Main\n-toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory\n-repository in the Tool Maker section. Open it and review the code and select the option to install it.\n-\n-If not already there please add:\n-\n-```\n-<datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />\n-```\n-\n-to your local config/data_types_conf.xml.\n-\n-\n-## Restricted execution\n-\n-The tool factory tool itself will ONLY run for admin users -\n-people with IDs in config/galaxy.yml "admin_users".\n-\n-*ONLY admin_users can run this tool*\n-\n-That doesn\'t mean it\'s safe to install on a shared or exposed instance - please don\'t.\n-\n-## Generated tool Security\n-\n-Once you install a generated tool, it\'s just\n-another tool - assuming the script is safe. They just run normally and their\n-user cannot do anything unusually insecure but please, practice safe toolshed.\n-Read the code before you install any tool. Especially this one - it is really scary.\n-\n-## Attribution\n-\n-Creating re-usable tools from scripts: The Galaxy Tool Factory\n-Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team\n-Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573\n-\n-http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref\n-\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/images/TFasIDE.png |
| b |
| Binary file toolfactory/images/TFasIDE.png has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/images/dynamicScriptTool.png |
| b |
| Binary file toolfactory/images/dynamicScriptTool.png has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/images/hello_toolfactory_form.png |
| b |
| Binary file toolfactory/images/hello_toolfactory_form.png has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/install_tf_demos.py --- a/toolfactory/install_tf_demos.py Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,47 +0,0 @@ -import argparse -import os -import subprocess -import sys -import urllib.request - -from bioblend import galaxy - -WF = "https://drive.google.com/uc?export=download&id=13xE8o7tucHGNA0qYkEP98FfUGl2wdOU5" -HIST = ( - "https://drive.google.com/uc?export=download&id=1V0ZN9ZBuqcGJvt2AP7s3g0q11uYEhdDB" -) -WF_FILE = "tf_workflow.ga" -HIST_FILE = "tf_history.tgz" - - -def _parser(): - parser = argparse.ArgumentParser() - parser.add_argument( - "-g", "--galaxy", help="URL of target galaxy", default="http://localhost:9090" - ) - parser.add_argument("-a", "--key", help="Galaxy admin key", default=None) - return parser - - -def main(): - """ - load the planemo tool_factory demonstration history and tool generating workflow - fails in planemo served galaxies because there seems to be no user in trans? - """ - args = _parser().parse_args() - urllib.request.urlretrieve(WF, WF_FILE) - urllib.request.urlretrieve(HIST, HIST_FILE) - assert args.key, "Need an administrative key for the target Galaxy supplied please" - wfp = os.path.abspath(WF_FILE) - hp = os.path.abspath(HIST_FILE) - gi = galaxy.GalaxyInstance( - url=args.galaxy, key=args.key, email="planemo@galaxyproject.org" - ) - x = gi.workflows.import_workflow_from_local_path(WF_FILE, publish=True) - print(f"installed {WF_FILE} Returned = {x}\n") - x = gi.histories.import_history(file_path=HIST_FILE) - print(f"installed {HIST_FILE} Returned = {x}\n") - - -if __name__ == "__main__": - main() |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/install_tf_demos_toolshed.tgz |
| b |
| Binary file toolfactory/install_tf_demos_toolshed.tgz has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/planemo_install_tfdemo.tar.gz |
| b |
| Binary file toolfactory/planemo_install_tfdemo.tar.gz has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/rgToolFactory2.py --- a/toolfactory/rgToolFactory2.py Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| [ |
| b'@@ -1,1021 +0,0 @@\n-# replace with shebang for biocontainer\n-# see https://github.com/fubar2/toolfactory\n-#\n-# copyright ross lazarus (ross stop lazarus at gmail stop com) May 2012\n-#\n-# all rights reserved\n-# Licensed under the LGPL\n-# suggestions for improvement and bug fixes welcome at\n-# https://github.com/fubar2/toolfactory\n-#\n-# July 2020: BCC was fun and I feel like rip van winkle after 5 years.\n-# Decided to\n-# 1. Fix the toolfactory so it works - done for simplest case\n-# 2. Fix planemo so the toolfactory function works\n-# 3. Rewrite bits using galaxyxml functions where that makes sense - done\n-\n-import argparse\n-import copy\n-import logging\n-import os\n-import re\n-import shutil\n-import subprocess\n-import sys\n-import tarfile\n-import tempfile\n-import time\n-\n-from bioblend import ConnectionError\n-from bioblend import toolshed\n-\n-import galaxyxml.tool as gxt\n-import galaxyxml.tool.parameters as gxtp\n-\n-import lxml\n-\n-import yaml\n-\n-myversion = "V2.1 July 2020"\n-verbose = True\n-debug = True\n-toolFactoryURL = "https://github.com/fubar2/toolfactory"\n-ourdelim = "~~~"\n-\n-# --input_files="$intab.input_files~~~$intab.input_CL~~~\n-# $intab.input_formats# ~~~$intab.input_label\n-# ~~~$intab.input_help"\n-IPATHPOS = 0\n-ICLPOS = 1\n-IFMTPOS = 2\n-ILABPOS = 3\n-IHELPOS = 4\n-IOCLPOS = 5\n-\n-# --output_files "$otab.history_name~~~$otab.history_format~~~\n-# $otab.history_CL~~~$otab.history_test"\n-ONAMEPOS = 0\n-OFMTPOS = 1\n-OCLPOS = 2\n-OTESTPOS = 3\n-OOCLPOS = 4\n-\n-\n-# --additional_parameters="$i.param_name~~~$i.param_value~~~\n-# $i.param_label~~~$i.param_help~~~$i.param_type\n-# ~~~$i.CL~~~i$.param_CLoverride"\n-ANAMEPOS = 0\n-AVALPOS = 1\n-ALABPOS = 2\n-AHELPPOS = 3\n-ATYPEPOS = 4\n-ACLPOS = 5\n-AOVERPOS = 6\n-AOCLPOS = 7\n-\n-\n-foo = len(lxml.__version__)\n-# fug you, flake8. Say my name!\n-FAKEEXE = "~~~REMOVE~~~ME~~~"\n-# need this until a PR/version bump to fix galaxyxml prepending the exe even\n-# with override.\n-\n-\n-def timenow():\n- """return current time as a string"""\n- return time.strftime("%d/%m/%Y %H:%M:%S", time.localtime(time.time()))\n-\n-\n-def quote_non_numeric(s):\n- """return a prequoted string for non-numerics\n- useful for perl and Rscript parameter passing?\n- """\n- try:\n- _ = float(s)\n- return s\n- except ValueError:\n- return \'"%s"\' % s\n-\n-\n-html_escape_table = {\n- "&": "&",\n- ">": ">",\n- "<": "<",\n- "#": "#",\n- "$": "$",\n-}\n-cheetah_escape_table = {"$": "\\\\$", "#": "\\\\#"}\n-\n-\n-def html_escape(text):\n- """Produce entities within text."""\n- return "".join([html_escape_table.get(c, c) for c in text])\n-\n-\n-def cheetah_escape(text):\n- """Produce entities within text."""\n- return "".join([cheetah_escape_table.get(c, c) for c in text])\n-\n-\n-def html_unescape(text):\n- """Revert entities within text. Multiple character targets so use replace"""\n- t = text.replace("&", "&")\n- t = t.replace(">", ">")\n- t = t.replace("<", "<")\n- t = t.replace("\\\\$", "$")\n- t = t.replace("$", "$")\n- t = t.replace("#", "#")\n- return t\n-\n-\n-def parse_citations(citations_text):\n- """"""\n- citations = [c for c in citations_text.split("**ENTRY**") if c.strip()]\n- citation_tuples = []\n- for citation in citations:\n- if citation.startswith("doi"):\n- citation_tuples.append(("doi", citation[len("doi") :].strip()))\n- else:\n- citation_tuples.append(\n- ("bibtex", citation[len("bibtex") :].strip())\n- )\n- return citation_tuples\n-\n-\n-class ScriptRunner:\n- """Wrapper for an arbitrary script\n- uses galaxyxml\n-\n- """\n-\n- def __init__(self, args=None):\n- """\n- prepare command line cl for running the tool here\n- and prepare elements needed for galaxyxml tool generation\n- """\n- self.ourcwd = os.getcwd()\n- self.ourenv = copy.deepcopy(os.environ)\n- self.infiles = [x.split(ourdelim) for x in args.input_files]\n- self.outfiles '..b'%s%s" % (entry.name.replace(".", "_"), ext)\n- dest = os.path.join(self.repdir, ofn)\n- src = os.path.join(self.tooloutdir, entry.name)\n- shutil.copyfile(src, dest)\n- with os.scandir(self.testdir) as outs:\n- for entry in outs:\n- if (\n- (not entry.is_file())\n- or entry.name.endswith("_sample")\n- or entry.name.endswith("_planemo_test_report.html")\n- ):\n- continue\n- if "." in entry.name:\n- nayme, ext = os.path.splitext(entry.name)\n- else:\n- ext = ".txt"\n- newname = f"{entry.name}{ext}"\n- dest = os.path.join(self.repdir, newname)\n- src = os.path.join(self.testdir, entry.name)\n- shutil.copyfile(src, dest)\n-\n-\n-def main():\n- """\n- This is a Galaxy wrapper.\n- It expects to be called by a special purpose tool.xml\n-\n- """\n- parser = argparse.ArgumentParser()\n- a = parser.add_argument\n- a("--script_path", default=None)\n- a("--history_test", default=None)\n- a("--cl_prefix", default=None)\n- a("--sysexe", default=None)\n- a("--packages", default=None)\n- a("--tool_name", default="newtool")\n- a("--tool_dir", default=None)\n- a("--input_files", default=[], action="append")\n- a("--output_files", default=[], action="append")\n- a("--user_email", default="Unknown")\n- a("--bad_user", default=None)\n- a("--make_Tool", default="runonly")\n- a("--help_text", default=None)\n- a("--tool_desc", default=None)\n- a("--tool_version", default=None)\n- a("--citations", default=None)\n- a("--command_override", default=None)\n- a("--test_override", default=None)\n- a("--additional_parameters", action="append", default=[])\n- a("--edit_additional_parameters", action="store_true", default=False)\n- a("--parampass", default="positional")\n- a("--tfout", default="./tfout")\n- a("--new_tool", default="new_tool")\n- a("--galaxy_url", default="http://localhost:8080")\n- a("--toolshed_url", default="http://localhost:9009")\n- # make sure this is identical to tool_sheds_conf.xml\n- # localhost != 127.0.0.1 so validation fails\n- a("--toolshed_api_key", default="fakekey")\n- a("--galaxy_api_key", default="fakekey")\n- a("--galaxy_root", default="/galaxy-central")\n- a("--galaxy_venv", default="/galaxy_venv")\n- args = parser.parse_args()\n- assert not args.bad_user, (\n- \'UNAUTHORISED: %s is NOT authorized to use this tool until Galaxy \\\n-admin adds %s to "admin_users" in the galaxy.yml Galaxy configuration file\'\n- % (args.bad_user, args.bad_user)\n- )\n- assert (\n- args.tool_name\n- ), "## Tool Factory expects a tool name - eg --tool_name=DESeq"\n- assert (\n- args.sysexe or args.packages\n- ), "## Tool Factory wrapper expects an interpreter \\\n-or an executable package in --sysexe or --packages"\n- args.input_files = [\n- x.replace(\'"\', "").replace("\'", "") for x in args.input_files\n- ]\n- # remove quotes we need to deal with spaces in CL params\n- for i, x in enumerate(args.additional_parameters):\n- args.additional_parameters[i] = args.additional_parameters[i].replace(\n- \'"\', ""\n- )\n- r = ScriptRunner(args)\n- r.writeShedyml()\n- r.makeTool()\n- if args.make_Tool == "generate":\n- retcode = r.run()\n- r.moveRunOutputs()\n- r.makeToolTar()\n- else:\n- retcode = r.planemo_test(genoutputs=True) # this fails :( - see PR\n- r.moveRunOutputs()\n- r.makeToolTar()\n- retcode = r.planemo_test(genoutputs=False)\n- r.moveRunOutputs()\n- r.makeToolTar()\n- print(f"second planemo_test returned {retcode}")\n- if args.make_Tool == "gentestinstall":\n- r.shedLoad()\n- r.eph_galaxy_load()\n-\n-\n-if __name__ == "__main__":\n- main()\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/rgToolFactory2.xml --- a/toolfactory/rgToolFactory2.xml Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| [ |
| b'@@ -1,536 +0,0 @@\n-<tool id="rgtf2" name="toolfactory" version="2.00" profile="16.04" >\n- <description>Scripts into tools v2.0</description>\n- <macros>\n- <xml name="tool_metadata">\n- <param name="tool_version" label="Tool Version - bump this to warn users trying to redo old analyses" type="text" value="0.01"\n- help="If you change your script and regenerate the \'same\' tool, you should inform Galaxy (and users) by changing (bumping is traditional) this number"/>\n- <param name="tool_desc" label="Tool Synopsis" type="text" value=""\n- help="Supply a brief tool description for the Galaxy tool menu entry" />\n- <param name="help_text" label="Tool form documentation and help text for users" type="text" area="true"\n- value="**What it Does**"\n- help="Supply user documentation to appear on the new tool form as reStructured text - http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html" >\n- <sanitizer>\n- <valid initial="string.printable">\n- </valid>\n- <mapping initial="none"/>\n- </sanitizer>\n- </param>\n- <repeat name="citations" title="Citation">\n- <conditional name="citation_type">\n- <param name="type" type="select" display="radio" label="Citation Type">\n- <option value="doi">DOI</option>\n- <option value="bibtex">BibTeX</option>\n- </param>\n- <when value="doi">\n- <param name="doi" label="DOI" type="text" value=""\n- help="Supply a DOI (e.g. doi: 10.1111/j.1740-9713.2007.00258.x) to be cited when this tool is used in published research." />\n- </when>\n- <when value="bibtex">\n- <param name="bibtex" label="BibTex" type="text" area="true"\n- help="Supply a BibTex entry that should be cited when this tool is used in published research." value="" >\n- <sanitizer>\n- <valid initial="string.printable">\n- </valid>\n- <mapping initial="none"/>\n- </sanitizer>\n- </param>\n- </when>\n- </conditional>\n- </repeat>\n- </xml>\n- <xml name="io">\n- <section name="io" title="Input and output files" expanded="true">\n- <repeat name="history_inputs" title="zero or more input data files to pass as parameters to the executable."\n- help="USE SMALL SAMPLES for the new tool\'s test. Prompts will form a history item selector as input for users of this new tool">\n- <param name="input_files" type="data" format="data" label="Select an input file from your history" optional="true" multiple="false"\n- help=""/>\n- <param name="input_formats" type="select" multiple="true" label="Select the datatype(s) that your tool/script accepts as input"\n- help="If your datatype is not listed here, it has to be added in galaxy\'s datatypes_conf.xml" value="tabular">\n- <options from_parameter="tool.app.datatypes_registry.upload_file_formats">\n- <column name="value" index="0"/>\n- </options>\n- </param>\n- <param name="input_label" type="text" value="" label="This will become the user prompt for the form so please make it informative"\n- help="Note that \'~~~\' is an internal delimiter so must not appear in this text field - please work around this technical limitation" >\n- <sanitizer invalid_char="">\n- <valid initial="string.printable"> <remove value=\'~~~\'/> </valid>\n- <mapping initial="none"/>\n- </sanitizer>\n- </param>\n- <param'..b'value="import sys; inp = sys.argv[1]; outp = sys.argv[2]; inlist = open(inp,\'r\').readlines(); o = open(outp,\'w\'); rs = [\'\'.join(list(reversed(x.rstrip()))) for x in inlist]; o.write(\'\\n\'.join(rs)); o.close()"/>\n- <param name="choosescript" value="yes" />\n- <param name="script_path" value="$runme"/>\n- <output name="new_tool" file="toolfactory_pyrevpos_tgz_sample" compare="sim_size" delta="6000" />\n-</test>\n-</tests>\n-<help>\n-\n-.. class:: warningmark\n-\n-**Details and attribution**\n-(see GTF_)\n-\n-**Local Admins ONLY**\n-Only users whose IDs found in the local admin_user configuration setting in universe_wsgi.ini can run this tool.\n-\n-**If you find a bug**\n-Please raise an issue, or even better, submit a pull request fixing it, on the github repository GTF_\n-\n-**What it does**\n-This tool optionally generates normal workflow compatible first class Galaxy tools\n-\n-Generated tools can run existing binary packages that become requirements, existing scripts, or new scripts pasted into this tool form.\n-Pasted scripts are written so they are part of the new tool and cannot be adjusted by the downstream user.\n-Binary packages are managed by the dependency subsystem - conda usually, so anything in bioconda or conda_forge is available for example.\n-\n-Any number of parameters can be built into the new tool form for passing in to the script or executable at runtime.\n-These can be editable by the downstream user or baked in.\n-\n-When you run this tool, your executable or script and supplied parameter values will be run to produce a canonical\n-set of outputs - these are used to construct a test for the new tool.\n-\n-If tool generation is required, a new tarball compatible with any Galaxy toolshed is created.\n-It can be unpacked in your galaxy/tools directory and manually added to tool_conf.xml, or\n-installed into any toolshed from where it can be installed into your Galaxy.\n-\n-\n-.. class:: warningmark\n-\n-**Note to system administrators**\n-This tool offers *NO* built in protection against malicious scripts. It should only be installed on private/personnal Galaxy instances.\n-Admin_users will have the power to do anything they want as the Galaxy user if you install this tool.\n-\n-.. class:: warningmark\n-\n-**Use on public servers** is STRONGLY discouraged for obvious reasons\n-\n-The tools generated by this tool will run just as securely as any other normal installed Galaxy tool but like any other new tools, should always be checked carefully before installation.\n-We recommend that you follow the good code hygiene practices associated with safe toolshed practices.\n-\n-Here\'s a sample python script that can be cut and pasted into the tool form, suitable for positional parameter passing:\n-\n-::\n-\n- # reverse order of text by row\n- import sys\n- inp = sys.argv[1]\n- outp = sys.argv[2]\n- i = open(inp,\'r\').readlines()\n- o = open(outp,\'w\')\n- for row in i:\n- rs = row.rstrip()\n- rs = list(rs)\n- rs.reverse()\n- o.write(\'\'.join(rs))\n- o.write(\'\\n\')\n- o.close()\n-\n-With argparse style parameters:\n-\n-::\n-\n- # reverse order of text by row\n- import argparse\n- parser = argparse.ArgumentParser()\n- a = parser.add_argument\n- a(\'--infile\',default=\'\')\n- a(\'--outfile\',default=None)\n- args = parser.parse_args()\n- inp = args.infile\n- outp = args.outfile\n- i = open(inp,\'r\').readlines()\n- o = open(outp,\'w\')\n- for row in i:\n- rs = row.rstrip()\n- rs = list(rs)\n- rs.reverse()\n- o.write(\'\'.join(rs))\n- o.write(\'\\n\')\n- o.close()\n-\n-\n-\n-Paper_\n-\n-*Licensing*\n-\n-Copyright Ross Lazarus (ross period lazarus at gmail period com) May 2012\n-All rights reserved.\n-Licensed under the LGPL_\n-\n-.. _LGPL: http://www.gnu.org/copyleft/lesser.html\n-.. _GTF: https://github.com/fubar2/toolfactory\n-.. _Paper: http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573\n-\n-\n-</help>\n-<citations>\n- <citation type="doi">10.1093/bioinformatics/bts573</citation>\n-</citations>\n-</tool>\n-\n-\n' |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/test-data/input1_sample --- a/toolfactory/test-data/input1_sample Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,166 +0,0 @@ -*WARNING before you start* - - Install this tool on a private Galaxy ONLY - Please NEVER on a public or production instance - -Updated august 2014 by John Chilton adding citation support - -Updated august 8 2014 to fix bugs reported by Marius van den Beek - -Please cite the resource at -http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref -if you use this tool in your published work. - -**Short Story** - -This is an unusual Galaxy tool capable of generating new Galaxy tools. -It works by exposing *unrestricted* and therefore extremely dangerous scripting -to all designated administrators of the host Galaxy server, allowing them to -run scripts in R, python, sh and perl over multiple selected input data sets, -writing a single new data set as output. - -*You have a working r/python/perl/bash script or any executable with positional or argparse style parameters* - -It can be turned into an ordinary Galaxy tool in minutes, using a Galaxy tool. - - -**Automated generation of new Galaxy tools for installation into any Galaxy** - -A test is generated using small sample test data inputs and parameter settings you supply. -Once the test case outputs have been produced, they can be used to build a -new Galaxy tool. The supplied script or executable is baked as a requirement -into a new, ordinary Galaxy tool, fully workflow compatible out of the box. -Generated tools are installed via a tool shed by an administrator -and work exactly like all other Galaxy tools for your users. - -**More Detail** - -To use the ToolFactory, you should have prepared a script to paste into a -text box, or have a package in mind and a small test input example ready to select from your history -to test your new script. - -```planemo test rgToolFactory2.xml --galaxy_root ~/galaxy --test_data ~/galaxy/tools/tool_makers/toolfactory/test-data``` works for me - -There is an example in each scripting language on the Tool Factory form. You -can just cut and paste these to try it out - remember to select the right -interpreter please. You'll also need to create a small test data set using -the Galaxy history add new data tool. - -If the script fails somehow, use the "redo" button on the tool output in -your history to recreate the form complete with broken script. Fix the bug -and execute again. Rinse, wash, repeat. - -Once the script runs sucessfully, a new Galaxy tool that runs your script -can be generated. Select the "generate" option and supply some help text and -names. The new tool will be generated in the form of a new Galaxy datatype -*toolshed.gz* - as the name suggests, it's an archive ready to upload to a -Galaxy ToolShed as a new tool repository. - -Once it's in a ToolShed, it can be installed into any local Galaxy server -from the server administrative interface. - -Once the new tool is installed, local users can run it - each time, the script -that was supplied when it was built will be executed with the input chosen -from the user's history. In other words, the tools you generate with the -ToolFactory run just like any other Galaxy tool,but run your script every time. - -Tool factory tools are perfect for workflow components. One input, one output, -no variables. - -*To fully and safely exploit the awesome power* of this tool, -Galaxy and the ToolShed, you should be a developer installing this -tool on a private/personal/scratch local instance where you are an -admin_user. Then, if you break it, you get to keep all the pieces see -https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home - -**Installation** -This is a Galaxy tool. You can install it most conveniently using the -administrative "Search and browse tool sheds" link. Find the Galaxy Main -toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory -repository. Open it and review the code and select the option to install it. - -If you can't get the tool that way, the xml and py files here need to be -copied into a new tools -subdirectory such as tools/toolfactory Your tool_conf.xml needs a new entry -pointing to the xml -file - something like:: - - <section name="Tool building tools" id="toolbuilders"> - <tool file="toolfactory/rgToolFactory.xml"/> - </section> - -If not already there, -please add: -<datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary" -mimetype="multipart/x-gzip" subclass="True" /> -to your local data_types_conf.xml. - - -**Restricted execution** - -The tool factory tool itself will then be usable ONLY by admin users - -people with IDs in admin_users in universe_wsgi.ini **Yes, that's right. ONLY -admin_users can run this tool** Think about it for a moment. If allowed to -run any arbitrary script on your Galaxy server, the only thing that would -impede a miscreant bent on destroying all your Galaxy data would probably -be lack of appropriate technical skills. - -**What it does** - -This is a tool factory for simple scripts in python, R and -perl currently. Functional tests are automatically generated. How cool is that. - -LIMITED to simple scripts that read one input from the history. Optionally can -write one new history dataset, and optionally collect any number of outputs -into links on an autogenerated HTML index page for the user to navigate - -useful if the script writes images and output files - pdf outputs are shown -as thumbnails and R's bloated pdf's are shrunk with ghostscript so that and -imagemagik need to be available. - -Generated tools can be edited and enhanced like any Galaxy tool, so start -small and build up since a generated script gets you a serious leg up to a -more complex one. - -**What you do** - -You paste and run your script, you fix the syntax errors and -eventually it runs. You can use the redo button and edit the script before -trying to rerun it as you debug - it works pretty well. - -Once the script works on some test data, you can generate a toolshed compatible -gzip file containing your script ready to run as an ordinary Galaxy tool in -a repository on your local toolshed. That means safe and largely automated -installation in any production Galaxy configured to use your toolshed. - -**Generated tool Security** - -Once you install a generated tool, it's just -another tool - assuming the script is safe. They just run normally and their -user cannot do anything unusually insecure but please, practice safe toolshed. -Read the code before you install any tool. Especially this one - it is really scary. - -**Send Code** - -Patches and suggestions welcome as bitbucket issues please? - -**Attribution** - -Creating re-usable tools from scripts: The Galaxy Tool Factory -Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team -Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573 - -http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref - -**Licensing** - -Copyright Ross Lazarus 2010 -ross lazarus at g mail period com - -All rights reserved. - -Licensed under the LGPL - -**Obligatory screenshot** - -http://bitbucket.org/fubar/galaxytoolmaker/src/fda8032fe989/images/dynamicScriptTool.png - |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/test-data/output2_sample --- a/toolfactory/test-data/output2_sample Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,165 +0,0 @@ -*trats uoy erofeb GNINRAW* - -YLNO yxalaG etavirp a no loot siht llatsnI -ecnatsni noitcudorp ro cilbup a no REVEN esaelP - -troppus noitatic gnidda notlihC nhoJ yb 4102 tsugua detadpU - -keeB ned nav suiraM yb detroper sgub xif ot 4102 8 tsugua detadpU - -ta ecruoser eht etic esaelP -fer=epytyek&JWYdwMrWs1hQzcl=yekji?375stb/tnirper/igc/gro.slanruojdrofxo.scitamrofnioib//:ptth -.krow dehsilbup ruoy ni loot siht esu uoy fi - -**yrotS trohS** - -.sloot yxalaG wen gnitareneg fo elbapac loot yxalaG lausunu na si sihT -gnitpircs suoregnad ylemertxe erofereht dna *detcirtsernu* gnisopxe yb skrow tI -ot meht gniwolla ,revres yxalaG tsoh eht fo srotartsinimda detangised lla ot -,stes atad tupni detceles elpitlum revo lrep dna hs ,nohtyp ,R ni stpircs nur -.tuptuo sa tes atad wen elgnis a gnitirw - -*sretemarap elyts esrapgra ro lanoitisop htiw elbatucexe yna ro tpircs hsab/lrep/nohtyp/r gnikrow a evah uoY* - -.loot yxalaG a gnisu ,setunim ni loot yxalaG yranidro na otni denrut eb nac tI - - -**yxalaG yna otni noitallatsni rof sloot yxalaG wen fo noitareneg detamotuA** - -.ylppus uoy sgnittes retemarap dna stupni atad tset elpmas llams gnisu detareneg si tset A -a dliub ot desu eb nac yeht ,decudorp neeb evah stuptuo esac tset eht ecnO -tnemeriuqer a sa dekab si elbatucexe ro tpircs deilppus ehT .loot yxalaG wen -.xob eht fo tuo elbitapmoc wolfkrow ylluf ,loot yxalaG yranidro ,wen a otni -rotartsinimda na yb dehs loot a aiv dellatsni era sloot detareneG -.sresu ruoy rof sloot yxalaG rehto lla ekil yltcaxe krow dna - -**liateD eroM** - -a otni etsap ot tpircs a deraperp evah dluohs uoy ,yrotcaFlooT eht esu oT -yrotsih ruoy morf tceles ot ydaer elpmaxe tupni tset llams a dna dnim ni egakcap a evah ro ,xob txet -.tpircs wen ruoy tset ot - -em rof skrow ```atad-tset/yrotcafloot/srekam_loot/sloot/yxalag/~ atad_tset-- yxalag/~ toor_yxalag-- lmx.2yrotcaFlooTgr tset omenalp``` - -uoY .mrof yrotcaF looT eht no egaugnal gnitpircs hcae ni elpmaxe na si erehT -thgir eht tceles ot rebmemer - tuo ti yrt ot eseht etsap dna tuc tsuj nac -gnisu tes atad tset llams a etaerc ot deen osla ll'uoY .esaelp reterpretni -.loot atad wen dda yrotsih yxalaG eht - -ni tuptuo loot eht no nottub "oder" eht esu ,wohemos sliaf tpircs eht fI -gub eht xiF .tpircs nekorb htiw etelpmoc mrof eht etaercer ot yrotsih ruoy -.taeper ,hsaw ,esniR .niaga etucexe dna - -tpircs ruoy snur taht loot yxalaG wen a ,yllufssecus snur tpircs eht ecnO -dna txet pleh emos ylppus dna noitpo "etareneg" eht tceleS .detareneg eb nac -epytatad yxalaG wen a fo mrof eht ni detareneg eb lliw loot wen ehT .seman -a ot daolpu ot ydaer evihcra na s'ti ,stseggus eman eht sa - *zg.dehsloot* -.yrotisoper loot wen a sa dehSlooT yxalaG - -revres yxalaG lacol yna otni dellatsni eb nac ti ,dehSlooT a ni s'ti ecnO -.ecafretni evitartsinimda revres eht morf - -tpircs eht ,emit hcae - ti nur nac sresu lacol ,dellatsni si loot wen eht ecnO -nesohc tupni eht htiw detucexe eb lliw tliub saw ti nehw deilppus saw taht -eht htiw etareneg uoy sloot eht ,sdrow rehto nI .yrotsih s'resu eht morf -.emit yreve tpircs ruoy nur tub,loot yxalaG rehto yna ekil tsuj nur yrotcaFlooT - -,tuptuo eno ,tupni enO .stnenopmoc wolfkrow rof tcefrep era sloot yrotcaf looT -.selbairav on - -,loot siht fo *rewop emosewa eht tiolpxe ylefas dna ylluf oT* -siht gnillatsni repoleved a eb dluohs uoy ,dehSlooT eht dna yxalaG -na era uoy erehw ecnatsni lacol hctarcs/lanosrep/etavirp a no loot -ees seceip eht lla peek ot teg uoy ,ti kaerb uoy fi ,nehT .resu_nimda -emoH/ikiw/yrotcaflootyxalag/rabuf/gro.tekcubtib//:sptth - -**noitallatsnI** -eht gnisu yltneinevnoc tsom ti llatsni nac uoY .loot yxalaG a si sihT -niaM yxalaG eht dniF .knil "sdehs loot esworb dna hcraeS" evitartsinimda -yrotcafloot eht rof hcraes dna /ude.usp.xb.2g.dehsloot//:sptth ta dehsloot -.ti llatsni ot noitpo eht tceles dna edoc eht weiver dna ti nepO .yrotisoper - -eb ot deen ereh selif yp dna lmx eht ,yaw taht loot eht teg t'nac uoy fI -sloot wen a otni deipoc -yrtne wen a sdeen lmx.fnoc_loot ruoY yrotcafloot/sloot sa hcus yrotceridbus -lmx eht ot gnitniop -::ekil gnihtemos - elif - ->"sredliubloot"=di "sloot gnidliub looT"=eman noitces< ->/"lmx.yrotcaFlooTgr/yrotcafloot"=elif loot< ->noitces/< - -,ereht ydaerla ton fI -:dda esaelp -"yraniB:yranib.sepytatad.yxalag"=epyt "zg.dehsloot"=noisnetxe epytatad< ->/ "eurT"=ssalcbus "pizg-x/trapitlum"=epytemim -.lmx.fnoc_sepyt_atad lacol ruoy ot - - -**noitucexe detcirtseR** - -- sresu nimda yb YLNO elbasu eb neht lliw flesti loot yrotcaf loot ehT -YLNO .thgir s'taht ,seY** ini.igsw_esrevinu ni sresu_nimda ni sDI htiw elpoep -ot dewolla fI .tnemom a rof ti tuoba knihT **loot siht nur nac sresu_nimda -dluow taht gniht ylno eht ,revres yxalaG ruoy no tpircs yrartibra yna nur -ylbaborp dluow atad yxalaG ruoy lla gniyortsed no tneb tnaercsim a edepmi -.slliks lacinhcet etairporppa fo kcal eb - -**seod ti tahW** - -dna R ,nohtyp ni stpircs elpmis rof yrotcaf loot a si sihT -.taht si looc woH .detareneg yllacitamotua era stset lanoitcnuF .yltnerruc lrep - -nac yllanoitpO .yrotsih eht morf tupni eno daer taht stpircs elpmis ot DETIMIL -stuptuo fo rebmun yna tcelloc yllanoitpo dna ,tesatad yrotsih wen eno etirw -- etagivan ot resu eht rof egap xedni LMTH detarenegotua na no sknil otni -nwohs era stuptuo fdp - selif tuptuo dna segami setirw tpircs eht fi lufesu -dna taht os tpircstsohg htiw knurhs era s'fdp detaolb s'R dna slianbmuht sa -.elbaliava eb ot deen kigamegami - -trats os ,loot yxalaG yna ekil decnahne dna detide eb nac sloot detareneG -a ot pu gel suoires a uoy steg tpircs detareneg a ecnis pu dliub dna llams -.eno xelpmoc erom - -**od uoy tahW** - -dna srorre xatnys eht xif uoy ,tpircs ruoy nur dna etsap uoY -erofeb tpircs eht tide dna nottub oder eht esu nac uoY .snur ti yllautneve -.llew ytterp skrow ti - gubed uoy sa ti nurer ot gniyrt - -elbitapmoc dehsloot a etareneg nac uoy ,atad tset emos no skrow tpircs eht ecnO -ni loot yxalaG yranidro na sa nur ot ydaer tpircs ruoy gniniatnoc elif pizg -detamotua ylegral dna efas snaem tahT .dehsloot lacol ruoy no yrotisoper a -.dehsloot ruoy esu ot derugifnoc yxalaG noitcudorp yna ni noitallatsni - -**ytiruceS loot detareneG** - -tsuj s'ti ,loot detareneg a llatsni uoy ecnO -rieht dna yllamron nur tsuj yehT .efas si tpircs eht gnimussa - loot rehtona -.dehsloot efas ecitcarp ,esaelp tub erucesni yllausunu gnihtyna od tonnac resu -.yracs yllaer si ti - eno siht yllaicepsE .loot yna llatsni uoy erofeb edoc eht daeR - -**edoC dneS** - -?esaelp seussi tekcubtib sa emoclew snoitseggus dna sehctaP - -**noitubirttA** - -yrotcaF looT yxalaG ehT :stpircs morf sloot elbasu-er gnitaerC -maeT yxalaG ehT ;nnameiZ kraM ;ipsaK ynotnA ;surazaL ssoR -375stb/scitamrofnioib/3901.01 :iod ;2102 scitamrofnioiB - -fer=epytyek&JWYdwMrWs1hQzcl=yekji?375stb/tnirper/igc/gro.slanruojdrofxo.scitamrofnioib//:ptth - -**gnisneciL** - -0102 surazaL ssoR thgirypoC -moc doirep liam g ta surazal ssor - -.devreser sthgir llA - -LPGL eht rednu desneciL - -**tohsneercs yrotagilbO** - -gnp.looTtpircScimanyd/segami/989ef2308adf/crs/rekamlootyxalag/rabuf/gro.tekcubtib//:ptth |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/test-data/pyrevpos.python --- a/toolfactory/test-data/pyrevpos.python Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| [ |
| @@ -1,13 +0,0 @@ -# reverse order of text by row -import sys -inp = sys.argv[1] -outp = sys.argv[2] -i = open(inp,'r').readlines() -o = open(outp,'w') -for row in i: - rs = row.rstrip() - rs = list(rs) - rs.reverse() - o.write(''.join(rs)) -o.close() - |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/test-data/test1_log.txt --- a/toolfactory/test-data/test1_log.txt Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,1 +0,0 @@ -## Executing Toolfactory generated command line = python /tmp/pyrevposq5dmcdy1.python /tmp/tmpqrksf8sd/files/5/b/9/dataset_5b952a86-87df-44ad-a415-ea549f3f0cee.dat output2 |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/test-data/toolfactory_pyrevpos_tgz_sample |
| b |
| Binary file toolfactory/test-data/toolfactory_pyrevpos_tgz_sample has changed |
| b |
| diff -r 8ea1133b9d9a -r 0c6c3e10a8f4 toolfactory/whoosh.sh --- a/toolfactory/whoosh.sh Tue Jan 05 00:34:48 2021 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| [ |
| @@ -1,16 +0,0 @@ -# using the galaxy venv this seems to work -. /galaxy_venv/bin/activate -python scripts/tool_shed/build_ts_whoosh_index.py -c config/tool_shed.yml --config-section tool_shed -# eeesh. /etc/galaxy is where the actual galaxy.yml lives - despite other configs being where they might be expected -# fix tool_shed.yml to 0.0.0.0 and admin email -# add tgz to datatypes :( -# need to source a venv in /export/tool_deps/toolfactorydeps/0.01/fubar/toolfactorydeps/9e9428fe9134/env.sh -# as nothing was done by the setup_virtualenv step apparently. -# gcc and friends for planemo pip installation -# File "/galaxy-central/lib/galaxy/tool_util/verify/interactor.py", line 595, in <listcomp> -# test_user = [user for user in all_users if user["email"] == email][0] -# add local to tool_sheds_conf.xml -# <tool_sheds> -# <tool_shed name="local" url="http://localhost:9009"/> -# mercurial > 5.5!! - |
