| Previous changeset 2:abdd608c869b (2013-04-24) Next changeset 4:1c36cf8ef133 (2014-07-30) |
|
Commit message:
Uploaded v0.0.6, automatic dependency on Biopython 1.62, new README file, citation information, MIT licence |
|
added:
tools/seq_filter_by_id/README.rst tools/seq_filter_by_id/repository_dependencies.xml tools/seq_filter_by_id/seq_filter_by_id.py tools/seq_filter_by_id/seq_filter_by_id.xml |
|
removed:
tools/filters/seq_filter_by_id.py tools/filters/seq_filter_by_id.txt tools/filters/seq_filter_by_id.xml |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/filters/seq_filter_by_id.py --- a/tools/filters/seq_filter_by_id.py Wed Apr 24 11:34:12 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| [ |
| b'@@ -1,233 +0,0 @@\n-#!/usr/bin/env python\n-"""Filter a FASTA, FASTQ or SSF file with IDs from a tabular file.\n-\n-Takes six command line options, tabular filename, ID column numbers (comma\n-separated list using one based counting), input filename, input type (e.g.\n-FASTA or SFF) and two output filenames (for records with and without the\n-given IDs, same format as input sequence file).\n-\n-If either output filename is just a minus sign, that file is not created.\n-This is intended to allow output for just the matched (or just the non-matched)\n-records.\n-\n-When filtering an SFF file, any Roche XML manifest in the input file is\n-preserved in both output files.\n-\n-Note in the default NCBI BLAST+ tabular output, the query sequence ID is\n-in column one, and the ID of the match from the database is in column two.\n-Here sensible values for the column numbers would therefore be "1" or "2".\n-\n-This tool is a short Python script which requires Biopython 1.54 or later\n-for SFF file support. If you use this tool in scientific work leading to a\n-publication, please cite the Biopython application note:\n-\n-Cock et al 2009. Biopython: freely available Python tools for computational\n-molecular biology and bioinformatics. Bioinformatics 25(11) 1422-3.\n-http://dx.doi.org/10.1093/bioinformatics/btp163 pmid:19304878.\n-\n-This script is copyright 2010-2013 by Peter Cock, The James Hutton Institute\n-(formerly the Scottish Crop Research Institute, SCRI), UK. All rights reserved.\n-See accompanying text file for licence details (MIT/BSD style).\n-\n-This is version 0.0.5 of the script, use -v or --version to get the version.\n-"""\n-import sys\n-\n-def stop_err(msg, err=1):\n- sys.stderr.write(msg.rstrip() + "\\n")\n- sys.exit(err)\n-\n-if "-v" in sys.argv or "--version" in sys.argv:\n- print "v0.0.5"\n- sys.exit(0)\n-\n-#Parse Command Line\n-try:\n- tabular_file, cols_arg, in_file, seq_format, out_positive_file, out_negative_file = sys.argv[1:]\n-except ValueError:\n- stop_err("Expected six arguments (tab file, columns, in seq, seq format, out pos, out neg), got %i:\\n%s" % (len(sys.argv)-1, " ".join(sys.argv)))\n-try:\n- columns = [int(arg)-1 for arg in cols_arg.split(",")]\n-except ValueError:\n- stop_err("Expected list of columns (comma separated integers), got %s" % cols_arg)\n-if min(columns) < 0:\n- stop_err("Expect one-based column numbers (not zero-based counting), got %s" % cols_arg)\n-\n-if out_positive_file == "-" and out_negative_file == "-":\n- stop_err("Neither output file requested")\n-\n-\n-#Read tabular file and record all specified identifiers\n-ids = set()\n-handle = open(tabular_file, "rU")\n-if len(columns)>1:\n- #General case of many columns\n- for line in handle:\n- if line.startswith("#"):\n- #Ignore comments\n- continue\n- parts = line.rstrip("\\n").split("\\t")\n- for col in columns:\n- ids.add(parts[col])\n- print "Using %i IDs from %i columns of tabular file" % (len(ids), len(columns))\n-else:\n- #Single column, special case speed up\n- col = columns[0]\n- for line in handle:\n- if not line.startswith("#"):\n- ids.add(line.rstrip("\\n").split("\\t")[col])\n- print "Using %i IDs from tabular file" % (len(ids))\n-handle.close()\n-\n-\n-def crude_fasta_iterator(handle):\n- """Yields tuples, record ID and the full record as a string."""\n- while True:\n- line = handle.readline()\n- if line == "":\n- return # Premature end of file, or just empty?\n- if line[0] == ">":\n- break\n-\n- no_id_warned = False\n- while True:\n- if line[0] != ">":\n- raise ValueError(\n- "Records in Fasta files should start with \'>\' character")\n- try:\n- id = line[1:].split(None, 1)[0]\n- except IndexError:\n- if not no_id_warned:\n- sys.stderr.write("WARNING - Malformed FASTA entry with no identifier\\n")\n- no_id_warned = True\n- id = None\n- li'..b' if identifier in wanted:\n- pos_count += 1\n- else:\n- neg_handle.write(record)\n- neg_count += 1\n- return pos_count, neg_count\n-\n-\n-if seq_format.lower()=="sff":\n- #Now write filtered SFF file based on IDs from BLAST file\n- try:\n- from Bio.SeqIO.SffIO import SffIterator, SffWriter\n- except ImportError:\n- stop_err("SFF filtering requires Biopython 1.54 or later")\n-\n- try:\n- from Bio.SeqIO.SffIO import ReadRocheXmlManifest\n- except ImportError:\n- #Prior to Biopython 1.56 this was a private function\n- from Bio.SeqIO.SffIO import _sff_read_roche_index_xml as ReadRocheXmlManifest\n- in_handle = open(in_file, "rb") #must be binary mode!\n- try:\n- manifest = ReadRocheXmlManifest(in_handle)\n- except ValueError:\n- manifest = None\n- #This makes two passes though the SFF file with isn\'t so efficient,\n- #but this makes the code simple.\n- pos_count = neg_count = 0\n- if out_positive_file != "-":\n- out_handle = open(out_positive_file, "wb")\n- writer = SffWriter(out_handle, xml=manifest)\n- in_handle.seek(0) #start again after getting manifest\n- pos_count = writer.write_file(rec for rec in SffIterator(in_handle) if rec.id in ids)\n- out_handle.close()\n- if out_negative_file != "-":\n- out_handle = open(out_negative_file, "wb")\n- writer = SffWriter(out_handle, xml=manifest)\n- in_handle.seek(0) #start again\n- neg_count = writer.write_file(rec for rec in SffIterator(in_handle) if rec.id not in ids)\n- out_handle.close()\n- #And we\'re done\n- in_handle.close()\n- #At the time of writing, Galaxy doesn\'t show SFF file read counts,\n- #so it is useful to put them in stdout and thus shown in job info.\n- print "%i with and %i without specified IDs" % (pos_count, neg_count)\n-elif seq_format.lower()=="fasta":\n- #Write filtered FASTA file based on IDs from tabular file\n- pos_count, neg_count = fasta_filter(in_file, out_positive_file, out_negative_file, ids)\n- print "%i with and %i without specified IDs" % (pos_count, neg_count)\n-elif seq_format.lower().startswith("fastq"):\n- #Write filtered FASTQ file based on IDs from tabular file\n- from galaxy_utils.sequence.fastq import fastqReader, fastqWriter\n- reader = fastqReader(open(in_file, "rU"))\n- if out_positive_file != "-" and out_negative_file != "-":\n- print "Generating two FASTQ files"\n- positive_writer = fastqWriter(open(out_positive_file, "w"))\n- negative_writer = fastqWriter(open(out_negative_file, "w"))\n- for record in reader:\n- #The [1:] is because the fastaReader leaves the > on the identifier.\n- if record.identifier and record.identifier.split()[0][1:] in ids:\n- positive_writer.write(record)\n- else:\n- negative_writer.write(record)\n- positive_writer.close()\n- negative_writer.close()\n- elif out_positive_file != "-":\n- print "Generating matching FASTQ file"\n- positive_writer = fastqWriter(open(out_positive_file, "w"))\n- for record in reader:\n- #The [1:] is because the fastaReader leaves the > on the identifier.\n- if record.identifier and record.identifier.split()[0][1:] in ids:\n- positive_writer.write(record)\n- positive_writer.close()\n- elif out_negative_file != "-":\n- print "Generating non-matching FASTQ file"\n- negative_writer = fastqWriter(open(out_negative_file, "w"))\n- for record in reader:\n- #The [1:] is because the fastaReader leaves the > on the identifier.\n- if not record.identifier or record.identifier.split()[0][1:] not in ids:\n- negative_writer.write(record)\n- negative_writer.close()\n- reader.close()\n-else:\n- stop_err("Unsupported file type %r" % seq_format)\n' |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/filters/seq_filter_by_id.txt --- a/tools/filters/seq_filter_by_id.txt Wed Apr 24 11:34:12 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| b |
| @@ -1,89 +0,0 @@ -Galaxy tool to filter FASTA, FASTQ or SFF sequences by ID -========================================================= - -This tool is copyright 2010-2013 by Peter Cock, The James Hutton Institute -(formerly SCRI, Scottish Crop Research Institute), UK. All rights reserved. -See the licence text below. - -This tool is a short Python script (using both the Galaxy and Biopython library -functions) which divides a FASTA, FASTQ, or SFF file in two, those sequences with -or without an ID present in the specified column(s) of a tabular file. Example uses -include filtering based on search results from a tool like NCBI BLAST before -assembly. - - -Installation -============ - -There are just two files to install: - -* seq_filter_by_id.py (the Python script) -* seq_filter_by_id.xml (the Galaxy tool definition) - -The suggested location is in the Galaxy folder tools/filters next to the tool -for calling sff_extract.py for converting SFF to FASTQ or FASTA + QUAL. - -You will also need to modify the tools_conf.xml file to tell Galaxy to offer the -tool. One suggested location is in the filters section. Simply add the line: - -<tool file="filters/sff_filter_by_id.xml" /> - -You will also need to install Biopython 1.54 or later. That's it. - - -History -======= - -v0.0.1 - Initial version, combining three separate scripts for each file format. -v0.0.4 - Record script version when run from Galaxy. - - Faster FASTA code which preserves the original line wrapping. - - Basic unit test included. -v0.0.5 - Check for errors using Python script's return code. - - Cope with malformed FASTA entries without an identifier. - - -Developers -========== - -This script and related tools are being developed on the following hg branch: -http://bitbucket.org/peterjc/galaxy-central/src/tools - -This incorporates the previously used hg branch: -http://bitbucket.org/peterjc/galaxy-central/src/fasta_filter - -For making the "Galaxy Tool Shed" http://toolshed.g2.bx.psu.edu/ tarball use -the following command from the Galaxy root folder: - -$ tar -czf seq_filter_by_id.tar.gz tools/filters/seq_filter_by_id.* test-data/k12_ten_proteins.fasta test-data/k12_hypothetical.fasta test-data/k12_hypothetical.tabular - -Check this worked: - -$ tar -tzf seq_filter_by_id.tar.gz -filter/seq_filter_by_id.py -filter/seq_filter_by_id.txt -filter/seq_filter_by_id.xml -test-data/k12_ten_proteins.fasta -test-data/k12_hypothetical.fasta -test-data/k12_hypothetical.tabular - - -Licence (MIT/BSD style) -======================= - -Permission to use, copy, modify, and distribute this software and its -documentation with or without modifications and for any purpose and -without fee is hereby granted, provided that any copyright notices -appear in all copies and that both those copyright notices and this -permission notice appear in supporting documentation, and that the -names of the contributors or copyright holders not be used in -advertising or publicity pertaining to distribution of the software -without specific prior permission. - -THE CONTRIBUTORS AND COPYRIGHT HOLDERS OF THIS SOFTWARE DISCLAIM ALL -WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED -WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL THE -CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY SPECIAL, INDIRECT -OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS -OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE -OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE -OR PERFORMANCE OF THIS SOFTWARE. |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/filters/seq_filter_by_id.xml --- a/tools/filters/seq_filter_by_id.xml Wed Apr 24 11:34:12 2013 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 |
| [ |
| @@ -1,113 +0,0 @@ -<tool id="seq_filter_by_id" name="Filter sequences by ID" version="0.0.5"> - <description>from a tabular file</description> - <version_command interpreter="python">seq_filter_by_id.py --version</version_command> - <command interpreter="python"> -seq_filter_by_id.py $input_tabular $columns $input_file $input_file.ext -#if $output_choice_cond.output_choice=="both" - $output_pos $output_neg -#elif $output_choice_cond.output_choice=="pos" - $output_pos - -#elif $output_choice_cond.output_choice=="neg" - - $output_neg -#end if - </command> - <stdio> - <!-- Anything other than zero is an error --> - <exit_code range="1:" /> - <exit_code range=":-1" /> - </stdio> - <inputs> - <param name="input_file" type="data" format="fasta,fastq,sff" label="Sequence file to filter on the identifiers" help="FASTA, FASTQ, or SFF format." /> - <param name="input_tabular" type="data" format="tabular" label="Tabular file containing sequence identifiers"/> - <param name="columns" type="data_column" data_ref="input_tabular" multiple="True" numerical="False" label="Column(s) containing sequence identifiers" help="Multi-select list - hold the appropriate key while clicking to select multiple columns"> - <validator type="no_options" message="Pick at least one column"/> - </param> - <conditional name="output_choice_cond"> - <param name="output_choice" type="select" label="Output positive matches, negative matches, or both?"> - <option value="both">Both positive matches (ID on list) and negative matches (ID not on list), as two files</option> - <option value="pos">Just positive matches (ID on list), as a single file</option> - <option value="neg">Just negative matches (ID not on list), as a single file</option> - </param> - <!-- Seems need these dummy entries here, compare this to indels/indel_sam2interval.xml --> - <when value="both" /> - <when value="pos" /> - <when value="neg" /> - </conditional> - </inputs> - <outputs> - <data name="output_pos" format="fasta" label="With matched ID"> - <!-- TODO - Replace this with format="input:input_fastq" if/when that works --> - <change_format> - <when input_dataset="input_file" attribute="extension" value="sff" format="sff" /> - <when input_dataset="input_file" attribute="extension" value="fastq" format="fastq" /> - <when input_dataset="input_file" attribute="extension" value="fastqsanger" format="fastqsanger" /> - <when input_dataset="input_file" attribute="extension" value="fastqsolexa" format="fastqsolexa" /> - <when input_dataset="input_file" attribute="extension" value="fastqillumina" format="fastqillumina" /> - <when input_dataset="input_file" attribute="extension" value="fastqcssanger" format="fastqcssanger" /> - </change_format> - <filter>output_choice_cond["output_choice"] != "neg"</filter> - </data> - <data name="output_neg" format="fasta" label="Without matched ID"> - <!-- TODO - Replace this with format="input:input_fastq" if/when that works --> - <change_format> - <when input_dataset="input_file" attribute="extension" value="sff" format="sff" /> - <when input_dataset="input_file" attribute="extension" value="fastq" format="fastq" /> - <when input_dataset="input_file" attribute="extension" value="fastqsanger" format="fastqsanger" /> - <when input_dataset="input_file" attribute="extension" value="fastqsolexa" format="fastqsolexa" /> - <when input_dataset="input_file" attribute="extension" value="fastqillumina" format="fastqillumina" /> - <when input_dataset="input_file" attribute="extension" value="fastqcssanger" format="fastqcssanger" /> - </change_format> - <filter>output_choice_cond["output_choice"] != "pos"</filter> - </data> - </outputs> - <tests> - <test> - <param name="input_file" value="k12_ten_proteins.fasta" ftype="fasta" /> - <param name="input_tabular" value="k12_hypothetical.tabular" ftype="tabular" /> - <param name="columns" value="1" /> - <param name="output_choice" value="pos" /> - <output name="output_pos" file="k12_hypothetical.fasta" ftype="fasta" /> - </test> - </tests> - <requirements> - <requirement type="python-module">Bio</requirement> - </requirements> - <help> - -**What it does** - -By default it divides a FASTA, FASTQ or Standard Flowgram Format (SFF) file in -two, those sequences with or without an ID present in the tabular file column(s) -specified. You can opt to have a single output file of just the matching records, -or just the non-matching ones. - -Note that the order of sequences in the original sequence file is preserved, as -is any Roche XML Manifest in an SFF file. Also, if any sequences share an -identifier (which would be very unusual in SFF files), duplicates are not removed. - -**Example Usage** - -You may have performed some kind of contamination search, for example running -BLASTN against a database of cloning vectors or bacteria, giving you a tabular -file containing read identifiers. You could use this tool to extract only the -reads without BLAST matches (i.e. those which do not match your contaminant -database). - -You may have a file of FASTA sequences which has been used with some analysis -tool giving tabular output, which has then been filtered on some criteria. -You can then use this tool to divide the original FASTA file into those entries -matching or not matching your criteria (those with or without their identifier -in the filtered tabular file). - -**Citation** - -This tool uses Biopython to read and write SFF files. If you use this tool in -scientific work leading to a publication, please cite the Biopython application -note (and Galaxy too of course): - -Cock et al 2009. Biopython: freely available Python tools for computational -molecular biology and bioinformatics. Bioinformatics 25(11) 1422-3. -http://dx.doi.org/10.1093/bioinformatics/btp163 pmid:19304878. - - </help> -</tool> |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/seq_filter_by_id/README.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/seq_filter_by_id/README.rst Fri Oct 11 04:37:12 2013 -0400 |
| b |
| @@ -0,0 +1,124 @@ +Galaxy tool to filter FASTA, FASTQ or SFF sequences by ID +========================================================= + +This tool is copyright 2010-2013 by Peter Cock, The James Hutton Institute +(formerly SCRI, Scottish Crop Research Institute), UK. All rights reserved. +See the licence text below. + +This tool is a short Python script (using both the Galaxy and Biopython library +functions) which divides a FASTA, FASTQ, or SFF file in two, those sequences with +or without an ID present in the specified column(s) of a tabular file. Example uses +include filtering based on search results from a tool like NCBI BLAST before +assembly. + +This tool is available from the Galaxy Tool Shed at: + +* http://toolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id + +See also sister tools: + +* http://toolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id +* http://toolshed.g2.bx.psu.edu/view/peterjc/seq_rename + + +Automated Installation +====================== + +This should be straightforward using the Galaxy Tool Shed, which should be +able to automatically install the dependency on Biopython, and then install +this tool and run its unit tests. + + +Manual Installation +=================== + +There are just two files to install to use this tool from within Galaxy: + +* seq_filter_by_id.py (the Python script) +* seq_filter_by_id.xml (the Galaxy tool definition) + +The suggested location is a dedicated tools/seq_filter_by_id folder. + +You will also need to modify the tools_conf.xml file to tell Galaxy to offer the +tool. One suggested location is in the filters section. Simply add the line:: + + <tool file="seq_filter_by_id/seq_filter_by_id.xml" /> + +If you wish to run the unit tests, also add this to tools_conf.xml.sample +and move/copy the test-data files under Galaxy's test-data folder. Then:: + + $ ./run_functional_tests.sh -id seq_filter_by_id + +You will also need to install Biopython 1.54 or later. That's it. + + +History +======= + +======= ====================================================================== +Version Changes +------- ---------------------------------------------------------------------- +v0.0.1 - Initial version, combining three separate scripts for each file format. +v0.0.4 - Record script version when run from Galaxy. + - Faster FASTA code which preserves the original line wrapping. + - Basic unit test included. +v0.0.5 - Check for errors using Python script's return code. + - Cope with malformed FASTA entries without an identifier. +v0.0.6 - Link to Tool Shed added to help text and this documentation. + - Automated installation of the Biopython dependency. + - Use reStructuredText for this README file. + - Adopt standard MIT License. + - Updated citation information (Cock et al. 2013). + - Development moved to GitHub, https://github.com/peterjc/pico_galaxy + - Renamed folder and adopted README.rst naming. +======= ====================================================================== + + + +Developers +========== + +This script and related tools were initially developed on the following hg branches: +http://bitbucket.org/peterjc/galaxy-central/src/fasta_filter +http://bitbucket.org/peterjc/galaxy-central/src/tools + +Development has now moved to a dedicated GitHub repository: +https://github.com/peterjc/pico_galaxy/tree/master/tools + +For making the "Galaxy Tool Shed" http://toolshed.g2.bx.psu.edu/ tarball use +the following command from the Galaxy root folder:: + + $ tar -czf seq_filter_by_id.tar.gz tools/seq_filter_by_id/README.rst tools/seq_filter_by_id/seq_filter_by_id.* tools/seq_filter_by_id/repository_dependencies.xml test-data/k12_ten_proteins.fasta test-data/k12_hypothetical.fasta test-data/k12_hypothetical.tabular + +Check this worked:: + + $ tar -tzf seq_filter_by_id.tar.gz + tools/seq_filter_by_id/README.rst + tools/seq_filter_by_id/seq_filter_by_id.py + tools/seq_filter_by_id/seq_filter_by_id.xml + tools/seq_filter_by_id/repository_dependencies.xml + test-data/k12_ten_proteins.fasta + test-data/k12_hypothetical.fasta + test-data/k12_hypothetical.tabular + + +Licence (MIT) +============= + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/seq_filter_by_id/repository_dependencies.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/seq_filter_by_id/repository_dependencies.xml Fri Oct 11 04:37:12 2013 -0400 |
| b |
| @@ -0,0 +1,6 @@ +<?xml version="1.0"?> +<repositories description="This requires Biopython as a dependency."> +<!-- Leave out the tool shed and revision to get the current + tool shed and latest revision at the time of upload --> +<repository changeset_revision="3e82cbc44886" name="package_biopython_1_62" owner="biopython" toolshed="http://toolshed.g2.bx.psu.edu" /> +</repositories> |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/seq_filter_by_id/seq_filter_by_id.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/seq_filter_by_id/seq_filter_by_id.py Fri Oct 11 04:37:12 2013 -0400 |
| [ |
| b'@@ -0,0 +1,260 @@\n+#!/usr/bin/env python\n+"""Filter a FASTA, FASTQ or SSF file with IDs from a tabular file.\n+\n+Takes six command line options, tabular filename, ID column numbers (comma\n+separated list using one based counting), input filename, input type (e.g.\n+FASTA or SFF) and two output filenames (for records with and without the\n+given IDs, same format as input sequence file).\n+\n+If either output filename is just a minus sign, that file is not created.\n+This is intended to allow output for just the matched (or just the non-matched)\n+records.\n+\n+When filtering an SFF file, any Roche XML manifest in the input file is\n+preserved in both output files.\n+\n+Note in the default NCBI BLAST+ tabular output, the query sequence ID is\n+in column one, and the ID of the match from the database is in column two.\n+Here sensible values for the column numbers would therefore be "1" or "2".\n+\n+This tool is a short Python script which requires Biopython 1.54 or later\n+for SFF file support. If you use this tool in scientific work leading to a\n+publication, please cite the Biopython application note:\n+\n+Cock et al 2009. Biopython: freely available Python tools for computational\n+molecular biology and bioinformatics. Bioinformatics 25(11) 1422-3.\n+http://dx.doi.org/10.1093/bioinformatics/btp163 pmid:19304878.\n+\n+This script is copyright 2010-2013 by Peter Cock, The James Hutton Institute\n+(formerly the Scottish Crop Research Institute, SCRI), UK. All rights reserved.\n+See accompanying text file for licence details (MIT license).\n+\n+This is version 0.1.0 of the script, use -v or --version to get the version.\n+"""\n+import os\n+import sys\n+\n+def stop_err(msg, err=1):\n+ sys.stderr.write(msg.rstrip() + "\\n")\n+ sys.exit(err)\n+\n+if "-v" in sys.argv or "--version" in sys.argv:\n+ print "v0.1.0"\n+ sys.exit(0)\n+\n+#Parse Command Line\n+if len(sys.argv) - 1 < 7 or len(sys.argv) % 2 == 1:\n+ stop_err("Expected 7 or more arguments, 5 required "\n+ "(in seq, seq format, out pos, out neg, logic) "\n+ "then one or more pairs (tab file, columns), "\n+ "got %i:\\n%s" % (len(sys.argv)-1, " ".join(sys.argv)))\n+\n+in_file, seq_format, out_positive_file, out_negative_file, logic = sys.argv[1:6]\n+\n+if not os.path.isfile(in_file):\n+ stop_err("Missing input file %r" % in_file)\n+if out_positive_file == "-" and out_negative_file == "-":\n+ stop_err("Neither output file requested")\n+if logic not in ["UNION", "INTERSECTION"]:\n+ stop_err("Fifth agrument should be \'UNION\' or \'INTERSECTION\', not %r" % logic)\n+\n+identifiers = []\n+for i in range((len(sys.argv) - 6) // 2):\n+ tabular_file = sys.argv[6+2*i]\n+ cols_arg = sys.argv[7+2*i]\n+ if not os.path.isfile(tabular_file):\n+ stop_err("Missing tabular identifier file %r" % tabular_file)\n+ try:\n+ columns = [int(arg)-1 for arg in cols_arg.split(",")]\n+ except ValueError:\n+ stop_err("Expected list of columns (comma separated integers), got %r" % cols_arg)\n+ if min(columns) < 0:\n+ stop_err("Expect one-based column numbers (not zero-based counting), got %r" % cols_arg)\n+ identifiers.append((tabular_file, columns))\n+\n+#Read tabular file(s) and record all specified identifiers\n+ids = None #Will be a set\n+for tabular_file, columns in identifiers:\n+ file_ids = set()\n+ handle = open(tabular_file, "rU")\n+ if len(columns)>1:\n+ #General case of many columns\n+ for line in handle:\n+ if line.startswith("#"):\n+ #Ignore comments\n+ continue\n+ parts = line.rstrip("\\n").split("\\t")\n+ for col in columns:\n+ file_ids.add(parts[col])\n+ else:\n+ #Single column, special case speed up\n+ col = columns[0]\n+ for line in handle:\n+ if not line.startswith("#"):\n+ file_ids.add(line.rstrip("\\n").split("\\t")[col])\n+ print "Using %i IDs from column %s in tabular file" % (len(file_ids), ", ".join(str(col+1) for col in col'..b' if identifier in wanted:\n+ pos_count += 1\n+ else:\n+ neg_handle.write(record)\n+ neg_count += 1\n+ return pos_count, neg_count\n+\n+\n+if seq_format.lower()=="sff":\n+ #Now write filtered SFF file based on IDs from BLAST file\n+ try:\n+ from Bio.SeqIO.SffIO import SffIterator, SffWriter\n+ except ImportError:\n+ stop_err("SFF filtering requires Biopython 1.54 or later")\n+\n+ try:\n+ from Bio.SeqIO.SffIO import ReadRocheXmlManifest\n+ except ImportError:\n+ #Prior to Biopython 1.56 this was a private function\n+ from Bio.SeqIO.SffIO import _sff_read_roche_index_xml as ReadRocheXmlManifest\n+ in_handle = open(in_file, "rb") #must be binary mode!\n+ try:\n+ manifest = ReadRocheXmlManifest(in_handle)\n+ except ValueError:\n+ manifest = None\n+ #This makes two passes though the SFF file with isn\'t so efficient,\n+ #but this makes the code simple.\n+ pos_count = neg_count = 0\n+ if out_positive_file != "-":\n+ out_handle = open(out_positive_file, "wb")\n+ writer = SffWriter(out_handle, xml=manifest)\n+ in_handle.seek(0) #start again after getting manifest\n+ pos_count = writer.write_file(rec for rec in SffIterator(in_handle) if rec.id in ids)\n+ out_handle.close()\n+ if out_negative_file != "-":\n+ out_handle = open(out_negative_file, "wb")\n+ writer = SffWriter(out_handle, xml=manifest)\n+ in_handle.seek(0) #start again\n+ neg_count = writer.write_file(rec for rec in SffIterator(in_handle) if rec.id not in ids)\n+ out_handle.close()\n+ #And we\'re done\n+ in_handle.close()\n+ #At the time of writing, Galaxy doesn\'t show SFF file read counts,\n+ #so it is useful to put them in stdout and thus shown in job info.\n+ print "%i with and %i without specified IDs" % (pos_count, neg_count)\n+elif seq_format.lower()=="fasta":\n+ #Write filtered FASTA file based on IDs from tabular file\n+ pos_count, neg_count = fasta_filter(in_file, out_positive_file, out_negative_file, ids)\n+ print "%i with and %i without specified IDs" % (pos_count, neg_count)\n+elif seq_format.lower().startswith("fastq"):\n+ #Write filtered FASTQ file based on IDs from tabular file\n+ from galaxy_utils.sequence.fastq import fastqReader, fastqWriter\n+ reader = fastqReader(open(in_file, "rU"))\n+ if out_positive_file != "-" and out_negative_file != "-":\n+ print "Generating two FASTQ files"\n+ positive_writer = fastqWriter(open(out_positive_file, "w"))\n+ negative_writer = fastqWriter(open(out_negative_file, "w"))\n+ for record in reader:\n+ #The [1:] is because the fastaReader leaves the > on the identifier.\n+ if record.identifier and record.identifier.split()[0][1:] in ids:\n+ positive_writer.write(record)\n+ else:\n+ negative_writer.write(record)\n+ positive_writer.close()\n+ negative_writer.close()\n+ elif out_positive_file != "-":\n+ print "Generating matching FASTQ file"\n+ positive_writer = fastqWriter(open(out_positive_file, "w"))\n+ for record in reader:\n+ #The [1:] is because the fastaReader leaves the > on the identifier.\n+ if record.identifier and record.identifier.split()[0][1:] in ids:\n+ positive_writer.write(record)\n+ positive_writer.close()\n+ elif out_negative_file != "-":\n+ print "Generating non-matching FASTQ file"\n+ negative_writer = fastqWriter(open(out_negative_file, "w"))\n+ for record in reader:\n+ #The [1:] is because the fastaReader leaves the > on the identifier.\n+ if not record.identifier or record.identifier.split()[0][1:] not in ids:\n+ negative_writer.write(record)\n+ negative_writer.close()\n+ reader.close()\n+else:\n+ stop_err("Unsupported file type %r" % seq_format)\n' |
| b |
| diff -r abdd608c869b -r 44ab4c0f7683 tools/seq_filter_by_id/seq_filter_by_id.xml --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/seq_filter_by_id/seq_filter_by_id.xml Fri Oct 11 04:37:12 2013 -0400 |
| [ |
| @@ -0,0 +1,125 @@ +<tool id="seq_filter_by_id" name="Filter sequences by ID" version="0.0.6"> + <description>from a tabular file</description> + <requirements> + <requirement type="package" version="1.62">biopython</requirement> + <requirement type="python-module">Bio</requirement> + </requirements> + <version_command interpreter="python">seq_filter_by_id.py --version</version_command> + <command interpreter="python"> +seq_filter_by_id.py "$input_file" "$input_file.ext" +#if $output_choice_cond.output_choice=="both" + $output_pos $output_neg +#elif $output_choice_cond.output_choice=="pos" + $output_pos - +#elif $output_choice_cond.output_choice=="neg" + - $output_neg +#end if +## TODO - Decide on best way to expose multiple ID files via the XML wrapper. +## Single tabular file, can call the Python script with either UNION or INTERSECTION +UNION "$input_tabular" "$columns" + </command> + <stdio> + <!-- Anything other than zero is an error --> + <exit_code range="1:" /> + <exit_code range=":-1" /> + </stdio> + <inputs> + <param name="input_file" type="data" format="fasta,fastq,sff" label="Sequence file to filter on the identifiers" help="FASTA, FASTQ, or SFF format." /> + <param name="input_tabular" type="data" format="tabular" label="Tabular file containing sequence identifiers"/> + <param name="columns" type="data_column" data_ref="input_tabular" multiple="True" numerical="False" label="Column(s) containing sequence identifiers" help="Multi-select list - hold the appropriate key while clicking to select multiple columns"> + <validator type="no_options" message="Pick at least one column"/> + </param> + <conditional name="output_choice_cond"> + <param name="output_choice" type="select" label="Output positive matches, negative matches, or both?"> + <option value="both">Both positive matches (ID on list) and negative matches (ID not on list), as two files</option> + <option value="pos">Just positive matches (ID on list), as a single file</option> + <option value="neg">Just negative matches (ID not on list), as a single file</option> + </param> + <!-- Seems need these dummy entries here, compare this to indels/indel_sam2interval.xml --> + <when value="both" /> + <when value="pos" /> + <when value="neg" /> + </conditional> + </inputs> + <outputs> + <data name="output_pos" format="fasta" label="With matched ID"> + <!-- TODO - Replace this with format="input:input_fastq" if/when that works --> + <change_format> + <when input_dataset="input_file" attribute="extension" value="sff" format="sff" /> + <when input_dataset="input_file" attribute="extension" value="fastq" format="fastq" /> + <when input_dataset="input_file" attribute="extension" value="fastqsanger" format="fastqsanger" /> + <when input_dataset="input_file" attribute="extension" value="fastqsolexa" format="fastqsolexa" /> + <when input_dataset="input_file" attribute="extension" value="fastqillumina" format="fastqillumina" /> + <when input_dataset="input_file" attribute="extension" value="fastqcssanger" format="fastqcssanger" /> + </change_format> + <filter>output_choice_cond["output_choice"] != "neg"</filter> + </data> + <data name="output_neg" format="fasta" label="Without matched ID"> + <!-- TODO - Replace this with format="input:input_fastq" if/when that works --> + <change_format> + <when input_dataset="input_file" attribute="extension" value="sff" format="sff" /> + <when input_dataset="input_file" attribute="extension" value="fastq" format="fastq" /> + <when input_dataset="input_file" attribute="extension" value="fastqsanger" format="fastqsanger" /> + <when input_dataset="input_file" attribute="extension" value="fastqsolexa" format="fastqsolexa" /> + <when input_dataset="input_file" attribute="extension" value="fastqillumina" format="fastqillumina" /> + <when input_dataset="input_file" attribute="extension" value="fastqcssanger" format="fastqcssanger" /> + </change_format> + <filter>output_choice_cond["output_choice"] != "pos"</filter> + </data> + </outputs> + <tests> + <test> + <param name="input_file" value="k12_ten_proteins.fasta" ftype="fasta" /> + <param name="input_tabular" value="k12_hypothetical.tabular" ftype="tabular" /> + <param name="columns" value="1" /> + <param name="output_choice" value="pos" /> + <output name="output_pos" file="k12_hypothetical.fasta" ftype="fasta" /> + </test> + </tests> + <help> +**What it does** + +By default it divides a FASTA, FASTQ or Standard Flowgram Format (SFF) file in +two, those sequences with or without an ID present in the tabular file column(s) +specified. You can opt to have a single output file of just the matching records, +or just the non-matching ones. + +Note that the order of sequences in the original sequence file is preserved, as +is any Roche XML Manifest in an SFF file. Also, if any sequences share an +identifier (which would be very unusual in SFF files), duplicates are not removed. + +**Example Usage** + +You may have performed some kind of contamination search, for example running +BLASTN against a database of cloning vectors or bacteria, giving you a tabular +file containing read identifiers. You could use this tool to extract only the +reads without BLAST matches (i.e. those which do not match your contaminant +database). + +You may have a file of FASTA sequences which has been used with some analysis +tool giving tabular output, which has then been filtered on some criteria. +You can then use this tool to divide the original FASTA file into those entries +matching or not matching your criteria (those with or without their identifier +in the filtered tabular file). + +**References** + +If you use this Galaxy tool in work leading to a scientific publication please +cite the following papers: + +Peter J.A. Cock, Björn A. Grüning, Konrad Paszkiewicz and Leighton Pritchard (2013). +Galaxy tools and workflows for sequence analysis with applications +in molecular plant pathology. PeerJ 1:e167 +http://dx.doi.org/10.7717/peerj.167 + +This tool uses Biopython to read and write SFF files, so you may also wish to +cite the Biopython application note (and Galaxy too of course): + +Cock et al (2009). Biopython: freely available Python tools for computational +molecular biology and bioinformatics. Bioinformatics 25(11) 1422-3. +http://dx.doi.org/10.1093/bioinformatics/btp163 pmid:19304878. + +This tool is available to install into other Galaxy Instances via the Galaxy +Tool Shed at http://toolshed.g2.bx.psu.edu/view/peterjc/seq_filter_by_id + </help> +</tool> |