Mercurial > repos > jjohnson > mothur_toolsuite

view mothur/README @ 6:ce6e81622c6a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Allow for output file ext to be either .chimera or .chimeras
author Jim Johnson <jj@umn.edu>
date Mon, 27 Jun 2011 09:34:14 -0500
parents e990ac8a0f58
children 7bfe1f843858
line wrap: on
line source

Provides galaxy tools for the Mothur metagenomics package -  http://www.mothur.org/wiki/Main_Page 

Install mothur v.1.19.0 on your galaxy system so galaxy can execute the mothur command
  ( This version of wrappers is designed for Mothur version 1.19 - it may work on later versions )
  http://www.mothur.org/wiki/Download_mothur
  http://www.mothur.org/wiki/Installation
  ( This Galaxy Mothur wrapper will invoke Mothur in command line mode: http://www.mothur.org/wiki/Command_line_mode )

TreeVector is also packaged with this Mothur package to view phylogenetic trees:
  TreeVector is a utility to create and integrate phylogenetic trees as Scalable Vector Graphics (SVG) files.
  TreeVector was written by Ralph_Pethica, Department_of_Computer_Science, University_of_Bristol
  TreeVector: http://supfam.cs.bris.ac.uk/TreeVector/about.html
  Install in galaxy:  tool-data/shared/jars/TreeVector.jar

Install reference data from silva and greengenes
 Silva reference:
  http://www.mothur.org/wiki/Silva_reference_files
  - Bacterial references (14,956 sequences)
    http://www.mothur.org/w/images/9/98/Silva.bacteria.zip
  - Archaeal references (2,297 sequences)
    http://www.mothur.org/w/images/3/3c/Silva.archaea.zip
  - Eukaryotic references (1,238 sequences)
    http://www.mothur.org/w/images/1/1a/Silva.eukarya.zip
  - Silva-based alignment of template file for chimera.slayer (5,181 sequences)
    http://www.mothur.org/w/images/f/f1/Silva.gold.bacteria.zip
 Alignment database rRNA gene sequences:
  http://www.mothur.org/wiki/Alignment_database
  - greengenes reference alignment
    http://www.mothur.org/w/images/7/72/Greengenes.alignment.zip
  - SILVA (Silva reference)
    http://www.mothur.org/w/images/f/f1/Silva.gold.bacteria.zip
 Secondary structure mapping files:
  http://www.mothur.org/wiki/Secondary_structure_map
    http://www.mothur.org/w/images/6/6d/Silva_ss_map.zip
    http://www.mothur.org/w/images/4/4b/Gg_ss_map.zip
 Lane masks:
  http://www.mothur.org/wiki/Lane_mask
  greengenes-compatible mask:
     - lane1241.gg.filter - A Lane Masks that comes with the greengenes arb database
       http://www.mothur.org/w/images/2/2a/Lane1241.gg.filter
     - lane1287.gg.filter - A Lane Masks that comes with the greengenes arb database
       http://www.mothur.org/w/images/a/a0/Lane1287.gg.filter
     - lane1349.gg.filter - Pat Schloss's transcription of the mask from the Lane paper
       http://www.mothur.org/w/images/3/3d/Lane1349.gg.filter
  SILVA-compatible mask:
     - lane1349.silva.filter - Pat Schloss's transcription of the mask from the Lane paper
       http://www.mothur.org/w/images/6/6d/Lane1349.silva.filter

 Example from UMN installation: (We also made these available in a Galaxy public data library)
    /project/db/galaxy/mothur/Silva.bacteria.zip
    /project/db/galaxy/mothur/silva.eukarya.fasta
    /project/db/galaxy/mothur/Greengenes.alignment.zip
    /project/db/galaxy/mothur/Silva.archaea.zip
    /project/db/galaxy/mothur/Silva_ss_map.zip
    /project/db/galaxy/mothur/silva.eukarya.ncbi.tax
    /project/db/galaxy/mothur/Silva.gold.bacteria.zip
    /project/db/galaxy/mothur/Silva.archaea/silva.archaea.silva.tax
    /project/db/galaxy/mothur/Silva.archaea/silva.archaea.gg.tax
    /project/db/galaxy/mothur/Silva.archaea/silva.archaea.rdp.tax
    /project/db/galaxy/mothur/Silva.archaea/nogap.archaea.fasta
    /project/db/galaxy/mothur/Silva.archaea/silva.archaea.ncbi.tax
    /project/db/galaxy/mothur/Silva.archaea/silva.archaea.fasta
    /project/db/galaxy/mothur/nogap.eukarya.fasta
    /project/db/galaxy/mothur/silva.eukarya.silva.tax
    /project/db/galaxy/mothur/silva.gold.align
    /project/db/galaxy/mothur/silva.ss.map
    /project/db/galaxy/mothur/gg.ss.map
    /project/db/galaxy/mothur/silva.bacteria/silva.bacteria.silva.tax
    /project/db/galaxy/mothur/silva.bacteria/silva.bacteria.rdp6.tax
    /project/db/galaxy/mothur/silva.bacteria/nogap.bacteria.fasta
    /project/db/galaxy/mothur/silva.bacteria/silva.bacteria.gg.tax
    /project/db/galaxy/mothur/silva.bacteria/silva.bacteria.ncbi.tax
    /project/db/galaxy/mothur/silva.bacteria/silva.bacteria.fasta
    /project/db/galaxy/mothur/silva.bacteria/silva.bacteria.rdp.tax
    /project/db/galaxy/mothur/Silva.eukarya.zip
    /project/db/galaxy/mothur/Gg_ss_map.zip
    /project/db/galaxy/mothur/core_set_aligned.imputed.fasta


Add tool-data:  (contains  pointers to silva and greengenes reference data)
  tool-data/mothur_aligndb.loc
  tool-data/mothur_calulators.loc
  tool-data/mothur_map.loc
  tool-data/mothur_taxonomy.loc
  tool-data/shared/jars/TreeVector.jar


add config files (*.xml) and wrapper code (*.py) from tools/mothur/*  to your galaxy installation 


add datatype definition file: lib/galaxy/datatypes/metagenomics.py

add the following import line to:  lib/galaxy/datatypes/registry.py
import metagenomics # added for metagenomics mothur


add datatypes to:  datatypes_conf.xml
        <!-- Start Mothur Datatypes -->
        <datatype extension="otu" type="galaxy.datatypes.metagenomics:Otu" display_in_upload="true"/>
        <datatype extension="list" type="galaxy.datatypes.metagenomics:OtuList" display_in_upload="true"/>
        <datatype extension="sabund" type="galaxy.datatypes.metagenomics:Sabund" display_in_upload="true"/>
        <datatype extension="rabund" type="galaxy.datatypes.metagenomics:Rabund" display_in_upload="true"/>
        <datatype extension="shared" type="galaxy.datatypes.metagenomics:SharedRabund" display_in_upload="true"/>
        <datatype extension="relabund" type="galaxy.datatypes.metagenomics:RelAbund" display_in_upload="true"/>
        <datatype extension="names" type="galaxy.datatypes.metagenomics:Names" display_in_upload="true"/>
        <datatype extension="design" type="galaxy.datatypes.metagenomics:Design" display_in_upload="true"/>
        <datatype extension="summary" type="galaxy.datatypes.metagenomics:Summary" display_in_upload="true"/>
        <datatype extension="groups" type="galaxy.datatypes.metagenomics:Group" display_in_upload="true"/>
        <datatype extension="oligos" type="galaxy.datatypes.metagenomics:Oligos" display_in_upload="true"/>
        <datatype extension="align" type="galaxy.datatypes.metagenomics:SequenceAlignment" display_in_upload="true"/>
        <datatype extension="accnos" type="galaxy.datatypes.metagenomics:AccNos" display_in_upload="true"/>
        <datatype extension="map" type="galaxy.datatypes.metagenomics:SecondaryStructureMap" display_in_upload="true"/>
        <datatype extension="align.check" type="galaxy.datatypes.metagenomics:AlignCheck" display_in_upload="true"/>
        <datatype extension="align.report" type="galaxy.datatypes.metagenomics:AlignReport" display_in_upload="true"/>
        <datatype extension="filter" type="galaxy.datatypes.metagenomics:LaneMask" display_in_upload="true"/>
        <datatype extension="dist" type="galaxy.datatypes.metagenomics:DistanceMatrix" display_in_upload="true"/>
        <datatype extension="pair.dist" type="galaxy.datatypes.metagenomics:PairwiseDistanceMatrix" display_in_upload="true"/>
        <datatype extension="square.dist" type="galaxy.datatypes.metagenomics:SquareDistanceMatrix" display_in_upload="true"/>
        <datatype extension="lower.dist" type="galaxy.datatypes.metagenomics:LowerTriangleDistanceMatrix" display_in_upload="true"/>
        <datatype extension="seq.taxonomy" type="galaxy.datatypes.metagenomics:SequenceTaxonomy" display_in_upload="true"/>
        <datatype extension="rdp.taxonomy" type="galaxy.datatypes.metagenomics:RDPSequenceTaxonomy" display_in_upload="true"/>
        <datatype extension="cons.taxonomy" type="galaxy.datatypes.metagenomics:ConsensusTaxonomy" display_in_upload="true"/>
        <datatype extension="tax.summary" type="galaxy.datatypes.metagenomics:TaxonomySummary" display_in_upload="true"/>
        <datatype extension="freq" type="galaxy.datatypes.metagenomics:Frequency" display_in_upload="true"/>
        <datatype extension="quan" type="galaxy.datatypes.metagenomics:Quantile" display_in_upload="true"/>
        <datatype extension="filtered.quan" type="galaxy.datatypes.metagenomics:FilteredQuantile" display_in_upload="true"/>
        <datatype extension="masked.quan" type="galaxy.datatypes.metagenomics:MaskedQuantile" display_in_upload="true"/>
        <datatype extension="filtered.masked.quan" type="galaxy.datatypes.metagenomics:FilteredMaskedQuantile" display_in_upload="true"/>
        <datatype extension="axes" type="galaxy.datatypes.metagenomics:Axes" display_in_upload="true"/>
        <datatype extension="tre" type="galaxy.datatypes.data:Newick" display_in_upload="true"/>
        <!-- End Mothur Datatypes -->

add mothur tools to:   tool_conf.xml
  <section name="Metagenomics Mothur" id="metagenomics_mothur">
    <label text="Mothur Utilities" id="mothur_utilities"/>
      <tool file="mothur/merge.files.xml"/>
      <tool file="mothur/make.group.xml"/>
      <tool file="mothur/get.groups.xml"/>
      <tool file="mothur/remove.groups.xml"/>
      <tool file="mothur/merge.groups.xml"/>
      <tool file="mothur/make.design.xml"/>
      <tool file="mothur/sub.sample.xml"/>
    <label text="Mothur Sequence Analysis" id="mothur_sequence_analysis"/>
      <tool file="mothur/sffinfo.xml"/>
      <tool file="mothur/make.fastq.xml"/>
      <tool file="mothur/fastq.info.xml"/>
      <tool file="mothur/summary.seqs.xml"/>
      <tool file="mothur/reverse.seqs.xml"/>
      <tool file="mothur/list.seqs.xml"/>
      <tool file="mothur/get.seqs.xml"/>
      <tool file="mothur/remove.seqs.xml"/>
      <tool file="mothur/trim.seqs.xml"/>
      <tool file="mothur/unique.seqs.xml"/>
      <tool file="mothur/deunique.seqs.xml"/>
      <tool file="mothur/chop.seqs.xml"/>
      <tool file="mothur/screen.seqs.xml"/>
      <tool file="mothur/filter.seqs.xml"/>
      <tool file="mothur/degap.seqs.xml"/>
      <tool file="mothur/consensus.seqs.xml"/>
      <tool file="mothur/align.seqs.xml"/>
      <tool file="mothur/align.check.xml"/>
      <tool file="mothur/dist.seqs.xml"/>
      <tool file="mothur/pairwise.seqs.xml"/>
      <tool file="mothur/split.abund.xml"/>
      <tool file="mothur/split.groups.xml"/>
      <tool file="mothur/pcoa.xml"/>
      <tool file="mothur/pca.xml"/>
      <tool file="mothur/nmds.xml"/>
      <tool file="mothur/corr.axes.xml"/>
      <tool file="mothur/classify.seqs.xml"/>
    <label text="Mothur Sequence Chimera Detection" id="mothur_sequence_chimera"/>
      <tool file="mothur/chimera.bellerophon.xml"/>
      <tool file="mothur/chimera.ccode.xml"/>
      <tool file="mothur/chimera.check.xml"/>
      <tool file="mothur/chimera.pintail.xml"/>
      <tool file="mothur/chimera.slayer.xml"/>
    <label text="Mothur Operational Taxonomy Unit" id="mothur_taxonomy_unit"/>
      <tool file="mothur/pre.cluster.xml"/>
      <tool file="mothur/cluster.fragments.xml"/>
      <tool file="mothur/cluster.xml"/>
      <tool file="mothur/hcluster.xml"/>
      <tool file="mothur/cluster.classic.xml"/>
      <tool file="mothur/cluster.split.xml"/>
      <tool file="mothur/metastats.xml"/>
      <tool file="mothur/sens.spec.xml"/>
      <tool file="mothur/classify.otu.xml"/>
      <tool file="mothur/parse.list.xml"/>
      <tool file="mothur/get.otus.xml"/>
      <tool file="mothur/remove.otus.xml"/>
      <tool file="mothur/remove.rare.xml"/>
      <tool file="mothur/get.otulist.xml"/>
      <tool file="mothur/get.oturep.xml"/>
      <tool file="mothur/otu.hierarchy.xml"/>
      <tool file="mothur/get.rabund.xml"/>
      <tool file="mothur/get.sabund.xml"/>
      <tool file="mothur/get.relabund.xml"/>
      <tool file="mothur/make.shared.xml"/>
      <tool file="mothur/get.group.xml"/>
      <tool file="mothur/bin.seqs.xml"/>
      <tool file="mothur/get.sharedseqs.xml"/>
    <label text="Mothur Single Sample Analysis" id="mothur_single_sample_analysis"/>
      <tool file="mothur/collect.single.xml"/>
      <tool file="mothur/rarefaction.single.xml"/>
      <tool file="mothur/summary.single.xml"/>
      <tool file="mothur/heatmap.bin.xml"/>
    <label text="Mothur Multiple Sample Analysis" id="mothur_multiple_sample_analysis"/>
      <tool file="mothur/collect.shared.xml"/>
      <tool file="mothur/rarefaction.shared.xml"/>
      <tool file="mothur/normalize.shared.xml"/>
      <tool file="mothur/summary.shared.xml"/>
      <tool file="mothur/dist.shared.xml"/>
      <tool file="mothur/heatmap.bin.xml"/>
      <tool file="mothur/heatmap.sim.xml"/>
      <tool file="mothur/venn.xml"/>
      <tool file="mothur/tree.shared.xml"/>
    <label text="Mothur Hypothesis Testing" id="mothur_hypothesis_testing"/>
      <tool file="mothur/parsimony.xml"/>
      <tool file="mothur/unifrac.weighted.xml"/>
      <tool file="mothur/unifrac.unweighted.xml"/>
      <tool file="mothur/libshuff.xml"/>
      <tool file="mothur/amova.xml"/>
      <tool file="mothur/homova.xml"/>
      <tool file="mothur/mantel.xml"/>
      <tool file="mothur/anosim.xml"/>
    <label text="Mothur Phylotype Analysis" id="mothur_phylotype_analysis"/>
      <tool file="mothur/get.lineage.xml"/>
      <tool file="mothur/remove.lineage.xml"/>
      <tool file="mothur/phylotype.xml"/>
      <tool file="mothur/phylo.diversity.xml"/>
      <tool file="mothur/clearcut.xml"/>
      <tool file="mothur/indicator.xml"/>
      <tool file="mothur/TreeVector.xml"/>
  </section> <!-- metagenomics_mothur -->


############ DESIGN NOTES #########################################################################################################
Each mothur command has it's own tool_config (.xml) file, but all call the same python wrapper code: mothur_wrapper.py

* Every mothur tool will call mothur_wrapper.py script with a --cmd= parameter that gives the mothur command name.
* Many mothur commands require date to be read into memory (using read.dist, read.otu, read.tree) before executed the command,  
  these are accomplished in the tool_config and mothur_wrapper.py with --READ_cmd= and --READ_<option> parameters. 
* Every tool will produce the logfile of the mothur run as an output.
* When the outputs of a mothur command could be determined in advance, they are included in the --result= parameter to mothur_wrapper.py
* When the number of outputs cannot be determined in advance, the name patterns and datatypes of the ouputs 
  are included in the --new_datasets parameter to mothur_wrapper.py
 
Here is an example call to the mothur_wrapper.py script with an explanation before each param :
 mothur_wrapper.py
 # name of a mothur command, this is required
 --cmd='summary.shared'
 # Galaxy output dataset list, these are output files that can be determined before the command is run
 # The items in the list are separated by commas
 # Each item contains a regex to match the output filename and a galaxy dataset filepath in which to copy the data (separated by :)
 --result='^mothur.\S+\.logfile$:'/home/galaxy/data/database/files/002/dataset_2613.dat,'^\S+\.summary$:'/home/galaxy/data/database/files/002/dataset_2614.dat
 # Galaxy output dataset extra_files_path direcotry in which to put all output files (usually the logfile extra_file path)
 --outputdir='/home/galaxy/data/database/files/002/dataset_2613_files'
 # The id of one of the galaxy outputs (e.g. the mothur logfile) used for dynamic dataset generation (when number of outputs not known in advance)
 #  see: ttp://bitbucket.org/galaxy/galaxy-central/wiki/ToolsMultipleOutput
 --datasetid='2578'
 # The galaxy directory in which to copy all output files for dynamic dataset generation (special galaxy tool param: $__new_file_path__)
 --new_file_path='$__new_file_path__'
 # specifies files to copy to the new_file_path
 # The list is separated by commas
 # Each item  conatins:   a regex pattern for matching filenames and  a galaxy datatype (separated by :)
 # The regex match.groups()[0] is used as the id name of the dataset, and must result in  unique name for each output
 --new_datasets='^\S+?\.((\S+)\.(unique|[0-9.]*)\.dist)$:lower.dist'
 # Many mothur commands first require data to be read into memory using: read.otu, read.dist, or read.tree
 # This prequisite command and its params are prefixed with 'READ_'
 --READ_cmd='read.otu'
 --READ_list=/home/galaxy/data/database/files/001/dataset_1557.dat
 --READ_group='/home/galaxy/data/database/files/001/dataset_1545.dat'
 --READ_label='unique,0.07'