comparison bigwig_outlier_bed.xml @ 6:eb17eb8a3658 draft

planemo upload commit 1baff96e75def9248afdcf21edec9bdc7ed42b1f-dirty

5:68cb8e7e266b

<tool name="bigwig_outlier_bed" id="bigwig_outlier_bed" version="0.04" profile="22.05">
  <!--Source in git at: https://github.com/fubar2/galaxy_tf_overlay-->
  <!--Created by [e-mail] at 30/06/2024 19:44:14 using the Galaxy Tool Factory.-->
  <description>Writes high and low bigwig regions as features in a bed file</description>
  <edam_topics>
      <edam_topic>topic_0157</edam_topic>
      <edam_topic>topic_0092</edam_topic>
  </edam_topics>
  <edam_operations>
      <edam_operation>operation_0337</edam_operation>
  </edam_operations>  
  <requirements>
    <requirement type="package" version="3.12.3">python</requirement>
    <requirement type="package" version="2.0.0">numpy</requirement>
    <requirement type="package" version="0.1.4">pybigtools</requirement>
  </requirements>
  <version_command><![CDATA[python -c "import pybigtools; from importlib.metadata import version; print(version('pybigtools'))"]]></version_command>
  <command><![CDATA[python
'$runme'
--bigwig
'$bigwig'
--bedouthilo
'$bedouthilo'
--minwin
'$minwin'
--qhi
'$qhi'
--qlo
'$qlo'
#if $tableout == "set"
 --tableout
#end if
--bigwiglabels
'$bigwiglabels']]></command>
  <configfiles>
    <configfile name="runme"><![CDATA[#raw
"""
Bigwigs are great, but hard to reliably "see" small low coverage or small very high coverage regions.
Colouring in JB2 tracks will need a new plugin, so this code will find bigwig regions above and below a chosen percentile point.
0.99 and 0.01 work well in testing with a minimum span of 10 bp.
Multiple bigwigs **with the same reference** can be combined - bed segments will be named appropriately
Combining multiple references works but is silly because display will rely on one reference so features mapped to other references will not appear.

Tricksy numpy method from http://gregoryzynda.com/python/numpy/contiguous/interval/2019/11/29/contiguous-regions.html
takes about 95 seconds for a 17MB test wiggle
JBrowse2 bed normally displays ignore the score, so could provide separate low/high bed file outputs as an option.
Update june 30 2024: wrote a 'no-build' plugin for beds to display red/blue if >0/<0 so those are used for scores
Bed interval naming must be short for JB2 but needs input bigwig name and (lo or hi).
"""

import argparse
import numpy as np
import pybigtools
import sys
from pathlib import Path


class findOut():
    def __init__(self, args):
        self.bwnames=args.bigwig
        self.bwlabels=args.bigwiglabels
        self.bedwin=args.minwin
        self.qlo=args.qlo
        self.qhi=args.qhi
        self.bedouthilo=args.bedouthilo
        self.bedouthi=args.bedouthi
        self.bedoutlo=args.bedoutlo
        self.tableout = args.tableout
        self.bedwin = args.minwin
        self.qhi = args.qhi
        self.qlo = args.qlo
        self.makeBed()

    def processVals(self, bw, isTop):
        # http://gregoryzynda.com/python/numpy/contiguous/interval/2019/11/29/contiguous-regions.html
        if isTop:
            bwex = np.r_[False, bw >= self.bwtop, False] # extend with 0s
        else:
            bwex = np.r_[False, bw <= self.bwbot, False]
        bwexd = np.diff(bwex)
        bwexdnz = bwexd.nonzero()[0]
        bwregions = np.reshape(bwexdnz, (-1,2))
        return bwregions

    def writeBed(self, bed, bedfname):
        """
        potentially multiple
        """
        bed.sort()
        beds = ['%s\t%d\t%d\t%s\t%d' % x for x in bed]
        with open(bedfname, "w") as bedf:
            bedf.write('\n'.join(beds))
            bedf.write('\n')
        print('Wrote %d bed regions to %s' % (len(bed), bedfname))

    def makeBed(self):
        bedhi = []
        bedlo = []
        bwlabels = self.bwlabels
        bwnames = self.bwnames
        print('bwnames=', bwnames, "bwlabs=", bwlabels)
        for i, bwname in enumerate(bwnames):
            bwlabel = bwlabels[i].replace(" ",'')
            p = Path('in.bw')
            p.symlink_to( bwname ) # required by pybigtools (!)
            bwf = pybigtools.open('in.bw')
            chrlist = bwf.chroms()
            chrs = list(chrlist.keys())
            chrs.sort()
            restab = ["contig\tn\tmean\tstd\tmin\tmax\tqtop\tqbot"]
            for chr in chrs:
                bw = bwf.values(chr)
                bw = bw[~np.isnan(bw)] # some have NaN if parts of a contig not covered
                if self.qhi is not None:
                    self.bwtop = np.quantile(bw, self.qhi)
                    bwhi = self.processVals(bw, isTop=True)
                    for i, seg in enumerate(bwhi):
                        if seg[1] - seg[0] >= self.bedwin:
                            bedhi.append((chr, seg[0], seg[1], '%s_hi' % (bwlabel), 1))
                if self.qlo is not None:
                    self.bwbot = np.quantile(bw, self.qlo)
                    bwlo = self.processVals(bw, isTop=False)            
                    for i, seg in enumerate(bwlo):
                        if seg[1] - seg[0] >= self.bedwin:
                            bedlo.append((chr, seg[0], seg[1], '%s_lo' % (bwlabel), -1))
                bwmean = np.mean(bw)
                bwstd = np.std(bw)
                bwmax = np.max(bw)
                nrow = np.size(bw)
                bwmin = np.min(bw)
                restab.append('%s\t%d\t%f\t%f\t%f\t%f\t%f\t%f' % (chr,nrow,bwmean,bwstd,bwmin,bwmax,self.bwtop,self.bwbot))        
        print('\n'.join(restab), '\n')
        if self.tableout:
            with open(self.tableout) as t:
                t.write('\n'.join(restab))
                t.write('\n')
        if self.bedoutlo:
            if self.qlo:
                self.writeBed(bedlo, self.bedoutlo)
        if self.bedouthi:
            if self.qhi:
                self.writeBed(bedhi, self.bedouthi)
        if self.bedouthilo:
            allbed = bedlo + bedhi
            self.writeBed(allbed, self.bedouthilo)
        return restab


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    a = parser.add_argument
    a('-m', '--minwin',default=10, type=int)
    a('-l', '--qlo',default=None, type=float)
    a('-i', '--qhi',default=None, type=float)
    a('-w', '--bigwig', nargs='+')
    a('-n', '--bigwiglabels', nargs='+')
    a('-o', '--bedouthilo', default=None, help="optional high and low combined bed")
    a('-u', '--bedouthi', default=None, help="optional high only bed")
    a('-b', '--bedoutlo', default=None, help="optional low only bed")
    a('-t', '--tableout', default=None)
    args = parser.parse_args()
    print('args=', args)
    if not (args.bedouthilo or args.bedouthi or args.bedoutlo):
        sys.stderr.write("bigwig_outlier_bed.py cannot usefully run - need a bed output choice - must be one of low only, high only or both combined")
        sys.exit(2)
    if not (args.qlo or args.qhi):
        sys.stderr.write("bigwig_outlier_bed.py cannot usefully run - need one or both of quantile cutpoints qhi and qlo")
        sys.exit(2)
    restab = findOut(args)
    if args.tableout:
        with open(args.tableout, 'w') as tout:
            tout.write('\n'.join(restab))
            tout.write('\n')
#end raw]]></configfile>
  </configfiles>
  <inputs>
    <param name="bigwig" type="data" optional="false" label="Bigwig file(s) to process. " help="If more than one, MUST all use the same reference sequence to be displayable. Feature names will include the bigwig label." format="bigwig" multiple="true"/>
    <param name="minwin" type="integer" value="10" label="Minimum continuous bases to count as a high or low bed feature" help="Actual run length will be found and used for continuous features as long or longer."/>
    <param name="qhi" type="float" value="0.99" label="Quantile cutoff for a high region - 0.99 will cut off at or above the 99th percentile" help=""/>
    <param name="qlo" type="float" value="0.01" label="Quantile cutoff for a low region - 0.01 will cut off at or below the 1st percentile." help=""/>
    <param name="tableout" type="select" label="Write a table showing contig statistics for each bigwig" help="" display="radio">
      <option value="notset">Do not set this flag</option>
      <option value="set">Set this flag</option>
    </param>
    <param name="bigwiglabels" type="text" value="outbed" label="Label to use in bed feature names to indicate source bigwig contents - such as coverage" help=""/>
  </inputs>
  <outputs>
    <data name="bedouthilo" format="bed" label="Both high and low contiguous regions as long or longer than window length into one bed " hidden="false"/>
  </outputs>
  <tests>
    <test>
      <output name="bedouthilo" value="bedouthilo_sample" compare="diff" lines_diff="0"/>
      <param name="bigwig" value="bigwig_sample"/>
      <param name="minwin" value="10"/>
      <param name="qhi" value="0.99"/>
      <param name="qlo" value="0.01"/>
      <param name="tableout" value="notset"/>
      <param name="bigwiglabels" value="outbed"/>
    </test>
  </tests>
  <help><![CDATA[
 **What it Does**
 
 Takes one or more bigwigs mapped to the same reference and finds all the minimum window sized or greater contiguous regions above or below an upper and lower quantile cutoff.
 A window size of 10 works well, and quantiles set at 0.01 and 0.99 will generally work well.
 
  ]]></help>
  <citations>
    <citation type="doi">10.1093/bioinformatics/btae350</citation>
  </citations>

6:eb17eb8a3658

<tool name="Bigwig extremes to bed features" id="bigwig_outlier_bed" version="@TOOL_VERSION@" profile="22.05">
  <description>Writes high and low bigwig runs as features in a bed file</description>
  <macros>
  <token name="@TOOL_VERSION@">0.2.0</token>
  <token name="@NUMPY_VERSION@">2.0.0</token>
  <token name="@PYTHON_VERSION@">3.12.3</token>
  </macros>
  <edam_topics>
      <edam_topic>topic_0157</edam_topic>
      <edam_topic>topic_0092</edam_topic>
  </edam_topics>
  <edam_operations>
      <edam_operation>operation_0337</edam_operation>
  </edam_operations>
  <xrefs>
    <xref type="bio.tools">bigtools</xref>
  </xrefs>
  <requirements>
    <requirement type="package" version="@PYTHON_VERSION@">python</requirement>
    <requirement type="package" version="@NUMPY_VERSION@">numpy</requirement>
    <requirement type="package" version="@TOOL_VERSION@">pybigtools</requirement>
  </requirements>
  <required_files>
      <include path="bigwig_outlier_bed.py"/>
  </required_files>
  <version_command><![CDATA[python -c "import pybigtools; from importlib.metadata import version; print(version('pybigtools'))"]]></version_command>
  <command><![CDATA[python '${__tool_directory__}/bigwig_outlier_bed.py'
--bigwig 
#for bw in $bigwig: 
 '$bw' 
#end for
--bigwiglabels
#for bw in $bigwig: 
  '$bw.name'
#end for
--outbeds '$outbeds'
#if $outbeds in ['outhilo', 'outall']:
  --bedouthilo '$bedouthilo'
#end if
#if $outbeds in ['outhi', 'outall', 'outlohi']:
  --bedouthi '$bedouthi'
#end if
#if $outbeds in ['outlo', 'outall', 'outlohi']:
  --bedoutlo '$bedoutlo'
#end if
--minwin '$minwin'
#if $qhi:
--qhi '$qhi'
#end if
#if $qlo:
--qlo '$qlo'
#end if
#if $tableout == "create" or $outbeds == "outtab":
 --tableoutfile '$tableoutfile'
#end if
]]></command>
  <inputs>
    <param name="bigwig" type="data" optional="false" label="Choose one or more bigwig file(s) to return outlier regions as a bed file" 
      help="If more than one, MUST all use the same reference sequence to be displayable. Feature names will include the bigwig label." format="bigwig" multiple="true"/>
    <param name="minwin" type="integer" value="10" label="Minimum continuous bases to count as a high or low bed feature" 
      help="Continuous features as long or longer than this window size will appear as bed features"/>
    <param name="qhi" type="float" value="0.99" label="Quantile cutoff for a high region - 0.99 will cut off at or above the 99th percentile" help="Required" optional="false"/>
    <param name="qlo" type="float" value="0.01" label="Quantile cutoff for a low region - 0.01 will cut off at or below the 1st percentile." help="Optional" optional="true"/>
    <param name="outbeds" type="select" label="Select the required bed file outputs" help="Any combination of the 3 different kinds of bed file output can be made">
      <option value="outhilo" selected="true">Make 1 bed output with both low and high regions</option>
      <option value="outhi">Make 1 bed output with high regions only</option>
      <option value="outlo">Make 1 bed output with low regions only</option>
      <option value="outall">Make 3 bed outputs with low and high together in one, high in one and low in the other</option>
      <option value="outlohi">Make 2 bed outputs with high in one and low in the other</option>
      <option value="outtab">NO bed outputs. Report bigwig value distribution only</option>
    </param>
    <param name="tableout" type="select" label="Write a table showing contig statistics for each bigwig input" help="">
      <option value="donotmake">Do not create this report</option>
      <option value="create" selected="true">Create this report</option>
    </param>
  </inputs>
  <outputs>
    <data name="bedouthilo" format="bed" label="High_and_low_bed" hidden="false">
      <filter>outbeds in ["outall", "outhilo"]</filter>
    </data>
    <data name="bedouthi" format="bed" label="High bed" hidden="false">
      <filter>outbeds in ["outall", "outlohi", "outhi"]</filter>
    </data>
    <data name="bedoutlo" format="bed" label="Low bed" hidden="false">
      <filter>outbeds in ["outall", "outlohi", "outlo"]</filter>
    </data>
    <data name="tableoutfile" format="tabular" label="Contig statistics" hidden="false">
      <filter>tableout == "create"</filter>
    </data>
  </outputs>
  <tests>
    <test expect_num_outputs="1">
      <output name="bedouthilo" value="bedouthilo_sample" compare="diff" lines_diff="0"/>
      <param name="outbeds" value="outhilo"/>
      <param name="bigwig" value="bigwig_sample"/>
      <param name="minwin" value="10"/>
      <param name="qhi" value="0.99"/>
      <param name="qlo" value="0.01"/>
      <param name="tableout" value="donotmake"/>
    </test>
    <test expect_num_outputs="1">
      <output name="tableoutfile" value="table_only_sample" compare="diff" lines_diff="0"/>
      <param name="outbeds" value="outtab"/>
      <param name="bigwig" value="bigwig_sample,1.bigwig"/>
      <param name="minwin" value="10"/>
      <param name="qhi" value="0.99"/>
      <param name="qlo" value="0.01"/>
      <param name="tableout" value="create"/>
    </test>
    <test expect_num_outputs="2">
      <output name="bedouthilo" value="bedouthilo_sample" compare="diff" lines_diff="0"/>
      <output name="tableoutfile" value="table_sample" compare="diff" lines_diff="0"/>
      <param name="outbeds" value="outhilo"/>
      <param name="bigwig" value="bigwig_sample"/>
      <param name="minwin" value="10"/>
      <param name="qhi" value="0.99"/>
      <param name="qlo" value="0.01"/>
      <param name="tableout" value="create"/>
    </test>
    <test expect_num_outputs="2">
      <output name="bedouthi" value="bedouthi_qlo_notset_sample" compare="diff" lines_diff="0"/>
      <output name="tableoutfile" value="table_qlo_notset_sample" compare="diff" lines_diff="0"/>
      <param name="outbeds" value="outhi"/>
      <param name="bigwig" value="bigwig_sample"/>
      <param name="minwin" value="10"/>
      <param name="qhi" value="0.99"/>
      <param name="qlo" value=""/>
      <param name="tableout" value="create"/>
    </test>
    <test expect_num_outputs="3">
      <output name="bedouthi" value="bedouthi_sample" compare="diff" lines_diff="0"/>
      <output name="bedoutlo" value="bedoutlo_sample" compare="diff" lines_diff="0"/>
      <output name="tableoutfile" value="table3_sample" compare="diff" lines_diff="0"/>
      <param name="outbeds" value="outlohi"/>
      <param name="bigwig" value="bigwig_sample"/>
      <param name="minwin" value="1"/>
      <param name="qhi" value="0.9"/>
      <param name="qlo" value="0.1"/>
      <param name="tableout" value="create"/>
    </test>
    <test expect_num_outputs="4">
      <output name="bedouthilo" value="bedouthilo2_sample" compare="diff" lines_diff="0"/>
      <output name="bedoutlo" value="bedoutlo2_sample" compare="diff" lines_diff="0"/>
      <output name="bedouthi" value="bedouthi2_sample" compare="diff" lines_diff="0"/>
      <output name="tableoutfile" value="table2_sample" compare="diff" lines_diff="0"/>
      <param name="outbeds" value="outall"/>
      <param name="bigwig" value="bigwig_sample,1.bigwig"/>
      <param name="minwin" value="1"/>
      <param name="qhi" value="0.9"/>
      <param name="qlo" value="0.1"/>
      <param name="tableout" value="create"/>
    </test>
  </tests>
  <help><![CDATA[

 **Purpose**
 
 *Combine bigwig outlier regions into bed files*
 
 Bigwigs allow quantative tracks to be viewed in an interactive genome browser like JBrowse2. 
 Peaks are easy to see. Unusually low regions can be harder to spot, even if they are relatively large, unless the view is zoomed right in.
 Automated methods for combining evidence from multiple bigwigs can be useful for constructing browseable *issues* or other kinds of summary bed format tracks.
 For example, combining coverage outlier regions, with the frequency of specific dicnucleotide short tandem repeats,
 for evaluating technical sequencing technology effects in the evaluation of a genome assembly described at https://github.com/arangrhie/T2T-Polish

 **What does it produce?**

 Bed format results are output, containing each continuous segment of at least *minwin* base pairs above a cut point, or below another cut point. 
 These can be viewed as features on the reference genome using a genome browser tool like JBrowse2.
 Three kinds of bed files can be created depending on the values included.  
 Both high and low regions in one bed output is the default. This can be displayed in JBrowse2 with colour indicating the high or low status, 
 one less track and a little easier to understand. High and low features can be output as separate bed files.

 **How is it controlled?**

 The cut points are calculated using a user supplied quantile, from each chromosome's bigwig value distribution. 
 The defaults are 0.99 and 0.01 and the default *minwin* is 10.  
 The probability of 10 values at or below the 1st percentile purely by chance is about 0.01**10, so false positives should be
 rare, even in a 3GB genome.  
 This data driven and non-parametric method is preferred for the asymmetrical distributions found in typical bigwigs, such as depth of coverage 
 for genome sequencing reads. Coverage values are truncated at zero, and regions with very high values often form a long sparse right tail. 

 **How do I choose the input data?**

 One or more bigwigs and can be selected as inputs.  
 Multiple bigwigs will be combined in bed files, so must share the reference genome to display
 using JBrowse2.
 
 .. class:: warningmark

 **Lower quantile may not behave as expected in bigwigs with large fractions of zero values**

 The lower cut point may be problematic for integer values like coverage if many values are zero. For example, if 5% of bases have zero coverage, the 1st percentile is also zero, 
 but that cut point will include the entire 5% *at or below 0*

 
  ]]></help>
  <citations>
    <citation type="doi">10.1093/bioinformatics/btae350</citation>
  </citations>