view egapx_runner.xml @ 8:1680e72e27be draft default tip

planemo upload for repository https://github.com/ncbi/egapx commit bdbe05027c2c40e217a2ff0c9e0556450c443e54

<tool name="egapx_runner" id="egapx_runner" version="@TOOL_VERSION@" profile="22.05">
  <description>Runs egapx</description>
  <macros>
    <token name="@TOOL_VERSION@">0.02-alpha</token>
  </macros>
  <requirements>
    <requirement version="3.12.3" type="package">python</requirement>
    <requirement version="24.04.4-0" type="package">nextflow</requirement>
    <requirement version="6.0.1" type="package">pyyaml</requirement>
  </requirements>
  <version_command><![CDATA[echo "@TOOL_VERSION@"]]></version_command>
  <command><![CDATA[mkdir -p ./egapx_config &&
#set econfigfile = $econfig + '.config'
cp  '$__tool_directory__/ui/assets/config/executor/$econfigfile' ./egapx_config/ &&
python '$__tool_directory__/ui/egapx.py' '$yamlconfig' -e '$econfig' -o 'egapx_out']]></command>
  <inputs>
    <param name="yamlconfig" type="data" optional="false" label="egapx configuration yaml file to execute" help="" format="yaml,txt" multiple="false"/>
    <param name="econfig" type="select" label="Workflow run configuration to suit the machine in use" help="Docker minimal will run the sample minimal dustmite yaml">
      <option value="docker_minimal">Docker_minimal: supports only the minimal dust mite example yaml using 6GB and 4 cores</option>
      <option value="singularity">Singularity: requires at least 128GB ram and 32 cores. 256GB and 64 cores recommended</option>
      <option value="docker">Docker: requires at least 128GB ram and 32 cores. 256GB and 64 cores recommended</option>
    </param>
  </inputs>
  <outputs>
    <collection name="egapx_out" type="list" label="Outputs from egapx">
      <discover_datasets pattern="__name_and_ext__" directory="egapx_out" visible="false"/>
    </collection>
  </outputs>
  

<tests>
    <test>
      <output_collection name="egapx_out" count="8"/>
      <param name="yamlconfig" value="yamlconfig_sample"/>
      <param name="econfig" value="docker_minimal"/>
    </test>
</tests>



  <help><![CDATA[
Galaxy tool wrapping the Eukaryotic Genome Annotation Pipeline (EGAPx)
=================================================================================================

.. class:: warningmark

**Proof of concept: a quick hack to run a NF workflow inside a specialised Galaxy tool wrapper** 

EGAPx is a big, complicated Nextflow workflow, challenging and costly to re-implement **properly**, requiring dozens of new tools and replicating a lot of 
complicated *groovy* workflow logic. 

It is also very new and in rapid development. Investing developer effort and keeping updated as EGAPx changes rapidly may be *inefficient of developer resources*. 

This wrapper is designed to allow measuring how *inefficient* it is in terms of computing resource utilisation, in comparison to the developer effort 
required to convert Nextflow DDL into tools and WF logic. Balancing these competing requirements is a fundamental Galaxy challenge.


EGAPx requires very substantial resources to run with real data. *128GB and 32 cores* are the minimum requirement; *256GB and 64 cores* are recommended. 

A special minimal example that can be run in 6GB with 4 cores is provided as a yaml configuration and is used for the tool test.

In this implementation, the user must supply a yaml configuration file as initial proof of concept.
History inputs and even a yaml editor might be provided in future. 

The NF workflow to tool model tested here may be applicable to other NF workflows that take a single configuration yaml.

.. class:: warningmark

The computational resource cost of typing the wrong SRA identifiers into a tool form is potentially enormous with this tool!


Sample yaml configurations
===========================

YAML sample configurations can be uploaded into your Galaxy history from the `EGAPx github repository <https://github.com/ncbi/egapx/tree/main/examples/>`_. 
The simplest possible example is shown below - can be cut/paste into a history dataset in the upload tool.


*./examples/input_D_farinae_small.yaml* is shown below and can be cut and pasted into the upload form to create a yaml file. 
RNA-seq data is provided as URI to the reads FASTA files.

input_D_farinae_small.yaml

::

  genome: https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/020/809/275/GCF_020809275.1_ASM2080927v1/GCF_020809275.1_ASM2080927v1_genomic.fna.gz
  taxid: 6954
  reads:
    - https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR8506572.1
    - https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR8506572.2
    - https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR9005248.1
    - https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR9005248.2


input_Gavia_stellata.yaml

::

  genome: https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/030/936/135/GCF_030936135.1_bGavSte3.hap2/GCF_030936135.1_bGavSte3.hap2_genomic.fna.gz
  reads: txid37040[Organism] AND biomol_transcript[properties] NOT SRS024887[Accession]
  taxid: 37040

input_C_longicornis.yaml

::

  genome: https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/029//603/195/GCF_029603195.1_ASM2960319v2/GCF_029603195.1_ASM2960319v2_genomic.fna.gz
  reads: txid2530218[Organism] AND biomol_transcript[properties] NOT SRS024887[Accession]
  taxid: 2530218
  
Purpose
========

**This is not intended for production** 

Just a proof of concept.
It is possibly too inefficient to be useful although it may turn out not to be a problem if run on a dedicated workstation.
At least the efficiency can now be more easily estimated.

This tool is not recommended for public deployment because of the resource demands.

EGAPx Overview
===============

.. image:: $PATH_TO_IMAGES/Pipeline_sm_ncRNA_CAGE_80pct.png

**Warning:**
The current version is an alpha release with limited features and organism scope to collect initial feedback on execution. Outputs are not yet complete and not intended for production use. Please open a GitHub [Issue](https://github.com/ncbi/egapx/issues)  if you encounter any problems with EGAPx. You can also write to [e-mail] to give us your feedback or if you have any questions.  

EGAPx is the publicly accessible version of the updated NCBI [Eukaryotic Genome Annotation Pipeline](https://www.ncbi.nlm.nih.gov/genome/annotation_euk/process/). 

EGAPx takes an assembly fasta file, a taxid of the organism, and RNA-seq data. Based on the taxid, EGAPx will pick protein sets and HMM models. The pipeline runs `miniprot` to align protein sequences, and `STAR` to align RNA-seq to the assembly. Protein alignments and RNA-seq read alignments are then passed to `Gnomon` for gene prediction. In the first step of `Gnomon`, the short alignments are chained together into putative gene models. 
In the second step, these predictions are further supplemented by *ab-initio* predictions based on HMM models. The final annotation for the input assembly is produced as a `gff` file. 

**Security Notice:**

EGAPx has dependencies in and outside of its execution path that include several thousand files from the [NCBI C++ toolkit](https://www.ncbi.nlm.nih.gov/toolkit), and more than a million total lines of code. Static Application Security Testing has shown a small number of verified buffer overrun security vulnerabilities. Users should consult with their organizational security team on risk and if there is concern, consider mitigating options like running via VM or cloud instance. 


*To specify an array of NCBI SRA datasets in yaml*

::

   reads:
     - SRR8506572
     - SRR9005248


*To specify an SRA entrez query*

::

  reads: 'txid6954[Organism] AND biomol_transcript[properties] NOT SRS024887[Accession] AND (SRR8506572[Accession] OR SRR9005248[Accession] )'
   

**Note:** Both the above examples will have more RNA-seq data than the `input_D_farinae_small.yaml` example. To make sure the entrez query does not produce a large number of SRA runs, please run it first at the [NCBI SRA page](https://www.ncbi.nlm.nih.gov/sra). If there are too many SRA runs, then select a few of them and list it in the input yaml.   

Output
=======

EGAPx output will appear as a collection in the user history. The main annotation file is called *accept.gff*. 

::

 accept.gff
 annot_builder_output
 nextflow.log
 run.report.html
 run.timeline.html
 run.trace.txt
 run_params.yaml


The *nextflow.log* is the log file that captures all the process information and their work directories. ``run_params.yaml`` has all the parameters that were used in the EGAPx run. More information about the process time and resources can be found in the other run* files.  

## Intermediate files

In the log, each line denotes the process that completed in the workflow. The first column (_e.g._ `[96/621c4b]`) is the subdirectory where the intermediate output files and logs are found for the process in the same line, _i.e._, `egapx:miniprot:run_miniprot`. To see the intermediate files for that process, you can go to the work directory path that you had supplied and traverse to the subdirectory `96/621c4b`: 

::

 $ aws s3 ls s3://temp_datapath/D_farinae/96/      
                           PRE 06834b76c8d7ceb8c97d2ccf75cda4/
                           PRE 621c4ba4e6e87a4d869c696fe50034/
 $ aws s3 ls s3://temp_datapath/D_farinae/96/621c4ba4e6e87a4d869c696fe50034/
                           PRE output/
 2024-03-27 11:19:18          0 
 2024-03-27 11:19:28          6 .command.begin
 2024-03-27 11:20:24        762 .command.err
 2024-03-27 11:20:26        762 .command.log
 2024-03-27 11:20:23          0 .command.out
 2024-03-27 11:19:18      13103 .command.run
 2024-03-27 11:19:18        129 .command.sh
 2024-03-27 11:20:24        276 .command.trace
 2024-03-27 11:20:25          1 .exitcode
 $ aws s3 ls s3://temp_datapath/D_farinae/96/621c4ba4e6e87a4d869c696fe50034/output/
 2024-03-27 11:20:24   17127134 aligns.paf


  ]]></help>
  <citations>
    <citation type="doi">10.1093/bioinformatics/bts573</citation>
  </citations>
</tool>