annotate README.md @ 2:9ca209477dfd draft default tip

planemo upload for repository https://github.com/Onnodg/Naturalis_NLOOR/tree/main/NLOOR_scripts/process_annotations_tool commit 4017d38cf327c48a6252e488ba792527dae97a70-dirty

# BLAST Annotations Processor Script

This script processes a single **annotated BLAST file** together with a **FASTA file containing the same reads but unannotated**, generating multiple output files for downstream visualization and reporting.

It is designed for BLAST-based taxonomic pipelines and provides a complete overview of annotation quality, distribution, and composition of the analyzed dataset.

---

## Usage

The script performs the following main tasks:

1. Parse command-line arguments.  
2. Load the annotated BLAST results and the unannotated FASTA headers.  
3. Group BLAST hits per read and filter them by specified thresholds.  
4. Resolve taxonomic conflicts with the lowest common ancestor method using predefined uncertainty rules.  
5. Generate a variety of outputs of statistics and annotations for downstream use.


### Command Line Interface
The BLAST annotations processor can be run as a Python script:

```bash
python blast_annotations_processor.py [options]
```

Below are detailed examples for common use case

#### General use case
This example shows the general use of the tool.

**Requirements**:
  
Requirements as listed in the blast_annotations_processor xml file:

- Python version=3.12.3
- Matplotlib version=3.12.3
- Pandas version=2.3.2
- Numpy version=2.3.2
- Openpyxl version=3.1.5


**Input requirements**

- BLAST tabular file with alignment metrics, source and taxa
- Fasta file with preprocessed reads
- Header correspondence: Query identifiers in the BLAST output and FASTA headers **must match**. The script relies on matching IDs to merge annotations with read headers.  



**Example: Analyzing BLAST annotation result using curated database**

```bash
python annotate_blast_results.py
--input-anno 'annotated_curated_results.tabular'
--input-unanno 'unannotated_reads.fasta'
--eval-plot 'eval_curated.png'
--taxa-output 'taxa_curated.txt'
--circle-data 'circle_curated.txt'
--header-anno 'anno_curated.xlsx'
--anno-stats 'stats_curated.txt' 
--eval-threshold '1e-5'
--uncertain-threshold '0.9'
--use-counts
```

This command will:

- Parse the BLAST and FASTA files.  
- Filter hits using `E-value ≤ 1e-5`, `uncertainty threshold ≥ 90%`, and use read count in the circular data output.  
- Resolve taxonomic conflicts and generate plots, reports, and spreadsheet outputs in the given output files.


**Example Input (`annotated_curated_results.tabular`)**


```
    #Query ID	#Subject	#Subject accession	#Subject Taxonomy ID	#Identity percentage	#Coverage	#evalue	#bitscore	#Source	#Taxonomy
    M01687:476:000000000-LL5F5:1:1102:12299:1165_CONS(1758)	source=NCBI   sequenceID=EU382995   superkingdom=Eukaryota   kingdom=Viridiplantae   phylum=Streptophyta   subphylum=Streptophytina   class=Magnoliopsida   subclass=NA   infraclass=NA   order=Ranunculales   suborder=NA   infraorder=NA   superfamily=NA   family=Ranunculaceae   genus=Ranunculus   species=Ranunculus repens   markercode=trnL   lat=NA   lon=NA	source=NCBI	N/A	100.000	100	1.24e-38	152	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Ranunculales / Ranunculaceae / Ranunculus / Ranunculus repens
    M01687:476:000000000-LL5F5:1:1102:12299:1165_CONS(1758)	source=NCBI   sequenceID=JQ041850   superkingdom=Eukaryota   kingdom=Viridiplantae   phylum=Streptophyta   subphylum=Streptophytina   class=Magnoliopsida   subclass=NA   infraclass=NA   order=Ranunculales   suborder=NA   infraorder=NA   superfamily=NA   family=Ranunculaceae   genus=Ranunculus   species=Ranunculus repens   markercode=trnL   lat=NA   lon=NA	source=NCBI	N/A	100.000	100	1.24e-38	152	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Ranunculales / Ranunculaceae / Ranunculus / Ranunculus repens
    M01687:476:000000000-LL5F5:1:1102:12299:1165_CONS(1758)	source=NCBI   sequenceID=DQ410740   superkingdom=Eukaryota   kingdom=Viridiplantae   phylum=Streptophyta   subphylum=Streptophytina   class=Magnoliopsida   subclass=NA   infraclass=NA   order=Ranunculales   suborder=NA   infraorder=NA   superfamily=NA   family=Ranunculaceae   genus=Ranunculus   species=Ranunculus muricatus   markercode=trnL   lat=NA   lon=NA	source=NCBI	N/A	98.780	100	5.79e-37	147	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Ranunculales / Ranunculaceae / Ranunculus / Ranunculus muricatus
    M01687:476:000000000-LL5F5:1:1102:14619:1181_CONS(6595)	source=NCBI   sequenceID=HM590330   superkingdom=Eukaryota   kingdom=Viridiplantae   phylum=Streptophyta   subphylum=Streptophytina   class=Magnoliopsida   subclass=NA   infraclass=NA   order=Malpighiales   suborder=NA   infraorder=NA   superfamily=NA   family=Salicaceae   genus=Populus   species=Populus tremula   markercode=trnL   lat=50.47   lon=-104.37	source=NCBI	N/A	100.000	100	2.16e-52	198	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Malpighiales / Salicaceae / Populus / Populus tremula
    M01687:476:000000000-LL5F5:1:1102:14619:1181_CONS(6595)	source=NCBI   sequenceID=MH573985   superkingdom=Eukaryota   kingdom=Viridiplantae   phylum=Streptophyta   subphylum=Streptophytina   class=Magnoliopsida   subclass=NA   infraclass=NA   order=Malpighiales   suborder=NA   infraorder=NA   superfamily=NA   family=Salicaceae   genus=Populus   species=Populus alba   markercode=trnL   lat=NA   lon=NA	source=NCBI	N/A	99.074	100	1.01e-50	193	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Malpighiales / Salicaceae / Populus / Populus alba
    ...
```

**Example FASTA (`unannotated_reads.fasta`)**


```
    >M01687:476:000000000-LL5F5:1:1102:12299:1165_CONS(1758) merged_sample={}; count=1758; direction=right; sminR=40.0;
    ali_length=82; seq_b_deletion=219; seq_b_insertion=0; mode=alignment; sminL=40.0; seq_a_single=0; seq_b_single=0;
    gggcaatcctgagccaaatcctgctttcagaaaacaaaaagagggttcagaaagcaaagg
    gataggtgcagagactcaatgg

    >M01687:476:000000000-LL5F5:1:1102:14619:1181_CONS(6595) merged_sample={}; count=6595; direction=right; sminR=40.0;
    ali_length=107; mode=alignment; sminL=40.0; seq_a_single=0; seq_b_single=0; 
    gggcaatcctgagccaaatcctatttttcgaaaacaaacaaaaaaacaaacaaaggttca
    taaagacagaataagaatacaaaaggataggtgcagagactcaatgg
    ...
```

**Outputs**


| Output Type                   | Format | Description |
|-------------------------------|--------|-------------|
| **E-value distribution plots**| `.png` | Histogram  of BLAST E-values across all queries; useful for choosing score cutoffs or spotting anomalies. |
| **Taxonomic composition**     | `.txt` | Summarized counts or proportions of reads assigned to each taxonomic level. |
| **Circular taxonomy data**    | `.txt` | JSON-formatted hierarchical taxonomy structure, used to generate circular taxonomic plots. |
| **Header annotations**        | `.xlsx` | Excel workbook with merged and per-read annotation information, and alignment statistics. |
| **Annotation statistics**     | `.txt` | Summary metrics such as number of annotated reads, unassigned reads, unique taxa detected, and filtering statistics. |


**Output files (example)**


outputs

├── eval.png
<img width="2100" height="900" alt="afbeelding" src="https://github.com/user-attachments/assets/75b8fac6-da31-4980-a535-f9dd7ffd15bb" />


├── taxa.txt
```
Uncertain count per taxonomie level{'K': 0, 'P': 0, 'C': 0, 'O': 18, 'F': 10, 'G': 615, 'S': 1285}
percentage_rooted	number_rooted	total_num	taxon_level	indentificatie
100.00	3373	3373	K	Viridiplantae
100.00	3373	3373	P	  Streptophyta
99.97	3372	3373	C	    Magnoliopsida
1.96	66	3373	O	      Apiales
1.96	66	3373	F	        Apiaceae
1.22	41	3373	G	          Aegopodium
1.22	41	3373	S	            Aegopodium podagraria
0.27	9	3373	G	          Apium
0.27	9	3373	S	            Apium graveolens
0.47	16	3373	G	          Uncertain taxa
4.77	161	3373	O	      Asterales
4.77	161	3373	F	        Asteraceae
0.06	2	3373	G	          Achillea
0.06	2	3373	S	            Achillea millefolium
0.15	5	3373	G	          Artemisia
0.15	5	3373	S	            Uncertain taxa
0.03	1	3373	G	          Calendula
...
4.57	154	3373	G	          Uncertain taxa
0.12	4	3373	F	        Uncertain taxa
0.53	18	3373	O	      Uncertain taxa
0.03	1	3373	C	    Pinopsida
0.03	1	3373	O	      Cupressales
0.03	1	3373	F	        Taxaceae
0.03	1	3373	G	          Taxus
0.03	1	3373	S	            Taxus baccata
```
├── circle.txt
```
[
  {
    "labels": [
      "Bacteria",
      "Uncertain taxa",
      "Viridiplantae"
    ],
    "sizes": [
      2,
      1,
      29
    ]
  },
  {
    "labels": [
      "Pseudomonadota",
      "Uncertain taxa",
      "Streptophyta"
    ],
    "sizes": [
      2,
      1,
      29
    ]
...
    ],
    "sizes": [
      1,
      1,
      1,
      1,
      1,
      1,
      1,
      1,
      1,
      1,
      1,
      3,
      1,
      1,
      2,
      2,
      1,
      1,
      1,
      1,
      4,
      1,
      1,
      1,
      1
    ]
  }
]
```
├── anno.xlsx
```
header	e_value	identity percentage	coverage	bitscore	count	source	taxa	kingdom	phylum	class	order	family	genus	species
M01687:476:000000000-LL5F5:1:1102:8926:6561_CONS	2.33E-41	98.889	100	161	12	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Asterales / Asteraceae / Achillea / Achillea millefolium	Viridiplantae	Streptophyta	Magnoliopsida	Asterales	Asteraceae	Achillea	Achillea millefolium
M01687:476:000000000-LL5F5:1:2114:16883:18620_CONS	1.08E-39	97.778	100	156	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Asterales / Asteraceae / Achillea / Achillea millefolium	Viridiplantae	Streptophyta	Magnoliopsida	Asterales	Asteraceae	Achillea	Achillea millefolium
M01687:476:000000000-LL5F5:1:1102:20658:7882_CONS	1.63E-37	98.795	100	148	29	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Apiales / Apiaceae / Aegopodium / Aegopodium podagraria	Viridiplantae	Streptophyta	Magnoliopsida	Apiales	Apiaceae	Aegopodium	Aegopodium podagraria
M01687:476:000000000-LL5F5:1:1102:3453:17892_CONS	3.51E-39	100	100	154	179	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Apiales / Apiaceae / Aegopodium / Aegopodium podagraria	Viridiplantae	Streptophyta	Magnoliopsida	Apiales	Apiaceae	Aegopodium	Aegopodium podagraria
M01687:476:000000000-LL5F5:1:1101:16634:16511_CONS	5.79E-37	98.795	100	147	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Apiales / Apiaceae / Aegopodium / Aegopodium podagraria	Viridiplantae	Streptophyta	Magnoliopsida	Apiales	Apiaceae	Aegopodium	Aegopodium podagraria
...
M01687:476:000000000-LL5F5:1:1119:27044:6653_CONS	2.69E-35	97.59	100	141	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Fabales / Fabaceae / Vicia / Vicia faba	Viridiplantae	Streptophyta	Magnoliopsida	Fabales	Fabaceae	Vicia	Vicia faba
M01687:476:000000000-LL5F5:1:1109:2464:14257_CONS	7.37E-36	100	95	143	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Fabales / Fabaceae / Vicia / Vicia faba	Viridiplantae	Streptophyta	Magnoliopsida	Fabales	Fabaceae	Vicia	Vicia faba
M01687:476:000000000-LL5F5:1:1106:26123:11458_CONS	1.63E-37	98.795	100	148	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Fabales / Fabaceae / Vicia / Vicia faba	Viridiplantae	Streptophyta	Magnoliopsida	Fabales	Fabaceae	Vicia	Vicia faba
M01687:476:000000000-LL5F5:1:1104:24402:7089_CONS	5E-43	100	100	167	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Fabales / Fabaceae / Vicia / Vicia hirsuta	Viridiplantae	Streptophyta	Magnoliopsida	Fabales	Fabaceae	Vicia	Vicia hirsuta
M01687:476:000000000-LL5F5:1:2114:19155:4308_CONS	1.07E-39	100	94	156	13	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Gentianales / Apocynaceae / Vinca / Vinca minor	Viridiplantae	Streptophyta	Magnoliopsida	Gentianales	Apocynaceae	Vinca	Vinca minor
M01687:476:000000000-LL5F5:1:1117:11316:6653_CONS	4.96E-38	98.81	94	150	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Gentianales / Apocynaceae / Vinca / Vinca minor	Viridiplantae	Streptophyta	Magnoliopsida	Gentianales	Apocynaceae	Vinca	Vinca minor
M01687:476:000000000-LL5F5:1:1106:28052:14441_CONS	8.25E-41	98.876	100	159	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Asterales / Asteraceae / Xanthium / Xanthium strumarium	Viridiplantae	Streptophyta	Magnoliopsida	Asterales	Asteraceae	Xanthium	Xanthium strumarium
M01687:476:000000000-LL5F5:1:2118:15258:6790_CONS	8.25E-41	98.876	100	159	1	NCBI	Viridiplantae / Streptophyta / Magnoliopsida / Asterales / Asteraceae / Xanthium / Xanthium strumarium	Viridiplantae	Streptophyta	Magnoliopsida	Asterales	Asteraceae	Xanthium	Xanthium strumarium
```


└── stats.txt
```
metric	value
percentage_annotated	71.3862433862434
annotated_sequences	3373
total_sequences	4725
percentage_unique_annotated	89.46585409571608
unique_annotated	99826
total_unique	111580
```

---

#### CLI Arguments (common)

| Argument | Description |
|----------|-------------|
| `--input-anno` | Path to the annotated BLAST results (tab-separated) |
| `--input-unanno` | Path to the unannotated reads FASTA file |
| `--eval-plot` | Output file where eval plot output will be written |
| `--taxa-output` | Output file where taxa output will be written |
| `--circle-data` | Output file where circle data output will be written |
| `--header-anno` | Output file where header annotation results will be written |
| `--anno-stats` | Output file where annotation statistics will be written |
| `--eval-treshold` | Maximum E-value to retain hits (default: `1e-5`) |
| `--uncertain-threshold` | percentage for which lca picks the majority taxon (default: `0.9 (90%)`) |
| `--use-counts` | Use read counts in the circle data output when true (default: `True`) |

---


### Galaxy integration

The tool is also available through the Galaxy platform:

- **Galaxy Toolshed**: The BLAST annotations processor tool is available in the Galaxy Toolshed, 
  enabling easy installation into any Galaxy instance.
- **Web-based interface**: Users can upload sequence files, configure validation parameters through the GUI, 
  run validations, and download results.
- **Workflow integration**: The tool can be incorporated into Galaxy workflows for automated processing pipelines.

To use the tool in Galaxy:
1. Install the tool from the Galaxy Toolshed (search for "blast_annotations_processor")
2. Upload your raw read and BLAST files to your Galaxy history
3. Configure parameters through the GUI
4. Run the tool
5. View results and download validation reports and tabular annotations

## License

No license yet

## Citation

If you use this software in your research, please cite this repository.

## Contact

For questions or issues:
- GitHub Issues: https://github.com/Onnodg/Naturalis_NLOOR/issues
- Email: [e-mail] (until Febuary 2026)

## Acknowledgments

This tool was developed to support the New lights on old remedies project, a PhD project by Anja Fischer.