Split file into a dataset collection

This tool splits a data set consisting of records into multiple data sets within a collection. A record can be for instance simply a line, a FASTA sequence (header + sequence), a FASTQ sequence (headers + sequence + qualities), etc. The important property is that the records either have a specific length (e.g. 4 lines for FASTQ) or that the beginning/end of a new record can be specified by a regular expression, e.g. ".*" for lines or ">.*" for FASTA. The tool has presets for text, tabular data sets (which are split after each line), FASTA (new records start with ">.*"), FASTQ (records consist of 4 lines), SDF (records start with "^BEGIN IONS") and MGF (records end with "^$$$$"). For other data types the text delimiting records or the number of lines making up a record can be specified manually using the generic splitter. If the generic splitter is used, an option is also available to split records either before or after the separator. If a preset filetype is used, this is selected automatically (after for SDF, before for all others).

If splitting by line (or by some other item, like a FASTA entry or an MGF record), the splitting can be either done alternatingly, in original record order, or at random.

If t records are to be distributed to n new data sets, then the i-th record goes to data set

• floor(i / t * n) (for batch),
• i % n (for alternating), or
• a random data set

For instance, t=5 records are distributed as follows on n=2 data sets

If the five records are distributed on n=3 data sets:

Note that there are no guarantees when splitting at random that every result file will be non-empty, so downstream tools should be able to gracefully handle empty files.

If a tabular file is used as input, you may choose to split by line or by column. If split by column, a new file is created for each unique value in the column. In addition, (Python) regular expressions may be used to transform the value in the column to a new value. Caution should be used with this feature, as it could transform all values to the same value, or other unexpected behavior. The default regular expression uses each value in the column without modifying it.

Two modes are available for the tool. For the main mode, the number of output files is selected. In this case, records are shared out between this number of files. Alternatively, 'chunking mode' can be selected, which puts a fixed number of records (the 'chunk size') into each output file.