Mercurial > repos > dawe > srf2fastq
view srf2fastq/io_lib-1.12.2/CHANGES @ 0:d901c9f41a6a default tip
Migrated tool version 1.0.1 from old tool shed archive to new tool shed repository
| author | dawe |
|---|---|
| date | Tue, 07 Jun 2011 17:48:05 -0400 |
| parents | |
| children |
line wrap: on
line source
Version 1.12.2 (15th Jan 2010) -------------- * Extra options in srf2fastq: -S to output split regions sequentially to stdout. -r to request a region to be reverse complemented before output. * API addition - Added pooled_alloc.h. This is a general purpose mechanism of pooling multiple fixed size memory allocations into fewer malloc() library calls. - HashTables now have a HASH_POOL_ITEMS option to use the above pooling system. This reduces memory wasted and speeds them up. * Bug fix: Fixed ztr_add_text() so that is leaves two nul bytes on the end of TEXT chunks instead of one, as documented in the ZTR specification. * Bug fix: Fixed buffer overrun in parse region chunks; srf2fastq and srf2fasta. * Bug fix: API read_sff_read_data() did not skip ahead to the next 8-byte boundary. Version 1.12.1 (7th August 2009) -------------- * Fixed the endianness detection in io_lib/os.h when used in conjuction with auto-conf. This fix allows for "fat" binaries to be built on MacOS X. * Fixed io_lib-config program to use -lstaden-read instead of -lread. Version 1.12.0 (29th July 2009) -------------- * Renamed the library from libread.so to libstaden-read.so. This was already the case for the Fedora bundled RPM. * Switched to using libtool to allow building of dynamic libraries. Note that this is tweaked to not use -rpath though. Proper library versioning has been added too. * Removed deprecated platform specific tools: illumina2srf, srf2illumina. * Srf_info now reports the compressed size of chunks, sorted by type, in addition to their counts. It also correctly sums to over 2Gb now for base-call counting. * Various SRF tools have had the maximum sequence length changed from 1024 to 10000. This allows for even the most gifting capillary traces. * API - The Array functions now take size_t instead of int for the array dimensions. (API CHANGE) - Removed the (unused?) pipe2 function from compress.h. This was intended to be internal only, and it now clashes with a new linux kernel function. (API CHANGE) - Added iterators to the HashTable* api. * Bug fixes - Fixed a memory allocation bug in the codes2codeset() function. - ztr2read() should now work better on ZTR structs with no BPOS chunk. - Fixed various srf tools when facing an SRF file containing zero chunks in the data block header. - index_tar handles some GNU tar extensions better (LongLink). Version 1.11.6.1 (9th December 2008) ---------------- * Identical except removal of a debugging printf statement in solexa2srf. Version 1.11.6 (9th December 2008) -------------- * illumina2srf, srf2illumina, srf2fastq - We no longer change from log-odds to phred when storing data in SRF, instead preferring to just mark it in correct input scale. srf2fastq now honours this scale information and so the conversion from log-odd to phred is done at the export stage instead. (Chris Saunders) - Bug fix to srf2illumina qcal conversion. Combined with above changes the qcal output should now be 100% identical to the original data input via illumina2srf. * API - New function srf_next_ztr_flags. This is like srf_next_ztr but also returns the SRF flags value (good/bad read, etc). * srf_filter, srf2fastq, srf_info (Steven Leonard) - Improved support for multiple index blocks in SRF files, eg from manually concatenated files. - srf2fastq now sports options for splitting the output into multiple fastq files when the input data is a paired-end run. Version 1.11.5 (3rd December 2008) -------------- * Illumina2srf - Fixed major bug with using *both* -qf and -qr together. The quality values for the reverse strand were shifted by one character. - Fixed qcal quality values so they're not shifted down by 64 (illumina format fastq). - Fixed bugs in parsing directory names if not matching the expected format. * Removed major memory leaks from srf_filter. * hash_sff now has support for outputting the table of contents to a new file rather than appending to an existing sff file or copying the entire contents to a new file. * Various man pages have been added. The list is still incomplete though. Additions are most welcome. * New program: srf_list. This lists and/or counts the number of sequences within an SRF file. Version 1.11.4 (11th September 2008) -------------- * New "make check" build target to perform some automated tested. Currently limited to testing the SRF tools. * Fixed machine endianness issues. Specifically this resolves known Intel MacOS-X problems. * New SRF tools - srf_info: reports simple metrics on the contents of an SRF file. - srf_filter: slices and dices the SRF file to produce a new one with various types of data removed. * illumina2srf - Minor float/int rounding change when storing int/nse/sig2 data. - Improved error detection such that it returns a failure code more often given a parsing issue. - Added -pf/pr parameters for storing Phasing files. - Reduced memory usage, especially on large numbers of clusters per tile. We may now produce multiple DBH blocks per tile. Also major reduction to memory when handling the .params files. - Added storage of 2nd .params file (firecrest). - Fixed bug in the automatic base-call version identification. - Fixed a bug with using -qf/qr when not providing all tiles (ie not starting from tile number 1). - Bug fix with storing the reverse matrix file in paired-end runs; a duplicate of the forward one was being used instead. * General SRF - Improved error checking in srf_index_hash. It now spots duplicate reads and also has a -c option to check an existing SRF file without writing the index. - Fixed a memory leak in srf_next_ztr(), triggered in srf2fastq -C. Version 1.11.3 (9th July 2008) -------------- * illumina2srf change: - IMPORTANT bug fix to illumina2srf when using the "-r" flag to store raw (.int and .nse) data. This could often result in corrupting the data ZTR meta-data for the SMP4 chunks resulting in confusion over which trace channels are raw and which are processed. Fortunately the corruption is reversable. For more details and a fix see the ssrformat announcement of the issue: http://www.bcgsc.ca/pipermail/ssrformat/2008-July/000531.html * General SRF changes: - Removed a memory leak in ztr_find_chunks(). - Added SRFB_NULL_INDEX as an SRF block type. This provides a more transparent way to skip over the 8 zero value bytes that may exist at the end of an SRF file missing an index block. * Other changes - Fixed a bug in extract_seq when operating on multiple files and outputting to a file rather than a pipe. An erroneous seek in the mFILE code lead to it repeatedly truncating the output, resulting in one sequence file at the end instead of multiple files. Version 1.11.2 (4th June 2008) -------------- * solexa2srf/srf2solexa changes: - Renamed to illumina2srf/srf2illumina. - Incorporated support the IPAR format (Come Raczy, Illumina). - Added support for qcal format data (Come Raczy). - Added -C option to tag data as failing the chastity filter, but it is still included in the SRF output (Camil Toma). - Many more additional features added to srf_dump_all provided by Camil Toma. It somewhat overlaps srf2solexa now, but may still have it's own use. - Ztr TEXT chunks now output in srf2solexa. - Improved ways to specify matrices (-mf/-mr) in solexa2srf. - solexa2srf is substantially faster when reading gzipped files. - The -N/-n naming scheme options for solexa2srf now default to the same conventions used by GERALD. Added additional %d, %m and %r format rules too. - Calibrated confidence values are now output if -qf or -qr paramaters are used, in addition to uncalibrated ones. These are stored in phred scale in a CNF1 ZTR chunk. * srf2fastq now has a -c option to output calibrated confidence values (if present). It also supports multiple archives on the command line. * SRF fixes: - Better handling of full pathnames in solexa2srf. - Use binary IO mode; fixes bugs on Windows. - Fixed an error where some chunks were not compressed properly (valid still, just not compressed). - Removed memory corruption in solexa2srf (in rare cases). - Fixed bug with binary formatted read_id suffixes (fixed by Cristian Goina). - Initialised memory in hash table code (used in indexing amongst other things). - Indexes very occasionally failed to find a trace that did infact exist. - Removed memory leak in construct_trace_name (patch from John Emhoff, Helicos). - Fixed reading of XML block in srf_read_xml(). From John Emhoff. * Added SRF= format string to TRACE_PATH to facilitate on-the-fly extraction from indexed SRF files. This means io_lib can now transparently pull traces from an archive or treat it as if it was a directory - eg "foo.srf/IL15_..._123:456". * Bug fix (SF-1898427) - now builds on Fedora. * Better handling of 64-bit file size sensing in autoconf. Version 1.11.1 (not officially released - internal testing only) -------------- Version 1.11.0 (20th February 2008) -------------- First official release of v1.11.0 and SRF support. * Further speed improvements to solexa2srf. * Added extract_qual program (analogous to extract_seq). * Added new srf2fasta program and also sped up srf2fastq by 25%. * Solexa2srf now supports storing the raw .int/.nse trace data instead of or in addition to the processed .sig2 data. * Solexa2srf now stores enough to reproduce sufficient firecrest output to rerun the solexa basecaller. Specifically that's a couple matrix files and 'region' data for paired end runs. * Minor changes / bug fixes: - extract_seq no longer attempts to gzip the output by default if the input was gzipped - ztr2read conversion (eg visible in trace_dump) now correctly handles ZTR files with multiple SMP4 chunks. - Fixed memory leaks in various bits of SRF code (srf_extract_linear mainly and srf_index_hash). Version 1.11.0b8 (25th January 2008) ---------------- (Hopefully final beta test of SRF code before official 1.11.0 release.) * Bug fixed the index format. We incorrectly handled null dbhFile and containerFile elements plus incorrectly computing the index size. * Improvements for solexa2srf code. - Can store raw vs processed data - Stores matrix and .params contents. - Optional chastity filtering. - Input data may now be gzipped. * Minor fixes to output of trace_dump and ztr_dump. * Minor srf_index_hash bug fixes (when dealing with concatenated indexed files). Version 1.11.0b7 (11th January 2008) ---------------- * IMPORTANT bug fix to the SRF format. The Data Block Header had the blocksize field 4 bytes too large. Now fixed. Old SRF files will not be readable by this new code (as they were in error). Version 1.11.0b6 (2nd January 2008) ---------------- * Changes to adhere to SRF v1.3: * Removal of the readID counter. * Added support for printf style name formatting. * Minor index format tweaks (64-bit data, dch/container filenames). Index format is therefore now 1.01. Version 1.11.0b5 (8th November 2007) ---------------- * Major reorganisation of directories. All library code is in subdir "io_lib". The code now uses "io_lib/xxx.h" in all include statements too. * Fixed memory leaks in ZTR code * Various SRF bug fixes and better support for sample OFFS metadata in both ZTR/ZTR. * Added srf_extract_hash program to perform random-access on a hash indexed SRF archive. Version 1.11.0b4 (26th October 2007) ---------------- * The SRF format now supported adheres to version 1.2. * More speedups, in particular focusing on uncompression this time, so srf2solexa is an order of magnitude faster. * ztr2read() now honours the read_sections() options and so is much faster when only decoding (say) base and quality values. * New program srf2fastq. * Internal changes to various ztr data structures. If you use these yourself take note of the new ztr_owns fields to avoid memory leaks. Version 1.11.0b3 (16th October 2007) ---------------- * Major speed improvements for compression. solexa2srf is now 30-35x faster. * Fixed various buffer overruns and memory leaks reported by valgrind in the new deflate interlaced and SRF code. Version 1.11.0b2 (2nd October 2007) ---------------- * Minor version change to fix typoes in Makefile system. Version 1.11.0b1 (28th September 2007) ---------------- Beta release 1. * Added preliminary SRF support. This consists of a new subdirectory 'srf' (yes these all really need merging into a single directory, but that's a later task), a substantial update to ZTR and a variety of SRF tools in progs. The old huffman_static.[ch] files were renamed and substantially worked upon to create deflate_interlaced.[ch]. Added new compression types. xrle2, tshift and qshift. The latter two of these are very specific to trace and quality packings. May need to rename to be more generic. Version 1.10.3 (???) -------------- * The HashTable interface now also allows for Bob Jenkins' lookup3 64-bit hash function. This allows for substantially larger hash tables. * Replaced tempnam() with tmpfile(). On systems without tmpfile (Windows) this is simply a wrapper to use the old tempnam calls. * hash_extract bug fix for windows: now operates in binary mode. * INCOMPATIBLE CHANGE: On windows we now use semi-colon as the path separator. The reason is that with the MinGW getenv() seems to do "clever things" with PATH variables and consequently ends up corrupting our clumsy attempt of escaping colons in paths. * Fasta format is semi-supported in "plain" format. It returns the first entry when reading. * Experimental support for static huffman (STHUFF) compression type. Version 1.10.2 (30th May 2007) -------------- Primarily this is a bug fix release. * Convert_trace now has -signed and -noneg options to control signed vs unsigned issues when shifting trace data about. * Include files now have C++ extern "C" style guards around them. * Various programs now accept -ztr command line arguments to force ZTR format reading. This is for consistencies sake only and it is recommended that users simply let the programs automatically detect the file formats. * Hash_exp now outputs to the same file containing the experiment files (in appended hash-table mode). It also has better Windows handling (stripping ^M and using binary mode). * hash_extract bug fix: now only needs at least 1 filename specified when fofn mode is not in use. * mFILE emulation: bug fixes when dealing with ftruncate, append mode, checking for read/write flags, new mfcreate_from() function. * ZTR: added an experimental ZTR_FORM_STHUFF compression scheme. This uses static huffman encoding on a predefined hard-coded set of huffman tables. The purpose (as yet not put into action) is to allow efficient compression of very small data sets for Illumina, AB SOLiD, etc style traces. Version 1.10.1 (20th June 2006) -------------- * Trace files are now opened in read-only mode by default (open_trace_file func). Version 1.10.0 (15th June 2006) -------------- * Two new environment variables are used, EXP_PATH and TRACE_PATH, to replace RAWDATA. EXP_PATH is used when the new open_exp_mfile() function is called and TRACE_PATH is used when open_trace_mfile() is called. Both default to using RAWDATA when EXP or TRACE env is now found. Also defined a trace type TT_ANYTR which is analogous to the existing TT_ANY except it will not look for experiment or plain format files. Modified the various example programs to use the appropriate open call. This allows for traces and experiment files to have identical names, such as is usually the case when querying named trace objects from a trace server. * New program: extract_fastq to generate FASTQ output format. * New program: hash_exp. This allows multiple experiment files to be contatenated together and then indexed so io_lib can still treat them as single files. * The URL based search path mechanism now by default uses libcurl instead of wget. This makes it considerably faster. * If an element in RAWDATA, EXP_PATH or TRACE_PATH now starts with the pipe symbol ("|") then the compressed file extension code is negated for that search element. (This prevents looking for foo.gz, foo.Z, foo.bz2, etc if it fails to find foo.) * Added HashTableDel() and HashTableRemove() functions to take items out of a hash table. * ZTR's compress_chunk() and uncompress_chunk() functions are now externally callable. * New program io_lib-config. This has --version, --cflags and --libs options to query the appropriate configuration when compiling and linking against io_lib. There's also a new io_lib.m4 file which provides an AC_CHECK_IO_LIB autoconf macro to use io_lib-config and generate appropriate Makefile substitutions. * Updated the autoconf code to support libcurl searching. * Renamed SCF's delta_samples[12] functions to be scf_delta_samples[12]. (From Saul Kravitz) * Added a '-error filename' option to convert_trace. (From Saul Kravitz) * Bug fix: HashTableAdd() now works properly with non-string keys. * Bug fix to read_dup(). * Bug fix to xrle which could read past the array bounds. It also now handles run-lengths of 256 or more. * Bug fix: the fwrite_* functions no longer close the FILE pointer given to them. * Bug fix to fdetermine_trace_type(); it now rewinds the file back. * Bug fix to mfseek and mrewind; they both now clear the EOF flag. * Bug fix to find_file_dir(). Version 1.9.2 (14th December 2005) ------------- * Added AC_CHECK_LIB calls for the nsl and socket libraries (gethostbyname / socket functions). Needed for Solaris compilations. * In extract_seq, used open_trace_mfile instead of open_trace_file. Functionally this is the same, but it is faster. * fwrite_reading() now frees the temporary mFILE it created. * mfreopen_compressed() no longer closes the original FILE pointer. This brings it back into line with the original functionality provided in 1.8.x. It also cures a bug where the old file pointer was often left opening meaning operates on many files could could cause a resource leak ending in the inability to open more trace files. * Added private_data and private_size to the Read struct. Populate these when reading SCF files. * Hash_extract now returns an error code to the calling process upon failure. * Major overhaul of hash_sff. It no longer loads the entire file into memory. It can now cope with adding a hash index to an archive that already contains an index. * Added support for 454's "sorted index" code. NB this is based on the extraction code from their getsff.c code and has not been tested with a genuine indexed SFF file yet. * Fixed an uninitialised memory access in mfload(). * Fixed a bug where hash query searches for items that do not exist and map to an empty bucket could cause hangs or crashes. * Fixed a hang in mfload() when reading a zero length file. Version 1.9.1 ------------- * Implemented the SFF (454) file structure, currently as read-only. This is supported both as an archive containing multiple files and also as a single SFF entry. * Allow for SFF=? components in RAWDATA search path. * Tar files, SFF archives and hashed archives (eg hashed tar, sff, or "solid" archives) may now be used as part of a pathname. Eg if a tar file foo.tar contains entry xyzzy.ztr then we can ask to fetch trace foo.tar/xyzzy.ztr instead of requiring setting of the RAWDATA environment variable. * Changed the HashFile format slightly. It's now format 1.00. The key difference is that it has a file footer pointing back to the hashfile header (so the hashfile can be appended to an archive) and it also has an offset in the header to apply to all seeks within the archive itself, so it can be prepending to an archive that's already been indexed without breaking the offsets. Extended the hash_tar program to allow control over these header options. * Fixed divide-by-zero buf when calling mfread for zero * Removed the warning for unknown ZTR chunk types. It now just silently stores them in memory. * mfopen now honours binary verses ascii differences (and so updated Read.c calls accordingly) so that Windows works better. * Removed file descriptor 'leak' in write_reading(). * Unset compression_used when opening uncompressed files instead of leaving as the last value. * Fixed a file descriptor (and some memory) leak in freopen_compressed. (Bug ID #1289095) * Fixed the hash file saving and loading so that it works on all platforms instead of just x86 linux. There were bugs in assuming the size of structures. The assumptions are still there in that I assume they pad the same internally (for ease of coding - we can change it when we finally see a system which operates differently), but the final "boundary" padding has been resolved. Version 1.9.0 ------------- * ***INCOMPATIBILITIES*** to 1.8.12 - The Exp_info structure now internally contains an "mFILE *" member instead of "FILE *" member. If you use the experiment file functions for I/O then hopefully it'll still work. However if you directly manipulated the Exp_info yourself using fprintf etc then you will need to modify your code. - Some functions no longer have external scope. Most of these did not previously have external function prototypes. If you have a burning need to use one of these, please contact me directly via sourceforge. The full list is: ctfType (global variable) ztr_encode_samples_C replace_nl ztr_encode_samples_G ctfDecorrelate ztr_encode_samples_T exp_print_line_ ztr_decode_samples find_file_tar ztr_encode_bases find_file_archive ztr_decode_bases find_file_url ztr_encode_positions ztr_write_header ztr_decode_positions ztr_write_chunk ztr_encode_confidence_1 ztr_read_header ztr_decode_confidence_1 ztr_read_chunk_hdr ztr_encode_confidence_4 compress_chunk ztr_decode_confidence_4 uncompress_chunk ztr_encode_text ztr_encode_samples_4 ztr_decode_text ztr_decode_samples_4 ztr_encode_clips ztr_encode_samples_common ztr_decode_clips ztr_encode_samples_A - Some external functions have changed prototypes to use mFILE instead of FILE. Most cases of these I've put in place a wrapper function with the old name, but not yet all. Functions changed are: ctfFRead write_scf_samples32 ctfFWrite write_scf_base exp_print_line write_scf_bases exp_print_mline write_scf_bases3 exp_print_seq write_scf_comment read_scf_header fcompress_file read_scf_sample1 fopen_compressed read_scf_samples1 freopen_compressed read_scf_samples31 be_write_int_1 read_scf_sample2 be_write_int_2 read_scf_samples2 be_write_int_4 read_scf_samples32 be_read_int_1 read_scf_base be_read_int_2 read_scf_bases be_read_int_4 read_scf_bases3 le_write_int_1 read_scf_comment le_write_int_2 write_scf_header le_write_int_4 write_scf_sample1 le_read_int_1 write_scf_samples1 le_read_int_2 write_scf_samples31 le_read_int_4 write_scf_samples2 fdetermine_trace_type - Removed support for the OLD unix "pack" program as a valid trace compression algorithm. - Removed CORBA support. (It wasn't enabled and I've no idea if it even worked as I cannot test it.) - The default search order for RAWDATA now has the current working directory at the end of RAWDATA instead of the start. * Significant speed ups, particularly when dealing with reading gzipped files or when extracting data from tar files. * New external functions for faster access via mFILE (memory-file) structs. These mimic the fread/fwrite calls, but with mfread/mfwrite etc. * Numerous minor tweaks and updates to fix compiler warnings on more stricter modes of the Intel C Compiler. * Preliminary support for storing pyrosequencing style traces. This has been modeled on the flowgram data from 454, but should be applicable to other platforms. ZTR has been updated to incorporate this too. The Read structure also has flow, flow_order, nflows and flow_raw elements too. Code to convert these into the more usual traceA/C/G/T arrays exists currently as part of Trev (in tk_utils in the Staden Package), but this may move into io_lib for the next official release. * New hash_tar and hash_extract programs. These replace the index_tar program for rast random access. For RAWDATA include "HASH=hashfile" as an element to get io_lib to use the archive hash. It's possible to create hash files of most archive formats as the hash itself contains the offset and size of each item in the archive. This means that extracting an item does not need to know the format of the original archive. Some benchmarks show that on ext3 it's actually faster to extract files from the hash than directly via the directory. This was testing with ~200,000 files, whereupon directory lookups become slow. I'd imagine ResierFS or similar to be faster. * Added an XRLE encoding for ZTR. This is similar to the existing RLE mechanism but it copes with run length encoding of items larger than a single byte. It's current use is for storing the 4-base repeating flow order in 454 data. Version 1.8.12 -------------- * The ABI format code now reads the confidence values from KB (via PCON field). * New program: trace_dump. Like scf_dump, but deals with generic input formats. * Slightly more sensible average spacing calculation in the ABI reading code. It's still not perfect, but is only used when the real spacing value is negative or zero. * Disabled the base-reordering fix for ABI files. We believe the bug causing this no longer exists. * Expriment file format: added FT (EMBL feature table) and LF (LiGation; a combination of LI and LE) records. * Experiment files: strip out digits from the sequence we read (for better support of EMBL files). * Experiment files: fixed a potential buffer overrun in the conversion of binary confidence values to ascii values. * Minor improvements to portability (INT_MAX vs MAXINT2) and removal of some compilation warnings. * Extract_seq now accepts a -fofn argument. * New functions: read_update_base_positions() and read_udpate_confidence_values() to replace read_update_opos(). These apply an edit buffer to the sequence details and are used (for example) within Trev for saving edits back to a trace file. * Better error handling in fcompress_file(). * New specifiers in RAWDATA. Added a generic URL format (eg "URL=http://some/where/trace=%s") implemented via use of wget. There is also an ARC= format to make use of the Sanger Trace Archive, although currently this will not work externally. * Zero memory used in read_alloc(). Fixes to read_dup(). Version 1.8.11 -------------- * Rewrote the background subtraction in convert_trace to deal with each channel independently. * Make install now install the include files (all of them, although not all are strictly required) in $prefix/include/io_lib/. * Moved the ABI filter wheel order (FWO) reading from outside the sample reading code into the general reading bit as this is needed for reading the comments too (it also applies to the order of the signal strengths). Hence when the READ_COMMENTS section only is defined it now works correctly. * Moved the DataCount #defines into static values and added a abi_set_data_counts function to change these. This allows reading of the raw data from ABI files. This is used within the new convert_trace -abi_data option. * Removed a one-byte write buffer overflow in the CTF writing code. * New Experiment file records WL and WR for indicating clip points within a WT trace. * Removed the saved copy of fp for exp_fread_info in 'e' structure as it doesn't belong to us. (If we do store it there then the exp_destroy_info function will free it and this causes bugs.). POTENTIAL INCOMPATIBILITY: if you assumed that exp_destroy_info closed the files that you opened and passed into exp_fread_info, then this is no longer true. * New function read_dup() to copy a Read structure. * get_read_conf() now deals with loading confidence values from any suitable format and not just SCF. * Fixed memory leak in ztr (ztr->text_segments). Version 1.8.10 -------------- * Added Steven Leonard's changes to index_tar. It no longer adds index entries for directories, unless -d is specified. It also now supports longer names using the @LongLink tar extension. * Fixed a bug in exp2read where the base positions were random if experiment files are loaded without referencing a trace and without having ON lines. * New program get_comment. This queries and extracts text fields held within the Read 'info' section * Overhaul of convert_trace to support the makeSCF options (normalise etc). Version 1.8.9 ------------- Sorry this isn't a proper changes-by-source listing. Any suggestions for how I collate the 'cvs log' output into something more concise? The below text is simply a list of changes, but more complete than in the NEWS file. * ZTR spec updated to v1.2. The chebyshev predictor has been rewritten in integer format. The old chebyshev still has a format type allocated to it (73), but the new ICHEB format (74) is now the default. The old floating point method was potentially unstable (eg when running on non IEEE fp systems). The new method also seems to save a bit more space. * The docs and code disagreed for CNF4 storage. Changed the docs to reflect the code (which does as intended). * ZTR speed increase. Follow1 is substantially faster, increasing write times by about 10%. * New named formats types. ZTR1, ZTR2 and ZTR3. ZTR defaults to ZTR2, but we can explicitly ask for another compression level if desired. Also explicit statement of format (TT_ZTR instead of TT_ANY) removes the need for a rewind() call and so ZTR can now work through a pipe. * General tidy up to remove a few compilation warnings (missing include files, signed vs unsigned issues, etc). * Initial support is included for BioLIMS integration, but this is not complete. (Unfortunately it requires access to a non-public library.) * New function compress_str2int - opposite of existing compress_int2str. * (Steven Leonard). Uses zlib for gzip compression and decompression. These are extracts from the full Staden Package change log. They may not be immediately obvious when taken out of context, but we feel this information may still be useful to the users of io_lib. 23rd August 2000, James ----------------------- 1. Removed find_trace_file and added an open_trace_file function. The idea is that searching for a files existance is better done by attempting to open it. This in turn allows for more possibilities of file searching. Makefile utils/open_trace_file.c read/Read.c read/scf_extras.c read/translate.[ch] progs/extract_seq.c 2. Added a TAR option to RAWDATA. We can now read trace files directly from tar files (although they cannot be written to directly). utils/open_trace_file.c utils/tar_format.h 3. Created an index_tar program to optimise tar reading, although it is not mandatory. progs/index_tar.c progs/Makefile 4. Fixed a bug when dealing with plain text files containing spaces. plain/seqIOPlain.c 31st July 2000, James --------------------- 1. Renamed TTFF to be ZTR. read/Read.[ch] utils/traceType.c utils/compress.c ttff/* -> ztr/* README 2. ZTR reading will now stop when it spots a ZTR magic number. This allows concatenation of ZTR files. ztr/ztr.[ch] 15th June 2000, James --------------------- 1. Added a TTFF_FOLLOW filter type to TTFF. This is enabled with compression level 2 for the chromatogram data. io_lib/ttff/ttff.[ch] io_lib/ttff/compression.[ch] 9th June 2000, James -------------------- * RELEASED 1.8.4 */ 1. Added zlib bits to windows compilation. io_lib/mk/windows.mk 2. Updated convert_trace. It can now reduce sample-size to 8-bit (with the "-8" option) and the formats may now be specified as either integer or text format. The text format is case insensitive. io_lib/progs/convert_trace.c io_lib/utils/traceType.c 3. More windows binary vs ascii fixes. When reading we switch to binary mode before attempting fdetermine_trace_type, otherwise it fails to auto-detect TTFF (which includes a newline as part of the magic number). Also added a _setmode() call to the fwrite_reading code too. io_lib/read/Read.c 4. Changed the default compression technique of TTFF to that used in 1.8.2. I accidently left it set to the experimental dynamic-delta method in 1.8.3, which currently doesn't have the uncompression function! Also removed lots of debugging output. io_lib/ttff/ttff.c io_lib/ttff/ttff_translate.c 5. Bug fix to exp2read - when no right hand quality cutoff is specified we were defaulting to the left end of the trace, instead of the right end. (This only happens when opening experiment files which do not have clip points.) io_lib/read/translate.c 6. Changed the strftime() format in ABI reading code to use %H:%M:%S instead of %T, as %T doesn't appear to be part of ANSI (I think it's probably XPG4-UNIX). It worked on Unix machines, but not on MS Windows. io_lib/abi/seqIOABI.c 8th June 2000, James -------------------- * RELEASED 1.8.3 */ 1. Updated the CTF support so that it includes a couple of new block types. This allows for base positions being non-sequentially ordered, as is possible in severe compressions. io_lib/ctf/ctfCompress.c 2. Overhaul of TTFF format - now more PNG based in style. Still highly experimental. io_lib/ttff/* 16th May 2000, James -------------------- * RELEASED 1.8.0 */ 1. Added szip support. Szip generally gives better compression ratios than gzip and often marginally better than bzip2, but is generally considerably slower at decompression. io_lib/utils/compress.[ch] 2. Merged in Jean Thierre-Mieg's CTF code. This is a compressed trace format which holds the same data as SCF, but in reduce space. io_lib/read/Read.[ch] io_lib/utils/traceType.c io_lib/ctf/* 3. Added my own highly experimental TTFF format. (Thanks to Jean Thierre-Mieg for re-awakining my interest in this.) TTFF files are typically equivalent in size to bzip2'ed SCF files, but are much quicker to write than any of the currently supported compressed formats. Depends on zlib. io_lib/read/Read.[ch] io_lib/utils/traceType.c io_lib/ttff/* 4. Reorganised the Makefiles for easier building. */Makefile 5. New program "convert_trace". Primarily a test tool at present as it needs a friendlier interface. progs/convert_trace.c 20th April 2000, James ---------------------- 1. Removed a file-descriptor leak in extract_seq. io_lib/progs/extract_seq.c 22nd March 2000, James ---------------------- 1. Fixed bug in time formatting from ABI files. We used strftime code %a without setting tm.tm_wday (number of days since sunday). It's not easy to work that out, so we convert from struct tm to time_t, which resets any errornous elements of struct tm. Also fixed a silly error where the end time was set to the start time (incorrectly). io_lib/abi/seqIOABI.c 25th February 2000, James ------------------------- 2. Added checks for QR <= QL in the exp2read conversion function. This caused trev to display incorrectly (blanking incorrect screen portions) when dealing with inconsistent experiment files. Also changed qclip so that it doesn't create this inconsistent case. io_lib/read/translate.c 1st February 2000, Kathryn -------------------------- 1. Fixed bug which caused init_exp to crash when QL was more than 5 digits. Increased it to handle 15 digits. io_lib/read/translate.c 27th January 2000, James ------------------------ 1. Moved Gap4's copy of scf_extras into io_lib, and renamed io_liub's scf_bits to be scf_extras (to avoid editing too many #include statements). Without this we were getting errors due to dynamic linking using odd copies. Eg loading libread.so and then libgap.so meant that find_trace_file called from edUtils2.c (libgap.so) would pick up the first copy from libread.so, despite the fact that there's also a copy in the same libgap.so. gap4/scf_extras.[ch] io_lib/scf_bits.[ch] 25th January 2000, Kathryn -------------------------- 1. Fixed crash in qclip due to insufficent arguments being passed to find_trace_file and also fixed an array bounds error in scan_right of qclip.c io_lib/read/scf_bits.c 19th January 2000, James ------------------------ 4. Copied bits of the fakii and cap2/3 scf/expFile reading code into io_lib. Not all of this is in there, just the things which seem to be common and sensibly fit there. This also helps qclip to build on Windows. FIXME: We should now remove some of this code from Gap4. Also fixed a small memory leak in fopen_compressed() - it wasn't freeing the result of tempnam(). io_lib/read/translate.c io_lib/read/scf_bits.[ch] io_lib/read/seqInfo.[ch] io_lib/utils/files.c io_lib/utils/compress.c 31st August 1999, James ----------------------- 1. -fasta_out mode of extract_seq now changes - to N. io_lib/progs/extract_seq.c 27th August 1999, James ----------------------- 1. The order of information items added by the abi to scf code has changed, to make it more sensible. Also fixed a bug in the textual (rather than numerical) date output, and wrote this to the DATE field. io_lib/abi/seqIOABI.c 2. makeSCF no longer adds a MACH field, as this was redudant. io_lib/abi/makeSCF.c 3. Extract_seq now has proper use of CL and CR when using -cosmid_only. It was assuming they were the same as QL/QR and SL/SR, which is not the case (rather it's like having a CS line of `CL`..`CR`). Extract_seq also now has a -fasta_out format option and can handle multiple files, which makes it easier to produce a fasta file from multiple experiment files. io_lib/progs/extract_seq.c 4th August 1999, James ---------------------- 1. The exp2read() function in io_lib now initialises the confidence arrays (eg r->prob_A) to zero, or to the experiment file AV line. io_lib/read/translate.c 2nd June 1999, James -------------------- 1. The MegaBACE sequencer creates ABI files. However it does so in a odd way. Sometimes the samples arrays are truncated such that bases are positioned above samples which are not stored in the ABI file. We now realloc the samples array in such cases and fill out the remainder with blank data. This removes a crash in trev when viewing such data. io_lib/abi/seqIOABI.c 2. Fixed a memory corruption of io-lib compression. The switch to use tempnam (for Windows) implies that the filename returned is no longer allocated by us. Unfortunately we forgot to remove the xfree(fname) calls. src/io_lib/utils/compress.c 18th May 1999, James -------------------- 1. Fixed the trace rescaling option of makeSCF. We now go through the rescale function twice. Once to work out the maximum value, and again to do the rescaling. This fixes a bug where the maximum value after rescaling was sometimes above 65536 and hence cause "trace wraparound" effects. io_lib/progs/makeSCF.c 26th April 1999, JohnT ---------------------- 1. Allow : to be entered in RAW_DATA by using :: Misc/find.c io_lib/utils/find.c 2. Support for fetching trace files using Corba Modified: Misc/find.c mk/misc.mk io_lib/utils/find.c init_exp/init_exp.c io_lib/read/Makefile io_lib/utils/find.c io_lib/utils/compress.c io_lib/utils/Makefile mk/global.mk Added: io_lib/utils/corba.cpp io_lib/utils/stcorba.h Generated from IDL: io_lib/utils/trace.h io_lib/utils/trace.cpp io_lib/utils/basicServer.h io_lib/utils/basicServer.cpp 3. Added ABI utility progs to NT port mk/abi.mk 4. Added Windows 95 support io_lib/utils/compress.c mk/WINNT.mk 5th March 1999, JohnT --------------------- Various changes for WINNT support as follows: io_lib/utils - Don't redirect to /dev/null on WINNT 3rd February 1999, James ------------------------ 1. Fixed problems reported by Insure on Windows NT. These are mainly lack of prototypes (malloc/memcpy) and not returning properly from 'int' functions. However one fix to seqed_translate.c (find_line_start3) was a array read overflow. io_lib/progs/makeSCF.c 18th January 1999, James ------------------------ 1. Changed the read2exp io_lib translation function so that it can accept lowercase a,c,g,t. Oddly enough it was already coded to accept lowercase IUB codes, but we missed out a,c,g and t! io_lib/read/translate.c 15th January 1999, JohnT ----------------------- Modified files thoughout for Windows NT Compatibility as follows: 8. need to explicitly set text or binary file mode under WINNT io_lib/exp_file/expFileIO.c 18. need to include stddef.h for size_t with Visual C++ io_lib/utils/array.h 19. need to have target LIBS (not LIB) and correct ordering for correct make on WINNT. Also need additional abstractions to allow for different compile and link calling conventions with Visual C++, and have rules for building Windows .def files. io_lib/abi/Makefile io_lib/alf/Makefile io_lib/exp_file/Makefile io_lib/plain/Makefile io_lib/progs/Makefile io_lib/read/Makefile io_lib/scf/Makefile io_lib/utils/Makefile 18th December 1998, James ------------------------- 1. Added bzip2 recognition to the (de)compression code of io_lib. This is now the latest bzip, and is recognised by phred (unlike bzip version 1). Bzip2 is approx the same as bzip1, but more or less twice as fast for decompression. io_lib/utils/compress.c 27th November 1998, James ------------------------- 1. Fixed the trace file searching mechanism in io_lib. When loading an experiment file with LN/LT lines, we now first search for the trace file relative to the location of the experiment file. io_lib/read/Read.c io_lib/read/translate.[ch] 16th November 1998, James ------------------------- 4. Added NT (NoTe) and GD (Gap4 Database) line types to the experiment file. io_lib/exp_file/expFile.[ch] 24th September 1998, James -------------------------- 1. The scf reading and writing code now handles traces with zero bases. Previously this failed after a malloc(0). io_lib/scf/read_scf.c io_lib/scf/write_scf.c 2. The ABI file reading code has been tidied up. It now also supports conversion of more ABI fields, including RUND, RUNT, SPAC(2), CMNT, LANE and MTXF. io_lib/abi/seqIOABI.c 17th July 1998, James --------------------- 1. Extract_seq now copes with sequences containing no SQ line (instead of just SEGV). io_lib/progs/extract_seq 9th July 1998, James -------------------- 1. Enforce IUBC code set in io_lib when converting from trace (any format) to experiment file. We leave the IUBC 'N' intact. io_lib/read/translate.c 28th May 1998, James -------------------- 1. Added a read_sections() function to io_lib so that programs can state which bits of a trace file they are interested in. The loading code only then parses those bits. This can give big increases to things like init_exp which only wants bases and does not care about the delta-delta format of SCF trace data. io_lib/read/Read.h io_lib/read/translate.c io_lib/scf/scf.h io_lib/scf/read_scf.c io_lib/abi/seqIOABI.c io_lib/alf/seqIOALF.c init_exp/init_exp.c 3. Extract GELN (gel name) from ABI file when converting to SCF. io_lib/abi/seqIOABI.[ch] 2. Improved the makeSCF -normalise option. Background subtraction is now cleaner (and simpler) and it also now scales the heights. Moved it to io_lib as it's now freely available. io_lib/progs/makeSCF.c 23rd March 1998, James ---------------------- 1. Removed the change made on 7th May 1997 to seqIOPlain.c. This code is used by extract_seq, and so clipping in seqIOPlain causes double clipping (and hence wrong sections). io_lib/plain/seqIOPlain.c 11th March 1998, James ---------------------- 2. Removed the requirement of EXP_FILE_LINE_LENGTH in exp_fread_info(). This allows for (eg) tags with very long comments to be read in without being truncated. io_lib/exp_file/expFileIO.c 4th March 1998, James --------------------- 1. Following advice from Leif Hansson <leif.hansson@mbox4.swipnet.se>, the ALF reading code now reads the "Raw data" subfile when the "Processed data" subfile is not present, as "Processed data" is apparently an optional output of the pharmacia software. Raw data is in the same format, although I do not know what processing takes place to convert it to Processed data. (Looking at some real traces, apparently none!) io_lib/alf/seqIOALF.c 24th February 1998, James ------------------------- 1. Added an ABI in MacBinary format file type detector so that these are now autodetected. io_lib/utils/traceType.c 15th January 1998, James ------------------------ 1. Rewrote the delta_samples1/2 functions to be faster. Times vary between 0.55 and 0.7 fractions of the original time. io_lib/scf/misc_scf.c 4th December 1997, James ------------------------ 1. First post-release bug fix. Io_lib incorrect sets read->trace_name when reading anything except SCF files. This means that when outputting to an experiment file no LN line is present. io_lib/read/Read.c 1st October 1997, James ----------------------- 1. Allow for SCF files to contain 0 bases. This mainly affects memory allocation, but also the display widget. io_lib/scf/read_scf.c io_lib/utils/read_alloc.c 28/29th August 1997, James -------------------------- 2. Added a few changes to make the code more portable for the Mac. Not really used at present. Misc/os.h Misc/files.c io_lib/utils/traceType.c io_lib/read/translate.c io_lib/utils/compress.c 30th June 1997, James --------------------- 1. The exp2read function produced invalid rightCutoff values (INT_MAX) when no QR line is present. It now correctly sets it to 0. io_lib/read/translate.c