Bioinformatics and Genomics File Formats Reference

This reference covers file formats used in genomics, transcriptomics, sequence analysis, and related bioinformatics applications.

Sequence Data Formats

.fasta / .fa / .fna - FASTA Format

Description: Text-based format for nucleotide or protein sequences Typical Data: DNA, RNA, or protein sequences with headers Use Cases: Sequence storage, BLAST searches, alignments Python Libraries:

Biopython: SeqIO.parse('file.fasta', 'fasta')
pyfaidx: Fast indexed FASTA access
screed: Fast sequence parsing EDA Approach:
Sequence count and length distribution
GC content analysis
N content (ambiguous bases)
Sequence ID parsing
Duplicate detection
Quality metrics for assemblies (N50, L50)

.fastq / .fq - FASTQ Format

Description: Sequence data with base quality scores Typical Data: Raw sequencing reads with Phred quality scores Use Cases: NGS data, quality control, read mapping Python Libraries:

Biopython: SeqIO.parse('file.fastq', 'fastq')
pysam: Fast FASTQ/BAM operations
HTSeq: Sequencing data analysis EDA Approach:
Read count and length distribution
Quality score distribution (per-base, per-read)
GC content and bias
Duplicate rate estimation
Adapter contamination detection
k-mer frequency analysis
Encoding format validation (Phred33/64)

.sam - Sequence Alignment/Map

Description: Tab-delimited text format for alignments Typical Data: Aligned sequencing reads with mapping quality Use Cases: Read alignment storage, variant calling Python Libraries:

pysam: pysam.AlignmentFile('file.sam', 'r')
HTSeq: HTSeq.SAM_Reader('file.sam') EDA Approach:
Mapping rate and quality distribution
Coverage analysis
Insert size distribution (paired-end)
Alignment flags distribution
CIGAR string patterns
Mismatch and indel rates
Duplicate and supplementary alignment counts

.bam - Binary Alignment/Map

Description: Compressed binary version of SAM Typical Data: Aligned reads in compressed format Use Cases: Efficient storage and processing of alignments Python Libraries:

pysam: Full BAM support with indexing
bamnostic: Pure Python BAM reader EDA Approach:
Same as SAM plus:
Compression ratio analysis
Index file (.bai) validation
Chromosome-wise statistics
Strand bias detection
Read group analysis

.cram - CRAM Format

Description: Highly compressed alignment format Typical Data: Reference-compressed aligned reads Use Cases: Long-term storage, space-efficient archives Python Libraries:

pysam: CRAM support (requires reference)
Reference genome must be accessible EDA Approach:
Compression efficiency vs BAM
Reference dependency validation
Lossy vs lossless compression assessment
Decompression performance
Similar alignment metrics as BAM

.bed - Browser Extensible Data

Description: Tab-delimited format for genomic features Typical Data: Genomic intervals (chr, start, end) with annotations Use Cases: Peak calling, variant annotation, genome browsing Python Libraries:

pybedtools: pybedtools.BedTool('file.bed')
pyranges: pyranges.read_bed('file.bed')
pandas: Simple BED reading EDA Approach:
Feature count and size distribution
Chromosome distribution
Strand bias
Score distribution (if present)
Overlap and proximity analysis
Coverage statistics
Gap analysis between features

.bedGraph - BED with Graph Data

Description: BED format with per-base signal values Typical Data: Continuous-valued genomic data (coverage, signals) Use Cases: Coverage tracks, ChIP-seq signals, methylation Python Libraries:

pyBigWig: Can convert to bigWig
pybedtools: BedGraph operations EDA Approach:
Signal distribution statistics
Genome coverage percentage
Signal dynamics (peaks, valleys)
Chromosome-wise signal patterns
Quantile analysis
Zero-coverage regions

.bigWig / .bw - Binary BigWig

Description: Indexed binary format for genome-wide signal data Typical Data: Continuous genomic signals (compressed and indexed) Use Cases: Efficient genome browser tracks, large-scale data Python Libraries:

pyBigWig: pyBigWig.open('file.bw')
pybbi: BigWig/BigBed interface EDA Approach:
Signal statistics extraction
Zoom level analysis
Regional signal extraction
Efficient genome-wide summaries
Compression efficiency
Index structure analysis

.bigBed / .bb - Binary BigBed

Description: Indexed binary BED format Typical Data: Genomic features (compressed and indexed) Use Cases: Large feature sets, genome browsers Python Libraries:

pybbi: BigBed reading
pybigtools: Modern BigBed interface EDA Approach:
Feature density analysis
Efficient interval queries
Zoom level validation
Index performance metrics
Feature size statistics

.gff / .gff3 - General Feature Format

Description: Tab-delimited format for genomic annotations Typical Data: Gene models, transcripts, exons, regulatory elements Use Cases: Genome annotation, gene prediction Python Libraries:

BCBio.GFF: Biopython GFF module
gffutils: gffutils.create_db('file.gff3')
pyranges: GFF support EDA Approach:
Feature type distribution (gene, exon, CDS, etc.)
Gene structure validation
Strand balance
Hierarchical relationship validation
Phase validation for CDS
Attribute completeness
Gene model statistics (introns, exons per gene)

.gtf - Gene Transfer Format

Description: GFF2-based format for gene annotations Typical Data: Gene and transcript annotations Use Cases: RNA-seq analysis, gene quantification Python Libraries:

pyranges: pyranges.read_gtf('file.gtf')
gffutils: GTF database creation
HTSeq: GTF reading for counts EDA Approach:
Transcript isoform analysis
Gene structure completeness
Exon number distribution
Transcript length distribution
TSS and TES analysis
Biotype distribution
Overlapping gene detection

.vcf - Variant Call Format

Description: Text format for genetic variants Typical Data: SNPs, indels, structural variants with annotations Use Cases: Variant calling, population genetics, GWAS Python Libraries:

pysam: pysam.VariantFile('file.vcf')
cyvcf2: Fast VCF parsing
PyVCF: Older but comprehensive EDA Approach:
Variant count by type (SNP, indel, SV)
Quality score distribution
Allele frequency spectrum
Transition/transversion ratio
Heterozygosity rates
Missing genotype analysis
Hardy-Weinberg equilibrium
Annotation completeness (if annotated)

.bcf - Binary VCF

Description: Compressed binary variant format Typical Data: Same as VCF but binary Use Cases: Efficient variant storage and processing Python Libraries:

pysam: Full BCF support
cyvcf2: Optimized BCF reading EDA Approach:
Same as VCF plus:
Compression efficiency
Indexing validation
Read performance metrics

.gvcf - Genomic VCF

Description: VCF with reference confidence blocks Typical Data: All positions (variant and non-variant) Use Cases: Joint genotyping workflows, GATK Python Libraries:

pysam: GVCF support
Standard VCF parsers EDA Approach:
Reference block analysis
Coverage uniformity
Variant density
Genotype quality across genome
Reference confidence distribution

RNA-Seq and Expression Data

.counts - Gene Count Matrix

Description: Tab-delimited gene expression counts Typical Data: Gene IDs with read counts per sample Use Cases: RNA-seq quantification, differential expression Python Libraries:

pandas: pd.read_csv('file.counts', sep='\t')
scanpy (for single-cell): sc.read_csv() EDA Approach:
Library size distribution
Detection rate (genes per sample)
Zero-inflation analysis
Count distribution (log scale)
Outlier sample detection
Correlation between replicates
PCA for sample relationships

.tpm / .fpkm - Normalized Expression

Description: Normalized gene expression values Typical Data: TPM (transcripts per million) or FPKM values Use Cases: Cross-sample comparison, visualization Python Libraries:

pandas: Standard CSV reading
anndata: For integrated analysis EDA Approach:
Expression distribution
Highly expressed gene identification
Sample clustering
Batch effect detection
Coefficient of variation analysis
Dynamic range assessment

.mtx - Matrix Market Format

Description: Sparse matrix format (common in single-cell) Typical Data: Sparse count matrices (cells × genes) Use Cases: Single-cell RNA-seq, large sparse matrices Python Libraries:

scipy.io: scipy.io.mmread('file.mtx')
scanpy: sc.read_mtx('file.mtx') EDA Approach:
Sparsity analysis
Cell and gene filtering thresholds
Doublet detection metrics
Mitochondrial fraction
UMI count distribution
Gene detection per cell

.h5ad - Anndata Format

Description: HDF5-based annotated data matrix Typical Data: Expression matrix with metadata (cells, genes) Use Cases: Single-cell RNA-seq analysis with Scanpy Python Libraries:

scanpy: sc.read_h5ad('file.h5ad')
anndata: Direct AnnData manipulation EDA Approach:
Cell and gene counts
Metadata completeness
Layer availability (raw, normalized)
Embedding presence (PCA, UMAP)
QC metrics distribution
Batch information
Cell type annotation coverage

.loom - Loom Format

Description: HDF5-based format for omics data Typical Data: Expression matrices with metadata Use Cases: Single-cell data, RNA velocity analysis Python Libraries:

loompy: loompy.connect('file.loom')
scanpy: Can import loom files EDA Approach:
Layer analysis (spliced, unspliced)
Row and column attribute exploration
Graph connectivity analysis
Cluster assignments
Velocity-specific metrics

.rds - R Data Serialization

Description: R object storage (often Seurat objects) Typical Data: R analysis results, especially single-cell Use Cases: R-Python data exchange Python Libraries:

pyreadr: pyreadr.read_r('file.rds')
rpy2: For full R integration
Conversion tools to AnnData EDA Approach:
Object type identification
Data structure exploration
Metadata extraction
Conversion validation

Alignment and Assembly Formats

.maf - Multiple Alignment Format

Description: Text format for multiple sequence alignments Typical Data: Genome-wide or local multiple alignments Use Cases: Comparative genomics, conservation analysis Python Libraries:

Biopython: AlignIO.parse('file.maf', 'maf')
bx-python: MAF-specific tools EDA Approach:
Alignment block statistics
Species coverage
Gap analysis
Conservation scoring
Alignment quality metrics
Block length distribution

.axt - Pairwise Alignment Format

Description: Pairwise alignment format (UCSC) Typical Data: Pairwise genomic alignments Use Cases: Genome comparison, synteny analysis Python Libraries:

Custom parsers (simple format)
bx-python: AXT support EDA Approach:
Alignment score distribution
Identity percentage
Syntenic block identification
Gap size analysis
Coverage statistics

.chain - Chain Alignment Format

Description: Genome coordinate mapping chains Typical Data: Coordinate transformations between genome builds Use Cases: Liftover, coordinate conversion Python Libraries:

pyliftover: Chain file usage
Custom parsers for chain format EDA Approach:
Chain score distribution
Coverage of source genome
Gap analysis
Inversion detection
Mapping quality assessment

.psl - Pattern Space Layout

Description: BLAT/BLAST alignment format Typical Data: Alignment results from BLAT Use Cases: Transcript mapping, similarity searches Python Libraries:

Custom parsers (tab-delimited)
pybedtools: Can handle PSL EDA Approach:
Match percentage distribution
Gap statistics
Query coverage
Multiple mapping analysis
Alignment quality metrics

Genome Assembly and Annotation

.agp - Assembly Golden Path

Description: Assembly structure description Typical Data: Scaffold composition, gap information Use Cases: Genome assembly representation Python Libraries:

Custom parsers (simple tab-delimited)
Assembly analysis tools EDA Approach:
Scaffold statistics (N50, L50)
Gap type and size distribution
Component length analysis
Assembly contiguity metrics
Unplaced contig analysis

.scaffolds / .contigs - Assembly Sequences

Description: Assembled sequences (usually FASTA) Typical Data: Assembled genomic sequences Use Cases: Genome assembly output Python Libraries:

Same as FASTA format
Assembly-specific tools (QUAST) EDA Approach:
Assembly statistics (N50, N90, etc.)
Length distribution
Coverage analysis
Gap (N) content
Duplication assessment
BUSCO completeness (if annotations available)

.2bit - Compressed Genome Format

Description: UCSC compact genome format Typical Data: Reference genomes (highly compressed) Use Cases: Efficient genome storage and access Python Libraries:

py2bit: py2bit.open('file.2bit')
twobitreader: Alternative reader EDA Approach:
Compression efficiency
Random access performance
Sequence extraction validation
Masked region analysis
N content and distribution

.sizes - Chromosome Sizes

Description: Simple format with chromosome lengths Typical Data: Tab-delimited chromosome names and sizes Use Cases: Genome browsers, coordinate validation Python Libraries:

Simple file reading with pandas
Built into many genomic tools EDA Approach:
Genome size calculation
Chromosome count
Size distribution
Karyotype validation
Completeness check against reference

Phylogenetics and Evolution

.nwk / .newick - Newick Tree Format

Description: Parenthetical tree representation Typical Data: Phylogenetic trees with branch lengths Use Cases: Evolutionary analysis, tree visualization Python Libraries:

Biopython: Phylo.read('file.nwk', 'newick')
ete3: ete3.Tree('file.nwk')
dendropy: Phylogenetic computing EDA Approach:
Tree structure analysis (tips, internal nodes)
Branch length distribution
Tree balance metrics
Ultrametricity check
Bootstrap support analysis
Topology validation

.nexus - Nexus Format

Description: Rich format for phylogenetic data Typical Data: Alignments, trees, character matrices Use Cases: Phylogenetic software interchange Python Libraries:

Biopython: Nexus support
dendropy: Comprehensive Nexus handling EDA Approach:
Data block analysis
Character type distribution
Tree block validation
Taxa consistency
Command block parsing
Format compliance checking

.phylip - PHYLIP Format

Description: Sequence alignment format (strict/relaxed) Typical Data: Multiple sequence alignments Use Cases: Phylogenetic analysis input Python Libraries:

Biopython: AlignIO.read('file.phy', 'phylip')
dendropy: PHYLIP support EDA Approach:
Alignment dimensions
Sequence length uniformity
Gap position analysis
Informative site calculation
Format variant detection (strict vs relaxed)

.paml - PAML Output

Description: Output from PAML phylogenetic software Typical Data: Evolutionary model results, dN/dS ratios Use Cases: Molecular evolution analysis Python Libraries:

Custom parsers for specific PAML programs
Biopython: Basic PAML parsing EDA Approach:
Model parameter extraction
Likelihood values
dN/dS ratio distribution
Branch-specific results
Convergence assessment

Protein and Structure Data

.embl - EMBL Format

Description: Rich sequence annotation format Typical Data: Sequences with extensive annotations Use Cases: Sequence databases, genome records Python Libraries:

Biopython: SeqIO.read('file.embl', 'embl') EDA Approach:
Feature annotation completeness
Sequence length and type
Reference information
Cross-reference validation
Feature overlap analysis

.genbank / .gb / .gbk - GenBank Format

Description: NCBI's sequence annotation format Typical Data: Annotated sequences with features Use Cases: Sequence databases, annotation transfer Python Libraries:

Biopython: SeqIO.parse('file.gb', 'genbank') EDA Approach:
Feature type distribution
CDS analysis (start codons, stops)
Translation validation
Annotation completeness
Source organism extraction
Reference and publication info
Locus tag consistency

.sff - Standard Flowgram Format

Description: 454/Roche sequencing data format Typical Data: Raw pyrosequencing flowgrams Use Cases: Legacy 454 sequencing data Python Libraries:

Biopython: SeqIO.parse('file.sff', 'sff')
Platform-specific tools EDA Approach:
Read count and length
Flowgram signal quality
Key sequence detection
Adapter trimming validation
Quality score distribution

.hdf5 (Genomics Specific)

Description: HDF5 for genomics (10X, Hi-C, etc.) Typical Data: High-throughput genomics data Use Cases: 10X Genomics, spatial transcriptomics Python Libraries:

h5py: Low-level access
scanpy: For 10X data
cooler: For Hi-C data EDA Approach:
Dataset structure exploration
Barcode statistics
UMI counting
Feature-barcode matrix analysis
Spatial coordinates (if applicable)

.cool / .mcool - Cooler Format

Description: HDF5-based Hi-C contact matrices Typical Data: Chromatin interaction matrices Use Cases: 3D genome analysis, Hi-C data Python Libraries:

cooler: cooler.Cooler('file.cool')
hicstraw: For .hic format EDA Approach:
Resolution analysis
Contact matrix statistics
Distance decay curves
Compartment analysis
TAD boundary detection
Balance factor validation

.hic - Hi-C Binary Format

Description: Juicer binary Hi-C format Typical Data: Multi-resolution Hi-C matrices Use Cases: Hi-C analysis with Juicer tools Python Libraries:

hicstraw: hicstraw.HiCFile('file.hic')
straw: C++ library with Python bindings EDA Approach:
Available resolutions
Normalization methods
Contact statistics
Chromosomal interactions
Quality metrics

.bw (ChIP-seq / ATAC-seq specific)

Description: BigWig files for epigenomics Typical Data: Coverage or enrichment signals Use Cases: ChIP-seq, ATAC-seq, DNase-seq Python Libraries:

pyBigWig: Standard bigWig access EDA Approach:
Peak enrichment patterns
Background signal analysis
Sample correlation
Signal-to-noise ratio
Library complexity metrics

.narrowPeak / .broadPeak - ENCODE Peak Formats

Description: BED-based formats for peaks Typical Data: Peak calls with scores and p-values Use Cases: ChIP-seq peak calling output Python Libraries:

pybedtools: BED-compatible
Custom parsers for peak-specific fields EDA Approach:
Peak count and width distribution
Signal value distribution
Q-value and p-value analysis
Peak summit analysis
Overlap with known features
Motif enrichment preparation

.wig - Wiggle Format

Description: Dense continuous genomic data Typical Data: Coverage or signal tracks Use Cases: Genome browser visualization Python Libraries:

pyBigWig: Can convert to bigWig
Custom parsers for wiggle format EDA Approach:
Signal statistics
Coverage metrics
Format variant (fixedStep vs variableStep)
Span parameter analysis
Conversion efficiency to bigWig

.ab1 - Sanger Sequencing Trace

Description: Binary chromatogram format Typical Data: Sanger sequencing traces Use Cases: Capillary sequencing validation Python Libraries:

Biopython: SeqIO.read('file.ab1', 'abi')
tracy tools: For quality assessment EDA Approach:
Base calling quality
Trace quality scores
Mixed base detection
Primer and vector detection
Read length and quality region
Heterozygosity detection

.scf - Standard Chromatogram Format

Description: Sanger sequencing chromatogram Typical Data: Base calls and confidence values Use Cases: Sequencing trace analysis Python Libraries:

Biopython: SCF format support EDA Approach:
Similar to AB1 format
Quality score profiles
Peak height ratios
Signal-to-noise metrics

.idx - Index Files (Generic)

Description: Index files for various formats Typical Data: Fast random access indices Use Cases: Efficient data access (BAM, VCF, etc.) Python Libraries:

Format-specific libraries handle indices
pysam: Auto-handles BAI, CSI indices EDA Approach:
Index completeness validation
Binning strategy analysis
Access performance metrics
Index size vs data size ratio

bioinformatics_genomics_formats.md 21 KB History Raw