Samtools sort: too many open files error


Hello everyone!
I am using a cluster to align my WGS data to indexed reference genome via BWA.
I receive this error on samtools sort part of the task;

[main] CMD: bwa mem -t 40 /save/refs/bwa_refs/Homo_sapiens.GRCh38.dna.toplevel.fa.gz /work/fastqs/557-male_$
[main] Real time: 125431.829 sec; CPU: 4892515.629 sec
[bam_sort_core] merging from 1260 files and 35 in-memory blocks...
[E::hts_open_format] Failed to open file /work/rekren/aligned/aligned_to_human_M_sortedbam.bam.tmp.1016.bam
samtools sort: fail to open "/work/aligned/aligned_to_human_M_sortedbam.bam.tmp.1016.bam": Too many open files

I have read about -m and @ parameters of samtools sort from the manual but playing around with different combinations for those I received out-of memory error from the cluster. Maybe I am overlooking something.
Can you suggest a solution for me please?


To submit this task to cluster I use SLURM and my script is below;

#SBATCH -J bwa_task #job name
#SBATCH -e error.out #error file name
#SBATCH --mem=200G #memory reservation
#SBATCH --cpus-per-task=40 #ncpu on the same node
#SBATCH --mail-type=BEGIN,END,FAIL ([email protected])
#Purge any previous modules
module purge
#Load the application
module load bioinfo/bwa-0.7.17
module load bioinfo/samtools-1.9
# My command lines I want to run on the cluster
bwa mem -t 40 /save/refs/bwa_refs/Homo_sapiens.GRCh38.dna.toplevel.fa.gz /work/fastqs/557-male_ACGCACCT-CCTTCACC-BHNJYMDSXY_L004_R1.fastq.gz /work/fastqs/557-male_ACGCACCT-CCTTCACC-BHNJYMDSXY_L004_R2.fastq.gz | samtools sort [email protected] 35 -o /work/aligned/aligned_to_human_M_sortedbam.bam




Source link