Hi,

I am running Picard's MarkDuplicates command on a bam file that I generated according to GATK's best practices guidelines.
The command has failed now multiple times at different stages without any error messages or warning. The last few lines of the last attempt are the following:

INFO    2021-06-28 13:43:06 MarkDuplicates  Read   665,000,000 records.  Elapsed time: 02:01:54s.  Time for last 1,000,000:    5s.  Last read position: chr8:30,463,617
INFO    2021-06-28 13:43:06 MarkDuplicates  Tracking 6745737 as yet unmatched pairs. 344808 records in RAM.
INFO    2021-06-28 13:43:18 MarkDuplicates  Read   666,000,000 records.  Elapsed time: 02:02:06s.  Time for last 1,000,000:   11s.  Last read position: chr8:32,749,364
INFO    2021-06-28 13:43:18 MarkDuplicates  Tracking 6744138 as yet unmatched pairs. 339607 records in RAM.
INFO    2021-06-28 13:43:23 MarkDuplicates  Read   667,000,000 records.  Elapsed time: 02:02:11s.  Time for last 1,000,000:    5s.  Last read position: chr8:35,027,712
INFO    2021-06-28 13:43:23 MarkDuplicates  Tracking 6742550 as yet unmatched pairs. 334065 records in RAM.
INFO    2021-06-28 13:44:39 MarkDuplicates  Read   668,000,000 records.  Elapsed time: 02:03:27s.  Time for last 1,000,000:   76s.  Last read position: chr8:37,295,885
INFO    2021-06-28 13:44:39 MarkDuplicates  Tracking 6740947 as yet unmatched pairs. 328694 records in RAM.
INFO    2021-06-28 13:44:47 MarkDuplicates  Read   669,000,000 records.  Elapsed time: 02:03:35s.  Time for last 1,000,000:    7s.  Last read position: chr8:39,537,579
INFO    2021-06-28 13:44:47 MarkDuplicates  Tracking 6739367 as yet unmatched pairs. 323206 records in RAM.
INFO    2021-06-28 13:45:57 MarkDuplicates  Read   670,000,000 records.  Elapsed time: 02:04:45s.  Time for last 1,000,000:   69s.  Last read position: chr8:41,790,226
INFO    2021-06-28 13:45:57 MarkDuplicates  Tracking 6737773 as yet unmatched pairs. 317727 records in RAM.
INFO    2021-06-28 13:46:03 MarkDuplicates  Read   671,000,000 records.  Elapsed time: 02:04:51s.  Time for last 1,000,000:    5s.  Last read position: chr8:43,092,877
INFO    2021-06-28 13:46:03 MarkDuplicates  Tracking 7007382 as yet unmatched pairs. 583427 records in RAM.
INFO    2021-06-28 13:46:11 MarkDuplicates  Read   672,000,000 records.  Elapsed time: 02:04:59s.  Time for last 1,000,000:    7s.  Last read position: chr8:43,092,916
INFO    2021-06-28 13:46:11 MarkDuplicates  Tracking 7094903 as yet unmatched pairs. 668562 records in RAM.
INFO    2021-06-28 13:46:22 MarkDuplicates  Read   673,000,000 records.  Elapsed time: 02:05:10s.  Time for last 1,000,000:   11s.  Last read position: chr8:43,094,783
INFO    2021-06-28 13:46:22 MarkDuplicates  Tracking 7144551 as yet unmatched pairs. 715971 records in RAM.
INFO    2021-06-28 13:46:29 MarkDuplicates  Read   674,000,000 records.  Elapsed time: 02:05:17s.  Time for last 1,000,000:    7s.  Last read position: chr8:43,095,887
INFO    2021-06-28 13:46:29 MarkDuplicates  Tracking 6905139 as yet unmatched pairs. 474456 records in RAM.
INFO    2021-06-28 13:46:35 MarkDuplicates  Read   675,000,000 records.  Elapsed time: 02:05:23s.  Time for last 1,000,000:    6s.  Last read position: chr8:43,820,929
INFO    2021-06-28 13:46:35 MarkDuplicates  Tracking 6757332 as yet unmatched pairs. 293186 records in RAM.
INFO    2021-06-28 13:46:45 MarkDuplicates  Read   676,000,000 records.  Elapsed time: 02:05:33s.  Time for last 1,000,000:    9s.  Last read position: chr8:46,856,120
INFO    2021-06-28 13:46:45 MarkDuplicates  Tracking 6757777 as yet unmatched pairs. 282813 records in RAM.
INFO    2021-06-28 13:46:50 MarkDuplicates  Read   677,000,000 records.  Elapsed time: 02:05:38s.  Time for last 1,000,000:    5s.  Last read position: chr8:48,968,815
INFO    2021-06-28 13:46:50 MarkDuplicates  Tracking 6752661 as yet unmatched pairs. 253411 records in RAM.

I have tried switching to the more recent gatk MarkDuplicatesSpark command, but it failed again a few hours in the analysis without any error message.

Does anybody have any suggestion as to what I could do to pinpoint the problem?

Thanks!



Source link