Hi, I have tried to run my RnBeads run using HPC (SLURM) and run into some issues. I suspect it may be related to the memory issue since our hpc one node has only 4GB RAM. To overcome this, I tried to set the rnb.cr <- setModuleResourceRequirements(rnb.cr,c(mem="32G"),"all") but it still showed the issues. I think I may need to do something more for allocating RAM to multiple nodes for total 32GB RAM. But, I couldn't find any script using SLURM yet. It would be nice if I can get any help since I am trying to import my cov files for 4 days and all of my trials were failed so far. Many thanks!!!

I did this:
* xml file *

<rnb.xml>
<analysis.name>Methylation_Seq</analysis.name>
<data.source>data/Methylation_cov/dataset/coverage,data/Methylation_cov/dataset/sample_annotation.csv</data.source>
<dir.reports>reports</dir.reports>
<data.type>bed.dir</data.type>
<assembly>hg38</assembly>
<region.types>tiling,genes,promoters,cpgislands</region.types>
<identifiers.column>filename_bed</identifiers.column>
<colors.category>#1B9E77,#D95F02,#7570B3,#E7298A,#66A61E,#E6AB02,#A6761D,#666666,#2166AC,#B2182B</colors.category>
<min.group.size>2</min.group.size>
<max.group.count>20</max.group.count>
<import.bed.style>bismarkCov</import.bed.style>
<qc.coverage.plots>true</qc.coverage.plots>
<filtering.sex.chromosomes.removal>true</filtering.sex.chromosomes.removal>
<filtering.missing.value.quantile>0.5</filtering.missing.value.quantile>
<filtering.coverage.threshold>5</filtering.coverage.threshold>
<filtering.low.coverage.masking>true</filtering.low.coverage.masking>
<filtering.high.coverage.outliers>true</filtering.high.coverage.outliers>
<filtering.greedycut>false</filtering.greedycut>
<exploratory.columns>Sample_ID,CellType,Sample_Group</exploratory.columns>
<exploratory.intersample>false</exploratory.intersample>
<exploratory.region.profiles>genes,promoters</exploratory.region.profiles>
<differential.site.test.method>limma</differential.site.test.method>
<differential.comparison.columns>CellType,Sample_Group</differential.comparison.columns>
<differential.enrichment>true</differential.enrichment>
<export.to.trackhub>bigBed</export.to.trackhub>
<logging.memory>true</logging.memory>
<disk.dump.big.matrices>true</disk.dump.big.matrices>
<enforce.memory.management>true</enforce.memory.management>
<gz.large.files>true</gz.large.files>
</rnb.xml>

Execute codes in R

library(RnBeads) 
xml.file <- "submit.xml"
arch <- new("ClusterArchitectureSLURM")
rnb.cr <- new("RnBClusterRun",arch)
rnb.cr <- setModuleResourceRequirements(rnb.cr,c(mem="32G"),"all")
rnb.cr <- setModuleNumCores(rnb.cr,4L,"all")
rnb.cr <- setModuleNumCores(rnb.cr,2L,"exploratory")
run(rnb.cr, "rnbeads_analysis", xml.file) 

* This is the import.log file *

2021-05-25 22:50:15     1.5  STATUS ...Started module: import
2021-05-25 22:50:15     1.5  STATUS STARTED Configuring Analysis
2021-05-25 22:50:15     1.5 WARNING     The option 'differential.enrichment' no longer exists. Note, that RnBeads now supports GO and LOLA enrichment. Your option setting will be applied to the new option 'differential.enrichment.go'
2021-05-25 22:50:15     1.5 WARNING     The option 'differential.enrichment' no longer exists. Note, that RnBeads now supports GO and LOLA enrichment. Your option setting will be applied to the new option 'differential.enrichment.go'
2021-05-25 22:50:15     1.5    INFO     Machine name: hpc-d36-5-4.local
2021-05-25 22:50:15     1.5  STATUS     STARTED Setting up Multicore
2021-05-25 22:50:15     1.5    INFO         Using 4 cores
2021-05-25 22:50:15     1.5  STATUS     COMPLETED Setting up Multicore
2021-05-25 22:50:15     1.5    INFO     Analysis Title: Methylation_Seq
2021-05-25 22:50:15     1.5    INFO     Number of cores: 4
2021-05-25 22:50:15     1.5  STATUS COMPLETED Configuring Analysis

2021-05-25 22:50:15     1.5  STATUS STARTED Loading Data
2021-05-25 22:50:15     1.5    INFO     Number of cores: 4
2021-05-25 22:50:15     1.5    INFO     Loading data of type "bed.dir"
2021-05-25 22:50:15     1.5  STATUS     STARTED Performing loading test
2021-05-25 22:50:15     1.5    INFO         The first 10000 rows will be read from each data file
2021-05-25 22:50:15     1.5    INFO         No column with file names specified: will try to find one
2021-05-25 22:50:15     1.5  STATUS         STARTED Loading Data From BED Files
2021-05-25 22:50:19     1.6  STATUS             STARTED Automatically parsing the provided sample annotation file
2021-05-25 22:50:19     1.6  STATUS                 Potential file names found in column 1 of the supplied annotation table
2021-05-25 22:50:19     1.6  STATUS             COMPLETED Automatically parsing the provided sample annotation file
2021-05-25 22:50:19     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDF1_1_bismark.cov
2021-05-25 22:50:33     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDF1_2_bismark.cov
2021-05-25 22:50:47     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDF2_1_bismark.cov
2021-05-25 22:51:00     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDF2_2_bismark.cov
2021-05-25 22:51:14     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDF3_1_bismark.cov
2021-05-25 22:51:28     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDF3_2_bismark.cov
2021-05-25 22:51:41     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDFPA_1_bismark.cov
2021-05-25 22:51:55     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/HDFPA_2_bismark.cov
2021-05-25 22:52:08     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/KPA_1_bismark.cov
2021-05-25 22:52:22     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/KPA_2_bismark.cov
2021-05-25 22:52:36     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P5_1_bismark.cov
2021-05-25 22:52:49     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P5_2_bismark.cov
2021-05-25 22:53:03     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P16_1_bismark.cov
2021-05-25 22:53:16     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P16_2_bismark.cov
2021-05-25 22:53:30     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P17_1_bismark.cov
2021-05-25 22:53:44     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P17_2_bismark.cov
2021-05-25 22:53:58     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P21_1_bismark.cov
2021-05-25 22:54:11     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P21_2_bismark.cov
2021-05-25 22:54:25     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P24_1_bismark.cov
2021-05-25 22:54:38     1.6    INFO             Reading BED file: data/Methylation_cov/dataset/coverage/P24_2_bismark.cov
2021-05-25 22:54:51     1.6  STATUS             Read 20 BED files
2021-05-25 22:54:52     1.6  STATUS             Matched chromosomes and strands to annotation
2021-05-25 22:54:52     1.6  STATUS             Checked for the presence of sites and coverage
2021-05-25 22:54:52     1.6  STATUS             Initialized meth/covg matrices
opening ff /tmp/Rtmpt5pObc/ff/ff2f4c7d78799e.ff
2021-05-25 22:55:06     1.6  STATUS             Combined a data matrix with 14820 sites and 20 samples
2021-05-25 22:55:06     1.6  STATUS             Processed all BED files
2021-05-25 22:55:19     1.6  STATUS             STARTED Creating RnBiseqSet object
2021-05-25 22:55:19     1.6    INFO                 Removed 14820 sites with unknown chromosomes
2021-05-25 22:55:19     1.6 WARNING                 All sites have been removed, returning NULL
2021-05-25 22:55:19     1.6  STATUS             COMPLETED Creating RnBiseqSet object
2021-05-25 22:55:19     1.6  STATUS         COMPLETED Loading Data From BED Files
2021-05-25 22:55:19     1.6  STATUS         STARTED Checking the loaded object
2021-05-25 22:55:19     1.6    INFO             The supplied object is not of class RnBiseqSet. Breaking the check...
2021-05-25 22:55:19     1.6 WARNING             The object loaded during the loading test contains invalid information (see details above). Please check the whether the data source arguments as well as the data import options, like table separator, BED style or BED column assignment, are set correctly
2021-05-25 22:55:19     1.6  STATUS         COMPLETED Checking the loaded object
2021-05-25 22:55:32     1.6  STATUS     COMPLETED Performing loading test
2021-05-25 22:55:32     1.6    INFO     No column with file names specified: will try to find one
2021-05-25 22:55:32     1.6  STATUS     STARTED Loading Data From BED Files
2021-05-25 22:55:32     1.6  STATUS         STARTED Automatically parsing the provided sample annotation file
2021-05-25 22:55:32     1.6  STATUS             Potential file names found in column 1 of the supplied annotation table
2021-05-25 22:55:32     1.6  STATUS         COMPLETED Automatically parsing the provided sample annotation file
2021-05-25 22:55:32     1.6    INFO         Reading BED file: data/Methylation_cov/dataset/coverage/HDF1_1_bismark.cov
/var/spool/slurmd/job1756391/slurm_script: line 4: 12108 Killed                  Rscript /gpfs/home/ys16b/R/x86_64-redhat-linux-gnu-library/4.0/RnBeads/extdata/Rscript/rscript_import.R -x submit.xml -o reports/cluster_run -c 4
slurmstepd: error: Detected 1 oom-kill event(s) in step 1756391.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.



Source link