How to fix C function R_nc4_get_vara_double returned error in ncdf4 parallel processing in R

Question

I want to download nc data through OPENDAP from a remote storage. I use parallel backend with foreach - dopar loop as follows:

# INPUTS
inputs=commandArgs(trailingOnly = T)
interimpath=as.character(inputs[1])
gcm=as.character(inputs[2])
period=as.character(inputs[3])
var=as.character(inputs[4])
source='MACAV2'

cat('\n\n EXTRACTING DATA FOR',var, gcm, period, '\n\n')

# CHANGING LIBRARY PATHS
.libPaths("/storage/home/htn5098/local_lib/R40") # local library for packages
setwd('/storage/work/h/htn5098/DataAnalysis')
source('./src/Rcodes/CWD_function_package.R') # Calling the function Rscript

# CALLING PACKAGES
library(foreach)
library(doParallel)
library(parallel)
library(filematrix)

# REGISTERING CORES FOR PARALLEL PROCESSING
no_cores <- detectCores() 
cl <- makeCluster(no_cores)
registerDoParallel(cl)
invisible(clusterEvalQ(cl,.libPaths("/storage/home/htn5098/local_lib/R40"))) # Really have to import library paths into the workers
invisible(clusterEvalQ(cl, c(library(ncdf4))))

# EXTRACTING DATA FROM THE .NC FILES TO MATRIX FORM
url <- readLines('./data/external/MACAV2_OPENDAP_allvar_allgcm_allperiod.txt')
links <- grep(x = url,pattern = paste0('.*',var,'.*',gcm,'_.*',period), value = T) 

start=c(659,93,1) # lon, lat, time
count=c(527,307,-1)

spfile <- read.csv('./data/external/SERC_MACAV2_Elev.csv',header = T)
grids <- sort(unique(spfile$Grid))

clusterExport(cl,list('ncarray2matrix','start','count','grids')) #exporting data into clusters for parallel processing

cat('\nChecking when downloading all grids\n')

# k <- foreach(x = links,.packages = c('ncdf4')) %dopar% {
    # nc <- nc_open(x)
    # nc.var=ncvar_get(nc,varid=names(nc$var),start=start,count=count)
    # return(nc.var)
    # nc_close(nc)
# }

k <- foreach(x = links,.packages = c('ncdf4'),.errorhandling = 'pass') %dopar% {
    nc <- nc_open(x)
    print(nc)
    nc.var=ncvar_get(nc,varid=names(nc$var),start=c(659,93,1),count=c(527,307,-1))
    nc_close(nc)
    return(dim(nc.var))
    Sys.sleep(10)
}

# k <- parSapply(cl,links,function(x) {
    # nc <- nc_open(x)
    # nc.var=ncvar_get(nc,varid=names(nc$var),start=start,count=count)
    # nc_close(nc)
    # return(nc.var)
# })

print(k)

However, I keep getting this error:

<simpleError in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset,     scaleFact, start = start, count = count, verbose = verbose,     signedbyte = signedbyte, collapse_degen = collapse_degen): C function R_nc4_get_vara_double returned error>

What could be the reason for this problem? Can you recommend a solution for this that is time-efficient (I have to repeat this for about 20 files)?

Thank you.

Answer 1

I had the same error in my code. The problem was not the code itself. It was one of the files that I wanted to read. It has something wrong, so R couldn't open it. I identified the file and downloaded it again, and the same code worked perfectly.

Answer 2

I also encountered the same error. For me, restarting R session did the trick.