ffappend Error in get(".rambytes") : object '.rambytes' not found when package ff not explicitly loaded

[chrisvwn]

Hi,

I am getting the error:

 Error in get(".rambytes") : object '.rambytes' not found

when I use the ffbase::ffappend function within the package Rnightlights.

The relevant part of the traceback when this happens is:

Error in get(".rambytes") : object '.rambytes' not found 
11. get(".rambytes") 
10. chunk.default(y) 
9. chunk(y) 
8. ffbase::ffappend(vals, rastVals, adjustvmode = T) at stats.R#132

rastVals is a numeric vector and vals is an ff_vector.

My sessionInfo:

> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 32 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblas-r0.3.10.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Rnightlights_0.2.4.5

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5         compiler_4.0.3     pillar_1.4.6       plyr_1.8.6        
 [5] class_7.3-17       R.methodsS3_1.8.1  R.utils_2.10.1     iterators_1.0.12  
 [9] tools_4.0.3        gdalUtils_2.0.3.2  bit_4.0.4          digest_0.6.27     
[13] lubridate_1.7.9    tibble_3.0.4       lifecycle_0.2.0    lattice_0.20-41   
[17] ff_4.0.2           doSNOW_1.0.18      pkgconfig_2.0.3    rlang_0.4.8       
[21] fastmatch_1.1-0    foreach_1.5.0      DBI_1.1.0          rstudioapi_0.11   
[25] parallel_4.0.3     yaml_2.2.1         rgdal_1.5-16       e1071_1.7-4       
[29] httr_1.4.2         stringr_1.4.0      dplyr_1.0.2        raster_3.3-13     
[33] xml2_1.3.2         generics_0.0.2     vctrs_0.3.4        rgeos_0.5-5       
[37] hms_0.5.3          classInt_0.4-3     grid_4.0.3         tidyselect_1.1.0  
[41] ffbase_0.13.1      glue_1.4.2         data.table_1.13.2  sf_0.9-6          
[45] R6_2.5.0           snow_0.4-3         sp_1.4-2           selectr_0.4-2     
[49] readr_1.3.1        reshape2_1.4.4     purrr_0.3.4        magrittr_1.5      
[53] units_0.6-7        codetools_0.2-16   settings_0.2.4     ellipsis_0.3.1    
[57] rvest_0.3.5        fasterize_1.0.3    KernSmooth_2.23-17 stringi_1.5.3     
[61] crayon_1.3.4       R.oo_1.24.0

get(".rambytes") seems to work when I load the ff package explicitly via library().

Not sure if I should post this here or on the ff package page.

[jwijffels]

Are you sure in your pkg you import ff in the namespace. We have

ffbase/pkg/R/pkg.R

Line 72 in ec716c9

#' @importFrom ff arrayIndex2vectorIndex nrow<- .rambytes set.ff hi ffindexordersize ffindexget

Explicitely mentioning .rambytes. I don't see ffbase or ff mentioned in your pkg namespace on github

[chrisvwn]

I see what you mean. I haven't had to explicitly import either ff or ffbase before which is why I raised this. Only data.table and %dopar% from foreach are imported in the pkg namespace because I could not find a way to use them without doing so.

I can go ahead and import ffbase and ff if this is not an issue. I just thought it was neat being able to get full pkg functionality without attaching the package.

[jwijffels]

.rambytes is defined here in ff: https://github.com/truecluster/ff/blob/8d109d251c15bc49618cf479f69029a5093aef5a/R/vmode.R#L416
Please test your package by adding something like (if you use roxygen2)
#' @importFrom .rambytes

[chrisvwn]

I am using roxygen2 so I will use the @importFrom if necessary. I think it would be #' @importFrom ff .rambytes.

But at the risk of belaboring the point, I was thinking this is something that can be altered at source. Shouldn't it be possible to simply run the following code without having to import or attach any packages/resources?

x <- ff::ff(initdata = 1:10, finalizer = "delete", overwrite = T)

x <- ffbase::ffappend(x, 11:20, adjustvmode = T)

I get the same error when I run this.

I am not an expert at this but would it maybe make sense to maybe use ff::.rambytes or something similar rather than get(".rambytes") since it specifies where to look for the resource even if pkg ff is not attached?

[jwijffels]

yes, you're right

[eshom]

I can confirm this. In my case however I get the following error additionally (after importing .rambytes) to the package:

Error in get("vmode") : object 'vmode' not found

traceback:

7: get("vmode")
6: chunk.default(y)
5: chunk(y)
4: ffappend(fltr, a)
3: ffwhich.ffdf(x, ss, envir = parent.frame())

sessionInfo():

R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux bullseye/sid

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=C.UTF-8          LC_NUMERIC=C             
 [3] LC_TIME=en_IL.utf8        LC_COLLATE=en_IL.utf8    
 [5] LC_MONETARY=en_IL.utf8    LC_MESSAGES=en_IL.utf8   
 [7] LC_PAPER=en_IL.utf8       LC_NAME=C                
 [9] LC_ADDRESS=C              LC_TELEPHONE=C           
[11] LC_MEASUREMENT=en_IL.utf8 LC_IDENTIFICATION=C      

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] covidAgeData_0.0.3

loaded via a namespace (and not attached):
 [1] RcppEigen_0.3.3.9.1      tidyselect_1.1.0         xfun_0.19               
 [4] ffbase_0.13.1            remotes_2.2.0            purrr_0.3.4             
 [7] lattice_0.20-41          vctrs_0.3.6              generics_0.1.0          
[10] testthat_3.0.1           usethis_2.0.0            rlang_0.4.10            
[13] pkgbuild_1.2.0           pillar_1.4.7             httpcode_0.3.0          
[16] glue_1.4.2               withr_2.3.0              sessioninfo_1.1.1       
[19] lifecycle_0.2.0          stringr_1.4.0            commonmark_1.7          
[22] devtools_2.3.2           memoise_1.1.0            knitr_1.30              
[25] RcppArmadillo_0.10.1.2.0 ff_4.0.4                 callr_3.5.1             
[28] ps_1.5.0                 curl_4.3                 parallel_4.0.3          
[31] fansi_0.4.1              Rcpp_1.0.5               osfr_0.2.8              
[34] desc_1.2.0               pkgload_1.1.0            jsonlite_1.7.2          
[37] fs_1.5.0                 bit_4.0.4                fastmatch_1.1-0         
[40] digest_0.6.27            stringi_1.5.3            processx_3.4.5          
[43] dplyr_1.0.2              collapse_1.5.0           rprojroot_2.0.2         
[46] grid_4.0.3               cli_2.2.0                tools_4.0.3             
[49] magrittr_2.0.1           tibble_3.0.4             crul_1.0.0              
[52] crayon_1.3.4             pkgconfig_2.0.3          ellipsis_0.3.1          
[55] Matrix_1.3-0             xml2_1.3.2               data.table_1.13.6       
[58] prettyunits_1.1.1        assertthat_0.2.1         httr_1.4.2              
[61] roxygen2_7.1.1           rstudioapi_0.13          R6_2.5.0                
[64] compiler_4.0.3          

Importing vmode as well as .rambytes to my package "fixes" this issue. But we shouldn't have to import anything.

[chrisvwn]

@eshom How are you importing vmode and .rambytes? When I import only .rambytes it gives the error about vmode not found. But when I import both it complains about .rambytes not found again. I cannot seem to get the imports to work.

What I have at the function that uses ffbase:

#' @importFrom ff vmode .rambytes

[eshom]

@chrisvwn
That's how I import as well: #' @importFrom ff .rambytes vmode

[chrisvwn]

Thanks. For some reason it won't work on mine.