How to use fairseq's Megatron in r-text?

Question

I am using R 4.1.2, reticulate 1.22 and text 0.9.50.

# install.packages("devtools")
devtools::install_github("oscarkjell/text")
library(text)
temp <- textEmbed("Hello!", model="anton-l/megatron-11b")

# Here comes an error message with traceback information. 
# My guess is huggingface's internal error.
Error in py_call_impl(callable, dots$args, dots$keywords) : 
KeyError: 'megatron' 
9.
stop(structure(list(message = "KeyError: 'megatron'", call = py_call_impl(callable, 
dots$args, dots$keywords), cppstack = structure(list(file = "", 
line = -1L, stack = 
c("/usr/local/lib/R/library/reticulate/libs/reticulate.so
(Rcpp::exception::exception(char const*, bool)+0x74) [0xffff9c3a97c4]", 
"/usr/local/lib/R/library/reticulate/libs/reticulate.so
(Rcpp::stop(std::__cxx11::basic_string<char, std::char_traits<char>, 
std::allocator<char> > const&)+0x28) [0xffff9c3a9838]",  ... 
8.
__getitem__ at configuration_auto.py#317
7.
from_pretrained at configuration_auto.py#600
6.
hgTransformerGetEmbedding at <string>#68
5.
hgTransformerGetEmbedding(text_strings = x[[i_variables]], model = model, 
layers = layers, return_tokens = return_tokens, device = device, 
tokenizer_parallelism = tokenizer_parallelism) 
4.
force(expr) 
3.
reticulate::py_capture_output(
hg_embeddings <- hgTransformerGetEmbedding(text_strings = x[[i_variables]], 
model = model, layers = layers, return_tokens = return_tokens, 
device = device, tokenizer_parallelism = tokenizer_parallelism), 
type = "stderr") 
2.
textEmbedLayersOutput(x, contexts = contexts, decontexts = decontexts, 
model = model, layers = layers, return_tokens = FALSE, device = device, 
print_python_warnings = print_python_warnings) 
1.
textEmbed("Hello!", model = "anton-l/megatron-11b") 

The following is my environment:

R version 4.1.2 (2021-11-01) Platform: aarch64-unknown-linux-gnu (64-bit) Running under: Ubuntu 18.04.6 LTS

Matrix products: default BLAS: /usr/local/lib/R/lib/libRblas.so LAPACK: /usr/local/lib/R/lib/libRlapack.so

locale: [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C [3] LC_TIME=C LC_COLLATE=C [5] LC_MONETARY=C LC_MESSAGES=C [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=C LC_IDENTIFICATION=C

attached base packages: [1] stats graphics grDevices utils [5] datasets methods base

other attached packages: [1] multilevel_2.7 MASS_7.3-54 [3] nlme_3.1-153 stylo_0.7.4 [5] magrittr_2.0.3 text_0.9.60 [7] forcats_0.5.1 stringr_1.4.0 [9] dplyr_1.0.9 purrr_0.3.4 [11] readr_2.1.1 tidyr_1.2.0 [13] tibble_3.1.6 ggplot2_3.3.5 [15] tidyverse_1.3.1 reticulate_1.22 [17] stringi_1.7.6

loaded via a namespace (and not attached): [1] colorspace_2.0-3 ellipsis_0.3.2 [3] class_7.3-19 fs_1.5.2 [5] rstudioapi_0.13 proxy_0.4-26 [7] listenv_0.8.0 furrr_0.2.3 [9] dials_0.1.1 prodlim_2019.11.13 [11] fansi_1.0.3 lubridate_1.8.0 [13] xml2_1.3.3 codetools_0.2-18 [15] splines_4.1.2 knitr_1.37 [17] jsonlite_1.8.0 workflows_0.2.6 [19] pROC_1.18.0 broom_0.8.0 [21] cluster_2.1.2 dbplyr_2.1.1 [23] png_0.1-7 yardstick_0.0.9 [25] tune_0.2.0 compiler_4.1.2 [27] httr_1.4.2 backports_1.4.1 [29] fastmap_1.1.0 assertthat_0.2.1 [31] Matrix_1.3-4 cli_3.3.0 [33] tcltk2_1.2-11 htmltools_0.5.2 [35] tools_4.1.2 gtable_0.3.0 [37] glue_1.6.2 tinytex_0.36 [39] Rcpp_1.0.8.3 cellranger_1.1.0 [41] DiceDesign_1.9 vctrs_0.4.1 [43] ape_5.6-2 iterators_1.0.14 [45] parsnip_0.2.1 timeDate_3043.102 [47] xfun_0.30 gower_1.0.0 [49] globals_0.14.0 brio_1.1.3 [51] testthat_3.1.4 rvest_1.0.2 [53] lifecycle_1.0.1 future_1.25.0 [55] scales_1.2.0 ipred_0.9-12 [57] hms_1.1.1 parallel_4.1.2 [59] yaml_2.2.1 rpart_4.1-15 [61] foreach_1.5.2 e1071_1.7-9 [63] lhs_1.1.5 hardhat_0.2.0 [65] lava_1.6.10 rlang_1.0.2 [67] pkgconfig_2.0.3 evaluate_0.15 [69] rsample_0.1.1 lattice_0.20-45 [71] recipes_0.2.0 cowplot_1.1.1 [73] tidyselect_1.1.2 parallelly_1.31.1 [75] plyr_1.8.7 R6_2.5.1 [77] generics_0.1.2 DBI_1.1.2 [79] pillar_1.7.0 haven_2.4.3 [81] withr_2.5.0 overlapping_1.7 [83] survival_3.2-13 nnet_7.3-16 [85] future.apply_1.9.0 tsne_0.1-3.1 [87] modelr_0.1.8 crayon_1.5.1 [89] pamr_1.56.1 utf8_1.2.2 [91] rmarkdown_2.14 tzdb_0.2.0 [93] grid_4.1.2 readxl_1.3.1 [95] reprex_2.0.1 digest_0.6.29 [97] GPfit_1.0-8 munsell_0.5.0 [99] tcltk_4.1.2

Answer 1

I have searched for this question online, though not having time to try these solutions.

Solution 1: One github issue page mentioned a similar issue for model name not found. The page thought one could extend the autoconfig class in HuggingFace.

https://huggingface.co/docs/transformers/main/en/model_doc/auto#extending-the-auto-classes

Solution 2: Follow the steps mentioned by the author anton-l: https://github.com/huggingface/transformers/pull/10301