Category "nlp"

How to setup LSTM to use n-grams instead of sequence length?

I currently have an LSTM which uses sequence length as input, but this only allows the LSTM to predict when the input length is equal to the used sequence lengt

Extracting education from Text data

I have bunch of text data describing people's eduation. I have already done some basic NLP processing to those text data. An example would be this : XXX receive

RoBERTa classifier: cannot generate single prediction

I have succesfully trained a text emotion classifier fine-tuning a RoBERTa language model, mostly using a helpful notebook found online. Now I am trying to writ

Use the polarity distribution of word to detect the sentiment of new words

I have just started a project in NLP. Suppose I have a graph for each word that shows the polarity distribution of sentiments for that word in different sentenc

Prodigy + Spacy to train dataset

I would like an example of using Prodigy to train a dataset (text file with some named entities). This file would be in Portuguese. The idea would be to train a

Why is my word lemmatization not working as expected?

Hi stackoverflow community! Long-time reader but first-time poster. I'm currently trying my hand at NLP and after reading a few forum posts touching upon this t

Convert from Prodigy's JSONL format for labeled NER to spaCy's training format?

I am new to Prodigy and spaCy as well as CLI coding. I'd like to use Prodigy to label my data for an NER model, and then use spaCy in python to create models.

FastText 0.9.2 - why is recall 'nan'?

I trained a supervised model in FastText using the Python interface and I'm getting weird results for precision and recall. First, I trained a model: model = fa

How to solve missing words in nltk.corpus.words.words()?

I have tried to remove non-English words from a text. Problem many other words are absent from the NLTK words corpus. My code: import pandas as pd lst = ['

Process and progress for natural language analysis of company communication?

Assume there is a large record of all different kinds of inter-employee and customer communications (e.g. mails, chat transcripts, OCRed letters) which should b

How can I use "NER" for German Language with stanford-corenlp?

I am trying to use nlp for german language but it does not work! I was making the pipeline and then NER to find the entity of each element in sentence which is

'CRF' object has no attribute 'keep_tempfiles'

I have imported ` from itertools import chain import nltk import sklearn import scipy.stats import sklearn_crfsuite from sklearn_crfsuite import scorers,CR

Tokenization of Compound Words not Working in Quanteda

I'm trying to create a dataframe containing specific keywords-in-context using the kwic() function, but unfortunately, I'm running into some error when attempti

How are the TokenEmbeddings in BERT created?

In the paper describing BERT, there is this paragraph about WordPiece Embeddings. We use WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocab

How do I know the order of the classes in a CatBoost classifier weights?

This is a pretty dumb question, but I couldn't find anywhere, so I will take my chances in here... I'm building a classifier using CatBoost. Since this is a NLP

TypeError: "hypothesis" expects pre-tokenized hypothesis (Iterable[str]):

I am trying to calculate the Meteor score for the following: print (nltk.translate.meteor_score.meteor_score( ["this is an apple", "that is an apple"], "an

NLP textEmbed function

I am trying to run the textEmbed function in R. Set up needed: require(quanteda) require(quanteda.textstats) require(udpipe) require(reticulate) #udpi

How to Vectorize python function

I have made a resume parser but to parse my resumes, I am using a for loop to run my parse function over each resume. Is there a way to vectorize this approach?

How to store Bag of Words or Embeddings in a Database

I would like to store vector features, like Bag-of-Words or Word-Embedding vectors of a large number of texts, in a dataset, stored in a SQL Database. What're t

R: Correct Way to Calculate Cosine Similarity?

I am working with the R programming language. I have the following data: text = structure(list(id = 1:8, reviews = c("I guess the employee decided to buy their