Does Fine-tunning Bert Model in multiple times with different dataset make it more accuracy?

Question

i'm totally new in NLP and Bert Model. What im trying to do right now is Sentiment Analysis on Twitter Trending Hashtag ("neg", "neu", "pos") by using DistilBert Model, but the accurazcy was about 50% ( I tried w Label data taken from Kaggle). So here is my idea: (1) First, I will Fine-tunning Distilbertmodel (Model 1) with IMDB dataset, (2) After that since i've got some data took from Twitter post, i will sentiment analysis them my Model 1 and get Result 2. (3) Then I will refine-tunning Model 1 with the Result 2 and expecting to have Model (3).

Im not really sure this process has any meaning to make the model more accuracy or not. Thanks for reading my post.

Answer 1

I'm a little skeptical about your first step. Since the IMDB database is different from your target database, I do not think it will positively affect the outcome of your work. Thus, I would suggest fine-tuning it on a dataset like a tweeter or other social media hashtags; however, if you are only focusing on hashtags and do not care about the text, that might work! My little experience with fine-tuning transformers like BART and BERT shows that the dataset that you are working on should be very similar to your actual data. But in general, you can fine-tune a model with different datasets, and if the datasets are structured for one goal, it can improve the model's accuracy.