Asking gpt-2 to finish sentence with huggingface transformers

Question

I am currently generating text from left context using the example script run_generation.py of the huggingface transformers library with gpt-2:

$ python transformers/examples/run_generation.py \
  --model_type gpt2 \
  --model_name_or_path gpt2 \
  --prompt "Hi, " --length 5

=== GENERATED SEQUENCE 1 ===
Hi,  could anyone please inform me

I would like to generate short complete sentences. Is there any way to tell the model to finish a sentence before length words?

---

Note: I don't mind changing model, but would prefer an auto-regressive one.

Answer 1

Unfortunately there is no way to do so. You can set the length parameter to a greater value and then just discard the incomplete part at the end.

Even GPT3 doesn't support completing a sentence before a specific length. GPT3 support "sequences" though. Sequences force the model to stop when certain condition is fulfilled. You can find more information about in thi article

Answer 2

This parameter helps you to get the results:

    '''
model_name is the model name, such as "124M" or "345M" and relies on 
models_dir.
• models_dir defines the directory containing the models.
• seed sets a random integer for random generators. The seed can be set to 
reproduce results.
• nsamples is the number of samples to return. If it is set to 0, it will continue 
to generate samples until you double-click on the run button of the cell or 
press Ctrl + M.
• batch_size determines the size of a batch and has an impact on memory 
and speed.
• length is the number of tokens of generated text. If set to none, it relies on 
the hyperparameters of the model.
• temperature determines the level of Boltzmann distributions. If the 
temperature is high, the completions will be more random. If the temperature 
is low, the results will become more deterministic.
• top_k controls the number of tokens taken into consideration by Top-k at 
each step. 0 means no restrictions. 40 is the recommended value.
• top_p controls Top-p
'''

Hope it will help you!