DistilBERT
Update Any Language Model: New Method to Update Pretrained Language Models
The ability to update language models is essential to incorporate new information and correct undesirable behaviors. Previous methods are unwieldy and often fail as the amount of new data increases. New work offers a workaround.
