In classical machine learning, a model is trained from scratch for every individual task. Transfer learning is a machine learning technique where a model trained on one task is re-purposed on a second related task. Transfer learning allows for knowledge to be transferred from a related task that has already been learnt by the model. BERT and GPT-3 are examples of Transfer learning models. There are various types of transfer learning but the most relevant one for industrial applications is sequential transfer learning. This involves two stages: 1. Pre-training 2. Fine-tuning
Transfer Learning is particularly important for industry applications such as Legal NLP. It can greatly reduce the need for annotated data and speed up the time it takes to get to a model with reasonable performance.