Pre-Training Language Models with Document Links: The New Way to Unlock Knowledge and Answer Questions!

BERT over Document Graphs

In this blog, we will be getting into the paper LinkBERT: Pre-training Language Models with Document Links.

Overview

A new language model pre-training method called LinkBERT has been proposed by researchers from Stanford University. It uses links between documents to capture inter-document dependencies and knowledge sharing, allowing for better question-answering performance than traditional language model based methods.

Document links (e.g. hyperlinks) can provide salient multi-hop knowledge.

Outline

Proposed Approach

Obtaining the Document Graph

Relevance, Salience, and Diversity

Experiments and Results

Conclusion

Proposed Approach

The LinkBERT approach is an innovative way of pre-training language models with document links. This approach is based on the concept of treating a corpus of…