Self-supervised approaches to abstractive summarization

Humans judge summaries based on how much key information they capture, not whether they match someone else’s summary word for word. Can we treat summarization as autoencoding rather than supervised text generation?

Abstractive summarization is a notoriously difficult task both to perform and to evaluate.  Nearly all existing approaches rely on supervised learning.  This imposes three important limitations:

  1. models trained this way can only train on moderate amounts of data, and only for genres like news articles or academic papers where summaries are readily available

  2. models can only learn the exact styles and lengths of the available summaries

  3. models must train on an inherently noisy proxy target: matching the exact wording of some human-generated summary (of many equally good possible summaries) is a very imperfect proxy for encapsulating the core information of a document

Summarization, at least as performed by humans, is inherently a kind of information compression.  It should therefore be possible to learn using the sort of self-supervised methods used to training autoencoders, inducing the summarizer model to condense the core information of a document into a small summary passage without requiring the passage to match the exact words a human summarizer chose.  The only paper I’ve seen attempt summarization-as-autoencoding (Wang and Lee, 2018) used a text-reconstruction loss, but several other alternatives could be possible.

In this project I explore several loss function alternatives for self-supervised abstractive summarization using current LLMs:

  1. Text-reconstruction loss: the model is rewarded based on how well a reconstructer can reconstruct the original text given the summary.  This is conceptually simple, but overspecific (humans can’t reconstruct a text from a summary).  Attempted with some success by Wang and Lee, 2018 using LSTM text generators, so it’s very plausible with a current LLM.

  2. Partial reconstruction loss: same as above, but with something analagous to teacher forcing: the model only generates a next paragraph based on a previous paragraph.  This is conceptually similar to the BottleSum approach to sentence summarization (West et al., 2019).

  3. Co-summarization loss: the model is rewarded based on text-reconstruction loss plus a (more heavily weighted) measure of how well its summary matches the summary produced by feeding a second, separately trained summarizer model the reconstructed text.  The idea here is that a good summary may not produce an identical full text, but it will produce a full text that yields an identical good summary from another summarizer (i.e., that contains the same core information).  In setting up two parallel learners to train each other, this approach is analagous to the Double DQN or Dueling DQN approaches in reinforcement learning (or, more roughly, to GANs).

  4. Text-matching loss: the summarizer model is rewarded based on a classifier model’s ability to identify which of several text passages is an excerpt from the original text, given the generated summary.  Essentially, the model generates features for a classifier, but the features have to be encoded in text.  An additional minimum-token-frequency-in-corpus constraint would be needed to induce the summarizer model to use plain language and restrict it from “cheating” by simply passing through highly distinctive words in the full text that would immediately identify the match.

In all cases, the generated summary can be constrained to remain human-readable either by using a discriminator to induce the summary sentences to be indistinguishable from human-written sentences (as in Wang and Lee, 2018) or by penalizing word-sequences a LLM judges improbable.  Depending on the use case, it may also help to implement a minimum-token-frequency-in-corpus constraint to induce the summarizer model to express the summary in plain, easily comprehensible language. 

If successful, this project will produce an abstractive summarization method that is far more flexible and generalizable than the current SOTA.  If less successful, it will still yield the interesting result of showing us what kind of text a model uses when it has to communicate with another model in English in order to accomplish a joint task.

Note: project is pending / no code written yet