Evaluating and Improving Text Summarization using Large Language Models

Figure 1. A framework for LLM reflective summarization through self-evaluation

Text summarization is the process of producing a concise and coherent summary while preserving key information and meaning of the source text. This technique is widely used in various fields; for example, it is commonly used to summarize scientific, medical, and legal documents, as it enables users to quickly grasp key points of lengthy texts and efficiently access relevant information.

This project aims to explore methods to effectively assess the quality of model-generated summaries using LLMs. The proposed evaluation methods are customizable to different quality attributes that users are interested in. Additionally, the developed methods are designed to work in the case when no reference summaries are available. Furthermore, this project will propose a framework to iteratively improve the quality of LLM-based summarization by leveraging the evaluation results. The proposed methods will be evaluated on multiple datasets with various domains, such as patent, legal, medical, finance, and others.

Related papers

Ding, J., Chen, H., Kolapudi, S., Pobbathi, L., & Nguyen, H. (2023, October). Quality Evaluation of Summarization Models for Patent Documents. In 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS) (pp. 250-259). IEEE.

Nguyen, H., Chen, H., Maganti, R., Hossain, K. T., & Ding, J. (2023, July). Measurement and Identification of Informative Reviews for Automated Summarization. In 2023 IEEE International Conference on Artificial Intelligence Testing (AITest) (pp. 146-151). IEEE.

Zhao, H., Chen, H., & Yoon, H. J. (2023, July). Enhancing Text Classification Models with Generative AI-aided Data Augmentation. In 2023 IEEE International Conference on Artificial Intelligence Testing (AITest) (pp. 138-145). IEEE.