TRANSFORMER-BASED ABSTRACTIVE TEXT SUMMARIZATION: MODELS, CHALLENGES, AND EVALUATION

Abstract

Text summarization has undergone significant transformation with the advent of transformer-based neural architectures. This research examines the landscape of transformer models for abstractive summarization, exploring how attention mechanisms and pre-training strategies have revolutionized the field. We investigate prominent models including BERT, GPT variants, T5, and PEGASUS, analyzing their architectural innovations and summarization capabilities. The study addresses persistent challenges including factual consistency, handling long documents, computational efficiency, and evaluation reliability. Through comprehensive analysis of existing approaches and evaluation methodologies, we identify critical gaps in current summarization systems and propose directions for advancement. Our findings reveal that while transformer models achieve impressive fluency and coherence, ensuring factual accuracy remains problematic. The research contributes a structured understanding of the transformer summarization ecosystem, highlighting that future progress depends on developing better evaluation metrics, improving factual grounding, and creating more efficient architectures suitable for practical deployment.