Tri-Level Relevance Learning with Ordered Ranking Constraints for Compulsory Education Quality Evaluation

Abstract

Recent advances in large language models (LLMs) have made retrieval-augmented generation (RAG) an effective means of enhancing LLMs by retrieving relevant documents from external knowledge bases, thereby improving retrieval accuracy. However, conventional binary retrieval cannot distinguish partially relevant evaluation texts-such as indicator-relevant but dimension-mismatched passages or policy fragments with only partial usability-leading to distorted evaluation conclusions and misaligned indicator matching. This paper proposes AAR-TRL, a tri-level relevance retrieval-augmented method for compulsory education quality evaluation, and validates it on a dedicated dataset covering indicator-definition and policy-basis queries. It leverages the cross-attention scores of the FiD fusion decoder to construct an automated tri-level labeling system that partitions documents into fully supportive, partially supportive, and non-supportive categories, and jointly optimizes the list-wise InfoNCE loss and ordered pairwise loss to sharpen fine-grained discrimination among easily confused evaluation documents. On our constructed compulsory education quality evaluation dataset, we evaluate AAR-TRL with Precision@10, Recall@10, and MRR. AAR-TRL attains 60.28%, 82.91%, and 0.6528 on the three metrics, respectively; compared with the vanilla AAR, it improves Precision@10 by 11.2 percentage points, Recall@10 by 6.82 percentage points, and MRR by 0.024; compared with ARL2, the improvements are 3.11 percentage points, 1.37 percentage points, and 0.012, respectively. In generation quality evaluation, AAR-TRL achieves an indicator matching accuracy of 87.0% and a policy citation validity of 82.0%, both superior to the baseline methods on the same test queries. These results, obtained within the compulsory education stage, provide preliminary evidence of AAR-TRL’s effectiveness in retrieval and question-answering generation for compulsory education quality evaluation; its generalizability to other education stages remains to be validated.