Automatic Assessment of Text Complexity Levels in European Portuguese

Abstract

The assessment of text readability and the classification of texts by complexity levels is essential for language education and language-related industries that rely on effective communication. The Common European Framework of Reference for Languages (CEFR) provides a widely recognized framework for classifying language proficiency levels. This framework can be used not only to assess the proficiency of learners of a given language, but also from a readability perspective, as a means to identify the proficiency required to understand specific pieces of text. This study aims to develop and evaluate automatic models capable of classifying texts in European Portuguese according to the complexity levels defined by the (CEFR). For that, we explore the fine-tuning of several foundation models on textual data used for proficiency evaluation purposes. Additionally, we explore approaches to leverage the information provided by the ordinal nature of the levels. Furthermore, we perform a preliminary analysis of the base capability of instruction-based models to perform this task. Our experiments show that the best models can achieve over 80% accuracy and 75% F1 score. However, they have difficulty in generalizing to different types of text, which reveals the need for additional and more diverse training data.