Processing of Syntactic Structure in Neural Language Models: Subject-Verb Agreement in Galician and Portuguese

Abstract

Combining knowledge from the areas of computational linguistics and psycholinguistics, this paper explores the syntactic knowledge of neural language models, specifically the principle of subject-verb agreement in Galician and Portuguese. In order to study whether the models process this phenomenon in a similar way to humans, two datasets were created (one in Galician and one in Portuguese) with 16 sentences and 8 variants for each of them, taking into account grammaticality, the presence or absence of a distractor and the distance between the subject and the main verb. With these elements, surveys were carried out with speakers of these languages to check the acceptability of the sentences in relation to the variables referred to. Using an adapted version of the datasets, neural language models for Galician and Portuguese were evaluated. The results show that although both humans and models distinguish between grammatical and ungrammatical sentences, speakers exhibit greater accuracy and even benefit from distractors, while these confuse LMs, especially in long contexts.