A Comprehensive Neural and Behavioral Task Taxonomy Method for Transfer Learning in NLP

Abstract

on the target task (Dwivedi and Roig, 2019; Song et al., 2019). Nevertheless, devising an effective Transfer learning is frequently utilized in scemethod to identify a suitable similar task remains a narios with limited labeled examples, where a challenging endeavor (Ramirez et al., 2023). crucial step is to identify a related task to the tarTo evaluate the relatedness between tasks, difget task. CogTaskonomy (Luo et al., 2022) was ferent methods have been employed. The first catproposed to acquire a taxonomy of NLP tasks, specifically focusing on assessing the similaregory of method is task embedding, which learns ities between tasks. This method, inspired by a dedicated high-dimensional representation for cognitive processes, exhibits notable time effieach task using a task encoder (James et al., 2018; ciency. Nevertheless, it does not fully exploit Lan et al., 2019; Achille et al., 2019; Vu et al., the task-related information present in cogni2020). Despite the low time complexity, modtive data and lacks a comprehensive evaluation ulating the model to adapt for a new task using of various types of cognitive data. To address this method is challenging. Another approach is these limitations, this paper proposes a comprehensive neural and behavioral method to invesTaskonomy method (Zamir et al., 2018), which tigate the relationship among NLP tasks. Our fine-tunes a model on each task, and transfers each approach utilizes cognitive data, encompassing fine-tuned model to other tasks in a fully superboth neural data such as fMRI and EEG, as well vised manner. This process can effectively capture as behavioral data including eye-tracking and task similarity, while it is demanding and timesemantic feature ratings. Each data modality is consuming. The third category encompasses the employed to establish a common representation cognitively inspired CogTaskonomy proposed by space with Representation Similarity Analysis Luo et al. (2022). This method projects task-related for projecting task-related representations. To fully leverage the cognitive information, we representations into a shared space based on fMRI effectively extract the task-relevant informaand subsequently evaluates the similarity between tion extracted from neural data through feature tasks. It only needs to fine-tune a model for each ranking. Experimental results on 12 NLP tasks task separately, which exhibits notable time effidemonstrate that our proposed method outperciency. However, shared spaces in CogTaskonomy forms state-of-the-art methods on evaluating are not strict shared spaces, and this method does task similarity. not effectively leverage task-related information from neural data. Our study falls under the third