WWW 2025 Papers

ACM Web Conference

Decision breakdown

409papers

Counted from the official paper list; may differ slightly from the number the conference announced.

  • Oral
    15537.9%
  • Poster
    25462.1%

Top 10 topics (click to filter)

No topic data yet.

Trends for all 207 research topics

Browse all papers

Highly-efficient minimization of network connectivity in large-scale graphs

Mingyang Zhou, Gang Liu, Kezhong Lu, Hao Liao, Rui Mao

Expand abstract

Network connectivity minimization is a fundamental problem in controlling the spread of epidemics and facilitating information propagation in social networks. The problem aims to identify a budget number of key nodes whose removal would minimize the connectivity of a network. However, the existing solutions heavily rely on the number of edges, making it challenging to handle large and densely connected social networks. In this study, we present a fast algorithm that is independent of the number of edges. To achieve this, we first introduce a surrogate matrix that approximates the residual adjacency matrix with arbitrary small predefined error. We then devise an efficient approach for calculating the key nodes by optimizing the eigenvalues of the surrogate matrix. Remarkably, the algorithm has a small time complexity , with a small tunable number. Our algorithm thereby maintains a linear scalability in terms of the number of nodes and is unaffected by the number of edges. Hence, it has the capability to efficiently handle large and dense social networks. At last, we evaluate its performance against state-of-the-art techniques using diverse real-world datasets. The experimental results demonstrate the superiority of our proposed method in terms of both solution quality and computational efficiency.

Poster
Graph Embeddings Meet Link Keys Discovery for Entity Matching

Chloé Khadija Jradeh, Ensiyeh Raoufi, Jerome David, Pierre LARMANDE, Francois Scharffe, Konstantin Todorov, Cassia Trojahn

Expand abstract

Entity Matching (EM) automates the discovery of identity links between entities within different Knowledge Graphs (KGs). Link keys are crucial for EM, serving as rules allowing to identify identity links across different KGs, possibly described using different ontologies. However, the approach for extracting link keys struggles to scale on large KGs. While embedding-based EM methods efficiently handle large KGs they lack explainability. This paper proposes a novel hybrid EM approach to guarantee the scalability link key extraction approach and improve the explainability of embedding-based EM methods. First, embedding-based EM approaches are used to sample the KGs based on the identity links they generate, thereby reducing the search space to relevant sub-graphs for link key extraction. Second, rules (in the form of link keys) are extracted to explain the generation of identity links by the embedding-based methods. Experimental results demonstrate that the proposed approach allows link key extraction to scale on large KGs, preserving the quality of the extracted link keys. Additionally, it shows that link keys can improve the explainability of the identity links generated by embedding-methods, allowing for the regeneration of 77% of the identity links produced for a specific EM task, thereby providing an approximation of the reasons behind their generation.

Poster
What’s in a Query: Polarity-aware Distribution-based Fair Ranking

Aparna Balagopalan, Kai Wang, Olawale Elijah Salaudeen, Asia Biega, Marzyeh Ghassemi

Expand abstract

Machine learning-driven rankings, where individuals (or items) are ranked in response to a query, mediate search exposure or attention in a variety of safety-critical settings. Thus, it is important to ensure that such rankings are fair. Under the goal of equal opportunity, attention allocated to an individual on a ranking interface should be proportional to their relevance across search queries. In this work, we examine amortized fair ranking -- where relevance and attention are cumulated over a sequence of user queries to make fair ranking more feasible. Unlike prior methods that operate on expected amortized attention for each individual, we define new divergence-based measures for attention distribution-aware fairness in ranking (DistFaiR), characterizing unfairness as the divergence between the distribution of attention and relevance corresponding to an individual over time. This allows us to propose new definitions of unfairness, which are more reliable at test time and outperform prior fair ranking baselines. Second, we prove that group fairness is upper-bounded by individual fairness under this definition for a useful sub-class of divergence measures, and experimentally show that maximizing individual fairness through an integer linear programming-based optimization is often beneficial to group fairness. Lastly, we find that prior research in amortized fair ranking ignores critical information about queries, potentially leading to a fairwashing risk in practice by making rankings appear more fair than they actually are.

Poster
Centralization in Decentralized Web: Challenges and Opportunities in IPFS’s Data Management

Ruizhe Shi, Ruizhi Cheng, Yuqi Fu, Bo Han, Yue Cheng, Songqing Chen

Expand abstract

The InterPlanetary File System (IPFS) is a pioneering effort for Web 3.0, well-known for its decentralized infrastructure. However, some recent studies have shown that IPFS exhibits a high degree of centralization and has integrated centralized components for better performance. While this change contradicts the core decentralized ethos of IPFS and introduces risks of hurting the data replication level and thus availability, it also opens some opportunities for better data management and cost savings through deduplication. To explore these challenges and opportunities, we start by collecting an extensive dataset of IPFS internal traffic spanning the last three years with 20+ billion messages. By analyzing this long-term trace, we obtain a more complete and accurate view of how the status of centralization evolves over an extended period. In particular, (1) IPFS shows a low replication level in general, with only about 2.71% of data files replicated more than 5 times. While increasing replication enhances lookup performance and data availability, it adversely affects downloading throughput due to the over- head involved in managing peer connections, (2) there is a clear growing trend in centralization within IPFS in the last 3 years, with just 5% of peers now hosting over 80% of the content, significantly decreasing from 21.38% 3 years ago, which is largely driven by the increase of cloud nodes, (3) the IPFS default deduplication strategy using Fixed-Size Chunking (FSC) is largely inefficient, especially with the current 256KB chunk size, achieving nearly zero efficiency. Although Content-Defined Chunking (CDC) with smaller chunks could save significant storage (about 1.8 PB) and cost, it could impact user performance negatively. We thus design and evaluate a new metadata format that optimizes deduplication without compromising performance.

Poster
A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems

lixi Zhu, Xiaowen Huang, Jitao Sang

Expand abstract

Conversational Recommender System (CRS) leverages real-time feedback from users to dynamically model their preferences, thereby enhancing the system's ability to provide personalized recommendations and improving the overall user experience. CRS has demonstrated significant promise, prompting researchers to concentrate their efforts on developing user simulators that are both more realistic and trustworthy. The emergence of Large Language Models (LLMs) has marked the onset of a new epoch in computational capabilities, exhibiting human-level intelligence in various tasks. Research efforts have been made to utilize LLMs for building user simulators to evaluate the performance of CRS. Although these efforts showcase innovation, they are accompanied by certain limitations. In this work, we introduce a Controllable, Scalable, and Human-Involved (CSHI) simulator framework that manages the behavior of user simulators across various stages via a plugin manager. CSHI customizes the simulation of user behavior and interactions to provide a more lifelike and convincing user interaction experience. Through experiments and case studies in two conversational recommendation scenarios, we show that our framework can adapt to a variety of conversational recommendation settings and effectively simulate users' personalized preferences. Consequently, our simulator is able to generate feedback that closely mirrors that of real users. This facilitates a reliable assessment of existing CRS studies and promotes the creation of high-quality conversational recommendation datasets.

Poster
Fact-based Counter Narrative Generation to Combat Hate Speech

Brian Wilk, Homaira Huda Shomee, Suman Maity, Sourav Medya

Expand abstract

Online hatred has become an increasingly pervasive issue, affecting individuals and communities across various digital platforms. To combat hate speech in such platforms, counter narratives (CNs) are regarded as an effective method. In recent years, there has been growing interest in using generative AI tools to construct CNs. However, most of the generative models produce generic responses to hate speech and can hallucinate, reducing their effectiveness. To address the above limitations, we propose a counter narrative generation method that enhances CNs by providing non-aggressive, fact-based narratives with relevant background knowledge from two distinct sources, including a web search module. Furthermore we conduct a comprehensive evaluation using multiple metrics, including LLM-based measures for persuasion, factuality, and informativeness, along with human and traditional NLP evaluations. Our method significantly outperforms baselines, achieving an average factuality score of 0.915, compared to 0.741 and 0.701 for competitive baselines, and performs well in human evaluations.

Poster
Biting Off More Than You Can Detect: Retrieval-Augmented Multimodal Experts for Short Video Hate Detection

Jian Lang, Rongpei Hong, Jin Xu, Yili Li, Xovee Xu, Fan Zhou

Expand abstract

Short Video Hate Detection (SVHD) is increasingly vital as hateful content — such as racial and gender-based discrimination — spreads rapidly across platforms like TikTok, YouTube Shorts, and Instagram Reels. Existing approaches face significant challenges: hate expressions continuously evolve, hateful signals are dispersed across multiple modalities (audio, text, and vision), and the contribution of each modality varies across different hate content. To address these issues, we introduce MoRE (Mixture of Retrieval-augmented multimodal Experts), a novel framework designed to enhance SVHD. MoRE employs specialized multimodal experts for each modality, leveraging their unique strengths to identify hateful content effectively. To ensure model's adaptability to rapidly evolving hate content, MoRE leverages contextual knowledge extracted from relevant instances retrieved by a powerful joint multimodal video retriever for each target short video. Moreover, a dynamic sample-sensitive integration network adaptively adjusts the importance of each modality on a per-sample basis, optimizing the detection process by prioritizing the most informative modalities for each instance. Our MoRE adopts an end-to-end training strategy that jointly optimizes both expert networks and the overall framework, resulting in nearly a twofold improvement in training efficiency, which in turn enhances its applicability to real-world scenarios. Extensive experiments on three benchmarks demonstrate that MoRE surpasses state-of-the-art baselines, achieving an average improvement of 6.91% in macro-F1 score across all datasets.

Poster
Toward Effective Digraph Representation Learning: A Magnetic Adaptive Propagation based Approach

Xunkai Li, Daohan Su, Zhengyu Wu, Guang Zeng, Hongchao Qin, Rong-Hua Li, Guoren Wang

Expand abstract

The $q$-parameterized magnetic Laplacian serves as the foundation of directed graph (digraph) convolution from a spectral perspective, enabling this kind of digraph neural network (MagDG) to encode node features and structural insights by complex-domain message passing. As a generalization of undirected methods, MagDG shows superior capability in modeling intricate web-scale topology and offers greater application potential. Despite the great success achieved by existing MagDGs, limitations still exist: (1) {Hand-crafted $q$}: The performance of MagDGs depends on selecting an appropriate $q$-parameter to construct suitable graph propagation equations in the complex domain. This parameter tuning, driven by downstream tasks, limits model flexibility and significantly increases manual effort. (2) {Coarse Message Passing}: Most approaches treat all nodes with the same complex-domain propagation and aggregation rules, neglecting their unique digraph contexts. This oversight results in sub-optimal performance. To address the above issues, we propose two key techniques: (1) MAP is crafted to be a plug-and-play complex-domain propagation optimization strategy in the context of digraph learning, enabling seamless integration into any MagDG to improve predictions while enjoying high running efficiency. (2) MAP++ is a new digraph learning framework, further incorporating a learnable mechanism to achieve adaptively edge-wise propagation and node-wise aggregation in the complex domain for better performance. Extensive experiments on 12 datasets demonstrate that MAP enjoys flexibility for it can be incorporated with any MagDG, and scalability as it can deal with web-scale digraphs. MAP++ achieves SOTA predictive performance on 4 different downstream tasks.

Poster
Detecting and Understanding the Promotion of Illicit Goods and Services on Twitter

Hongyu Wang, Ying Li, Ronghong Huang, xianghang mi

Expand abstract

In this study, we reveal, for the first time, popular online social networks (especially Twitter) are being extensively abused by miscreants to promote illicit goods and services of diverse categories. This study is made possible by multiple machine learning tools that are designed to detect and analyze Posts of Illicit Promotion (PIPs) as well as revealing their underlying promotion campaigns. Particularly, we observe that PIPs are prevalent on Twitter, along with extensive visibility on other three popular OSNs including YouTube, Facebook, and TikTok. For instance, applying our PIP hunter to the Twitter platform for 6 months has led to the discovery of 12 million distinct PIPs which are widely distributed in 5 major natural languages and 10 illicit categories, e.g., drugs, data leakage, gambling, and weapon sales. Along the discovery of PIPs are 580K Twitter accounts publishing PIPs as well as 37K distinct instant messaging accounts that are embedded in PIPs and serve as next hops of communication with prospective customers. Also, an arms race between Twitter and illicit promotion operators is also observed. Especially, 90% PIPs can survice the first two months since getting published on Twitter, which is likely due to the diverse evasion tactics adopted by miscreants to masquerade PIPs.

Poster
FP-Rainbow : Fingerprint-based Browser Configuration Identification

Maxime Huyghe, Walter Rudametkin, Clément Quinton

Expand abstract

Browser fingerprinting is a tracking technique that collects attributes and calls functions from the browser’s APIs. Unlike cookies, browser fingerprints are difficult to evade or delete, raising significant privacy concerns for users as they can be used to re-identify individuals over browsing sessions without their consent. Yet, there has been limited research on the impact of browser configuration settings on these fingerprints. This paper introduces FP-Rainbow, a novel approach to systematically explore and map the configuration space of Chromium-based web browsers aiming to identify the impact of configuration parameters on browser fingerprints and their changes over time. We explore 1,748 configuration parameters (switches) and identify their impact on the browser’s BOM (Browser Object Model). By collecting and analyzing over 61,000 fingerprints from 18 versions of Chromium, our study reveals that 32 to 56 of these configuration parameters (depending on versions), such as disable-3d-apis or disable-notifications, influence the fingerprint of a web browser. FP-Rainbow also proves efficient in identifying browser configuration parameters from unknown fingerprints, achieving an average successful identification rate of 84% when considering a single configuration parameter and 78% when multiple parameters are involved, across all evaluated browser versions. These findings emphasize the importance of measuring the impact of configuration parameters on browsers to develop safer and more ethical web browsers.

Poster
HeatSnap: A Hot Page-Aware Continuous Snapshots System for Virtual Machines in Web Infrastructure

Kangyue Gao, Chuangyu Ouyang, Xinkui Zhao, Miao Ye, Chen Zhi, Guanjie Cheng, Yueshen Xu, Shuiguang Deng, Jianwei Yin

Expand abstract

Snapshot technology is crucial for data protection and system recovery in virtualized environments, particularly with the growing need for continuous snapshots to maintain the integrity of long-running web-based and distributed applications. However, traditional snapshot methods often suffer from performance bottlenecks, and inefficient storage usage. These challenges are closely tied to the way memory pages are accessed during VM execution, where memory access patterns show significant disparities between frequently accessed "hot" pages and less-used "cold" pages.In this paper, we introduce HeatSnap, a continuous snapshot system designed to address these issues by leveraging the uneven access frequencies of memory pages. HeatSnap distinguishes between intensive hot pages and dirty pages, applying specialized snapshotting and storage strategies to optimize the handling of both hot and cold memory regions. This approach aims to optimize snapshot efficiency, minimize performance impact on the VM, and decrease storage costs.Our implementation of HeatSnap on QEMU/KVM demonstrates significant improvements in VM performance loss, snapshot duration, and storage efficiency compared to existing methods, as evidenced by evaluations on common web and cloud-based workloads.

Poster
Beyond the Crawl: Unmasking Browser Fingerprinting in Real User Interactions

Meenatchi Sundaram Muthu Selva Annamalai, Emiliano De Cristofaro, Igor Bilogrevic

Expand abstract

Browser fingerprinting is a pervasive online tracking technique increasingly used for profiling and targeted advertising. Existing research on fingerprinting prevalence relies heavily on automated web crawls, which inherently struggle to replicate the nuances of human-computer interaction. This raises concerns about the accuracy of current understandings of real-world fingerprinting deployments. To that end, this paper presents a user study involving 30 participants over a 10-week period, capturing telemetry data from real browsing sessions across 3,000 top-ranked websites. Our findings reveal that automated crawls miss nearly half (47.8%) of the fingerprinting websites encountered by real users. This discrepancy mainly stems from crawlers' inability to access authentication-protected pages, circumvent bot detection mechanisms, and trigger fingerprinting scripts activated by specific user interactions. We also identify potential new fingerprinting vectors present in real user data but absent from automated crawls. Finally, we evaluate the effectiveness of federated learning for training browser fingerprinting detection models on real user data, demonstrating superior performance to models trained solely on automated crawl data.

Poster
InfoMAE: Pairing-Efficient Cross-Modal Alignment with Informational Masked Autoencoders for IoT Signals

Tomoyoshi Kimura, Xinlin Li, Osama Hanna, Yatong Chen, Yizhuo Chen, Denizhan Kara, Tianshi Wang, Jinyang Li, Xiaomin Ouyang, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, Tarek F. Abdelzaher

Expand abstract

Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, where the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross- modal alignment of pretrained unimodal representations. InfoMAE achieves efficient cross-modal alignment with limited data pairs through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level align- ment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE’s pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%

Poster
Dual-level Mixup for Graph Few-shot Learning with Fewer Tasks

Yonghao Liu, Mengyu Li, Fausto Giunchiglia, Lan Huang, Ximing Li, Xiaoyue Feng, Renchu Guan

Expand abstract

Graph neural networks have been demonstrated as a powerful paradigm for effectively learning graph-structured data for downstream task analysis. Current leading graph models require a large number of labeled samples for training, which unavoidably leads to overfitting in few-shot scenarios. Recent research has sought to alleviate this issue by simultaneously leveraging graph learning and meta-learning paradigms. However, these graph meta-learning models assume the availability of numerous meta-training tasks to learn transferable meta-knowledge. Such an assumption may not be feasible in the real world due to the difficulty of constructing tasks and the substantial costs involved. Therefore, we propose a SiMple yet effectIve approach for graph few-shot Learning with fEwer tasks, named SMILE. We introduce a dual-level mixup strategy, encompassing both within-task and across-task mixup, to simultaneously enrich the available nodes and tasks in meta-learning. Moreover, we explicitly leverage the prior information provided by the node degrees in the graph to encode expressive node representations. Theoretically, we demonstrate that SMILE can enhance the model generalization ability. Empirically, SMILE consistently outperforms other competitive models by a large margin across all evaluated datasets with in-domain and cross-domain settings. Our anonymous code can be found here.

Poster
Decoupling Knowledge and Context: An Efficient and Effective Retrieval Augmented Generation Framework via Cross Attention

Qian Dong, Qingyao Ai, Hongning Wang, Yiding Liu, Haitao Li, Weihang Su, Yiqun Liu, Tat-Seng Chua, Shaoping Ma

Expand abstract

Retrieval-Augmented Generation (RAG) systems have become acrucial tool to augment large language models (LLMs) with external knowledge for better task performance. However, existing traditional RAG methods inject knowledge directly in the context, resulting in several limitations. First, these methods highly rely on the in-context learning capability of LLMs, which often leads to excessively long contexts. This is inefficient due to the quadratic complexity of self-attention, leading to significant increases in inference time. Second, the extended context and the nature of self-attention can cause the LLMs to lose important information in the context, thereby degrading the original capabilities of LLMs. Furthermore, the effectiveness of knowledge injection is perturbed by the permutation of knowledge within the extended context, reducing the robustness of existing RAG methods. To tackle the above problems, we propose DecoupledRAG, a method that decouples external knowledge from the context within the RAG framework. Specifically, we introduce a cross-attention based method that injects retrieved knowledge directly to the inference process of LLM on the fly, without modifying its parameters or the input context. The external knowledge could be utilized robustly in a permutation-independent manner. To the best of our knowledge, this is the first work that explore how to utilize cross-attention to inject knowledge with low training cost in decoder-only LLM era. By leveraging cross-attention operation, DecoupledRAG enables seamless knowledge aggregation without creating extended context. Experimental results demonstrate that our method achieves high efficiency while maintaining strong performance, which indicates that RAG frameworks have the potential to benefit further from more knowledge.

Poster
Breaking the Shield: Analyzing and Attacking Canvas Fingerprinting Defenses in the Wild

Hoang Dai Nguyen, Phani Vadrevu

Expand abstract

Canvas fingerprinting has become one of the most effective techniques for tracking users online, allowing websites to identify and track visitors without their consent. In this paper, we investigate four primary defense techniques designed to counter canvas fingerprinting, systematically analyzing their adoption across 18 browser extensions in Chrome and Firefox, as well as built-in protections from five major browsers: Chrome, Firefox, Brave, Tor, and Safari. Our analysis reveals significant disparities in the implementation and effectiveness of these defenses, with randomization-based techniques being the most widely adopted, particularly across nine extensions and in the privacy-focused browser, Brave. Despite their sophistication, we demonstrate successful attacks on all these randomization mechanisms, revealing that their supposed non-deterministic behavior can, in fact, be predicted and exploited. In summary, we demonstrate that, unfortunately, no fully deployable defense against canvas fingerprinting attacks exists currently. We conclude by proposing recommendations to strengthen existing defenses and enhance their resistance to future attacks.

Poster
Virtual Stars, Real Fans: Understanding the VTuber Ecosystem

Yiluo Wei, Gareth Tyson

Expand abstract

Livestreaming by VTubers -- animated 2D/3D avatars controlled by real individuals -- have recently garnered substantial global followings and achieved significant monetary success. Despite prior research highlighting the importance of realism in audience engagement, VTubers deliberately conceal their identities, cultivating dedicated fan communities through virtual personas. While previous studies underscore that building a core fan community is essential to a streamer's success, we lack an understanding of the characteristics and behaviors of viewers of this new type of streamer. Gaining a deeper insight into these viewers is critical for VTubers to enhance audience engagement, foster a more robust fan base, and attract a larger viewership. To address this gap, we conduct a comprehensive analysis of VTuber viewers on Bilibili, a leading livestreaming platform where nearly all VTubers in China stream. By compiling a first-of-its-kind dataset covering 2.7M livestreaming sessions, we investigate the characteristics, engagement patterns, and influence of VTuber viewers. Our research yields several valuable insights, which we then leverage to develop a tool to "recommend" future subscribers to VTubers. By reversing the typical approach of recommending streams to viewers, this tool assists VTubers in pinpointing potential future fans to pay more attention to, and thereby effectively growing their fan community.

Poster
Private Order Flows and Builder Bidding Dynamics: The Road to Monopoly in Ethereum’s Block Building Market

Shuzheng Wang, Yue Huang, Wenqin Zhang, Huang Yuming, Xuechao Wang, Jing Tang

Expand abstract

Ethereum, as a representative of Web3, adopts a novel framework called Proposer Builder Separation (PBS) to prevent the centralization of block profits in the hands of institutional Ethereum stakers. Introducing builders to generate blocks based on public transactions, PBS aims to ensure that block profits are distributed among all stakers. Through the auction among builders, only one will win the block in each slot. Ideally, the equilibrium strategy of builders under public information would lead them to bid all block profits. However, builders are now capable of extracting profits from private order flows. In this paper, we explore the effect of PBS with private order flows. Specifically, we propose the asymmetry auction model of MEV-Boost auction. Moreover, we conduct empirical study on Ethereum blocks from January 2023 to May 2024. Our analysis indicates that private order flows contribute to 54.59% of the block value, indicating that different builders will build blocks with different valuations. Interestingly, we find that builders with more private order flows (i.e., higher block valuations) are more likely to win the block, while retain larger proportion of profits. In return, such builders will further attract more private order flows, resulting in a monopolistic market gradually. Our findings reveal that PBS in current stage is unable to balance the profit distribution, which just transits the centralization of block profits from institutional stakers to the monopolistic builder.

Poster
2D-TPE: Two-Dimensional Positional Encoding Enhances Table Understanding for Large Language Models

Jia-Nan Li, Jian Guan, Wei Wu, Zhengtao Yu, Rui Yan

Expand abstract

Tables are ubiquitous across various domains for concisely representing structured information. Empowering large language models (LLMs) to reason over tabular data represents an actively explored direction. However, since typical LLMs only support one-dimensional (1D) inputs, existing methods often flatten the two-dimensional (2D) table structure into a sequence of tokens, which can severely disrupt the spatial relationships and result in an inevitable loss of vital contextual information. In this paper, we first empirically demonstrate the detrimental impact of such flattening operations on the performance of LLMs in capturing the spatial information of tables through two elaborate proxy tasks. Subsequently, we introduce a simple yet effective positional encoding method, termed ``2D-TPE'' (Two-Dimensional Table Positional Encoding), to address this challenge. 2D-TPE enables each attention head to dynamically select a permutation order of tokens within the context for attending to them, where each permutation represents a distinct traversal mode for the table, such as column-wise or row-wise traversal. 2D-TPE effectively mitigates the risk of losing essential spatial information while preserving computational efficiency, thus better preserving the table structure. Extensive experiments across five benchmarks demonstrate that 2D-TPE outperforms strong baselines, underscoring the importance of preserving the table structure for accurate table comprehension. Comprehensive analysis further reveals the substantially better scalability of 2D-TPE to large tables than baselines.

Poster
Mitigating Forgetting in Adapting Pre-trained Language Models to Text Processing Tasks via Consistency Alignment

Jianqi Gao, Hao Wu, Yiu-ming Cheung, Jian Cao, Hang Yu, Yonggang Zhang

Expand abstract

There are a large number of text processing tasks in web applications, such as sentiment classification, summary extraction, and question answering. Recently, fine-tuning pre-trained language models (PLMs) to adapt to downstream text-processing tasks has attracted much attention. However, due to the differences in data, model, and tasks between the pre-training and fine-tuning processes, the fine-tuning process may suffer from catastrophic forgetting of pre-training knowledge, which may implicitly limit the model’s performance and generalization ability. To address these challenges, we propose a novel dual-model framework, termed as \emph{co}nsistency \emph{a}l\emph{i}gnment (CoAi). The insight of CoAi lies in building an auxiliary model that simulates the distribution of pre-training knowledge in real-time according to the current task, and co-training the task-specific model and the auxiliary model to balance the pre-training knowledge and task-specific knowledge during fine-tuning. Specifically, the auxiliary model is constructed on-the-fly to maintain the pre-training knowledge. Subsequently, CoAi simulates the pre-training process by performing distributional exploration in the parameter space, which is built upon our novel insight into the transformation between data and model parameter space. However, the objectives leveraged to construct the auxiliary model lead to the misalignment between the pre-training and task-specific knowledge. To alleviate the inconsistency, we employ an auxiliary variable to align the prediction distribution of the task-specific and the auxiliary models, inspired by constrastive clustering. We validate the effectiveness of CoAi on ten classic classification tasks and three generation tasks, showing consistent and significant improvements compared with state-of-the-art methods.

Poster