Megh Thakkar

2023

1. EMNLP2023

   Self-Influence Guided Data Reweighting for Language Model Pre-training

   Megh Thakkar, Sriram Ganapathy, Shikhar Vashishth, and 3 more authors

   In Proceedings of The 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023
2. ACL2023

   Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations

   Linlin Liu*, Xingxuan Li*, Megh Thakkar, and 4 more authors

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abs

   Due to the huge amount of parameters, finetuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level lowresource NLP tasks.

2022

1. ACL2022

   DMix: Adaptive Distance-aware Interpolative Mixup

   Ramit Sawhney*, Megh Thakkar*, Shrey Pandit*, and 4 more authors

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), May 2022

   Abs

   Interpolation-based regularisation methods such as Mixup, which generate virtual training samples, have proven to be effective for various tasks and modalities. We extend Mixup and propose DMix, an adaptive distance-aware interpolative Mixup that selects samples based on their diversity in the embedding space. DMix leverages the hyperbolic space as a similarity measure among input samples for a richer encoded representation.DMix achieves state-of-the-art results on sentence classification over existing data augmentation methods on 8 benchmark datasets across English, Arabic, Turkish, and Hindi languages while achieving benchmark F1 scores in 3 times less number of iterations. We probe the effectiveness of DMix in conjunction with various similarity measures and qualitatively analyze the different components.DMix being generalizable, can be applied to various tasks, models and modalities.

2. ACL2022

   Chart-to-Text: A Large-Scale Benchmark for Chart Summarization

   Shankar Kantharaj*, Rixie Tiffany Leong*, Xiang Lin*, and 4 more authors

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022

   Abs

   Charts are commonly used for exploring data and communicating insights. Generating natural language summaries from charts can be very helpful for people in inferring key insights that would otherwise require a lot of cognitive and perceptual efforts. We present Chart-to-text, a large-scale benchmark with two datasets and a total of 44,096 charts covering a wide range of topics and chart types. We explain the dataset construction process and analyze the datasets. We also introduce a number of state-of-the-art neural models as baselines that utilize image captioning and data-to-text generation techniques to tackle two problem variations: one assumes the underlying data table of the chart is available while the other needs to extract data from chart images. Our analysis with automatic and human evaluation shows that while our best models usually generate fluent summaries and yield reasonable BLEU scores, they also suffer from hallucinations and factual errors as well as difficulties in correctly explaining complex patterns and trends in charts.

2021

1. EMNLP2021

   HypMix: Hyperbolic Interpolative Data Augmentation

   Ramit Sawhney*, Megh Thakkar*, Shivam Agarwal, and 3 more authors

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021

   Abs

   Interpolation-based regularisation methods for data augmentation have proven to be effective for various tasks and modalities. These methods involve performing mathematical operations over the raw input samples or their latent states representations - vectors that often possess complex hierarchical geometries. However, these operations are performed in the Euclidean space, simplifying these representations, which may lead to distorted and noisy interpolations. We propose HypMix, a novel model-, data-, and modality-agnostic interpolative data augmentation technique operating in the hyperbolic space, which captures the complex geometry of input and hidden state hierarchies better than its contemporaries. We evaluate HypMix on benchmark and low resource datasets across speech, text, and vision modalities, showing that HypMix consistently outperforms state-of-the-art data augmentation techniques. In addition, we demonstrate the use of HypMix in semi-supervised settings. We further probe into the adversarial robustness and qualitative inferences we draw from HypMix that elucidate the efficacy of the Riemannian hyperbolic manifolds for interpolation-based data augmentation.