1. What machine learning approaches are used for co-reference resolution?
Machine learning approaches for co-reference resolution include pairwise classification, clustering algorithms, conditional random fields, and co-training. Initial methodologies redefined the task as a pairwise classification problem and proposed a decision-tree approach. Clustering algorithms represent noun phrases as feature vectors to group them together. Conditional random fields and co-training are also applied to this task. Additionally, machine learning approaches examined by researchers used annotated co-reference chains to generate additional co-reference data. In the provided section, an end-to-end co-reference resolution model with Coarse-to-fine inference was employed, trained on English OntoNotes 5.0. The model uses 3 highway LSTMs and 2 types of GloVe embeddings for context and head embeddings. To mitigate gender bias in these embeddings, SoftWEAT and HardWEAT debiasing methods were used, resulting in reduced bias in anti-stereotyped and type-2 pro-stereotyped data.
read more
2. What are the techniques used for model compression in NLP?
Model compression in NLP involves techniques such as Pruning, Quantization, Knowledge Distillation, Parameter Sharing, Tensor Decomposition, and Sub-quadratic Transformer-based methods. Pruning involves setting insignificant weights to zero, reducing the number of weights and potentially improving inference time. Quantization reduces the number of bits used to represent weights, but may result in loss of accuracy. Knowledge Distillation involves training a smaller model to mimic the output of a larger model. Parameter Sharing and Tensor Decomposition reduce the number of parameters and memory requirements. Sub-quadratic Transformer-based methods aim to reduce computational complexity while maintaining performance. These techniques aim to reduce the size and computational cost of overparametrized models while retaining as much performance as possible.
read more
3. What are the techniques for bias mitigation in NLP?
Bias mitigation techniques in NLP have evolved over the years, focusing on various aspects such as gender bias. Bolukbasi et al. explored word embedding debiasing, identifying and compensating for gender deviations in vector representations. Zhao et al. introduced vector space manipulation and data augmentation by gender swapping, creating a new benchmark (WinoBias) for gender bias-focused co-reference resolution. They also used data augmentation and embedding neutralization methods to mitigate gender bias in word embeddings. Zhao et al. proposed a novel training approach for learning gender-neutral word embeddings through attribute protection, generating a gender-neutral version of GloVE (GN-GloVe) that successfully separates gender information without compromising embedding model functionality.
read more
4. What are the two variants of debiasing evaluated in the Sentiment Analysis system?
The two variants of debiasing evaluated in the Sentiment Analysis system are HardWEAT and SoftWEAT. HardWEAT debiasing uses a neutralizing technique that removes non-gender words from a gender subspace, while SoftWEAT enables gradual bias removal by optimizing a transformation matrix with a tuning parameter l. Both methods aim to identify and reduce gender bias in word embeddings.
read more