Exploring Methods and Resources for Discriminating Similar Languages

Question

1. What contributions have the authors mentioned in the paper "Exploring methods and resources for discriminating similar languages" ?

2. What future works have the authors mentioned in the paper "Exploring methods and resources for discriminating similar languages" ?

3. What is the main assumption for the.pt TLD?

4. What was the first use of de-lexicalized text representations?

Accepted Answer

In this paper, the authors describe the submissions made by team UniMelb-NLP, which took part in both the closed and open categories.. The authors present the text representations and modeling techniques used, including cross-lingual POS tagging as well as fine-grained tags extracted from a deep grammar of English, and discuss additional data they collected for the open submissions, utilizing custombuilt web corpora based on top-level domains as well as existing corpora.

Accepted Answer

Overall, these results highlight the need for further research into discriminating between varieties of English.

Accepted Answer

For instance, the authors assume that Portuguese text found when crawling the .pt TLD will primarily be European Portuguese, while the Portuguese found in .br will be primarily Brazilian Portuguese.

Accepted Answer

De-lexicalized text representations through POS tagging were first considered for native language identification (NLI), where they were used as a proxy for syntax in order to capture certain types of grammatical errors (Wong and Dras, 2009).

Accepted Answer

To classify test data, the same group-level classifier used in AC2 was used to map sentences to language groups, and then the per-group POS tagger was applied to derive the corresponding stream of POS tags for each sentence.

Accepted Answer

One of the challenges in a machine learning approach to discriminating similar languages is to learn differences between languages that are truly representative of the distinction between varieties, rather than differences that are merely representative of peculiarities of the training data (Kilgarriff, 2001).

Accepted Answer

In their internal testing on the provided development data, the group-level classifier achieved 100% accuracy in classifying sentences at the group level, essentially reducing the problem to within-group disambiguation.

Accepted Answer

UMPOS generates POS taggers for new languages in an unsupervised fashion, by making use of parallel data and an existing POS tagger.

Accepted Answer

After deduplication, the seed URLs were then fed to a Heritrix 3.1.16 instance with default settings other than constraining the crawled content to the relevant TLD.

Accepted Answer

The process of creating a corpus for each sub-language involved translating a sample of 200 of the original ukWaC queries into each language using Panlex (Baldwin et al., 2010).

Accepted Answer

Each sentence in the training data is mapped to a POS-tag sequence in the 12-tag universal tagset, using the per-group POS tagger for the language group.

Accepted Answer

Their most successful methods implement straightforward hierarchical LangID, firstly identifying the language group that a sentence belongs to, before identifying the specific language.

Accepted Answer

their best result was obtained by applying two-level hierarchical LangID, firstly identifying the language group that a sentence belongs to, and then disambiguating within each group.

Accepted Answer

In order to have a resource that covers all the languages in the shared task drawn from a single source, the authors constructed web corpora for each language.

Accepted Answer

Since the ERG models a sharp notion of grammaticality, not all inputs receive an analysis — whether due to gaps in the coverage of the grammar or genuinely ungrammatical input.

Accepted Answer

One strategy that has been proposed to mitigate the effect of such topical differences is the use of a de-lexicalized text representation (Lui and Cook, 2013).

Exploring Methods and Resources for Discriminating Similar Languages

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What contributions have the authors mentioned in the paper "Exploring methods and resources for discriminating similar languages" ?

2. What future works have the authors mentioned in the paper "Exploring methods and resources for discriminating similar languages" ?

3. What is the main assumption for the.pt TLD?

4. What was the first use of de-lexicalized text representations?

5. How many per-group classifiers were used to classify test data?

6. What is the challenge in a machine learning approach to discriminating similar languages?

7. How did the group-level classifier achieve 100% accuracy?

8. What is the main idea behind the use of POS taggers?

9. What was the default setting for the seed URLs?

10. What was the process of creating a corpus for each sub-language?

11. How many per-group classifiers are used in the training data?

12. What is the method for identifying a language?

13. How did the authors achieve the result?

14. What did the authors do to create a corpus for each language?

15. Why did the ERG model a sharp notion of grammaticality?

16. What is the way to mitigate topical differences?

Figures

Citations

Discriminating between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task

Automatic Language Identification in Texts: A Survey

A Report on the DSL Shared Task 2014

Discriminating Similar Languages: Evaluations and Explorations

Discriminating Similar Languages: Evaluations and Explorations.

References

A systematic comparison of various statistical alignment models

Head-driven phrase structure grammar

Europarl: A Parallel Corpus for Statistical Machine Translation

Feature-rich part-of-speech tagging with a cyclic dependency network

N-gram-based text categorization

Related Papers (5)

Merging Comparable Data Sources for the Discrimination of Similar Languages : The DSL Corpus Collection

The NRC System for Discriminating Similar Languages

A Report on the DSL Shared Task 2014

Classifying English Documents by National Dialect

Automatic Identification of Language Varieties: The Case of Portuguese