Choosing the optimal threshold for the collocations extraction remains a manual task performed by experts. Until today, there is no serious work, based on deep studies, which explores possible solutions to automate the learning of the threshold in the statistical terminology field. In this paper, the authors try to spotlight on this problem by exploring, firstly, the evaluation performance techniques used in several scientific areas such as biomedical and biometric and applying them, subsequently, on the statistical terminology field. The experimental study gives promoters results. First, it shows the effectiveness of usual techniques such as ROC and Precision-Recall curves used to evaluate the performance of binary classification systems. Second, it provides a practical solution for automatic estimation of optimal thresholds for collocation extraction systems.

Estimation of a Priori Decision Threshold for Collocations Extraction: An Empirical Study

Detection of adulterations in a valuable Brazilian honey by using spectrofluorimetry and multiway classification.

Collocation is defined as a sequence of lexical tokens which habitually co-occur. This type of information is widely used in various applications such as Information Retrieval, document indexing, machine translation, lexicography, etc. Therefore, many techniques are developed for the automatic retrieval of collocations from textual documents. These techniques use statistical measures based on a joint frequency calculation to quantify the connection strength between the tokens of a candidate collocation. The discrimination between relevant and irrelevant collocations is performed using a priori fixed threshold. Generally, the discrimination threshold estimation is performed manually by a domain expert. This supervised estimation is considered as an additional cost which reduces system performance. In this paper, the authors propose a new technique for the threshold automatic learning to retrieve information from web text document. This technique is mainly based on the usual performance evaluation measures (such as ROC and Precision-Recall curves). The results show the ability to automatically estimate a statistical threshold independently of the treated corpus.

Information Retrieval from Unstructured Web Text Document Based on Automatic Learning of the Threshold

Bayesian estimation of a threshold time (hereafter simply threshold) for the receipt of impulse signals is accomplished given the following: 1) data, consisting of the number of impulses received in a time interval from zero to one and the time of the largest time impulse; 2) a model, consisting of a uniform probability density of impulse time from zero to the threshold, and constituting the probability density from which the data impulses are drawn randomly and independently; and 3) a prior probability density of threshold which is linear from a positive initial threshold to one and that is zero otherwise, or which is uniform from zero to one. The posterior probability density of threshold is found with the number of impulses, the time of the largest time impulse, and the initial threshold as parameters, and with the first two parameters fixed, the initial threshold is found that maximizes the entropy of the posterior probability density. It is shown that for some values of initial threshold, including the value that maximizes entropy, the following three estimates of threshold from the posterior probability density are all different: the maximum (maximum a posteriori or MAP estimate), the mean (mean squared error or MMSE estimate), and the maximum for a uniform prior probability density (maximum likelihood or ML estimate). This easily understood and practical Bayesian threshold estimation problem has two aspects of pedagogical interest: 1) the problem is unusual in that the MAP, MMSE, and ML estimates are generally different-in typical problems with Gaussian or uniform models and prior probability densities, at least two of these three estimates are equal; 2) the problem constitutes a clear illustration of how an unknown parameter in the prior probability density (here, initial threshold) may be specified so as to achieve a desired property for the posterior probability density (here, maximum entropy).

Bayesian Threshold Estimation

Popular Tasks used by Researchers

How can I help with your research?

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

We encountered an error while loading the panel. Please try refreshing the page.

Something went wrong

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

Anyone with the link can view this chat. You can switch it back to private anytime.

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

E.C. Like

Author Tools

Chat about Author

Papers

Bayesian Threshold Estimation