Open AccessProceedings Article10.1109/DCC.2008.78

Text Pre-processing for Lossless Compression

- 25 Mar 2008

- pp 506-506

TL;DR: Textual data holds a number of properties that can be taken into account in order to improve compression, and capital conversion is one of the most commonly used concepts in text pre-processing.

Abstract: Textual data holds a number of properties that can be taken into account in order to improve compression. Pre-processing deals with these properties by applying a number of transformations that make the redundancy "more visible" to the compressor. One of the most commonly used concepts in text pre-processing is called capital conversion. Words with capital letters are converted to their lowercase versions while signaling the change with a flag. This way not only context similarities are increased but also dictionaries used for word replacement only need to contain words in their lowercase versions. Word replacement consists of replacing words with shorter codes which are references to their location in a dictionary.

AI Agents for this Paper

Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps

Most frequently asked questions

1. What are the contributions in "Text pre-processing for lossless compression" ?

In this paper the authors present several pre-processing techniques developed to help general-purpose compressors achieve better results in the task of ( lossless ) text compression.. In their experiments, the authors achieved an improvement representing almost 3 % over existing techniques on a large ( 100Mbyte ) file.

Table 2: Small file tests. Sizes after compression with PAQ8h. Best results are in bold. Sizes are in bytes.

Table 1: Large file tests. Sizes after compression with PAQ8h. The column ENWIK8A represents the results obtained with the new group sizes for codewords. Best results are in bold. Sizes are in bytes.

Citations

Proceedings Article•10.1109/KSE.2019.8919368

Effective Text Data Preprocessing Technique for Sentiment Analysis in Social Media Data

Saurav Pradha, +2 more

- 24 Oct 2019

TL;DR: An algorithm that weights the sentiment score in terms of weight of hashtag and cleaned text to obtain the sentiment and an algorithm to train the Support Vector Machine, Deep Learning, and Naïve Bayes classifiers to process Twitter data.

...read moreread less

Proceedings Article•10.1109/ICCITECHN.2011.6164801

An enhanced scheme for lossless compression of short text for resource constrained devices

Md. Rafiqul Islam, +1 more

- 01 Dec 2011

TL;DR: The main contribution of this paper is the integration of statistical text ranking or statistical component categorization with static coding in obtaining the compression and a mathematical model of the proposed scheme in terms of compression parameters is presented.

...read moreread less

Patent

Compression of genomic data

Sharmila S. Mande, +4 more

- 19 Mar 2012

TL;DR: In this paper, a system and a method for compression of genomic data is described, which includes obtaining modified genomic data from genomic data based at least in part on intermediary data identified from the genomic data.

...read moreread less

•Journal Article•10.14569/ijacsa.2023.0140290

Building a Machine Learning Powered Chatbot for KSU Blackboard Users

Qubayl Alqahtani, +1 more

- 01 Jan 2023

- International Journal of Advanced Comput...

TL;DR: In this article , the authors developed a chatbot engine that helps with frequently asked questions about the Blackboard system, which could be embedded into the website of the university of Saudi Arabia.

...read moreread less

•Journal Article•10.3906/ELK-1410-124

A new dictionary-based preprocessor that uses radix-190 numbering

Mete Eray Şenergin, +1 more

- 20 Jun 2016

- Turkish Journal of Electrical Engineerin...

TL;DR: A new word-based preprocessor named METEHAN190 (M190) is proposed and its performance is contrasted with four other state-of-the-art preprocessors and shows that the semistatic byte-oriented methods are much faster in comparison to the static dictionary-based methods that encode words with characters.

...read moreread less

References

Journal Article•10.1002/SPE.678

Revisiting dictionary‐based compression

Przemysław Skibiński, +2 more

- 01 Dec 2005

- Software - Practice and Experience

TL;DR: This paper discusses several aspects of dictionary‐based compression, including compact dictionary representation, and presents a PPM/BWCA‐oriented scheme, word replacing transformation, achieving compression ratios higher by 2–6% than the state‐of‐the‐art StarNT (2003) text preprocessor.

...read moreread less

Journal Article•10.1109/TC.2005.85

Universal text preprocessing for data compression

J. Abel, +1 more

- 01 May 2005

- IEEE Transactions on Computers

TL;DR: Several preprocessing algorithms for text files are presented which complement each other and which are performed prior to the compression scheme and the compression gain is compared along with the costs of speed for the BWT, PPM, and LZ compression schemes.

...read moreread less

Text Pre-processing for Lossless Compression

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What are the contributions in "Text pre-processing for lossless compression" ?

Figures

Citations

Effective Text Data Preprocessing Technique for Sentiment Analysis in Social Media Data

An enhanced scheme for lossless compression of short text for resource constrained devices

Compression of genomic data

Building a Machine Learning Powered Chatbot for KSU Blackboard Users

A new dictionary-based preprocessor that uses radix-190 numbering

References

Revisiting dictionary‐based compression

Universal text preprocessing for data compression

Related Papers (5)

Data compression using text encryption

Tag based models of English text

A systematic approach to compressing a full-text retrieval system

Static compression for dynamic texts

Dictionary-Based Data Compression: An Algorithmic Perspective