Big5

Abstract: Named entities form the major components in a document. When we catch the fundamental entities, we can understand a document to some degree. This paper employs different types of information from different levels of text to extract named entities, including character conditions, statistic information, titles, punctuation marks, organization and location keywords, speech-act and locative verbs, cache and n-gram model. In the formal run of MET-2, the F-measures P&R, 2P&R and P&2R are 79.61%, 77.88% and 81.42%, respectively. INTRODUCTION People, affairs, time, places and things are five basic entities in a document. When we catch the fundamental entities, we can understand a document to some degree. Natural Language Processing Laboratory (NLPL) in Department of Computer Science and Information Engineering (CSIE), National Taiwan University (NTU) starts to study named entity extraction problem in 1993. At first, we focus on the extraction of Chinese person names, transliterated person names [1] and organization names [2]. The training data and the testing data in these experiments are selected from three Taiwan newspaper corpora (China Times, Liberty Times News and United Daily News). Chen and Lee [3] reported the precision rate and the recall rate for the extraction of Chinese person names, transliterated person names and organization names are (88.04%, 92.56%), (50.62%, 71.93%) and (61.79%, 54.50%), respectively in the 16th International Conference on Computational Linguistics. We employ these results to several applications. Chen and Wu [4] considered person names as one of clues in sentence alignment. Chen and Lee [3] show its application to anaphora resolution. Chen and Bian [5] proposed a method to construct white pages for Internet/Intranet users automatically. We extract information from World Wide Web documents, including proper nouns, E-mail addresses and home page URLs, and find the relationship among these data. Chen, Ding and Tsai [6,7] dealt with proper noun extraction for information retrieval. In MUC-7 and MET-2, we attend named entity extraction tasks for both English and Chinese. We extend our previous work on this problem to cover more named entity types such as locations, date/time expressions and monetary and percentage expressions. Several issues have to be addressed during extension. One of the major differences between Chinese and English language processing is that segmentation is required for Chinese. That is, we have to identify word boundary in Chinese sentences beforehand. That makes Chinese named entity extraction tasks more changeable. Besides, the vocabulary set and the Chinese coding set used in Taiwan and in China are not the same. The documents adopted in MET-2 are selected from newspapers in China, thus we have to transform simplified Chinese characters in GB coding set to traditional Chinese characters in Big-5 coding set before testing. A word that is known may become unknown due to transformation. For example, the character "u" in "‡u" (early morning) is used in traditional Chinese characters. However, "D" is used in simplified Chinese characters and it is also a legal traditional Chinese character that denotes another meaning. In other words, the mapping from GB to Big5 is "‡D", which is an unknown word based on our dictionary. The different vocabulary set between China and Taiwan results in different segmentation. This paper is organized as follows. Section 2 illustrates the flow of named entity extraction and the summary scores of our team in MET-2 formal run. Sections 3, 4 and 5 propose methods to extract named people, organizations and locations. Section 6 deals with the rest of named entities, i.e., date/time expressions and monetary and percentage expressions. After each section, we discuss the sources of errors in the formal run. Section 7 concludes the remarks. FLOW OF NAMED ENTITY EXTRACTION The following shows the flow of named entity extraction in MET-2 formal run. (1) Transform Chinese texts in GB codes into texts in Big-5 codes. (2) Segment Chinese texts into a sequence of tokens.

Abstract: The invention provides a stroke and phonetic text input entry system that has substantially the same definition of stroke match as that used in T9, where the input is a phrasal input rather than a character input. The invention solves the problem of Chinese phrasal stroke and phonetic text input by allowing users to enter an arbitrary number of strokes for each character in a phrase, where each character is separated by a delimiter. In this way, the invention provides a system that is easily learned and efficiently applied. Thus, the invention makes it possible for users to enter multiple characters while keeping their single character input habits. Each Chinese character has a standard stroke sequence in Guo Biao (GB), which is the standard for mainland China, or multiple sequences for BIG5 Chinese Character Encoding for Traditional (Complex) Characters, which is the de facto standard in Taiwan but not used in mainland China. With the invention, users do not have to enter the complete sequence for a single character, but instead can stop at any point and enter a delimiter, which indicates the end of the previous character and the start of the next character. The whole stroke sequence entered by the user can then be split into a few groups that are separated by zero or more delimiters. Phrases can then be identified by user entry of groups of characters. The presently preferred phrase matching criteria are as follows: the first stroke group matches the leading stroke sequence of the first character of the phrase; the second stroke group matches the leading stroke sequence of the second character of the phrase, etc; the phrases that match the entered stroke sequence are presented to the user for selection. A user interface design for Chinese phrasal stroke text input is also provided.

Abstract: ISO 10646 Universal Character Set (UCS) or Unicode covers symbols in most of the World's written languages. There are various UCS transformation formats (UTF). UTF-8 is compatible with systems that assume 8-bit characters. One of the problems with UTF-8 is its space efficiency. For files containing most Asian characters such as Han ideographs, the file sizes increase by about 50% by using UTF-8. Although the Standard Compression Scheme for Unicode (SCSU) can compress Unicode strings to the size of a locale-specific character set, it is complicated and is not intended to serve as a general purpose interchange format. This paper proposes a page-shift transformation format of ISO 10646, called UTF-S. There are four pages: 1-byte, 2-byte, 3-byte and 4-byte. Shift to page 0 uses a special code ; shift to page 1, 2, and 3 uses ISO 2022 shift codes SO, SS2, and SS3, respectively. We test several text files and compare these UTF with Big5, a locale-specific character set. The result shows that the space efficiency of UTF-S is better than that of UTF-16 and UTF-8 and is close to that of SCSU. UTF-S is suitable for replacing locale-specific character sets with ISO 10646 in Internet applications, such as the World Wide Web. Copyright © 2001 John Wiley & Sons, Ltd.