FACTOID # 18: Alaska spends more money per capita on elementary and secondary education than any other state.
 
 Home   Encyclopedia   Statistics   States A-Z   Flags   Maps   FAQ   About 
   
 
WHAT'S NEW
 

SEARCH ALL

FACTS & STATISTICS    Advanced view

Search encyclopedia, statistics and forums:

 

 

(* = Graphable)

 

 


Encyclopedia > Character encoding

A character encoding consists of a code that pairs a sequence of characters from a given character set (sometimes referred to as code page) with something else, such as a sequence of natural numbers, octets or electrical pulses, in order to facilitate the storage of text in computers and the transmission of text through telecommunication networks. Common examples include Morse code, which encodes letters of the Latin alphabet as series of long and short depressions of a telegraph key; and ASCII, which encodes letters, numerals, and other symbols, both as integers and as 7-bit binary versions of those integers, generally extended with an extra zero-bit to facilitate storage in 8-bit bytes (octets). In information theory, a code is a function mapping an alphabet to non-negative real numbers, satisfying a generalization of Krafts inequality. ... Code page is the traditional IBM term used for a specific character encoding table: a mapping in which a sequence of bits, usually a single octet representing integer values 0 through 255, is associated with a specific character. ... For other uses, see Number (disambiguation). ... In computing, an octet is a grouping of eight bits. ... This article does not cite any references or sources. ... In computer and machine-based telecommunications terminology, a character is a unit of information that roughly corresponds to a grapheme or a grapheme-like unit or symbol, such as in an alphabet or syllabary in the written form of a natural language. ... This article is about the machine. ... 1922 Chart of the Morse Code Letters and Numerals Morse code is a method for transmitting telegraphic information, using standardized sequences of short and long elements to represent the letters, numerals, punctuation and special characters of a message. ... Abcdefghijklmnopqrstuvwxyz redirects here. ... Telegraph key Telegraph key (also known as the Morse key) is a generic term for any switching device used primarily to send Morse code. ... Image:ASCII fullsvg There are 95 printable ASCII characters, numbered 32 to 126. ... The integers are commonly denoted by the above symbol. ... This article is about the unit of information. ... The binary numeral system, or base-2 number system, is a numeral system that represents numeric values using two symbols, usually 0 and 1. ... For the computer industry magazine, see Byte (magazine). ... Look up octet in Wiktionary, the free dictionary. ...


In earlier days of computing, the introduction of coded character sets such as ASCII (1963) and EBCDIC (1964) began the process of standardization. The limitations of such sets soon became apparent, and a number of ad-hoc methods developed to extend them. The need to support multiple writing systems (Languages), including the CJK family of East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ad hoc approaches. Image:ASCII fullsvg There are 95 printable ASCII characters, numbered 32 to 126. ... EBCDIC (Extended Binary Coded Decimal Interchange Code) is an 8-bit character encoding (code page) used on IBM mainframe operating systems, like z/OS, OS/390, VM and VSE, as well as IBM minicomputer operating systems like OS/400 and i5/OS. It is also employed on various non-IBM... Look up Ad hoc in Wiktionary, the free dictionary. ... Writing systems of the world today. ... CJK is a collective term for Chinese, Japanese, and Korean, which comprise the main East Asian languages. ...

Contents

Simple character sets

Conventionally character set and character encoding were considered synonymous, as the same standard would specify both what characters were available and how they were to be encoded into a stream of code units (usually with a single character per code unit). For historical reasons, MIME and systems based on it use the term charset to refer to the complete system for encoding a sequence of characters into a sequence of octets. For mime as an art form, see mime artist. ... Look up octet in Wiktionary, the free dictionary. ...


Modern encoding model

Unicode and its parallel standard, ISO 10646 Universal Character Set, which together constitute the most modern character encoding, broke away from this idea, and instead separated the ideas of what characters are available, their numbering, how those numbers are encoded as a series of "code units" (limited-size numbers), and finally how those units are encoded as a stream of octets (bytes). The idea behind this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. To correctly describe this model needs more precise terms than "character set" and "character encoding". The terms used in the modern model follow: The Unicode Standard, Version 5. ... The international standard ISO/IEC 10646 defines the Universal Character Set (UCS) as a character encoding. ... Look up octet in Wiktionary, the free dictionary. ...


A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, that is no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the Windows code pages). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into linear information units. The basic variants of the Latin, Greek, and Cyrillic alphabets, can be broken down into letters, digits, punctuation, and a few special characters like the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. Even with these alphabets however diacritics pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known in modern terminology as a precomposed character), or as separate characters. The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined together in different ways for different situations. Microsoft uses two main groups of code pages in Windows. ... Abcdefghijklmnopqrstuvwxyz redirects here. ... The Cyrillic alphabet (pronounced also called azbuka, from the old name of the first two letters) is actually a family of alphabets, subsets of which are used by certain Slavic languages — Belarusian, Bulgarian, Macedonian, Russian, Rusyn, Serbian, and Ukrainian—as well as many other languages of the former Soviet Union... Example of a letter with a diacritic A diacritic or diacritical mark, also called an accent, is a small sign added to a letter to alter pronunciation or to distinguish between similar words. ... variant glyphs representing the character a (allographs of a) in the Zapfino typeface. ...


A coded character set specifies how to represent a repertoire of characters using a number of non-negative integer codes called code points. For example, in a given repertoire, a character representing the capital letter "A" in the Latin alphabet might be assigned to the integer 65, the character for "B" to 66, and so on. A complete set of characters and corresponding integers is a coded character set. Multiple coded character sets may share the same repertoire; for example ISO-8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different codes. In a coded character set, each code point only represents one character. ISO 8859-1, more formally cited as ISO/IEC 8859-1 or less formally as Latin-1, is part 1 of ISO/IEC 8859, a standard character encoding defined by ISO. It encodes what it refers to as Latin alphabet no. ...


A character encoding form (CEF) specifies the conversion of a coded character set's integer codes into a set of limited-size integer code values that facilitate storage in a system that represents numbers in binary form using a fixed number of bits (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units would only be able to directly represent integers from 0 to 65,535 in each unit, but larger integers could be represented if more than one 16-bit unit could be used. This is what a CEF accommodates: it defines a way of mapping single code point from a range of, say, 0 to 1.4 million, to a series of one or more code values from a range of, say, 0 to 65,535.


The simplest CEF system is simply to choose large enough units that the values from the coded character set can be encoded directly (one code point to one code value). This works well for coded character sets that fit in 8 bits (as most legacy non-CJK encodings do) and reasonably well for coded character sets that fit in 16 bits (such as early versions of Unicode). However, as the size of the coded character set increases (e.g. modern Unicode requires at least 21 bits/character), this becomes less and less efficient, and it is difficult to adapt existing systems to use larger code values. Therefore, most systems working with later versions of Unicode use either UTF-8, which maps Unicode code points to variable-length sequences of octets, or UTF-16, which maps Unicode code points to variable-length sequences of 16-bit words. UTF-8 (8-bit UCS/Unicode Transformation Format) is a variable-length character encoding for Unicode. ... Look up octet in Wiktionary, the free dictionary. ... In computing, UTF-16 is a 16-bit Unicode Transformation Format, a character encoding form that provides a way to represent a series of abstract characters from Unicode and ISO/IEC 10646 as a series of 16-bit words suitable for storage or transmission via data networks. ...


Next, a character encoding scheme (CES) specifies how the fixed-size integer codes should be mapped into an octet sequence suitable for saving on an octet-based file system or transmitting over an octet-based network. With Unicode, a simple character encoding scheme is used in most cases, simply specifying whether the bytes for each integer should be in big-endian or little-endian order (even this isn't needed with UTF-8). However, there are also compound character encoding schemes, which use escape sequences to switch between several simple schemes (such as ISO 2022), and compressing schemes, which try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and Punycode). Look up octet in Wiktionary, the free dictionary. ... Look up octet in Wiktionary, the free dictionary. ... Look up octet in Wiktionary, the free dictionary. ... Endian may refer to: Endian Firewall, a Linux Routing/Firewall distribution Endianess, the byte (and sometimes bit) ordering in memory used to represent some kind of data in computing Category: ... ISO 2022, more formally ISO/IEC 2022, is an ISO standard (equivalent to the ECMA standard ECMA-35) specifying a technique for including multiple character sets in a single character encoding. ... The Standard Compression Scheme for Unicode (SCSU) is a Unicode Technical Standard to reduce the number of bytes needed to represent text, especially if that text uses mostly characters from a small number of Unicode blocks. ... BOCU-1 is a MIME compatible Unicode compression scheme. ... This article or section may be confusing for some readers, and should be edited to be clearer. ...


Finally, there may be a higher level protocol which supplies additional information that can be used to select the particular variant of a Unicode character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the XML attribute xml:lang. The Unicode Standard, Version 5. ...


History of character encodings

Early binary repertoires:

Alternative meaning: I Ching (monk) The I Ching (Traditional Chinese: 易經, pinyin y jīng; Cantonese IPA: jɪk6gɪŋ1; Cantonese Jyutping: jik6ging1; alternative romanizations include I Jing, Yi Ching, Yi King) is the oldest of the Chinese classic texts. ... The sixteen figures of Western geomancy (divination by signs in the earth) are: Via (Way/Road) The figure resembles a road or path. ... Listen to this article ( info/dl) This audio file was created from a revision dated 2006-09-06, and may not reflect subsequent edits to the article. ...

Popular character encodings

ISO 646 is an ISO standard that specifies a 7-bit character code from which several national standards are derived, the best known of which is ASCII. Since the portion of ISO 646 shared by all countries specified only the letters used in the English alphabet, other countries using the... Image:ASCII fullsvg There are 95 printable ASCII characters, numbered 32 to 126. ... EBCDIC (Extended Binary Coded Decimal Interchange Code) is an 8-bit character encoding (code page) used on IBM mainframe operating systems, like z/OS, OS/390, VM and VSE, as well as IBM minicomputer operating systems like OS/400 and i5/OS. It is also employed on various non-IBM... ISO 8859, more formally ISO/IEC 8859, is a joint ISO and IEC standard for 8-bit character encodings for use by computers. ... ISO 8859-1, more formally cited as ISO/IEC 8859-1 or less formally as Latin-1, is part 1 of ISO/IEC 8859, a standard character encoding of the Latin alphabet. ... ISO 8859-2, more formally cited as ISO/IEC 8859-2 or less formally as Latin-2, is part 2 of ISO/IEC 8859, a standard character encoding defined by ISO. It encodes what it refers to as Latin alphabet no. ... ISO 8859-3, also known as Latin-3 or South European is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-4, also known as Latin-4 or North European, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-5, also known as Cyrillic is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-6, also known as Arabic, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-7, also known as Greek, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-8, more formally cited as ISO/IEC 8859-8 (but not as Latin-8!), is part 8 of ISO/IEC 8859, a standard character encoding defined by ISO. ISO 8859-8 contains all the Hebrew letters (consonants only, no Hebrew vowel signs). ... ISO 8859-9, also known as Latin-5 or Turkish, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-10, also known as Latin-6, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-11 is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-13, also known as Latin-7 or Baltic Rim, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-14, also known as Latin-8 or Celtic, is an 8-bit character encoding, part of the ISO 8859 standard. ... ISO 8859-15 is part 15 of ISO 8859, a standard character encoding defined by International Organization for Standardization. ... ISO 8859-16, also known as Latin-10 or South-Eastern European, is an 8-bit character encoding, part of the ISO 8859 standard. ... IBM PC or MS-DOS code page 437, often abbreviated CP437 and also known as DOS-US or OEM-US, is the original character set of the IBM PC, circa 1981. ... Code page 737 (CP 737, IBM 737, OEM 737) is a code page to be used under MS-DOS to write Greek language. ... The code page 850 is a code page which was used in occidental Europe, under systems such as DOS. It has been largely replaced with ISO 8859-1 and UTF-8, but is still sometimes used. ... Code page 852 (CP 852, IBM 852, OEM 852) is a code page to be used under MS-DOS with Eastern European languages that use Latin script. ... CP855 is a Cyrillic codepage to be used under MS-DOS. This codepage is not much used. ... Code page 857 (CP 857, IBM 857, OEM 857) is a code page to be used under MS-DOS to write Turkish language. ... Code page 858 (CP 858, IBM 858, OEM 858) is a code page to be used under MS-DOS to write Western European languages. ... Code page 860 (CP 860, IBM 860, OEM 860) is a code page to be used under MS-DOS to write Portuguese language. ... Code page 861 (CP 861, IBM 861, OEM 861) is a code page to be used under MS-DOS to write Icelandic language (as well as other Nordic languages). ... Code page 863 (CP 863, IBM 863, OEM 863) is a code page to be used under MS-DOS to write French language (mainly in Canada). ... Code page 865 (CP 865, IBM 865, OEM 865) is a code page to be used under MS-DOS with Nordic languages (except Icelandic, for which CP861 is used). ... CP866 is a Cyrillic code page to be used with MS-DOS. It is based on the alternative character set of GOST 19768-87. ... Code page 869 (CP 869, IBM 869, OEM 869) is a code page to be used under MS-DOS to write Greek language. ... Code page is the traditional IBM term used for a specific character encoding table: a mapping in which a sequence of bits, usually a single octet representing integer values 0 through 255, is associated with a specific character. ... Windows-1250 is a code page used under Microsoft Windows to represent texts in Eastern European languages that use Latin script, such as Polish, Czech, Slovak, Hungarian, Slovene, Croatian, Romanian and Albanian. ... Windows-1251 is an 8-bit character encoding, designed to cover languages that use the Cyrillic alphabet such as Russian and other languages. ... ISO 8859-1, more formally cited as ISO/IEC 8859-1 or less formally as Latin-1, is part 1 of ISO/IEC 8859, a standard character encoding defined by ISO. It encodes what it refers to as Latin alphabet no. ... Windows-1253 is a Windows code page used to write modern Greek (but not polytonic Greek). ... Windows-1254 is a code page used under Microsoft Windows to write Turkish. ... Windows-1255 is a codepage used under Microsoft Windows to write Hebrew. ... Windows-1256 is a codepage used to write Arabic (and possibly some other languages that use Arabic script) under Microsoft Windows. ... Windows-1257 (Windows Baltic) is a codepage used to write Estonian, Latvian and Lithuanian languages under Microsoft Windows. ... Windows-1258 is a codepage used in Microsoft Windows to represent Vietnamese texts. ... Mac OS Roman is a character encoding primarily used by Mac OS to represent text. ... KOI8-R is an 8-bit character encoding, designed to cover Russian, which uses the Cyrillic alphabet. ... KOI8-U is an 8-bit character encoding, designed to cover Ukrainian, which uses the Cyrillic alphabet. ... KOI7 is a 7-bit character encoding, designed to cover Russian, which uses the Cyrillic alphabet. ... MIK is a Cyrillic codepage to be used with MS-DOS. It is based on the character set used in the Bulgarian Pravets 16 PC. Codepage layout Only the upper half (128–255) of the table is shown, the lower half (0–127) being plain ASCII. Categories: | ... The Cork or T1 encoding is a character encoding. ... ISCII (Indian Script Code for Information Interchange) is a coding scheme for representing various Indic scripts as well as a Latin-based script with diacritic marks used to depict Romanised Indic languages. ... VISCII stands for Vietnamese Standard Code for Information Interchange. ... For other uses, see Big five. ... Code page 950 is Microsofts implementation of the defacto standard Big5. ... The Hong Kong Supplementary Character Set (commonly abbreviated to HKSCS) is a set of Chinese characters -- 4,702 in total in the initial release -- used exclusively in Cantonese. ... Guobiao code is a collective term of the national standard encoding of the Peoples Republic of China. ... GB2312 is the registered internet name for a key official character set of the Peoples Republic of China, used for simplified Chinese characters. ... GBK is an extension of the GB2312 character set for simplified Chinese characters, used in the Peoples Republic of China. ... GBK is an extension of the GB2312 character set for simplified Chinese characters, used in the Peoples Republic of China. ... GB18030 is the registered Internet name for the official character set of the Peoples Republic of China (PRC). ... Shift_JIS (SJIS) is a character encoding for the Japanese language developed by a Japanese company called ASCII and adopted by, amongst others, Microsoft. ... Code page 932 (aka CP932, Windows-31J) is Microsofts extension of Shift_JIS to include NEC special characters (Row 13), NEC selection of IBM extensions (Rows 89 to 92), and IBM extensions (Rows 115 to 119). ... Extended Unix Coding Equipment under Control IEC 61508 ... Code page 949 is Microsofts implementation that appears similar to KSC 5601. ... ISO 2022, more formally ISO/IEC 2022, is an ISO standard (equivalent to the ECMA standard ECMA-35) specifying a technique for including multiple character sets in a single character encoding. ... Extended Unix Coding Equipment under Control IEC 61508 ... CJK is a collective term for Chinese, Japanese, and Korean, which comprise the main East Asian languages. ... The Unicode Standard, Version 5. ... UTF-8 (8-bit UCS/Unicode Transformation Format) is a variable-length character encoding for Unicode. ... ANSEL, American National Standard for Extended Latin Alphabet Coded Character Set for Bibliographic Use, is a character set used in text encoding, and may also be known as ANSI/NISO Z39. ... ISO/IEC 6937 is a multibyte extension of ASCII, or rather of ISO/IEC 646-IRV. Certain byte codes are used as lead bytes for letters with diacritics (accents). ...

Character conversion tools

Cross-platform: A cross-platform (or platform independent) programming language, software application or hardware device works on more than one system platform (e. ...

  • iconv – program and standardized API to convert encodings
  • convert_encoding.py – Python based utility to convert text files between arbitrary encodings and line endings.[1]
  • decodeh.py - algorithm and module to heuristically guess the encoding of a string [2]

Linux: iconv is a computer program and a standardized API used to convert between different character encodings. ... This article is about operating systems that use the Linux kernel. ...

  • recode – convert file contents from one encoding to another [3]
  • utrac – convert file contents from one encoding to another.[4]
  • cstocs – convert file contents from one encoding to another
  • convmv – convert a filename from one encoding to another.[5]
  • enca – analyzes encodings for given text files/[6]

See also

  • Category:Character encoding — articles related to character encoding in general
  • Category:Character sets — articles detailing specific character encodings
  • Code page — various character set encodings used by Microsoft
  • Windows code page — various character set encodings used by Microsoft Windows
  • Mojibake — character set mismap.

Code page is the traditional IBM term used for a specific character encoding table: a mapping in which a sequence of bits, usually a single octet representing integer values 0 through 255, is associated with a specific character. ... Microsoft uses two main groups of code pages in Microsoft Windows (known as character encodings in other operating systems). ... The UTF-8-encoded Japanese Wikipedia article for mojibake, as displayed in ISO-8859-1 encoding. ...

References

  1. ^ Homepage of Michael Goerz - convert_encoding.py
  2. ^ decodeh - heuristically decode a string or text file
  3. ^ recode - GNU Project - Free Software Foundation (FSF)
  4. ^ Utrac Homepage
  5. ^ convmv - converts filenames from one encoding to another
  6. ^ Extremely Naive Charset Analyser

External links

SIL International is a worldwide non-profit evangelical Christian organization whose main purpose is to study, develop and document lesser-known languages in order to expand linguistic knowledge, promote literacy and aid minority language development. ... The Cyrillic alphabet (pronounced also called azbuka, from the old name of the first two letters) is actually a family of alphabets, subsets of which are used by certain Slavic languages — Belarusian, Bulgarian, Macedonian, Russian, Rusyn, Serbian, and Ukrainian—as well as many other languages of the former Soviet Union...

  Results from FactBites:
 
Character encoding - Wikipedia, the free encyclopedia (724 words)
Conventionally character set and character encoding were considered synonymous, as the same standard would specify both what characters were available and how they were to be encoded into a stream of code units (usually with a single character per code unit).
With Unicode in most cases a simple character encoding scheme is used, simply specifying if the bytes for each integer should be in big-endian or little-endian order (even this isn't needed with UTF-8).
However, there are also compound character encoding schemes, which use escape sequences to switch between several simple schemes (such as ISO 2022), and compressing schemes, which try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and Punycode).
Chinese character encoding - Wikipedia, the free encyclopedia (501 words)
In computing, Chinese character encodings can be used to represent text written in the CJK languages — Chinese, Japanese, Korean — and (rarely) Vietnamese, all of which use Chinese characters.
The opposite conversion often results in a data loss when converting to early forms of the GB character set (namely GB2312 80): in mapping one-to-many when assigning traditional glyphs to the simplified glyphs, some characters will inevitably be the wrong choices in some of the usages.
The issue of which encoding to use can also have political implications, as GB is the official standard of the People's Republic of China and Big5 is a de facto standard of Taiwan.
  More results at FactBites »

 
 

COMMENTARY     


Share your thoughts, questions and commentary here
Your name
Your comments

Want to know more?
Search encyclopedia, statistics and forums:

 


Press Releases |  Feeds | Contact
The Wikipedia article included on this page is licensed under the GFDL.
Images may be subject to relevant owners' copyright.
All other elements are (c) copyright NationMaster.com 2003-5. All Rights Reserved.
Usage implies agreement with terms, 1022, m