--- Python-Docs-2.4/./howto/unicode.rst	1970-01-01 09:00:00.000000000 +0900
+++ Python-Docs-2.5/./howto/unicode.rst	2006-05-12 01:32:24.000000000 +0900
@@ -0,0 +1,766 @@
+Unicode HOWTO
+================
+
+**Version 1.02**
+
+This HOWTO discusses Python's support for Unicode, and explains various 
+problems that people commonly encounter when trying to work with Unicode.
+
+Introduction to Unicode
+------------------------------
+
+History of Character Codes
+''''''''''''''''''''''''''''''
+
+In 1968, the American Standard Code for Information Interchange,
+better known by its acronym ASCII, was standardized.  ASCII defined
+numeric codes for various characters, with the numeric values running from 0 to
+127.  For example, the lowercase letter 'a' is assigned 97 as its code
+value.
+
+ASCII was an American-developed standard, so it only defined
+unaccented characters.  There was an 'e', but no 'é' or 'Í'.  This
+meant that languages which required accented characters couldn't be
+faithfully represented in ASCII.  (Actually the missing accents matter
+for English, too, which contains words such as 'naïve' and 'café', and some
+publications have house styles which require spellings such as
+'coöperate'.)
+
+For a while people just wrote programs that didn't display accents.  I
+remember looking at Apple ][ BASIC programs, published in French-language
+publications in the mid-1980s, that had lines like these::
+
+	PRINT "FICHER EST COMPLETE."
+	PRINT "CARACTERE NON ACCEPTE."
+
+Those messages should contain accents, and they just look wrong to
+someone who can read French.  
+
+In the 1980s, almost all personal computers were 8-bit, meaning that
+bytes could hold values ranging from 0 to 255.  ASCII codes only went
+up to 127, so some machines assigned values between 128 and 255 to
+accented characters.  Different machines had different codes, however,
+which led to problems exchanging files.  Eventually various commonly
+used sets of values for the 128-255 range emerged.  Some were true
+standards, defined by the International Standards Organization, and
+some were **de facto** conventions that were invented by one company
+or another and managed to catch on.
+
+255 characters aren't very many.  For example, you can't fit
+both the accented characters used in Western Europe and the Cyrillic
+alphabet used for Russian into the 128-255 range because there are more than
+127 such characters.
+
+You could write files using different codes (all your Russian
+files in a coding system called KOI8, all your French files in 
+a different coding system called Latin1), but what if you wanted
+to write a French document that quotes some Russian text?  In the
+1980s people began to want to solve this problem, and the Unicode
+standardization effort began.
+
+Unicode started out using 16-bit characters instead of 8-bit characters.  16
+bits means you have 2^16 = 65,536 distinct values available, making it
+possible to represent many different characters from many different
+alphabets; an initial goal was to have Unicode contain the alphabets for
+every single human language.  It turns out that even 16 bits isn't enough to
+meet that goal, and the modern Unicode specification uses a wider range of
+codes, 0-1,114,111 (0x10ffff in base-16).
+
+There's a related ISO standard, ISO 10646.  Unicode and ISO 10646 were
+originally separate efforts, but the specifications were merged with
+the 1.1 revision of Unicode.  
+
+(This discussion of Unicode's history is highly simplified.  I don't
+think the average Python programmer needs to worry about the
+historical details; consult the Unicode consortium site listed in the
+References for more information.)
+
+
+Definitions
+''''''''''''''''''''''''
+
+A **character** is the smallest possible component of a text.  'A',
+'B', 'C', etc., are all different characters.  So are 'È' and
+'Í'.  Characters are abstractions, and vary depending on the
+language or context you're talking about.  For example, the symbol for
+ohms (Ω) is usually drawn much like the capital letter
+omega (Ω) in the Greek alphabet (they may even be the same in
+some fonts), but these are two different characters that have
+different meanings.
+
+The Unicode standard describes how characters are represented by
+**code points**.  A code point is an integer value, usually denoted in
+base 16.  In the standard, a code point is written using the notation
+U+12ca to mean the character with value 0x12ca (4810 decimal).  The
+Unicode standard contains a lot of tables listing characters and their
+corresponding code points::
+
+	0061    'a'; LATIN SMALL LETTER A
+	0062    'b'; LATIN SMALL LETTER B
+	0063    'c'; LATIN SMALL LETTER C
+        ...
+	007B	'{'; LEFT CURLY BRACKET
+
+Strictly, these definitions imply that it's meaningless to say 'this is
+character U+12ca'.  U+12ca is a code point, which represents some particular
+character; in this case, it represents the character 'ETHIOPIC SYLLABLE WI'.
+In informal contexts, this distinction between code points and characters will
+sometimes be forgotten.
+
+A character is represented on a screen or on paper by a set of graphical
+elements that's called a **glyph**.  The glyph for an uppercase A, for
+example, is two diagonal strokes and a horizontal stroke, though the exact
+details will depend on the font being used.  Most Python code doesn't need
+to worry about glyphs; figuring out the correct glyph to display is
+generally the job of a GUI toolkit or a terminal's font renderer.
+
+
+Encodings
+'''''''''
+
+To summarize the previous section: 
+a Unicode string is a sequence of code points, which are
+numbers from 0 to 0x10ffff.  This sequence needs to be represented as
+a set of bytes (meaning, values from 0-255) in memory.  The rules for
+translating a Unicode string into a sequence of bytes are called an 
+**encoding**.
+
+The first encoding you might think of is an array of 32-bit integers.  
+In this representation, the string "Python" would look like this::
+
+       P           y           t           h           o           n
+    0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00 
+       0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 
+
+This representation is straightforward but using
+it presents a number of problems.
+
+1. It's not portable; different processors order the bytes 
+   differently. 
+
+2. It's very wasteful of space.  In most texts, the majority of the code 
+   points are less than 127, or less than 255, so a lot of space is occupied
+   by zero bytes.  The above string takes 24 bytes compared to the 6
+   bytes needed for an ASCII representation.  Increased RAM usage doesn't
+   matter too much (desktop computers have megabytes of RAM, and strings
+   aren't usually that large), but expanding our usage of disk and
+   network bandwidth by a factor of 4 is intolerable.
+
+3. It's not compatible with existing C functions such as ``strlen()``,
+   so a new family of wide string functions would need to be used.
+
+4. Many Internet standards are defined in terms of textual data, and 
+   can't handle content with embedded zero bytes.
+
+Generally people don't use this encoding, choosing other encodings
+that are more efficient and convenient.
+
+Encodings don't have to handle every possible Unicode character, and
+most encodings don't.  For example, Python's default encoding is the
+'ascii' encoding.  The rules for converting a Unicode string into the
+ASCII encoding are simple; for each code point:
+
+1. If the code point is <128, each byte is the same as the value of the 
+   code point.
+
+2. If the code point is 128 or greater, the Unicode string can't 
+   be represented in this encoding.  (Python raises  a 
+   ``UnicodeEncodeError`` exception in this case.)
+
+Latin-1, also known as ISO-8859-1, is a similar encoding.  Unicode
+code points 0-255 are identical to the Latin-1 values, so converting
+to this encoding simply requires converting code points to byte
+values; if a code point larger than 255 is encountered, the string
+can't be encoded into Latin-1.
+
+Encodings don't have to be simple one-to-one mappings like Latin-1.
+Consider IBM's EBCDIC, which was used on IBM mainframes.  Letter
+values weren't in one block: 'a' through 'i' had values from 129 to
+137, but 'j' through 'r' were 145 through 153.  If you wanted to use
+EBCDIC as an encoding, you'd probably use some sort of lookup table to
+perform the conversion, but this is largely an internal detail.
+
+UTF-8 is one of the most commonly used encodings.  UTF stands for
+"Unicode Transformation Format", and the '8' means that 8-bit numbers
+are used in the encoding.  (There's also a UTF-16 encoding, but it's
+less frequently used than UTF-8.)  UTF-8 uses the following rules:
+
+1. If the code point is <128, it's represented by the corresponding byte value.
+2. If the code point is between 128 and 0x7ff, it's turned into two byte values
+   between 128 and 255.
+3. Code points >0x7ff are turned into three- or four-byte sequences, where
+   each byte of the sequence is between 128 and 255.
+    
+UTF-8 has several convenient properties:
+
+1. It can handle any Unicode code point.
+2. A Unicode string is turned into a string of bytes containing no embedded zero bytes.  This avoids byte-ordering issues, and means UTF-8 strings can be processed by C functions such as ``strcpy()`` and sent through protocols that can't handle zero bytes.
+3. A string of ASCII text is also valid UTF-8 text. 
+4. UTF-8 is fairly compact; the majority of code points are turned into two bytes, and values less than 128 occupy only a single byte.
+5. If bytes are corrupted or lost, it's possible to determine the start of the next UTF-8-encoded code point and resynchronize.  It's also unlikely that random 8-bit data will look like valid UTF-8.
+
+
+
+References
+''''''''''''''
+
+The Unicode Consortium site at <http://www.unicode.org> has character
+charts, a glossary, and PDF versions of the Unicode specification.  Be
+prepared for some difficult reading.
+<http://www.unicode.org/history/> is a chronology of the origin and
+development of Unicode.
+
+To help understand the standard, Jukka Korpela has written an
+introductory guide to reading the Unicode character tables, 
+available at <http://www.cs.tut.fi/~jkorpela/unicode/guide.html>.
+
+Roman Czyborra wrote another explanation of Unicode's basic principles; 
+it's at <http://czyborra.com/unicode/characters.html>.
+Czyborra has written a number of other Unicode-related documentation, 
+available from <http://www.cyzborra.com>.
+
+Two other good introductory articles were written by Joel Spolsky
+<http://www.joelonsoftware.com/articles/Unicode.html> and Jason
+Orendorff <http://www.jorendorff.com/articles/unicode/>.  If this
+introduction didn't make things clear to you, you should try reading
+one of these alternate articles before continuing.
+
+Wikipedia entries are often helpful; see the entries for "character
+encoding" <http://en.wikipedia.org/wiki/Character_encoding> and UTF-8
+<http://en.wikipedia.org/wiki/UTF-8>, for example.
+
+
+Python's Unicode Support
+------------------------
+
+Now that you've learned the rudiments of Unicode, we can look at
+Python's Unicode features.
+
+
+The Unicode Type
+'''''''''''''''''''
+
+Unicode strings are expressed as instances of the ``unicode`` type,
+one of Python's repertoire of built-in types.  It derives from an
+abstract type called ``basestring``, which is also an ancestor of the
+``str`` type; you can therefore check if a value is a string type with
+``isinstance(value, basestring)``.  Under the hood, Python represents
+Unicode strings as either 16- or 32-bit integers, depending on how the
+Python interpreter was compiled.
+
+The ``unicode()`` constructor has the signature ``unicode(string[, encoding, errors])``.
+All of its arguments should be 8-bit strings.  The first argument is converted 
+to Unicode using the specified encoding; if you leave off the ``encoding`` argument, 
+the ASCII encoding is used for the conversion, so characters greater than 127 will 
+be treated as errors::
+
+    >>> unicode('abcdef')
+    u'abcdef'
+    >>> s = unicode('abcdef')
+    >>> type(s)
+    <type 'unicode'>
+    >>> unicode('abcdef' + chr(255))
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in ?
+    UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6: 
+                        ordinal not in range(128)
+
+The ``errors`` argument specifies the response when the input string can't be converted according to the encoding's rules.  Legal values for this argument 
+are 'strict' (raise a ``UnicodeDecodeError`` exception), 
+'replace' (add U+FFFD, 'REPLACEMENT CHARACTER'), 
+or 'ignore' (just leave the character out of the Unicode result).  
+The following examples show the differences::
+
+    >>> unicode('\x80abc', errors='strict')
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in ?
+    UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: 
+                        ordinal not in range(128)
+    >>> unicode('\x80abc', errors='replace')
+    u'\ufffdabc'
+    >>> unicode('\x80abc', errors='ignore')
+    u'abc'
+
+Encodings are specified as strings containing the encoding's name.
+Python 2.4 comes with roughly 100 different encodings; see the Python
+Library Reference at
+<http://docs.python.org/lib/standard-encodings.html> for a list.  Some
+encodings have multiple names; for example, 'latin-1', 'iso_8859_1'
+and '8859' are all synonyms for the same encoding.
+
+One-character Unicode strings can also be created with the
+``unichr()`` built-in function, which takes integers and returns a
+Unicode string of length 1 that contains the corresponding code point.
+The reverse operation is the built-in `ord()` function that takes a
+one-character Unicode string and returns the code point value::
+
+    >>> unichr(40960)
+    u'\ua000'
+    >>> ord(u'\ua000')
+    40960
+
+Instances of the ``unicode`` type have many of the same methods as 
+the 8-bit string type for operations such as searching and formatting::
+
+    >>> s = u'Was ever feather so lightly blown to and fro as this multitude?'
+    >>> s.count('e')
+    5
+    >>> s.find('feather')
+    9
+    >>> s.find('bird')
+    -1
+    >>> s.replace('feather', 'sand')
+    u'Was ever sand so lightly blown to and fro as this multitude?'
+    >>> s.upper()
+    u'WAS EVER FEATHER SO LIGHTLY BLOWN TO AND FRO AS THIS MULTITUDE?'
+
+Note that the arguments to these methods can be Unicode strings or 8-bit strings.  
+8-bit strings will be converted to Unicode before carrying out the operation;
+Python's default ASCII encoding will be used, so characters greater than 127 will cause an exception::
+
+    >>> s.find('Was\x9f')
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in ?
+    UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 3: ordinal not in range(128)
+    >>> s.find(u'Was\x9f')
+    -1
+
+Much Python code that operates on strings will therefore work with
+Unicode strings without requiring any changes to the code.  (Input and
+output code needs more updating for Unicode; more on this later.)
+
+Another important method is ``.encode([encoding], [errors='strict'])``, 
+which returns an 8-bit string version of the
+Unicode string, encoded in the requested encoding.  The ``errors``
+parameter is the same as the parameter of the ``unicode()``
+constructor, with one additional possibility; as well as 'strict',
+'ignore', and 'replace', you can also pass 'xmlcharrefreplace' which
+uses XML's character references.  The following example shows the
+different results::
+
+    >>> u = unichr(40960) + u'abcd' + unichr(1972)
+    >>> u.encode('utf-8')
+    '\xea\x80\x80abcd\xde\xb4'
+    >>> u.encode('ascii')
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in ?
+    UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in position 0: ordinal not in range(128)
+    >>> u.encode('ascii', 'ignore')
+    'abcd'
+    >>> u.encode('ascii', 'replace')
+    '?abcd?'
+    >>> u.encode('ascii', 'xmlcharrefreplace')
+    '&#40960;abcd&#1972;'
+
+Python's 8-bit strings have a ``.decode([encoding], [errors])`` method 
+that interprets the string using the given encoding::
+
+    >>> u = unichr(40960) + u'abcd' + unichr(1972)   # Assemble a string
+    >>> utf8_version = u.encode('utf-8')             # Encode as UTF-8
+    >>> type(utf8_version), utf8_version
+    (<type 'str'>, '\xea\x80\x80abcd\xde\xb4')
+    >>> u2 = utf8_version.decode('utf-8')            # Decode using UTF-8
+    >>> u == u2                                      # The two strings match
+    True
+ 
+The low-level routines for registering and accessing the available
+encodings are found in the ``codecs`` module.  However, the encoding
+and decoding functions returned by this module are usually more
+low-level than is comfortable, so I'm not going to describe the
+``codecs`` module here.  If you need to implement a completely new
+encoding, you'll need to learn about the ``codecs`` module interfaces,
+but implementing encodings is a specialized task that also won't be
+covered here.  Consult the Python documentation to learn more about
+this module.
+
+The most commonly used part of the ``codecs`` module is the 
+``codecs.open()`` function which will be discussed in the section
+on input and output.
+            
+            
+Unicode Literals in Python Source Code
+''''''''''''''''''''''''''''''''''''''''''
+
+In Python source code, Unicode literals are written as strings
+prefixed with the 'u' or 'U' character: ``u'abcdefghijk'``.  Specific
+code points can be written using the ``\u`` escape sequence, which is
+followed by four hex digits giving the code point.  The ``\U`` escape
+sequence is similar, but expects 8 hex digits, not 4.  
+
+Unicode literals can also use the same escape sequences as 8-bit
+strings, including ``\x``, but ``\x`` only takes two hex digits so it
+can't express an arbitrary code point.  Octal escapes can go up to
+U+01ff, which is octal 777.
+
+::
+
+    >>> s = u"a\xac\u1234\u20ac\U00008000"
+               ^^^^ two-digit hex escape
+                   ^^^^^^ four-digit Unicode escape 
+                               ^^^^^^^^^^ eight-digit Unicode escape
+    >>> for c in s:  print ord(c),
+    ... 
+    97 172 4660 8364 32768
+
+Using escape sequences for code points greater than 127 is fine in
+small doses, but becomes an annoyance if you're using many accented
+characters, as you would in a program with messages in French or some
+other accent-using language.  You can also assemble strings using the
+``unichr()`` built-in function, but this is even more tedious.
+
+Ideally, you'd want to be able to write literals in your language's
+natural encoding.  You could then edit Python source code with your
+favorite editor which would display the accented characters naturally,
+and have the right characters used at runtime.
+
+Python supports writing Unicode literals in any encoding, but you have
+to declare the encoding being used.  This is done by including a
+special comment as either the first or second line of the source
+file::
+
+    #!/usr/bin/env python
+    # -*- coding: latin-1 -*-
+    
+    u = u'abcdé'
+    print ord(u[-1])
+    
+The syntax is inspired by Emacs's notation for specifying variables local to a file.
+Emacs supports many different variables, but Python only supports 'coding'.  
+The ``-*-`` symbols indicate that the comment is special; within them,
+you must supply the name ``coding`` and the name of your chosen encoding, 
+separated by ``':'``.  
+
+If you don't include such a comment, the default encoding used will be
+ASCII.  Versions of Python before 2.4 were Euro-centric and assumed
+Latin-1 as a default encoding for string literals; in Python 2.4,
+characters greater than 127 still work but result in a warning.  For
+example, the following program has no encoding declaration::
+
+    #!/usr/bin/env python
+    u = u'abcdé'
+    print ord(u[-1])
+
+When you run it with Python 2.4, it will output the following warning::
+
+    amk:~$ python p263.py
+    sys:1: DeprecationWarning: Non-ASCII character '\xe9' 
+         in file p263.py on line 2, but no encoding declared; 
+         see http://www.python.org/peps/pep-0263.html for details
+  
+
+Unicode Properties
+'''''''''''''''''''
+
+The Unicode specification includes a database of information about
+code points.  For each code point that's defined, the information
+includes the character's name, its category, the numeric value if
+applicable (Unicode has characters representing the Roman numerals and
+fractions such as one-third and four-fifths).  There are also
+properties related to the code point's use in bidirectional text and
+other display-related properties.
+
+The following program displays some information about several
+characters, and prints the numeric value of one particular character::
+
+    import unicodedata
+    
+    u = unichr(233) + unichr(0x0bf2) + unichr(3972) + unichr(6000) + unichr(13231)
+    
+    for i, c in enumerate(u):
+        print i, '%04x' % ord(c), unicodedata.category(c),
+        print unicodedata.name(c)
+    
+    # Get numeric value of second character
+    print unicodedata.numeric(u[1])
+
+When run, this prints::
+
+    0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
+    1 0bf2 No TAMIL NUMBER ONE THOUSAND
+    2 0f84 Mn TIBETAN MARK HALANTA
+    3 1770 Lo TAGBANWA LETTER SA
+    4 33af So SQUARE RAD OVER S SQUARED
+    1000.0
+
+The category codes are abbreviations describing the nature of the
+character.  These are grouped into categories such as "Letter",
+"Number", "Punctuation", or "Symbol", which in turn are broken up into
+subcategories.  To take the codes from the above output, ``'Ll'``
+means 'Letter, lowercase', ``'No'`` means "Number, other", ``'Mn'`` is
+"Mark, nonspacing", and ``'So'`` is "Symbol, other".  See
+<http://www.unicode.org/Public/UNIDATA/UCD.html#General_Category_Values>
+for a list of category codes.
+
+References
+''''''''''''''
+
+The Unicode and 8-bit string types are described in the Python library
+reference at <http://docs.python.org/lib/typesseq.html>.
+
+The documentation for the ``unicodedata`` module is at 
+<http://docs.python.org/lib/module-unicodedata.html>.
+
+The documentation for the ``codecs`` module is at
+<http://docs.python.org/lib/module-codecs.html>.
+
+Marc-André Lemburg gave a presentation at EuroPython 2002
+titled "Python and Unicode".  A PDF version of his slides
+is available at <http://www.egenix.com/files/python/Unicode-EPC2002-Talk.pdf>,
+and is an excellent overview of the design of Python's Unicode features.
+
+
+Reading and Writing Unicode Data
+----------------------------------------
+
+Once you've written some code that works with Unicode data, the next
+problem is input/output.  How do you get Unicode strings into your
+program, and how do you convert Unicode into a form suitable for
+storage or transmission?  
+
+It's possible that you may not need to do anything depending on your
+input sources and output destinations; you should check whether the
+libraries used in your application support Unicode natively.  XML
+parsers often return Unicode data, for example.  Many relational
+databases also support Unicode-valued columns and can return Unicode
+values from an SQL query.
+
+Unicode data is usually converted to a particular encoding before it
+gets written to disk or sent over a socket.  It's possible to do all
+the work yourself: open a file, read an 8-bit string from it, and
+convert the string with ``unicode(str, encoding)``.  However, the
+manual approach is not recommended.
+
+One problem is the multi-byte nature of encodings; one Unicode
+character can be represented by several bytes.  If you want to read
+the file in arbitrary-sized chunks (say, 1K or 4K), you need to write
+error-handling code to catch the case where only part of the bytes
+encoding a single Unicode character are read at the end of a chunk.
+One solution would be to read the entire file into memory and then
+perform the decoding, but that prevents you from working with files
+that are extremely large; if you need to read a 2Gb file, you need 2Gb
+of RAM.  (More, really, since for at least a moment you'd need to have 
+both the encoded string and its Unicode version in memory.)
+
+The solution would be to use the low-level decoding interface to catch
+the case of partial coding sequences.   The work of implementing this
+has already been done for you: the ``codecs`` module includes a
+version of the ``open()`` function that returns a file-like object
+that assumes the file's contents are in a specified encoding and
+accepts Unicode parameters for methods such as ``.read()`` and
+``.write()``.
+
+The function's parameters are 
+``open(filename, mode='rb', encoding=None, errors='strict', buffering=1)``.  ``mode`` can be
+``'r'``, ``'w'``, or ``'a'``, just like the corresponding parameter to the
+regular built-in ``open()`` function; add a ``'+'`` to 
+update the file.  ``buffering`` is similarly
+parallel to the standard function's parameter.  
+``encoding`` is a string giving 
+the encoding to use; if it's left as ``None``, a regular Python file
+object that accepts 8-bit strings is returned.  Otherwise, a wrapper
+object is returned, and data written to or read from the wrapper
+object will be converted as needed.  ``errors`` specifies the action
+for encoding errors and can be one of the usual values of 'strict',
+'ignore', and 'replace'.
+
+Reading Unicode from a file is therefore simple::
+
+    import codecs
+    f = codecs.open('unicode.rst', encoding='utf-8')
+    for line in f:
+        print repr(line)
+
+It's also possible to open files in update mode, 
+allowing both reading and writing::
+
+    f = codecs.open('test', encoding='utf-8', mode='w+')
+    f.write(u'\u4500 blah blah blah\n')
+    f.seek(0)
+    print repr(f.readline()[:1])
+    f.close()
+
+Unicode character U+FEFF is used as a byte-order mark (BOM), 
+and is often written as the first character of a file in order
+to assist with autodetection of the file's byte ordering.
+Some encodings, such as UTF-16, expect a BOM to be present at 
+the start of a file; when such an encoding is used,
+the BOM will be automatically written as the first character 
+and will be silently dropped when the file is read.  There are 
+variants of these encodings, such as 'utf-16-le' and 'utf-16-be'
+for little-endian and big-endian encodings, that specify 
+one particular byte ordering and don't
+skip the BOM.
+
+
+Unicode filenames
+'''''''''''''''''''''''''
+
+Most of the operating systems in common use today support filenames
+that contain arbitrary Unicode characters.  Usually this is
+implemented by converting the Unicode string into some encoding that
+varies depending on the system.  For example, MacOS X uses UTF-8 while
+Windows uses a configurable encoding; on Windows, Python uses the name
+"mbcs" to refer to whatever the currently configured encoding is.  On
+Unix systems, there will only be a filesystem encoding if you've set
+the ``LANG`` or ``LC_CTYPE`` environment variables; if you haven't,
+the default encoding is ASCII.
+
+The ``sys.getfilesystemencoding()`` function returns the encoding to
+use on your current system, in case you want to do the encoding
+manually, but there's not much reason to bother.  When opening a file
+for reading or writing, you can usually just provide the Unicode
+string as the filename, and it will be automatically converted to the
+right encoding for you::
+
+    filename = u'filename\u4500abc'
+    f = open(filename, 'w')
+    f.write('blah\n')
+    f.close()
+
+Functions in the ``os`` module such as ``os.stat()`` will also accept
+Unicode filenames.
+
+``os.listdir()``, which returns filenames, raises an issue: should it
+return the Unicode version of filenames, or should it return 8-bit
+strings containing the encoded versions?  ``os.listdir()`` will do
+both, depending on whether you provided the directory path as an 8-bit
+string or a Unicode string.  If you pass a Unicode string as the path,
+filenames will be decoded using the filesystem's encoding and a list
+of Unicode strings will be returned, while passing an 8-bit path will
+return the 8-bit versions of the filenames.  For example, assuming the
+default filesystem encoding is UTF-8, running the following program::
+
+	fn = u'filename\u4500abc'
+	f = open(fn, 'w')
+	f.close()
+
+	import os
+	print os.listdir('.')
+	print os.listdir(u'.')
+
+will produce the following output::
+
+	amk:~$ python t.py
+	['.svn', 'filename\xe4\x94\x80abc', ...]
+	[u'.svn', u'filename\u4500abc', ...]
+
+The first list contains UTF-8-encoded filenames, and the second list
+contains the Unicode versions.
+
+
+	
+Tips for Writing Unicode-aware Programs
+''''''''''''''''''''''''''''''''''''''''''''
+
+This section provides some suggestions on writing software that 
+deals with Unicode.
+
+The most important tip is: 
+
+    Software should only work with Unicode strings internally, 
+    converting to a particular encoding on output.  
+
+If you attempt to write processing functions that accept both 
+Unicode and 8-bit strings, you will find your program vulnerable to 
+bugs wherever you combine the two different kinds of strings.  Python's 
+default encoding is ASCII, so whenever a character with an ASCII value >127
+is in the input data, you'll get a ``UnicodeDecodeError``
+because that character can't be handled by the ASCII encoding.  
+
+It's easy to miss such problems if you only test your software 
+with data that doesn't contain any 
+accents; everything will seem to work, but there's actually a bug in your
+program waiting for the first user who attempts to use characters >127.
+A second tip, therefore, is:
+
+    Include characters >127 and, even better, characters >255 in your
+    test data.
+
+When using data coming from a web browser or some other untrusted source,
+a common technique is to check for illegal characters in a string
+before using the string in a generated command line or storing it in a 
+database.  If you're doing this, be careful to check 
+the string once it's in the form that will be used or stored; it's 
+possible for encodings to be used to disguise characters.  This is especially
+true if the input data also specifies the encoding; 
+many encodings leave the commonly checked-for characters alone, 
+but Python includes some encodings such as ``'base64'``
+that modify every single character.
+
+For example, let's say you have a content management system that takes a 
+Unicode filename, and you want to disallow paths with a '/' character.
+You might write this code::
+
+    def read_file (filename, encoding):
+        if '/' in filename:
+            raise ValueError("'/' not allowed in filenames")
+        unicode_name = filename.decode(encoding)
+        f = open(unicode_name, 'r')
+        # ... return contents of file ...
+        
+However, if an attacker could specify the ``'base64'`` encoding,
+they could pass ``'L2V0Yy9wYXNzd2Q='``, which is the base-64
+encoded form of the string ``'/etc/passwd'``, to read a 
+system file.   The above code looks for ``'/'`` characters 
+in the encoded form and misses the dangerous character 
+in the resulting decoded form.
+
+References
+''''''''''''''
+
+The PDF slides for Marc-André Lemburg's presentation "Writing
+Unicode-aware Applications in Python" are available at
+<http://www.egenix.com/files/python/LSM2005-Developing-Unicode-aware-applications-in-Python.pdf>
+and discuss questions of character encodings as well as how to
+internationalize and localize an application.
+
+
+Revision History and Acknowledgements
+------------------------------------------
+
+Thanks to the following people who have noted errors or offered
+suggestions on this article: Nicholas Bastin, 
+Marius Gedminas, Kent Johnson, Ken Krugler,
+Marc-André Lemburg, Martin von Löwis, Chad Whitacre.
+
+Version 1.0: posted August 5 2005.
+
+Version 1.01: posted August 7 2005.  Corrects factual and markup
+errors; adds several links.
+
+Version 1.02: posted August 16 2005.  Corrects factual errors.
+
+
+.. comment Additional topic: building Python w/ UCS2 or UCS4 support
+.. comment Describe obscure -U switch somewhere?
+.. comment Describe use of codecs.StreamRecoder and StreamReaderWriter
+
+.. comment 
+   Original outline:
+
+   - [ ] Unicode introduction
+       - [ ] ASCII
+       - [ ] Terms
+	   - [ ] Character
+	   - [ ] Code point
+	 - [ ] Encodings
+	    - [ ] Common encodings: ASCII, Latin-1, UTF-8
+       - [ ] Unicode Python type
+	   - [ ] Writing unicode literals
+	       - [ ] Obscurity: -U switch
+	   - [ ] Built-ins
+	       - [ ] unichr()
+	       - [ ] ord()
+	       - [ ] unicode() constructor
+	   - [ ] Unicode type
+	       - [ ] encode(), decode() methods
+       - [ ] Unicodedata module for character properties
+       - [ ] I/O
+	   - [ ] Reading/writing Unicode data into files
+	       - [ ] Byte-order marks
+	   - [ ] Unicode filenames
+       - [ ] Writing Unicode programs
+	   - [ ] Do everything in Unicode
+	   - [ ] Declaring source code encodings (PEP 263)
+       - [ ] Other issues
+	   - [ ] Building Python (UCS2, UCS4)
