The Unicode Standard

To a computer, everything is a number. A photo is a grid of numbers that give the colour of each dot, and a song is a long sequence of numbers that describe the pressure of the air at each instant. Text is no different: every letter, every accent, every Chinese ideograph and every emoji that appears on your screen is kept in memory as a number. Type anything in the field below and follow the path of the text down to the bytes:

text

In the first stage are the characters, the way you see them. In the second, each one gets its number, called a code point and written in hexadecimal after “U+”. In the third, those numbers are turned into bytes, to be saved in a file or sent over the network. Notice that the characters do not all take the same space: the letters without accents use one byte, the accented ones use two, the Chinese ideographs three, and the emoji four.

The number of each character is defined by Unicode, a standard kept by a consortium of companies, governments, universities and experts. Version 18.0, published in September 2026, has 172,808 characters from 175 writing systems, from the letters of the Latin alphabet to Egyptian hieroglyphs, by way of mathematical symbols, musical notes and thousands of emoji. Practically all the text that travels today between computers and phones is Unicode.

But it was not always so. In this guide, we will see how the first tables of characters worked and why they multiplied until they became a problem, how Unicode organizes its enormous space of numbers and how those numbers become bytes. Then we will see that “character” is a more slippery idea than it seems: there are accents that combine with letters, emoji made of several pieces, scripts that run from right to left and letters that pass for others.

Letters as numbers

The idea of representing letters by numbers is older than the computer. In the telegraph that the Frenchman Émile Baudot created in the 1870s, each letter was sent as a combination of five electrical signals, each one on or off. Five signals of that kind, which today we would call bits, make 2⁵ = 32 combinations, too few for 26 letters, ten digits and punctuation. The code got around the problem with two special signals, which switched between the mode of letters and that of figures, like the shift key of a keyboard.

With computers came dozens of different codes, almost one for each manufacturer, until, in 1963, the American standards association published ASCII, short for American Standard Code for Information Interchange. ASCII uses 7 bits, which gives 128 codes, numbered from 0 to 127. Since the revision of 1967, which added the small letters, the table has hardly changed. Hover over or tap the pieces to see the number of each one:

The numbers appear in hexadecimal, base 16, in which the digits go from 0 to 9 and carry on from A to F. Each hexadecimal digit stands for exactly 4 bits, which is why this notation is so common among people who deal with bytes. In the table, the column gives the last digit and the row the first: “A”, in row 4 and column 1, has the code 41 in hexadecimal, which is 4 × 16 + 1 = 65 in decimal.

The first 32 codes, and also the last, are not letters but control characters, commands for the teletypes and terminals of the time. 7, BEL, rang the bell of the teletype. 13, CR, for carriage return, took the print carriage back to the start of the line, and 10, LF, for line feed, moved the paper up a line. To this day, Windows ends each line of a text file with the pair CR LF, while Linux and macOS use only LF, and that difference still causes small confusions when files go from one system to another.

The table was planned with care. The digits begin at code 48, so that the four bits on the right of each one form its own value: “7” is 011 0111, and 0111 is 7 in binary. The capitals begin at 65 and the small letters at 97, exactly 32 positions later. Since 32 is a power of 2, each letter differs from its small version in a single bit, and changing between capitals and small letters is just a matter of turning that bit on or off. The last code, 127, DEL, has all seven bits on, and not by chance: on punched paper tape, where each hole was a 1 bit, any character could be erased by punching all the positions.

For English, 128 codes were enough. But ASCII has no “é”, no “ñ”, no “ç”. To write in French, in Spanish or in Portuguese, and in almost any other language, one had to go further.

One byte, many tables

When computers began to keep each character in a byte, of 8 bits, there was a bit to spare, and with it the table could double in size. The codes from 0 to 127 remained those of ASCII, and the 128 new ones, from 128 to 255, were free for what was missing. The problem is that everyone filled that half in their own way.

The IBM PC, launched in 1981, used the upper half for some accented letters, Greek letters and lines for drawing frames on the screen. ISO defined a family of tables, ISO 8859, with a variant for each region: 8859-1, also called Latin-1, for the languages of Western Europe; 8859-5 for Cyrillic; 8859-7 for Greek; and so on. Microsoft created its own variants for Windows, such as Windows-1252, almost the same as Latin-1, and Windows-1251, for Cyrillic. Apple had others, and in the Soviet Union the KOI8 family had been in use since the 1970s. Each of those tables is a different encoding or, in the jargon of IBM and Microsoft, a code page.

A text file, however, is just a sequence of bytes, and it does not say in which table it was saved. Whoever opens it has to guess and, when they guess wrong, the right bytes become the wrong characters. Try it: choose how the text is saved and then how it is read.

The result even has a name: mojibake, from the Japanese 文字化け, something like “transformed characters”. If you have ever seen “cafÃ©” in place of “café”, you have seen a text saved in UTF-8, an encoding we will meet further on, and read as Windows-1252. The other way round, a byte that does not form a valid sequence of UTF-8 is replaced by the replacement character, U+FFFD, the diamond with a question mark. The Japanese named the phenomenon because they suffered from it more than anyone. Japanese has thousands of characters, which do not fit in 256 positions, and Japanese computers used encodings with several bytes per character, such as Shift JIS and EUC-JP, incompatible with each other. Chinese had its own, GB 2312 in mainland China and Big5 in Taiwan and Hong Kong, and Korean had others still.

Some tables had clever tricks to survive the chaos. In KOI8-R, the Russian letters sit in the positions of the Latin letters of similar sound, plus 128. If an old e‑mail server wiped out the eighth bit of each byte, as many did, “Привет, мир” (“hello, world”) became “pRIWET, MIR”, with capitals and small letters swapped, but still readable for a Russian. It was a workaround for a problem that would only be solved by a single table for all the scripts of the world.

A number for each character

In 1987, Joe Becker, of Xerox, and Lee Collins and Mark Davis, of Apple, began to study what a universal character set would look like. The following year, Becker published a draft called Unicode 88, which explained the name: the new code was to be “unique, unified and universal”. The proposal was to give each character of each script of the world a number of its own, which would not depend on a table, a manufacturer or a country.

At the start of 1991, companies such as Apple, IBM, Microsoft, NeXT and Sun formally set up the Unicode Consortium, which published the first version of the standard in October of that year. At the same time, ISO was working on a similar project, the standard ISO/IEC 10646. Instead of competing, the two efforts merged, and since 1993 Unicode and ISO/IEC 10646 have assigned exactly the same numbers to the same characters.

The number of a character, its code point, is written in hexadecimal, with at least four digits, after the prefix “U+”: “A” is U+0041, “ç” is U+00E7, “€” is U+20AC and “😀” is U+1F600. The first 128 code points are those of ASCII, in the same order, and the first 256 those of Latin-1. So texts in ASCII or in Latin-1 could be converted to Unicode without a single number changing.

Besides the number, each character gets an official name, in English and in capitals: “ç” is LATIN SMALL LETTER C WITH CEDILLA. Since 1996, names never change once published, not even when they are wrong. U+FE18 is called PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRAKCET, with “bracket” misspelt, and so it will stay forever; the consortium could only add a corrected alias. That stability is a golden rule of Unicode: programs written today must keep working with the texts of tomorrow, and vice versa. Type or paste any text to see the number and the name of each character:

Some names do not even need to be stored. Those of the Chinese ideographs are made from the number itself, such as CJK UNIFIED IDEOGRAPH-4E16 for 世, and those of the Korean syllables are put together from the letters that make them up: 한 is HANGUL SYLLABLE HAN. To stay light, this page carries the names of a selection of blocks; the rest are in the consortium's official charts.

Characters and glyphs

Unicode encodes characters, not drawings. A character is an abstract unit of writing, such as “the small letter g”. The shape it takes on the screen, the glyph, is up to the font. The same U+0067 can have an open or a closed tail, with or without serifs, and it is still the same character:

In some scripts, the shape depends on the context. In Arabic, which is cursive, most letters have four forms, according to whether they appear alone or at the beginning, in the middle or at the end of a word, but each letter is still a single character. The program that draws the text chooses the right form, following the rules of the font; in the second card, an invisible joining character, which we will meet further on, forces each of the forms.

The most delicate case is that of the Chinese ideographs, also used in Japanese, in Korean and, in the past, in Vietnamese. Many of them have regional variants, with small differences of stroke. So as not to encode the same ideograph several times, Unicode adopted Han unification: variants considered the same character get a single code point, and the difference is left to the font. That is why, in the third card, 骨 and 直 look one way in a Chinese font and another in a Japanese one. The decision was, and still is, controversial, especially in Japan, because it forces the program to know what language the text is in to choose the right font.

The separation between character and glyph also explains a familiar sight. When the font in use has no drawing for a certain character, the system looks for another one that has; if none has, an empty rectangle appears, nicknamed tofu for its likeness to the bean curd. The text is not corrupted: the character is there, there is just no way to draw it. It was to put an end to tofu that Google created the Noto family of fonts, whose name comes from no more tofu, and which today covers most of the scripts of Unicode.

The code space

Unicode's creators imagined, at first, that 16 bits would be enough: 65,536 positions seemed to be plenty for all the scripts in use. They were not. The Chinese ideographs alone, counting the rare and historical variants, number over 100,000. In version 2.0, of 1996, the space was enlarged to 1,114,112 code points, from U+0000 to U+10FFFF, divided into 17 planes of 65,536 positions each.

Below, each plane is a plate, and each tower a group of 256 consecutive code points. The height of the tower shows how many of them already have a character assigned, and the colour shows the region where the script comes from. Drag the slider to watch the space fill up, version by version, and hover over or tap the towers to see what each one holds:

Plane 0, the Basic Multilingual Plane, or BMP, gathers the characters of everyday use: the Latin, Greek, Cyrillic, Arabic and Hebrew alphabets, the scripts of India and Southeast Asia, Korean Hangul, the Japanese syllabaries and the most common Chinese ideographs. It is almost full, and much of what is not a script is reserved for two special uses. The private use area is free for each company or program to use as it likes: that is where some fonts keep icons, for example. And the surrogates serve a trick of UTF-16, which we will see in the next section.

Plane 1, the Supplementary Multilingual Plane, brings together historical scripts, such as Egyptian hieroglyphs and cuneiform, musical notation, mathematical letters and most of the emoji. Planes 2 and 3 are almost entirely ideographs; the unified Chinese ideographs alone add up to 101,985. Planes 4 to 13 are empty. Plane 14 holds a few invisible characters for special uses, and planes 15 and 16 are entirely private use.

In recent years, a new version of Unicode has come out every September. The largest of all was 3.1, of 2001, which added 44,946 characters and was the first to go beyond the BMP. The most recent, 18.0, brought 13,007, the largest batch since then. Of the new characters, 11,328 belong to the seal script, a form of the Chinese characters that goes back to the Qin dynasty, around 200 BC, and that still appears today on stamps and seals. The version also brought Jurchen, the script used in northeast China during the Jin dynasty, three currency symbols and nine emoji. Even so, after 35 years, the characters take up about 15% of the space: there are over 800,000 free positions, which should be enough for a long time.

From numbers to bytes

A code point is an abstract number. To save it in a file or send it over the network, one has to decide how to turn it into bytes, and that rule is called an encoding form. Unicode defines three: UTF-32, UTF-16 and UTF-8. The letters UTF stand for Unicode Transformation Format, and the number gives the size, in bits, of the basic unit of each one.

UTF-32 is the simplest: each code point takes 32 bits, or 4 bytes, always. Since the largest code point, U+10FFFF, has 21 bits, the other 11 stay at zero. UTF-32 is convenient for programs, because all characters have the same size and it is easy to jump to any of them in the middle of the text. But it wastes a lot of space: in a text in English, three bytes out of every four are zeros.

UTF-16 uses units of 16 bits. The code points of the BMP, which fit in 16 bits, take one unit, with the number itself. Those of the other planes do not fit, and that is where the surrogates come in: the 2,048 positions from U+D800 to U+DFFF, which Unicode set aside and will never assign to any character. From the code point, 10000 in hexadecimal is subtracted, and the result, which fits in 20 bits, is split into two halves of 10 bits. The first half is added to D800, and the second to DC00, making a surrogate pair. In the bits, that amounts to putting the prefixes 110110 and 110111 in front of each half:

“😀”, U+1F600, lies outside the BMP. Taking away 10000 leaves F600, and the resulting pair is D83D DE00. Since the units in that range mean nothing on their own, a program reading UTF-16 recognizes the pair without ambiguity. The other examples, such as “€”, take a single unit; and, in UTF-32, any of them takes one unit of 32 bits with the upper bits at zero.

Windows, Java and JavaScript adopted Unicode early, with units of 16 bits, when it was believed that this would be enough, and that is why they still treat text as a sequence of UTF-16 units. In a browser, "😀".length is 2, because JavaScript counts UTF-16 units, not characters.

UTF-8

One evening in September 1992, in a diner in New Jersey, Ken Thompson sketched on a placemat, in front of his colleague Rob Pike, a third encoding form. The two worked at Bell Labs, on the Plan 9 operating system, and in less than a week the system was running entirely on it. Today it is by far the most used of all: UTF-8.

In UTF-8, each code point takes from 1 to 4 bytes, according to the size of the number. Up to U+007F, one byte is enough, the same as in ASCII. Up to U+07FF, a range that includes the accented letters and the Greek, Cyrillic, Hebrew and Arabic alphabets, it is two bytes; up to U+FFFF, the rest of the BMP, three; and from there on, four. The first bits of each byte say what its role is. A byte that begins with 0 is a complete ASCII character; those that begin with 110, 1110 or 11110 open a sequence of 2, 3 or 4 bytes; and those that begin with 10 are continuations. The bits of the code point fill the positions that are left:

“ç”, U+00E7, is 1110 0111 in binary and does not fit in the 7 free bits of a single byte. So it goes into the pattern of two bytes, 110xxxxx 10xxxxxx, which has 11 free positions. Padded with zeros on the left, the bits make the bytes C3 A7, the same ones that, read as Windows-1252, appeared as “Ã§” in the panel of the tables. This design has several virtues:

Any text in ASCII is already valid UTF-8, byte for byte, and old programs kept working.
The bytes of the longer sequences are all greater than 127 and are never mistaken for an ASCII character. A slash or a zero byte in the name of a file keeps meaning what it always meant, and that made it possible to adopt UTF-8 without rewriting the operating systems.
It is easy to find your place: it is enough to look at the first bits of a byte to know whether it begins or continues a character. If a byte is lost, or if reading begins in the middle of the text, only one character is spoilt, and the rest is read normally.
There is no doubt about the order of the bytes. In UTF-16 and UTF-32, each unit has several bytes, and some computers store the most significant byte first, while others store the least significant first. UTF-8 is read byte by byte.
Sorting texts in UTF-8 byte by byte gives the same order as sorting the code points.

The price is that the size of the text varies from script to script. Below, the opening sentence of the Universal Declaration of Human Rights in several languages, with the average number of bytes per code point in each encoding. Hover over or tap the bars to see the sentence and the totals, and try a text of your own:

your text

In English, and in Portuguese without accents, as in that sentence, UTF-8 spends exactly one byte per character; each accented letter adds just one more. In Russian, Greek and Arabic, it comes close to two, because the letters take two bytes, but spaces and punctuation still take one. Hindi, Chinese and Japanese are the cases where UTF-16 has the edge: their characters take three bytes in UTF-8 and only two in UTF-16. In practice, though, text seldom travels alone. A web page in Chinese comes full of HTML markup, which is ASCII, and in UTF-8 it ends up smaller.

For all these reasons, UTF-8 won. According to W3Techs, it is used by 99.1% of the websites whose encoding is known. It is the encoding required by the current HTML standard and by the JSON exchanged between systems, and the default on Linux, on macOS and in most modern programming languages. UTF-16 lives on inside programs, but it is seldom used to store files or exchange data.

One curiosity remains: the character U+FEFF, the byte order mark, or BOM. In UTF-16, it goes at the start of the file to show in what order the bytes of each unit come. UTF-8 does not need it, but some programs wrote it anyway, as a signature; Windows Notepad did so for many years. In UTF-8, the BOM becomes the bytes EF BB BF, and anyone who has opened a file that began with “ï»¿” has seen those three bytes read as Windows-1252.

Accents that combine

So far, we have treated each character as a number. But there are letters that can be written in more than one way. “á” exists in Unicode as a ready-made character, U+00E1, inherited from Latin-1. And it also exists as the sum of two pieces: an ordinary “a”, U+0061, followed by U+0301, the combining acute accent, a character that does not appear on its own and is drawn over the character before it.

Combining marks make it possible to write any combination of letter and accent, including those that no language uses. Choose a letter and add marks:

The marks pile up in the order in which they are typed, and the font tries to arrange them above or below the letter. There are hundreds of them, used by languages all over the world. Vietnamese, for example, often puts two marks on the same vowel, as in “ệ”. And nothing stops dozens of marks from being stacked on the same letter, a game that became known on the internet as zalgo text.

Having two spellings for the same text creates a problem. If you search for “José” in a document where the name was saved with a separate accent, a byte-by-byte comparison will not find it: to the computer, they are different sequences. Unicode's solution is normalization, which rewrites the text in a standard form. In the NFC form, letters are composed whenever a ready-made character exists; in NFD, all are decomposed into a letter plus marks. Two equivalent spellings become identical after being normalized to the same form. The code points that change are highlighted:

NFC is the most common form, recommended for the web and produced by most keyboards. But it is not universal. The HFS+ file system, the standard on Macs until the arrival of APFS in 2017, stored the names of files decomposed, and files created on those computers and copied to other systems can still carry their names in the NFD form, which confuses programs that compare names byte by byte.

There are also the compatibility forms, NFKC and NFKD, which go further and replace characters that exist for historical or typographic reasons with their plain equivalents. The ligature “ﬁ” becomes “f” and “i”, “²” becomes “2”, and the wide “Ａ”, used in East Asian texts, becomes an ordinary “A”. These forms lose information, but they are useful in searches and when comparing user names, which should not be different just because one of them uses wide letters.

What is a character?

By now, the word “character” is stretched too far. The “á” of the previous section can be one code point or two, but the reader sees a single letter. What the reader perceives as a unit is what Unicode calls a grapheme, and there are official rules for grouping code points into graphemes. With emoji, the distance between the two things becomes enormous. Choose an example and pull the pieces apart:

Emoji appeared on Japanese mobile phones at the end of the 1990s, and each carrier had its own set. When companies such as Google and Apple needed to exchange messages with those phones, the emoji were brought into Unicode, and most of them came in version 6.0, of 2010. Since then, new emoji have appeared not only with new code points, but also by combining existing ones.

The family is made of four emoji of people, joined by U+200D, the zero width joiner, or ZWJ. It is an invisible character that tells the font: “if you have a drawing for this combination, use it”. If the font does not, the four people appear side by side. Skin tones, which arrived in 2015, are modifiers based on the Fitzpatrick scale, used by dermatologists, and they change the emoji that comes before them. And country flags have no code points of their own: each one is a pair of regional indicators, special letters that spell the two-letter code of the country, “GB” in the case of the United Kingdom. That way, Unicode does not have to decide which countries deserve a flag.

Graphemes also matter far from emoji. In Korean, the syllable 각 can be stored as a single code point or as three pieces, the letters ㄱ, ㅏ and ㄱ, which the font assembles into a block. In Hindi, the syllable “नि” is stored as the consonant “न” followed by the vowel “ि”, but the vowel is drawn before the consonant. The order in memory is not always the order on paper.

All this has practical consequences. Each programming language counts characters in its own way: for the family, Python answers that the text has 7 characters, because it counts code points, and JavaScript answers 11, because it counts UTF-16 units. In UTF-8, it is 25 bytes, and only the count of graphemes gives 1. A program that cuts a long text to fit on the screen, or that limits the length of a name, has to cut between graphemes, and not in the middle of them, or it may separate a skin tone from its emoji or an accent from its letter.

Right to left

Arabic and Hebrew are written from right to left. Numbers, however, are still written from left to right, even in the middle of an Arabic text, and it is common to mix words of both directions in the same sentence. How should such a text be stored?

Unicode stores everything in logical order, the order in which the text is read and typed, and leaves to the program that draws it the task of arranging each stretch in the right direction, following the bidirectional algorithm, a set of rules published by the consortium. Each character has a direction: Latin letters go to the right, Arabic and Hebrew ones go to the left, digits form blocks of their own, and spaces and punctuation are neutral and take on the direction of their neighbours. In the figure, the text appears at the top as you would see it on the screen, and below are the code points in the order in which they are stored:

In the first example, the Arabic word, “hello”, is stored in the order in which it is read, from the first letter to the last, but it appears reversed with respect to memory, with the first letter on the right. In the second, the whole sentence is Hebrew, and the number 4521, though surrounded by words that run to the left, is still read from left to right.

For the cases that the automatic rules cannot handle, there are invisible control characters that force a direction. U+202E, RIGHT-TO-LEFT OVERRIDE, makes everything after it be drawn from right to left, and it was once widely used to disguise files. In the third example, the name of a program, which ends in “.exe”, appears on the screen as photo‮gpj.exe, an innocent “.jpg” image.

In 2021, two researchers at the University of Cambridge showed that the trick also works in source code. With these characters hidden in comments and in quoted strings, a program can be displayed one way in the editor and compiled another, which would make it possible to hide a back door in innocent-looking code. The attack was named Trojan Source, and since then compilers, editors and sites such as GitHub have issued warnings when they find these characters.

Look-alike letters

With over 170,000 characters, it is inevitable that some look identical to others on the screen. The Cyrillic “а”, U+0430, is indistinguishable from the Latin “a”, U+0061, in most fonts, and the same goes for “е”, “о”, “р” and “с”, among others. Different characters with the same appearance are called homoglyphs.

That would be a mere curiosity, were it not for internet addresses. Since 2003, domain names can have letters of any script, which allows addresses in Chinese, in Arabic or with accents, such as “café.co.uk”. But it also allows registering a name in Cyrillic letters that imitates that of a bank or a shop, and using it to steal passwords. In 2017, a researcher registered “аррӏе.com”, written only in Cyrillic letters, and showed that Chrome, Firefox and Opera displayed it exactly like “apple.com”. See which script each letter of an address belongs to:

In fact, the internet's naming system, the DNS, is still limited to ASCII. Before querying it, the browser converts a name with other characters into a form called Punycode, which begins with “xn--”: “аррӏе.com” becomes “xn--80ak6aa92e.com”. To protect users, browsers show the Punycode in place of the letters when a name mixes scripts or when it is written entirely in letters that imitate another script. And the consortium keeps an official list of characters that can be confused with one another, used by browsers, domain registries and social networks.

Case and alphabetical order

Operations that seem trivial, such as turning a text into capitals, depend on the language. In German, “ß” becomes “SS” in capitals, and the word gets longer. In Turkish, the dotted “i” and the dotless “ı” are different letters, each with its own capital, “İ” and “I”; a program that does not know the text is Turkish turns “i” into “I” and swaps one letter for the other. And in Greek, the small sigma has a special form at the end of words. See what your browser does with each case:

Sorting words is harder still. By code points, “Z” comes before “a”, and accented letters go after “z”: “É”, 201, comes after “z”, 122. No dictionary works like that. Compare:

To sort like a dictionary, Unicode defines a collation algorithm, which compares first the letters, then the accents and only then capitals and small letters. The particularities of each language, such as the “ch” of traditional Spanish, which came after all the words with “c”, or the Swedish “å”, which comes after “z”, are kept in a large database maintained by the consortium, the CLDR, which also records how each country writes dates, numbers and currencies. When your phone sorts your contacts or writes “28 September 2026”, it is very likely using the CLDR.

Unicode in practice

For those who write programs, the most important lesson is simple: use UTF-8 everywhere and say so explicitly. In a web page, the head should begin with <meta charset="utf-8">. In a database, choose UTF-8 when creating the tables. And when reading or writing files, state the encoding instead of relying on the system's default, which changes from one computer to another. In Python:

import unicodedata

text = "Café 👍🏽"
print(len(text))                    # 7: code points, not graphemes
print(len(text.encode("utf-8")))    # 14 bytes
print(hex(ord("ç")))                # 0xe7
print(unicodedata.name("ç"))        # LATIN SMALL LETTER C WITH CEDILLA

a, b = "café", "café"    # ready-made é and combining accent
print(a == b)                       # False
nfc = lambda s: unicodedata.normalize("NFC", s)
print(nfc(a) == nfc(b))             # True

with open("data.txt", encoding="utf-8") as f:
    content = f.read()

Python's len function counts code points: “Café 👍🏽” has 7, because the thumb and the skin tone count as two. The same text takes 14 bytes in UTF-8. And the two spellings of “café”, with the ready-made “é” or with the combining accent, only become equal after being normalized. In JavaScript, Array.from(text) splits a text into code points, text.normalize("NFC") normalizes it and Intl.Segmenter splits it into graphemes.

Some pitfalls are classics. In MySQL, the encoding called utf8 accepts at most 3 bytes per character and does not store emoji; the complete version is called utf8mb4. Many programs tested only with English text fail with accents, with emoji or with Arabic, and it is worth testing them with the examples of this guide. And, when text comes from outside, such as user names or addresses, one must remember that two different sequences can look the same.

What if a character is missing? Anyone can send a proposal to the Unicode Consortium. For a new script, one must show that it is or was used, with examples from documents, and many proposals take years between research and approval. For a new emoji, one must show that it would be widely used. Today, much of the work is on historical scripts and on those of minority peoples, many still without support on computers. With the Adopt a Character programme, anyone can “adopt” a character, and the donation helps to fund that work.

Final words

Unicode is one of the most successful infrastructures of computing, and one of the least visible. It solved a problem that seemed to have no solution, that of making computers all over the world agree on the meaning of each number, without erasing the diversity of scripts: from Latin letters to hieroglyphs, from Arabic to Hangul, each one fits in the same space, with its own rules.

Almost all it asks of those who deal with text is not to assume: not that a character fits in a byte, not that a code point is a letter, not that text runs to the right, not that two letters that are the same on the screen are the same in memory. The next time an “Ã©” appears in place of an “é”, you will know exactly what happened: two bytes of UTF-8 were read, one by one, in the wrong table.