Unicode Without the Folklore

A character is what a person thinks of as a written unit. A Unicode code point is a numbered entry such as U plus zero zero four one for A, or U plus one F six zero zero for the grinning face. Bytes are the stored or transmitted numbers. Those layers overlap for simple ASCII and diverge everywhere interesting.

Text seen:       A        é        😀
Code point:      U+0041   U+00E9   U+1F600
UTF-8 bytes:     41       C3 A9    F0 9F 98 80
Byte count:      1        2        4

# Python makes the boundary visible
s = "é"
ord(s)                  # 233
s.encode("utf-8").hex()  # 'c3a9'

“Unicode character” is often too vague to debug with. Ask which code points exist, which bytes arrived, and which encoding was used. Hex output settles arguments that screenshots cannot.

UTF-8 represents every Unicode code point with one to four bytes. ASCII keeps its familiar single-byte values, which made UTF-8 easy to adopt. Non-ASCII text uses multiple bytes. It is variable-width but self-synchronizing: continuation bytes have a recognizable pattern, helping decoders find boundaries.

U+0000..U+007F      0xxxxxxx                         1 byte
U+0080..U+07FF      110xxxxx 10xxxxxx                2 bytes
U+0800..U+FFFF      1110xxxx 10xxxxxx 10xxxxxx       3 bytes
U+10000..U+10FFFF   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx  4 bytes

"café".encode("utf-8") → 63 61 66 C3 A9

My default is blunt: choose UTF-8 and state it. Do not choose a legacy encoding to save a few bytes of local text. Compression handles repetition better, while encoding ambiguity creates bugs that cross systems and survive backups.

Decoding maps bytes to text using an encoding. Encoding maps text to bytes. Files, sockets, database drivers, and subprocesses are boundaries. Inside the application, keep values as Unicode strings. An encoding is not something you “apply to a string for safety”; it is the contract used when crossing a byte boundary.

# explicit file boundaries in Python
with open("names.txt", "r", encoding="utf-8") as f:
    text = f.read()             # bytes → text

payload = text.encode("utf-8") # text → bytes
round_trip = payload.decode("utf-8")

Mojibake is readable-looking garbage caused by decoding bytes with the wrong character encoding. The word café becoming café is the classic case: UTF-8 bytes were interpreted as Windows-1252 or Latin-1. The replacement character, �, means a decoder could not map the bytes and substituted U plus F F F D. Once originals are replaced, information may already be lost.

Original text:              café
Correct UTF-8 bytes:        63 61 66 C3 A9
Wrong Windows-1252 decode:  café

Danger: decode with replacement → save → original bad bytes are gone
Repair: recover original bytes → identify encoding → decode once

A byte order mark was useful for signaling byte order in UTF-16 and UTF-32. UTF-8 has no byte-order problem, but some tools prepend the three bytes E F B B B F as a UTF-8 signature. Some readers consume it. Others expose U plus F E F F as the first character, breaking headers, JSON, and command scripts.

UTF-8 BOM bytes: EF BB BF
Visible mojibake if decoded wrongly: 
Python reader that consumes it: encoding="utf-8-sig"
Plain UTF-8 output:              encoding="utf-8"

# Diagnose, do not blindly strip every file
raw = open("data.csv", "rb").read(4)
print(raw.hex())  # efbbbf...

The same visible text can use different code-point sequences. The letter é may be one precomposed code point or the letter e followed by a combining acute accent. Unicode normalization converts equivalent sequences into a chosen form. NFC is a sound default for ordinary storage and comparison. NFKC changes compatibility characters too, so use it only when that semantic folding is intended.

import unicodedata

a = "é"             # U+00E9
b = "e\u0301"        # U+0065 + U+0301
a == b                      # False
unicodedata.normalize("NFC", a) == \
unicodedata.normalize("NFC", b)  # True

Normalize at a defined boundary, not randomly before every comparison. Preserve the original when exact spelling has legal or forensic value. Normalization is not case folding, transliteration, accent removal, or security validation; those are separate decisions with separate losses.

A grapheme cluster is roughly one user-perceived character. It may contain one code point, a base plus combining marks, a flag made from regional indicators, or an emoji sequence joined by invisible controls. The family emoji can contain several people and joiners while appearing as one glyph. Code-point indexing is therefore not a safe cursor model.

Visible unit       Possible code points
é                  U+00E9  or  U+0065 U+0301
🇯🇵                 U+1F1EF U+1F1F5
👩🏽‍💻                woman + skin tone + joiner + laptop

JavaScript:
"😀".length             // 2 UTF-16 code units
[..."😀"].length        // 1 code point
// Still use Intl.Segmenter for grapheme clusters.

Use a maintained Unicode segmentation implementation such as the platform's grapheme iterator or Intl.Segmenter. Do not write an emoji regex from a blog post and call it done. Unicode evolves, and flags, modifiers, variation selectors, and joiner sequences defeat simple ranges.

Length may mean bytes for a storage limit, code units for a runtime API, code points for protocol rules, grapheme clusters for an editor, or display width for a terminal. None is universally correct. Name the unit in the variable and the error message. “Maximum length one hundred” is an incomplete requirement.

LIMIT                         MEASURE
Database byte column          encoded bytes in database charset
Username policy               normalized code points or graphemes, documented
Text input counter            grapheme clusters users perceive
Terminal table                display columns, with a width library
Network packet                encoded bytes

Never: encoded[:10].decode("utf-8")
Instead: truncate text at a supported boundary, then encode

Filenames are Unicode strings at many APIs, but normalization and case behavior differ by filesystem. Databases have server, database, table, column, and connection encodings or collations. HTTP sends bytes; the media type and protocol rules say how to decode them. Test the whole path, not just a string literal inside one process.

FILES      open(path, encoding="utf-8") for text content
DATABASE   UTF-8-capable schema + UTF-8 client connection
MYSQL      prefer utf8mb4, not historical three-byte utf8
HTTP       Content-Type: text/plain; charset=utf-8
JSON       send UTF-8 bytes; do not guess from rendered text
TEST       "José · 東京 · हिंदी · 😀 · e\u0301" round trip

Do not repair encoding bugs by repeatedly calling encode and decode until the screen looks right. Freeze the input, find the first broken boundary, and prove every transformation.

DEBUG IN ORDER
1. Preserve original bytes; stop destructive imports or saves.
2. Locate the first boundary where correct text becomes wrong.
3. Print bytes as hex and text as escaped code points.
4. Find the declared encoding at the producer and transport.
5. Decode exactly once with that encoding.
6. Normalize only if the product's comparison rules require it.
7. Keep text internally; encode once at output.
8. Round-trip accents, combining marks, CJK, emoji, and empty text.

SYMPTOM                         LIKELY CAUSE
café                           UTF-8 decoded as Windows-1252/Latin-1
�                               invalid bytes replaced; possible data loss
 at the start                BOM bytes decoded as legacy text
Unexpected UTF-8 BOM            reader expects plain UTF-8
Incorrect string value \xF0…    database/connection lacks four-byte support
Different strings look equal    normalization mismatch
Broken emoji after truncation   code-unit, code-point, or byte split

DEFAULTS
Text encoding: UTF-8 · normalization when needed: NFC
MySQL Unicode: utf8mb4 · user-visible slicing: grapheme clusters

Unicode is not mysterious; hidden boundaries are. Make bytes visible, require UTF-8 contracts, and choose the right unit for comparison and truncation. The worst fix is one that makes today's sample look correct while destroying tomorrow's original bytes.