charsets and encoding

the picture

you know how a file that looks fine on your mac shows up as café on someone's windows box? nothing got corrupted. the bytes are identical. the two machines just read them with different codebooks.

think of it like a cipher wheel. the message is the text (code points, abstract numbers like U+00E9 for é). the wheel is the charset (UTF-8, ISO-8859-1...). the scribbles on paper are the bytes. hand the paper to someone with a different wheel and they'll read something, confidently, and it'll be wrong.

code point

a character's number in unicode. U+0041 is A, U+00E9 is é, U+1F600 is 😀. there's room for 1,114,112 of them.

charset

an encoding: the rule from code points to bytes. UTF-8, UTF-16, ISO-8859-1, windows-1252. in java, a Charset object.

char

java's 16-bit type. one UTF-16 code unit. fine for most text, but anything above U+FFFF (emoji, rare CJK) needs two of them.

surrogate pair

those two chars. a "high" one (d800 to dbff) and a "low" one (dc00 to dfff) that only mean something together.

three different lengths

one string, three honest answers to "how long is it?":

import java.nio.charset.StandardCharsets;
 
void main() {
    String s = "😀";
    IO.println(s.length());                                // chars
    IO.println(s.codePointCount(0, s.length()));           // code points
    IO.println(s.getBytes(StandardCharsets.UTF_8).length); // bytes
    IO.println(Integer.toHexString(s.charAt(0)) + " " + Integer.toHexString(s.charAt(1)));
}

2
1
4
d83d de00

length() counts chars, and 😀 is a surrogate pair: d83d + de00. that's a leftover from java 1.0, when unicode was a 16-bit standard and java bet on it. unicode outgrew 16 bits (the first characters past U+FFFF arrived in 2001), so java 5 kept char at 16 bits and adopted UTF-16's trick of splitting big code points across two of them.

String s = "hé😀";
IO.println(s.length());

how UTF-8 works

UTF-8 is variable length: small numbers get fewer bytes. it's the reason it won the web.

every byte after the first starts with 10. three nice properties fall out of that:

• ASCII is valid UTF-8, byte for byte. old english-only files just work.
• self-synchronizing. land in the middle of a stream and you can find the next character start: skip bytes that begin with 10.
• no byte of a multi-byte character is ever a plain ASCII byte, so searching UTF-8 for / or \n never matches half a character.

UTF-16 is the opposite deal: 2 bytes for almost everything, 4 for the rest. great for CJK-heavy text, wasteful for english, and it has a byte-order question (big-endian or little?) that UTF-8 doesn't. getBytes(UTF_16) answers it by writing a 2-byte mark, fe ff, at the front.

mojibake

decode bytes with the wrong charset and you get mojibake (文字化け, "character transformation"). it has a fingerprint: Ã followed by something is almost always UTF-8 read as latin-1 or windows-1252.

import static java.nio.charset.StandardCharsets.*;
 
void main() {
    byte[] bytes = "naïve café".getBytes(UTF_8);
    IO.println(new String(bytes, UTF_8));
    IO.println(new String(bytes, ISO_8859_1));
 
    // undo it: encode the wrong way, decode the right way
    byte[] back = new String(bytes, ISO_8859_1).getBytes(ISO_8859_1);
    IO.println(new String(back, UTF_8));
}

naïve café
naïve café
naïve café

the last line is the useful trick. ISO-8859-1 maps all 256 byte values to a character, so a UTF-8-read-as-latin-1 string still holds every original byte. turn it back into bytes with the same wrong charset and you can decode it properly. writing the other way is not recoverable: if you encode ’ or 日 as ISO-8859-1, java writes ? and the character is simply gone.

the default charset

every API that takes text but doesn't ask for a charset uses the default charset: new String(bytes), getBytes(), FileReader, FileWriter, Scanner(InputStream), InputStreamReader without a second argument.

before java 18 the default came from the OS: UTF-8 on linux and mac, usually windows-1252 on windows (cp1252), or whatever locale the server was set up with. same jar, same file, different text depending on where it ran. JEP 400 fixed it: since java 18 the default is UTF-8 on every platform.

the habit that makes all of this go away: always pass the charset. Files.readString(path, UTF_8), new InputStreamReader(in, UTF_8), new String(bytes, UTF_8). the default being UTF-8 is a safety net, not a design.

chars you shouldn't cut

because a char can be half a character, char-based string surgery can split one in two. the half is a lone surrogate: not a valid character, so encoders replace it.

import java.util.HexFormat;
import static java.nio.charset.StandardCharsets.*;
 
void main() {
    String s = "hi😀";
    String cut = s.substring(0, 3);           // keeps h, i, and half the emoji
    IO.println(cut.length() + " " + Character.isHighSurrogate(cut.charAt(2)));
    IO.println(HexFormat.of().formatHex(cut.getBytes(UTF_8)));   // 3f is '?'
 
    int end = s.offsetByCodePoints(0, 3);     // 3 code points, however many chars
    IO.println(s.substring(0, end) + " " + end);
}

3 true
68693f
hi😀 4

truncating usernames to "20 chars", splitting text for SMS, reversing strings by hand: all places this bites. use the code point APIs (codePointAt, offsetByCodePoints, codePoints()), and know that StringBuilder.reverse() already keeps surrogate pairs together.

import java.text.Normalizer;
 
void main() {
    String a = "é";
    String b = "é";
    IO.println(a + " " + b + " " + a.equals(b));
    IO.println(a.length() + " " + b.length());
    IO.println(Normalizer.normalize(b, Normalizer.Form.NFC).equals(a));
}

é é false
1 2
true