Unicode and UTF-8 in Go

Unicode assigns code points to text symbols. UTF-8 encodes those code points into one to four bytes, making text portable across files and network protocols.

A Go range loop over a string decodes successive UTF-8 code points. Its index is the byte offset where a rune begins; its value is the decoded rune. The byte indexes can therefore skip numbers.

Invalid UTF-8 sequences are represented by the Unicode replacement rune during decoding. Functions in unicode/utf8 help validate, count and decode UTF-8. A code-point count is different from a grapheme-cluster count, which is often closer to what users perceive as characters.

Read the labels and compare which positions are part of the visible value and which belong to the underlying representation.

This example is a complete program. Save it as main.go or paste it into the Go Playground.

Go code

package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	text := "Aé😀"
	fmt.Println("Valid UTF-8:", utf8.ValidString(text))
	fmt.Println("Bytes:", len(text))
	for offset, letter := range text {
		fmt.Printf("Offset %d: %c\n", offset, letter)
	}
}

Output

Valid UTF-8: true
Bytes: 7
Offset 0: A
Offset 1: é
Offset 3: 😀

A uses one byte, é uses two and 😀 uses four, so the string has seven bytes. The range loop emits offsets 0, 1 and 3, where the three runes begin. It does not yield offsets 2, 4, 5 or 6 as separate decoded characters.

• A range index on a string is a byte offset, not a consecutive rune number.
• Valid UTF-8 does not imply that every visually perceived character is exactly one rune.

Change the string to "Goé😀" and predict the starting byte offsets for every rune.

Tip: Edit the example above, predict the output, and run the modified code to test your reasoning.

You have practiced Unicode and UTF-8 in Go in a small, self-contained program. The important habit is to predict what runs, inspect the actual output and distinguish the language rule from the particular example.

Before continuing, make sure you can explain the program to someone who has not seen it, including the line that changes its result.