CSV Encoding Nightmares: UTF-8, Shift-JIS, and Mojibake

Every team that handles CSV files eventually meets a customer payload that looks like "?????" or "\\\\u00e9\\\\u00e9\\\\u00e9" in the warehouse. The encoding got mangled somewhere between the producer and your code, and now nobody can tell whether the original was UTF-8, Shift-JIS, Latin-1, or something more exotic. This post is the field guide we wish we had on day one: the four failure modes you will see, what each one looks like, and the recipe to fix each.

The four kinds of broken

There are basically four different ways a CSV file's encoding can be wrong on arrival. Each looks different on screen, and each has a different fix. Recognising the visual signature is the first step.

Mojibake from misidentified UTF-8. The producer wrote UTF-8 with Japanese / Chinese / Korean characters; you read it as Latin-1 or Windows-1252. The bytes are correct UTF-8, the bytes are wrongly labelled. What you see is a run of high-byte characters that look like garbage — æ—¥æœ¬æ–‡ for the Japanese word "日本語", or ä¸æ–‡ for "中文" — but each visible character is actually two or three bytes of UTF-8 mis-decoded as a single Latin-1 character. Fix: detect the encoding (the encoding-japanese library has a good detector for ja/zh/ko; mozilla's chardet for everything else), re-decode as UTF-8.
Question marks for anything non-ASCII. The producer wrote UTF-8; you or your downstream tool replaced every non-ASCII byte with ?. This is what happens when a Windows tool reads UTF-8 through a non-Unicode code path — for example, cmd.exe in CP936 reading a UTF-8 file and silently dropping the high bytes. There is no recovery; the original characters are gone. Fix: track down the producer and fix the read path. If you cannot, ask the producer to re-export.
The leading BOM appears as a literal character in the first column. Excel saves UTF-8 files with a three-byte BOM (0xEF 0xBB 0xBF). A naive CSV reader that does not strip the BOM sees the first column header as "\uFEFFname" instead of "name". Fix: strip the BOM bytes at the start of the stream before parsing.
Yen-sign and backslash confusion in Japanese files. Microsoft CP932 (the variant of Shift-JIS Windows uses) maps byte 0x5C to ¥ rather than \. A Japanese CSV file with Windows path separators like C:¥Users¥me imported into a UTF-8 pipeline that splits on \ will give you empty path segments. Fix: either treat the file as Shift-JIS, normalising the 0x5C to \, or re-export from Excel with UTF-8 encoding (and stick to POSIX paths in the file).

The pipeline that actually works

After enough production incidents, we have settled on the following order of operations for every CSV we ingest:

Read raw bytes, not strings. Never let your CSV reader open the file with a default text mode. The first thing in the file is a byte sequence that may or may not be a BOM; only the byte stream tells the truth.
Run an encoding detector. encoding-japanese is reliable for ja/zh/ko payloads; chardet (Python) or jschardet (Node.js) is the right call for European and Latin alphabets. Trust the detector's confidence score, but also inspect the file manually (open it in a hex editor or run xxd file.csv | head); the detector is usually right but not always.
Decode the bytes once. Convert the raw bytes to a JS / Python / Java string using the detected encoding. From this point on, work with strings; never touch bytes again.
Strip the BOM from the first character if present. After decoding, the BOM becomes the single Unicode character U+FEFF at position 0. Remove it.
Sniff the delimiter on the first 8 KB of decoded text. Count occurrences of ,, ;, \t, and | outside double-quoted fields. Whichever produces a consistent column count across several lines wins.
Parse character-by-character through quoted fields. Never split on the delimiter with regex; treat quoted regions as opaque until you see the matching close-quote. Most hand-written parsers get this wrong on the first try and quietly corrupt fields containing the literal delimiter.
Validate row count and column count at the end. Every row must have the same number of fields. If not, the file is broken — either the producer's quoting is inconsistent, the delimiter is wrong, or the encoding step mangled a byte that turned out to be ASCII.

The Japanese specifics, in detail

If you are processing data from a Japanese customer — accounting exports, POS logs, EDI feeds — the encoding story is its own special case and worth the extra paragraph.

Japanese Windows ships three encodings that all claim to be "Japanese":

Shift-JIS (Microsoft's name: CP932): the historical default of Japanese Windows; covers the JIS X 0208 character set plus extensions for vendor-specific kanji and IBM's NEC extensions.
EUC-JP: the Unix-world default of the 1990s; rarely seen in modern exports.
UTF-8: the right answer but not the default; Excel will not save UTF-8 CSV unless the user explicitly picks it in the Save As dialog.

Detecting among the three is the encoding-japanese library's entire reason to exist. Once decoded, the most common downstream bug is the ¥ vs \ confusion described above. Two further glitches appear regularly:

Full-width digits and ASCII digits in the same column. Excel often writes ２０２６ (full-width) for products designed for print but 2026 (ASCII) for technical SKUs. Downstream validation needs to NFKC-normalise strings before any numeric parsing.
Wave-dash vs tilde. 〜 (U+301C, wave dash) is what Japanese Windows writes; ~ (U+007E, tilde) is what the rest of the world uses. Most comparison code treats them as equal — verify yours, because the difference trips up URL matching and email validation.

Tooling and round-trip

The CSV Cleaner implements every step in the pipeline above; it auto-detects encoding on file load, lets you confirm or override the delimiter, and offers both UTF-8 and Shift-JIS download buttons. This means a Japanese accounting file dropped into the tool comes out as a clean UTF-8 stream you can hand to a SQL loader without further work.

Always prefer round-tripping through a tool that lets you download the same file in two encodings. If you cannot find one, the next best is a one-off script that does the steps above — and crucially, that script must be tested against a file with non-ASCII characters in the first row, because encoding bugs hide in the second row until the second row contains something the loader cares about.

Closing

If you only remember one thing from this post: never let your CSV reader open a file in text mode. The byte stream is the source of truth; strings are downstream. Tools: CSV Cleaner, the CSV encoding guide, and the encoding-japanese detector for non-ASCII payloads.