Garbled character repair/analysis & Excel CSV character code mutual conversion tool | ZeroTools

This feature provides a low-layer analysis infrastructure that directly reads the internal byte representations of various character encodings and performs mapping to precise code points.

For example, in Shift_JIS, the first byte is checked to see if it is in the range 0x81 to 0x9F and 0xE0 to 0xFC, and in combination with the second byte that follows it identifies a specific kanji or symbol.

As a specific example, the hiragana character "a" is expressed as a continuous byte string of 0x82 and 0xA0. UTF-8 uses a variable length encoding method from 1 to 4 bytes, and by analyzing the bit pattern of the first byte, the number of subsequent bytes is calculated and Unicode code points are assembled.

In the EUC-JP environment, half-width kana expressions starting with 0x8E and 3-byte supplementary kanji expressions starting with 0x8F are accurately identified and relocated to the appropriate character area.

Furthermore, in encoding systems that involve state transitions such as ISO-2022-JP, the internal state is switched to interpret the subsequent stream as a JISX0208 2-byte character set when the escape sequence 0x1B, 0x24, 0x42 byte string is detected.

At the same time, for UTF-16 and UTF-32, it detects both little-endian and big-endian byte order marks, performs direct binary analysis in 2-byte or 4-byte units according to the specified endianness, and reconstructs the character array expanded in memory.

In order to solve the incompatibility of certain symbols that frequently occur during mutual conversion between character code standards, this mechanism includes a dedicated mapping correction table.

For the wave dash problem, which is known as the most typical problem, we apply strict code point conversion rules. Automatically detects visual and semantic discrepancies between the Unicode standard wave dash U+301C and the full-width tilde U+FF5E assigned when converting from Shift_JIS in a Windows environment.

Depending on the user's target OS environment and output format requirements, it dynamically switches between a mode in which U+301C is forcibly replaced with U+FF5E to maintain Windows compatibility, and a mode in which it is strictly saved as the original U+301C.

This correction process is not limited to wave dashes, but comprehensively prevents garbled characters caused by the unique mapping of the CP932, such as the minus sign U+2212, the full-width hyphen minus U+FF0D, the double-column symbol U+2016 and the parallel symbol U+2225, as well as the cent sign, pound sign, and knot sign.

It monitors these specific code points during the byte string parsing stage, and if they do not meet the specifications of the output destination encoding, it has a processing mechanism that either falls back to a predefined safe alternative character or catches the exception and issues a warning to the user, preventing data loss.

We have implemented advanced ligature detection processing to correct problems caused by differences in Unicode normalization formats that occur when linking text data between different operating systems.

The standard file system in the macOS environment and some text editors use a decomposed normalization format called NFD.

In this format, characters with voiced or handakuten marks are recorded as two separate code points: a base character and a combined character.

For example, the hiragana `ga'' is expressed as a continuous sequence of the base character U+304B `KA'' and the conjunctive dakuten U+3099.

In contrast, Windows and general web environments predominate in a synthesized normalized format called NFC, which is treated as a single code point U+304C.

Due to this architectural difference, when text from macOS is displayed as is in a Windows environment, it will result in visual garbled characters where the voiced mark appears separated from the following character.

The system scans the entire input stream for ligature combinations, and if it finds a byte pattern where a base character is immediately followed by a U+3099 or U+309A combination character, it immediately runs a reconstruction algorithm that converts it into a corresponding single synthesized code point.

This fundamentally eliminates the problem of voiced mark separation that occurs when exchanging text data between OSes and ensures visual integrity.

For text data that has already been garbled, it is equipped with an inverse conversion repair algorithm that infers the misinterpretation process that caused the garbled character string and restores it to the original character string.

A typical pattern of garbled characters is when the system incorrectly interprets a byte string originally encoded in Shift_JIS as UTF-8.

If the combination of the first and second bytes of Shift_JIS accidentally matches the UTF-8 variable length encoding rules, it will be drawn on the screen as a completely unintended list of kanji and symbols.

This function extracts the current Unicode code point array from the input gibberish string and re-encodes them into the original raw byte array according to the specified misread encoding rules.

It then performs a two-step conversion process in which the resulting raw bytes are decoded again based on the original correct character encoding rules.

Furthermore, for complex garbled characters with multiple candidates, we apply a statistical heuristic evaluation function based on the frequency of Japanese characters and the continuity of character types.

We provide a self-repair mechanism that determines the decoding results in which hiragana, katakana, and commonly used kanji appear consecutively with a natural probability as having the highest likelihood, and automatically selects the optimal repair string and presents it to the user.

All character code parsing and binary array reconstruction processing uses a local processing architecture that is completed within the client-side web browser.

The manuscript data and garbled text files specified by the user to be converted are never sent to an external server, from the reading process through the file reader to the operation as a typed array on the internal memory, to the final export of the repaired file.

Even when handling large CSV files or database export data containing complex multi-byte characters, we make full use of array buffers and data views, which are JavaScript binary operation interfaces, to achieve high-speed encoding conversion through direct bit operations in memory space.

With this design, even when handling files containing personal information or highly confidential business data, there is no intervening network communication, completely eliminating the physical risk of information leakage.

In addition, full access to all functions is guaranteed even in an offline environment, providing a safe, delay-free and reliable character encoding environment for data conversion work within closed networks of companies with extremely stringent security requirements.

This tool functions as a core component of data cleansing when extracting data from legacy systems and linking large-scale text data between different devices.

Old CSV data in Shift_JIS or EUC-JP format discharged from mainframes or old business applications can be used as pre-processing before importing into the latest cloud database.

Load the target CSV file on the browser, accurately identify the original character code using the encoding determination engine, and then perform the batch conversion process to UTF-8.

At this time, while monitoring data corruption due to incompatibility with the above-mentioned wave dash and unique extended characters, we simultaneously apply safe character replacement and normalization processing.

Furthermore, the phenomenon of separation of voiced marks that occurs when sharing files between different operating systems is repaired in advance, thereby preventing inconsistencies in search indexes and incorrect sort orders within the database.

Even if text records contain garbled characters, the automatic inverse conversion repair function is applied in batches, greatly reducing the amount of human effort required for visual confirmation and manual correction.

The end result is clean data encoded as fully standardized UTF-8 strings, eliminating technical barriers to character encoding during system migration.