Text difference (Diff) comparison/addition/deletion text extraction tool | ZeroTools

As a basic function of a text difference extraction tool, it is equipped with two independent text input fields: the comparison source (Original) and the comparison target (Modified).

In addition to direct input and pasting from the clipboard, you can use the function to load locally saved files to quickly reflect the text content of existing documents in the system.

To ensure stable system operation, an upper limit is set on the string data to be input, restricted to a maximum of 20,000 UTF-16 code units for each of the source and target.

Since the maxLength attribute is set on the text areas (textarea) for direct manual input or pasting, character input exceeding this limit is systematically restricted by the browser's functionality itself.

On the other hand, when loading data from the outside using the file upload function, the target file size limit is set to 1MiB. Only when the text extracted from the specified file exceeds the aforementioned limit of 20,000 code units, a warning message notifying the excess is displayed on the interface, and the loading process of the corresponding file is interrupted.

As an internal processing mechanism, the system uses the readAsText method of the FileReader API provided by the browser to read the target file asynchronously, but the system automatically interprets the input byte data as a UTF-8 format text file.

Therefore, if you specify a file saved in a different character encoding such as Shift-JIS or EUC-JP, there is a high probability of text garbling.

This series of file loading and text expansion operations is entirely completed solely within the client-side browser environment.

Because the input text data is expanded on the local memory for processing, there is no process of sending or uploading data to an external server via the internet.

Furthermore, even if the input field is empty, the system will continue to execute the process normally without throwing an exception error.

In the match rate calculation process described later, comparing empty strings is interpreted as an "identical state with no changes," and the system is designed to continue the subsequent processing as is.

The difference comparison process can be executed by selecting from three different modes: "Line by line", "Word by word", and "Character unit", depending on the granularity of confirmation intended by the user.

In the line-by-line comparison mode, each line break code (CRLF, CR, or LF) included in the text is treated as a delimiter, and the entire text is divided into an array.

At this time, the string of each line divided and arrayed is evaluated as one independent item. In the word-by-word comparison mode, it is important to note that semantic context division using advanced natural language processing or morphological analysis using a dictionary is not performed at all.

As a specific implementation, consecutive chunks of alphanumeric characters and underscores, consecutive whitespace characters, or other single characters are sequentially tokenized using the u flag (Unicode mode) against a regular expression pattern (such as \s+|[A-Za-z0-9_]+|[^\s]).

Due to this specification, when word-by-word comparison is applied to Japanese text, it exhibits completely different behavior from word extraction separated by spaces like in English.

Japanese kanji, hiragana, katakana, etc., are not recognized as word chunks, and will generally be divided and treated as items of one Unicode code point each, that is, in character units.

Also, emoji sequences composed of multiple code points are not always treated entirely as a single grouped item. In the character unit comparison mode, methods such as Array.from() are applied internally to strictly break down the entire text into an array of one character (one code point) at a time.

The fundamental logic for comparison is based on Eugene W. Myers' difference extraction algorithm, which compares the token arrays constructed by the aforementioned rules and calculates the most efficient route of additions and deletions.

This creates a mechanism where only the modified parts are reflected as visual highlights. When using the word mode or character mode, it is necessary to utilize them with the understanding that, due to specific character combination rules of each language or characteristics of regular expressions, differences are not always detected at boundaries that perfectly match "one character" or "one word" as intuitively recognized by humans.

When performing text comparisons, option settings are provided to intentionally ignore the presence or absence of irregular surrounding margins or differences between uppercase and lowercase letters, focusing only on the logical differences in the strings.

When the "Ignore leading and trailing margins" option is enabled, the trim() method is applied to the string of each item before it is passed to the comparison algorithm, removing whitespace characters present at the beginning and end of the string.

Furthermore, a preprocessing step is performed where consecutive whitespace groups such as spaces and tabs included inside the item's string are merged into a single half-width space.

It should be noted here that these whitespace treatments (trim and whitespace integration) are not applied uniformly to the entire text, but are individually applied to "each item" divided according to the selected mode (line, word, character).

Therefore, there is no guarantee that any internal whitespace differences existing in the original text or differences in indentation depth seen in source code will always be completely ignored by this option.

Depending on the division granularity and the position of the whitespace, there are cases where whitespace differences are still detected as changes.

On the other hand, when the "Ignore case" option is enabled, the arrayed strings are temporarily converted to lowercase by the toLowerCase() method only within the internal processing of the comparison determination, and the comparison calculation is executed assuming there is no difference between uppercase and lowercase letters.

However, since support for Unicode lowercase extension processing may contain restrictions for each item, characters of special languages might not be equated as intended.

An important implementation specification in this function is the fact that the lowercase conversion processing of strings is limited only to the moment when the calculation of the internal match determination is performed.

The text of the difference result finally rendered on the screen and presented to the user is output while maintaining the original uppercase and lowercase relationships entered by the user in the input field exactly as they are.

This makes it possible to check the changed parts without visually impairing the format and notation of the original document.

The extracted text difference result is quantified as the number of added items, the number of deleted items, and a percentage indicating the overall match rate, which is displayed in real time on the status panel on the interface.

The "Match rate" metric displayed here does not indicate the semantic similarity of the sentences or the closeness of the context.

It purely represents an arithmetic calculation result based on the "item count ratio" of the tokens generated by the aforementioned division rules.

As a specific calculation formula, first, the number of items determined to perfectly match between the source and target arrays is obtained as the numerator.

Then, the total value combining all three numbers—the number of matched items, the number of added items, and the number of deleted items—is used as the denominator.

The ratio derived by this calculation is displayed as a percentage, rounded to an integer. For example, when comparing two lines composed of completely identical strings, the number of matched items is 2, and the added items and deleted items are 0, so the calculated match rate is 100%.

If there is input in only one text input area and the other is empty, or if all strings have been rewritten to completely different content, the matched item count becomes 0, so the displayed match rate shows 0%.

Also, if both the source and target input fields are empty strings, the match rate is exceptionally treated as 100% based on the determination that it is a state with absolutely no changes.

In actual visual representation, deleted items—that is, items that existed in the comparison source but do not exist in the comparison target—are highlighted with a red background color, and a minus (-) sign is displayed at the beginning of the element as a visual aid.

Conversely, added items—that is, items newly inserted in the comparison target—are highlighted with a green background color, and a plus (+) sign is appended.

Items that are determined to match without these differences being recognized do not have special background colors or signs added, and are designed to be displayed as they are in the standard normal text color.

Regarding the difference display format, you can seamlessly switch and check between two layouts using a tab interface according to user preference: "Split" mode and "Unified" mode.

When the Split display mode is selected, the source text (Original) is placed on the left side of the screen, and the target text (Modified) is placed on the right side, each divided into independent columns.

In this mode, deleted items are highlighted in red in the source view area, and added items are highlighted in green in the target view area.

For changed parts where a new line was added immediately after being deleted, such as when using the line-by-line comparison mode, appropriate blank lines are inserted into the opponent's column to align the vertical line position, allowing for smooth visual contrast side by side in the left and right columns.

However, a function to automatically synchronize and track the scroll positions of the left and right view areas is not currently implemented, so when checking differences in long texts, users must individually scroll the left and right views themselves to find the relevant section.

In the Unified display mode, the modification history is combined and displayed as a single continuous text stream. Because deleted red lines and added green lines are placed vertically adjacent, you can check the differences in a format similar to the changelists commonly seen in code reviews for software development.

Furthermore, the tool is equipped with a "Copy the difference" button to retrieve the extracted difference result to the clipboard.

This copy button is placed in the header section, and it is not exclusive to the Unified display mode; it can be used from either display mode, whether Split display or Unified display is currently selected.

The copying process will display a success notification after completion (await), and failure notifications are also supported.

When you use this button, the data is saved to the clipboard as plain text, with "+ " added to the beginning of added lines, "- " to deleted lines, and two blank spaces " " to unchanged lines.

Although the output format can be universally used as a general difference text, it is not perfectly identical to the specifications of the standard patch format used in Git and the like.

Myers' algorithm, adopted for difference extraction in this tool, has a characteristic where the computational workload increases rapidly for data with complexly intersecting changes or extremely long texts.

To prevent processing from becoming excessive, a cutoff limit is set for the internal search process. Specifically, it searches for the edit depth, and if a difference path is found within a depth of 500, it outputs the result as a success.

However, if the search depth exceeds 500, or if the number of operations in the internal loop reaches a work limit such as 2,000,000 times, the search process transitions to a fallback mode.

When this resource limit is reached, the tool simply outputs the combined result to the screen, treating all source text elements as "deleted" and all target text elements as "added".

This fallback is not guaranteed to represent the shortest possible difference. When a fallback occurs, a warning message in orange is displayed just below the UI input areas stating that the search limit has been reached.

In this state, because detailed identification of the changed parts has not been performed, the calculated match rate also no longer reflects the actual modification ratio.

The tool does not provide excessive speed guarantees. As a confirmation step to avoid exceeding the search limit, it is recommended to first divide the input data size or comparison range into smaller sections before running them through the tool again.

Also, by enabling the "Ignore case" or "Ignore leading and trailing margins" options, you can reduce unnecessary differences from the algorithm's perspective, which may prevent reaching the upper limit of operations.

By using the tool while understanding these limit values and fallback specifications—which are set in consideration of the load on the browser's operating environment—it is possible to achieve a smooth text comparison workflow.