Text difference (Diff) comparison/addition/deletion text extraction tool | ZeroTools

The core of the text difference extraction tool is an algorithm that mathematically minimizes and extracts the difference between two sets of strings.

This system uses Myers' minimum edit distance algorithm proposed by Eugene Myers to detect differences quickly and precisely.

This algorithm models the difference as a shortest path problem on a two-dimensional grid graph with two text strings placed on the X and Y axes, respectively.

Moving diagonally forward means matching elements in both texts, moving to the right means deleting text, and moving down means adding text.

There is a close relationship between the length of the longest common subsequence, or LCS, and the minimum edit distance, and by calculating both at the same time, it is possible to completely reconstruct the structural change history between texts.

In the search process, a greedy method-based wide-first search is applied to efficiently calculate all reachable terminal nodes with the same edit distance.

Even when the length of the text is N and M, the amount of calculation is kept to an order space proportional to the size of the difference, so even log files with a huge number of lines or huge source code can be parsed in milliseconds without blocking the main thread of the browser.

This allows users to receive an extremely fast response while inputting data, providing a stress-free difference confirmation environment.

The granularity required for text comparison varies greatly depending on the target format and purpose of use.

Therefore, this extraction engine implements a multi-layered parsing mechanism in three layers: line by word, character by character.

As a first step, the entire text is divided into line-by-line arrays and the hash value of each line is calculated to perform a fast global comparison.

Add/delete change flags are mapped here

In the second stage, the lines marked with change flags are tokenized using word boundary regular expressions to identify differences at the word level.

In this phase, it becomes possible to accurately separate and compare reserved words and operators in programming languages.

As a third step, we apply character-by-character LCS calculations to blocks that require more detailed analysis, highlighting minute differences such as typo corrections and particle changes.

Through these multi-layered parsing processes, it accurately recognizes logical changes rather than simple combinations of additions and deletions, and generates differential information objects in a format that is easiest for users to understand the context.

Each token retains offset position information within the original text, allowing extremely accurate alignment during subsequent rendering phases.

The differential information objects generated by the parsing mechanism need to be transformed into an intuitive visual representation.

This system is equipped with an advanced rendering engine that utilizes virtual DOM, and provides two types of split comparison rendering: inline display and side-by-side two-screen parallel display.

In side-by-side display, a synchronous scrolling algorithm synchronizes the left and right scroll positions, so when one viewport moves, the other dynamically follows the corresponding row position.

Even if the row heights are different, the line of sight is always kept horizontal by inserting dummy spacer blocks.

In addition, we use a hue design based on visual psychology for color highlighting to clearly indicate changes.

A red background is applied to the deleted text blocks as a warning color, and a green background is placed to the added text to indicate progress.

Furthermore, by overlaying similar colors with higher saturation for minute changes in words or characters, we have achieved hierarchical highlighting that allows you to clearly recognize changes to the entire line and local changes at the same time.

Sending data to external servers poses serious security risks when handling highly confidential text data such as source code, unpublished contract manuscripts, or internal company specifications.

To fundamentally solve this problem, the text difference extraction tool employs a completely local processing architecture that completes all parsing and rendering within the client-side web browser.

We are building a system in which data is expanded and discarded only on the user's local machine by using WebWorkers to execute differential calculations on a background thread.

Temporary cache memory usage is also limited to session storage, and memory management is thorough so that all differential information evaporates the moment the tab is closed.

This physically blocks the possibility of input text data leaking outside via the network, and it is guaranteed that even in zero trust network environments or projects that require strict information management, you can safely perform differential comparisons of confidential materials without having to worry about external API communication monitoring.

The text data that is compared in actual work is often created using different operating systems and various editors, so there are many cases where the handling of line feed codes and blank characters becomes a noise of difference.

To eliminate this noise, this tool implements advanced options to ignore whitespace and line breaks and a non-destructive character code normalization process.

Detects a mixture of carriage returns and line feeds in the text input stream and virtually unifies them into a single newline token as an internal representation.

It also applies a normalization filter that treats consecutive spaces and tab characters as a single white space just before the parsing phase.

At this time, in the parsing of programming languages ​​that depend on the depth of indentation, a heuristic judgment that protects only leading spaces is also executed at the same time.

The important point is that this normalization is performed only on the virtual buffer for calculating the difference, and the original text data itself that the user sees on the screen is not modified in any way, using a non-destructive model.

This makes it possible to check differences while maintaining the integrity of the original data while extracting only essential context changes.

The range of applications for text difference extraction tools ranges from development sites to legal departments.

In the field of source code review, it functions as a tool that complements the difference display of version control systems.

In particular, the word-by-word and character-by-character analysis functions demonstrate their true value for change histories that are difficult to track with standard GitDiff, such as batch replacement of variable names and reordering of function blocks during refactoring, and clearly separate logical changes from simple format changes.

In addition, when resolving conflicts caused by simultaneous editing by multiple people, it provides detailed visualization of which characters conflict and how.

On the other hand, side-by-side display is extremely effective when checking contract revisions and revising manuscripts for publications.

Legal personnel can place the previous draft and the latest draft side by side and check any changes in legal interpretation caused by the addition or deletion of clauses or even the change of a single particle through hierarchical highlights extracted in milliseconds.

This combination of advanced mathematical algorithms and visual rendering serves as a foundational tool that dramatically improves the accuracy and speed of document quality control processes in a variety of professions.