From search to substance: How DeepDive transforms search results into data for EDD

After DeepDive's search tools identify relevant sources across multiple languages and engines, the next challenge begins: retrieving and processing web pages efficiently. This step transforms cluttered, unstructured web content into clean, analysis-ready text.

For investigators and compliance teams conducting manual Enhanced Due Diligence (EDD), this process is painfully familiar: clicking through dozens of links, waiting for pages to load, scrolling past advertisements, and extracting relevant information from varying formats.

The challenges of manual web content extraction

Here's what makes manual web page review so time-consuming for compliance analysts and investigators:

• Modern website architecture: Today's websites rely heavily on JavaScript and dynamic loading, making information harder to extract systematically.
• Content buried in noise: Relevant information is often hidden beneath advertisements, menus, and auxiliary content.
• Inconsistent formatting: Every website organizes content differently, requiring constant adaptation by researchers.
• Throughput limitations: Manually processing hundreds of pages is time-consuming and error prone.
• Documentation burden: Tracking source URLs and maintaining the connection between information and source adds significant time.

DeepDive's intelligent web data retrieval system streamlines this process through five key capabilities:

1. Page rendering

DeepDive renders web pages much like a manual browser view would and captures dynamically loaded content that scraping tools would miss.  

The browser render emulation follows ethical access principles:

• Respects websites' robots.txt files.
• Does not circumvent security measures such as CAPTCHAs or paywalls
• Only accesses content that is freely available to any visitor
• Focuses on factual, publicly disclosed information relevant to legitimate due diligence and investigation purposes

Our sophisticated HTML-to-Text conversion identifies content zones to distinguish between primary content, navigation elements, and advertisements. It removes the clutter and eliminates repetitive elements like headers, footers, and menus. What remains is normalised with formatting that standardises spacing, breaks, and character encoding.

The result is a clean text file containing only what matters—precisely what a human analyst should focus on, undistracted by noisy adverts, but produced automatically and consistently across all sources.

While stripping away clutter, DeepDive maintains critical context. Date, author, and publisher information is preserved. Important structural elements like headings and paragraph breaks are maintained.

The extraction process also aligns with Data Protection principles and DeepDive's commitment to data minimisation, ensuring that only information necessary for the specific purpose is processed.

3. Preparation for Natural Language Processing (NLP) analysis

The system optimizes the extracted content for the next stage of processing.

1. Language detection: Automatic identification of the content's language
2. Text normalization: Handling of special characters, acronyms, and formatting variations for easy NLP
3. Benchmarking: Extraction accuracy is benchmarked against human-reviewed samples

 4.Multi-language NLP

DeepDive processes text in any language or alphabet without requiring translation first:

• Native language analysis: Extracts meaning directly from source language rather than relying on translation
• Contextual understanding: Preserves cultural and linguistic nuances that might be lost in translation
• Character set handling: Processes non-Latin scripts such as Cyrillic, Arabic, Chinese, and others with equal efficiency

This capability ensures comprehensive coverage regardless of the subject's geographical

footprint, or the languages used in source documents.

5.Named Entity Recognition (NER)

DeepDive's NLP identifies and categorizes key elements within text:

• People: Full names, partial names, titles, roles, and aliases
• Organizations: Companies, government bodies, NGOs, SOE’s and other entities
• Locations: Countries, cities, addresses, and regions
• Values: Monetary amounts, percentages, dates, and other numeric data
• Events: Transactions, legal proceedings, appointments, and other significant occurrences

This creates a structured layer of meaning over raw text, transforming unstructured content into categorized data points that can be systematically analysed.

The Knowledge Engine…beyond NLP

DeepDive's NLP capabilities create the foundation for the next critical stages:

• Entity resolution to filter out false positives and focus on the correct individual
• Body of Knowledge creation through LLM-powered statement extraction
• Report generation with full source citations and structured sections
• Interactive chatbot interrogation of the structured data

 By transforming the overwhelming process of content analysis into a structured, systematic procedure, DeepDive enables compliance teams and investigators to focus on what matters: making informed decisions based on comprehensive intelligence.

 The result? Investigations that extract more meaningful information from source documents, in less time, with greater consistency than manual methods could ever achieve.

Want to read more?  Entity Resolution: How DeepDive eliminates false positives