AI-powered Document Search Service for B2B Customers

Ability to open and parse closed-format documents
Providing a universal, comprehensive document search and analysis service for organizations across various industries necessitates the ability to open and parse closed-format documents, including complex and proprietary file formats.

Effective processing of unstructured data
For analyzing diverse file types on a corporate scale, the solution must excel in managing semi-structured or unstructured files, ensuring comprehensive and accurate data handling.

Ability to work with domain-specific data (specific terminology and style)
Given the broad application scope, the solution needs to efficiently process and adapt to business-specific data, catering to the unique requirements of each corporate client.

Ability to open and parse closed-format documents
The initial step in our solution involved designing and developing specialized connectors to access various closed-format files, each requiring a unique approach. Our engineering team focused on the distinct characteristics of each file type, tailoring these connectors accordingly. For example, in the case of CAD files, a detailed analysis led our engineers to choose VBScript, despite its limited documentation, to prioritize developmental flexibility and optimal outcomes. Each connector underwent rigorous testing, a step taken to ensure reliability and efficiency, where real-world scenarios were simulated to validate their performance and adaptability in diverse industrial environments.

Effective processing of unstructured data
The next critical step in our solution was the effective processing of unstructured data. To achieve this, our team thoroughly tested several Large Language Models (LLMs) to identify the most effective one for handling unstructured data types. Alongside this, we developed specialized parsers designed to interpret and organize data according to various document templates, enhancing our data analysis precision.

Selecting and customizing a set of chunkers, which are specialized tools for breaking text into manageable pieces, was another crucial step in the development process. These chunkers underwent rigorous testing to identify the optimal ones for each document template, a process that significantly improved efficiency. After fine tuning, our service became capable of effectively processing large sets of unstructured data, such as 1000-page PDFs – a task that extends beyond the capabilities of ChatGPT on its own.

Ability to work with business-specific data
Equally important in our development process was ensuring the effective ability to process business-specific data. To achieve optimal effectiveness for each type of business-specific data, we developed corresponding templates and made special adjustments. These adjustments were crucial to account for each business area's specific terminology and nuances.

Chatbot functionality
Finally, we integrated our service with OpenAI's ChatGPT API through Azure Endpoint to process queries and return answers. This integration leveraged ChatGPT's advanced natural language processing capabilities, enabling our service to comprehend and respond to complex inquiries with increased accuracy and contextual relevance. The Azure Endpoint offered a robust and scalable platform, ensuring seamless communication between our service and the ChatGPT API, thereby enhancing the overall efficiency and effectiveness of the query-handling process.

The resulting AI-powered document search and analysis service demonstrates excellent answer relevancy and accuracy, significantly enhancing the document management capabilities of our clients across various business sectors. The service continually evolves with each new client, expanding its support for a broader range of closed-format documents and enhancing our insights into different business sectors. This progressive enhancement ensures that the service remains adaptable and forward-looking, ready to meet present and future document management and analysis demands.