Expert Insights

Unstructured Data is Everywhere

Document processing is a critical function that directly affects the efficiency of all businesses, regardless of industry or size. Yet one of the biggest challenges these companies face is dealing with large amounts of unstructured data.

According to an estimate from Computer World magazine, unstructured data might account for more than 70%–80% of all data in organizations.

The biggest problem with unstructured data clearly is not lack of abundance, but rather a lack of available tools to extract the business value from such a readily available resource. Within the current digital landscape, the ability to process document data quickly and correctly is essential for enterprises to remain competitive – this means being able to locate and interpret unstructured data from places like invoices, emails, images, log files, and even video and audio files. For businesses seeking a solution, upgrading your automations with high-powered Intelligent Document Processing technologies is the key to efficiency and optimization of their business operations.

A New Approach

Traditionally, OCR technology has been used with RPA to digitize offline data into a machine readable format for document processing, but the technology has its constraints. OCR cannot “understand” data on its own – it can digitize text from documents but is incapable of actually interpreting this data without additional technology or tooling. Another limitation is that OCR is only as effective as the quality of the document being processed, meaning the output text of OCR solutions may contain typos, recognition mistakes (these arise when the software cannot distinguish between characters that appear similar, like ‘D’ versus ‘0’), non-text symbols, and other inaccuracies. Challenges with OCR are magnified by documents that vary in structure and format, since traditional OCR document processing solutions depend upon templates and positional extraction of data, rendering it ineffective for semi-structured and unstructured document processing. Ultimately, pure OCR lacks the reliability and cognitive capabilities to handle complex processes. To enable the most efficient and accurate document processing, OCR and RPA must be combined with the power of AI.

This cognitive approach offers more flexible document processing, increased operational efficiency, reduced risk of human error – and thus increased efficiency, and end-to-end automation.

Why AI?

To clear up a common misconception, Intelligent Document Processing (IDP) technologies are not the same as OCR. Rather, Intelligent Document Processing is an advanced technology that uses a combination of traditional OCR and Machine Learning Models to digitize and extract text from documents. It enables applications to recognize the type of document and understand which pieces of information are relevant.

Most IDP platforms provide out of the box, pretrained models for common document types such as invoices, receipts, utility bills etc. For other document types, these platforms provide the ability for humans to label data and train a custom model. These Machine Learning models allow RPA bots to look deeper into documents to classify document types and extract relevant information in a structured manner. These models can be trained to find certain pieces of data even when no static rules or templates can be applied. This is particularly advantageous for businesses with a wider variety of document processing use cases. While templates are a fine place to start, they are static. Applying cognitive technology lets the business create its own models for finding specific fields within a document based on details like recurring patterns or position within the document. This training, however, is hidden from the user’s view. All a human does is label the relevant fields on a document, and the AI model infers the rules. Once enough documents are labelled, the AI Model can begin to extract relevant information, and it can deal with changes and variants in your documents.

One key feature that makes Intelligent Document Platforms so valuable is that it includes a feedback loop. UiPath’s IDP Platform known as “Document Understanding”(DU) implements this using what they call a Human Validation Station. DU sits within AI Center, a product of the UiPath enterprise platform that enables deploying, consuming, improving and managing ML models. The Validation Station feature enables users to review and correct document classification and data extraction results, retraining the model to make it smarter over time and ensuring that the accuracy of your data continually improves.

We have done multiple DU implementations for clients across industries, one of the most notable being a Major North American Retailer struggling with processing a large volume of AP invoices. The solution for this client included pretrained ML models for invoices. The Dubotz team retrained these models with UiPath AI Fabric to be able to capture additional information fields specific to the vendors. With a 95% confidence score for the extracted data, 93% of the invoices were able to be processed without needing any manual inspection. For this client, the ML feedback loop was key in ensuring the speedy and accurate processing of up to 7,000 invoices per month.

Beyond Invoice Processing

While invoice processing remains one of the most common use cases for this high-powered document processing technology, there’s no reason its capabilities should be confined solely to a company’s accounting and finance functions. In the past year we’ve seen major a boom in RPA adoption for the public sector. Organizations in this sector are leveraging Intelligent Document Processing Platforms to bring greater efficiency to processing a varying number of use cases, including residency verification (utility bills and lease documents), identity verification (passports and driver’s licenses), and income verification (tax forms and pay stubs) to name a few. In healthcare and medicine, organizations are applying the technology for smarter processing of health records, drug prescriptions, medical forms, and bills. In the financial services and insurance area we are seeing an increase in firms using Intelligent Document Processing for IRS forms, loan application and mortgage processing, and various compliance-related processes.

Regardless of industry or company size, there is tremendous value in applying Intelligent Document Processing as you scale your document processing efforts. Given the sheer quantity of unstructured data in the business world and the increasing complexity of document-based enterprise processes, AI makes the execution of these tasks possible. If you’re interested in learning more about how AI can help improve and scale your automations, click here.