Uncategorized

Compare Algorithm Outputs to Reveal Underlying DataSet Structure

Problem: Data has noise. Data sets must be cleansed, normalized and curated. But you never really can tell for sure which algorithms to apply unless you compare results.

Steps:

  1. Plan to generate different visualizations  of your data set,
  2. Plan to create and various transforms on your data set
  3. Run through a number of typical algorithms on each view of your dataset.

Purpose:  Clarify which data transformations will reveal the structure of the dataset that underlies your problem.

Uncategorized

OCR text classification

One of the main decisions we have to make upfront is once we apply OCR to incoming unstructured data how do we classify it according to ontologies in that specific business, domain or even industry.

Domain ontologies serve as metadata or mid ontologies where we can consolidate unstructured and structured data that may have very little overlapping or mapping ontological schemas but can be used to consolidate and join them for subsequent congregation aggregation and data lake consolidation efforts.