Businesses face the constant challenge of leveraging organisational (institutional) memory created over time by teams and individuals – often captured in documents such as contracts, manuals, transcripts, emails, notes, and more. Easy and intuitive access to this organisational memory (accumulated body of data, information, and knowledge) can unlock significant business value – for example by reducing decision cycle times through quick access, or by bringing insights that were earlier unavailable.

The three main facets of organisational memory are data, information, and knowledge. Data is a fact depicted as a figure or a statistic, while data in context—such as in a historical time frame—is information and knowledge is the relevant and objective information that helps in drawing conclusions. Knowledge is interpretative and predictive. Its deductive character allows a person with knowledge to understand and act on the implications of information.

Organisational memory resides within teams or individuals, or within documents. To remain accessible, organisational memory residing in teams or with individuals needs to be passed to others or be captured in documentation.

How accessible is documented organisational memory? Access to organisational data is pretty much a solved problem. Accessing historical organisational information requires additional processing and synthesising efforts. The most daunting challenge is to leverage or organisational knowledge. Knowledge is perspective dependent and evolves in the light of new information. Let us look at this problem from the perspective of a National Security agency.

National Security and Intelligence organisations generate many reports daily, containing mission critical and time-sensitive information. These reports are typically studied, analysed, and used to generate insights and later stored for records and future reference. Recent documents are likely to be stored digitally however historical data still exists in physical form. Retrieving insights from these archived documents requires tremendous manual effort and time. The problem is compounded by the sensitive nature of the information and strict controls on its accessibility.

A simple text report like “10 porters and 5 ponies seen moving with sacks and iron sheets from Alpha to Bravo at 290600” could have multiple contexts: operational, historical, seasonal etc. An analyst’s work would be facilitated is such information could be converted into structured format for slicing, dicing, temporal, and geographical correlational analysis. Can contemporary systems convert the below text into semantic, structured data which can be queried for insights?

“A JCB and dozer seem moving west of post Juliet at 0800. Five blasts heard between Juliet and Zulu between 1200 and 1500.” A logical counter-question is “Why should data be captured in an unstructured form in the first place”? Is it possible to fit a communication into a structure? Without going into the complexities of information theory the simple answer is yes: if we can predefine all that’s to be reported. A tightly controlled menu-driven input option is limiting and counter institutive, and thus a non-starter. This is obvious from the fact that unstructured textual reports remain the mainstay of organisational communication.

Returning to the question of converting unstructured data to structured form, contemporary statistical/ NN (Neural Network) systems produce uninspiring results, and they get further mired in the colossal data requirements, opacity, and brittleness of the system. What if a system could laboriously read all data/information line by line, understand it in the light of common sense, historical perspective, institutional knowledge, and within the framework of tweak-able rules, comes up with a structured output which can be queried, sliced, diced, geo-linked and displayed on a timeline. Voila! Symbolic-Knowledge based AI provides a framework to achieve all these asks. AUI Systems has applied knowledge based symbolic AI to extract hitherto unavailable insights from situation-reports for a National Defence Agency.

References: Institutional/Organisational Memory, Greedy, Brittle, Opaque, and Shallow: The Downsides to Deep Learning by Jason Pontin