Bringing the single intelligence environment to life

This is the story and learning lessons of Janes successful transformation from a legacy publisher into a modern data provider in under three years. Exploring the triumphs and challenges of building an assured platform-agnostic OSINT asset of 50 million highly interconnected intelligence objects that can be utilised within a broad range of low- and high-side systems. To achieve this Janes brought together technology, tradecraft, and data engineering disciplines to deliver a transformational change. And yes, it uses Graphs and AI… 

In 2021 Janes started a transformational journey to create a state-of-the-art platform for the management of open-source intelligence, with the goal of creating an interconnected data asset that could be used across a range of low- and high-side systems.

Today, the Janes Single Intelligence Environment is fully operational: used by over 250 Janes analysts daily. It powers our new customer portal and feeds interconnected object-centric OSINT into customer high-side systems and third^-party solutions such as ESRI, BAE GXP, and i2. The solution contains millions of intelligence objects and over 180 million relationships connecting them, aiding analysis and insight, and was recently awarded Data Project of the Year at the BCS UK IT Industry Awards.

Delivering the Single Intelligence Environment

To deliver this capability, Janes had to transform both its use of technology and its organisation. The journey was executed in three phases:

Phase 1: Let there be graph – The first step was to implement an RDF Graph to interconnect our existing data assets through an ontological model. This created an intelligence schema and data payload that could be loaded into customer systems via API.

Phase 2 (A): Committing to a core data platform – The ‘heavy lift’ was building a core object-centric data platform that could create, manage, and orchestrate Janes intelligence to deliver a ‘higher-quality’ output. This platform sources external information; processes it leveraging combined analyst/AI augmented workflows; creating analyst-led research, which is structured, categorised, and assembled into OSINT objects within a rich, tightly defined object schema.

Phase 2 (B): Transforming our technical capability – To achieve this we undertook a digital transformation, moving to a disciplined enterprise agile methodology that emphasised hybrid teaming of analyst, data, and technology staff; discovery and learning; and iterative development.  Goal-driven rather than scope-driven delivery enabled us to learn through doing, resulting in optimised outcomes at each stage of the development.

Phase 3: A dynamic new portal – Our final phase is to create a new portal that fully leverages the interconnected asset to provide highly efficient analyst discovery, analysis, and presentation capabilities to users. With dynamic object-driven navigation and search, this portal showcases the power of the asset and enables users to find and interact with information in new ways.

Some learning lessons – critical success enablers

1. We had to transform ourselves before we could transform our solutions. Specifically, Janes established new ways of working using hybrid teams, leveraging agile development techniques, focusing on an evolutionary learning mindset. 
2. To enable hybrid data/technology design processes we needed to break down departmental barriers between research, data, and technical teams.
3. Focus on building the data management regime – data schema, inter- and intra-object validation rules, data classification etc. – is a challenge and needs to be matured before iterative code development commences.
4. Use the right technology for each problem – we believe relational databases are not well suited to OSINT – we use a mix of graph databases, object databases, hierarchical databases, and serverless compute. We use each for the specific capability they are well suited to.
5. We try to use proven generic rather than esoteric technologies where we can. We seek to overlay techniques and capabilities to maximise cost/performance outcomes.
6. We use a variety of techniques to identify and interconnect data (rather than relying on one), leveraging analyst definition, ML/AI, and graph inference – in that order of importance.
7. An evolutionary mindset – build something ‘good enough’ and then iteratively innovate to improve and expand. Our journey is not complete – rather this is our new way of working.

The outcomes

Today, Janes is a business operating within the Single Intelligence Environment, with all analysts using it daily to create research, using a simple, easy-to-use interface, with their research activities focused on their sphere of expertise. The platform then post-processes this high-quality research to automatically categorise, link to a defined object, interconnect, and then package into products for consumption by users.  This delivers the following key outcomes:

1. A highly leveragable, interconnected data asset that can be used by our customers in a wide variety of intelligence scenarios.
2. A highly optimised internal multimedia research tool for use by our analysts that manages quality, lifecycle management, and traceability back to source.
3. Out-of-the-box integrations to many leading intelligence platforms.

   Janes believes it has managed to deliver something exceptional through disciplined execution in an unpredictable engineering challenge, through self-reflection and transformation, leveraging its strengths in tradecraft, data analysis, and technology. The current platform creates a base for long-term evolution, being both extensible in terms of function and expandable in terms of data coverage.