The INTERSECT Federated Architecture for the Laboratory of the Future

Summary: The open Interconnected Science Ecosystem (INTERSECT) architecture connects scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud or the high-performance computing center to enable autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence driven design, discovery and evaluation. Its a novel approach consists of science use case design patterns, a system of systems architecture, and a microservice architecture.

Connecting scientific instruments and robot-controlled laboratories with computing and data resources at the edge, the Cloud, or the high-performance computing (HPC) center enables autonomous experiments, self-driving laboratories, smart manufacturing, and artificial intelligence (AI) driven design, discovery and evaluation. The goal is to autonomously collect, transfer, store, process, curate, and archive scientific data and reduce human-in-the-loop needs for controlling, steering and designing experiments.

Figure 1: The INTERSECT Interconnected Science Ecosystem (INTERSECT) Initiative at Oak Ridge National Laboratory enables autonomous experiments, self-driving laboratories, smart manufacturing, and AI-driven design, discovery and evaluation

A federated instrument-to-edge-to-center hardware/software ecosystem needs to provide (a) uniform interfaces that leverage community and custom software; (b) pluggability that permits adaptable solutions, reuse of existing solutions, and digital twins for testing and evaluation; and (c) an open architecture to enable adoption by science facilities world-wide. The Interconnected Science Ecosystem (INTERSECT) architecture (Figure 1) enables science breakthroughs using intelligent networked systems, instruments, and facilities for the research laboratory of the future. It roughly follows the U.S. Department of Defense Architecture Framework (DoDAF) with its different architectural viewpoints, such as (i) operational scenarios, (ii) composition, interconnectivity and context, (iii) services and their capabilities, (iv) policies, standards and guidance, and (v) capability. The major difference is that the INTERSECT open architecture splits these views over three different components: (1) science use case design patterns, (2) a system of systems (SoS) architecture, and (3) a microservice architecture.

Figure 2: Elements of the INTERSECT architecture in the context of the ecosystem for interconnected smart laboratories

Autonomous experiments, self-driving laboratories, smart manufacturing, and AI-driven design, discovery and evaluation are described as science use case design patterns that identify and abstract the involved hardware/software components and their interactions in terms of control, work and data flow. The basic template for a science use case design pattern is defined in a loop control problem paradigm. There are two classes in the catalog of science use case design patterns: strategic patterns and architectural patterns. Strategic patterns define high-level solution methods using experiment control architecture features at a very coarse granularity. Architectural patterns define more specific solution methods using hardware and software architecture features at a finer granularity. While the architectural patterns do inherit the features of certain parent strategic patterns, they also address additional problems that are not exposed at the high abstraction level of the strategic patterns. A specific solution may require pattern compositions.

Figure 3: Classification of INTERSECT science use case design patterns and their orthogonal relationship to the Integrated Research Infrastructure (IRI) science workflow execution patterns

The SoS architecture clarifies used terms, architectural elements, the interactions between them, and compliance. It decomposes the federated hardware/software ecosystem into smaller and less complex systems and components within these systems. It permits the development of individual systems and components with clearly defined interfaces, data formats and communication protocols. This not only separates concerns and functionality for reusability, but also promotes pluggability and extensibility with uniform protocols and system/component life cycles. Instead of developing individual monolithic solutions for each science use case, the SoS architecture provides one solution that can be easily adapted to different use cases using different compositions of systems. It offers operational and managerial independence of systems and of components within systems, geographical distribution with a physically distributed and federated ecosystem, emergent behavior based on the interplay between systems and components, and evolutionary development through pluggability and extensibility. Similar to the DoDAF, the SoS architecture offers different architectural viewpoints: a logical view, an operational view a user view, a data view, a physical view, and a standards view.

Figure 4: In the INTERSECT SoS architecture, infrastructure systems contain multiple logical systems and logical systems span aross multiple infrastructure systems, providing services

The microservice architecture maps the science use case design patterns to the SoS architecture with loosely coupled microservices and uniform interfaces. It defines microservice interaction patterns and provides a classification of microservices that includes microservice capabilities for infrastructure and experiment services. The microservices are defined to facilitate composition within the federated SoS Architecture. Infrastructure microservices represent common service functionality and capabilities, such as data management, computing, messaging, and workflow orchestration that are likely to be generally useful across many science ecosystems without the need for customization. Experiment-specific microservices, on the other hand, represent services whose implementation may require detailed application knowledge, such as experiment planning or steering services that require knowledge of experiment-specific control parameters and their associated constraints. The microservice architecture also clarifies orchestration and deployment of microservices.

Figure 5: The classification of INTERSECT microservices with capabilities for infrastructure and experiment services.

The INTERSECT federated architecture is currently used by a number of research experiments and laboratories. One example is autonomous additive manufacturing, a 3D metal printing process with a thermomechanical simulation in a live feedback loop to control the residual stress in a printed part to address a grand challenge — building parts that are ready and safe to use immediately (i.e., “born qualified”).

Figure 6: The INTERSECT autonomous additive manufacturing experiment uses a thermomechanical simulation in a live feedback loop to control the residual stress in a printed part

Latest INTERSECT architecture documentation: intersect-architecture.readthedocs.io

Research Projects

• 2021-…: An Open Federated Architecture for the Laboratory of the Future

Funding Sources

• Interconnected Science Ecosystem (INTERSECT) Initiative, Laboratory Directed Research and Development, Oak Ridge National Laboratory

Participating Institutions

• Oak Ridge National Laboratory

Peer-reviewed Conference Publications

1. Christian Engelmann and Suhas Somnath. Science Use Case Design Patterns for Autonomous Experiments. In Proceedings of the 28th European Conference on Pattern Languages of Programs (EuroPLoP) 2023, pages 1-14, Kloster Irsee, Germany, July 5-9, 2023. ACM Press, New York, NY, USA. ISBN 979-8-4007-0040-8. DOI 10.1145/3628034.3628060.
2. Christian Engelmann, Olga Kuchar, Swen Boehm, Michael J. Brim, Thomas Naughton, Suhas Somnath, Scott Atchley, Jack Lange, Ben Mintz, and Elke Arenholz. The INTERSECT Open Federated Architecture for the Laboratory of the Future. In Communications in Computer and Information Science (CCIS): Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation. 18th Smoky Mountains Computational Sciences & Engineering Conference (SMC) 2022, pages 173-190, August 24-25, 2022. Springer, Cham. ISBN 978-3-031-23605-1. DOI 10.1007/978-3-031-23606-8_11. Acceptance rate 32.4% (24/74).

Peer-reviewed Workshop Publications

1. Michael J. Brim, Lance Drane, Marshall McDonnell, Christian Engelmann, and Addi Malviya Thakur. A Microservices Architecture Toolkit for Interconnected Science Ecosystems. In Proceedings of the 37th International Conference on High Performance Computing, Networking, Storage and Analysis (SC) Workshops 2024: 19th Workshop on Workflows in Support of Large-Scale Science (WORKS) 2024, Atlanta, GA, USA, November 18, 2024. IEEE Computer Society, Los Alamitos, CA, USA. To appear.

Peer-reviewed Conference Posters

1. Christian Engelmann, Swen Boehm, Michael Brim, Jack Lange, Thomas Naughton, Patrick Widener, Ben Mintz, and Rohit Srivastava. INTERSECT: The Open Federated Architecture for the Laboratory of the Future. Poster at the 52nd International Conference on Parallel Processing (ICPP) 2023, Salt Lake City, UT, USA, August 7-10, 2023.

White Papers

1. Ryan Adamson and Christian Engelmann. Cybersecurity and Privacy for Instrument-to-Edge-to-Center Scientific Computing Ecosystems. White paper accepted at the U.S. Department of Energy's ASCR Workshop on Cybersecurity and Privacy for Scientific Computing Ecosystems, November 3-5, 2021.
2. Hal Finkel, Pete Beckman, Christian Engelmann, Shantenu Jha, and Jack Lange. Research Opportunities in Operating Systems for Scientific Edge Computing. White paper by the U.S. Department of Energy's ASCR Roundtable Discussions on Operating-Systems Research 2021, January 25, 2021.
3. Hal Finkel, Pete Beckman, Ron Brightwell, Rudi Eigenmann, Christian Engelmann, Roberto Gioiosa, Kamil Iskra, Shantenu Jha, Jack Lange, Tapasya Patki, and Kevin Pedretti. Research Opportunities in Operating Systems for High-Performance Scientific Computing. White paper by the U.S. Department of Energy's ASCR Roundtable Discussions on Operating-Systems Research 2021, January 25, 2021.

Technical Reports

1. Michael Brim and Christian Engelmann. INTERSECT Architecture Specification: Microservice Architecture (Version 0.9). Technical Report, ORNL/TM-2023/3171, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2333815.
2. Christian Engelmann and Suhas Somnath. INTERSECT Architecture Specification: Use Case Design Patterns (Version 0.9). Technical Report, ORNL/TM-2023/3133, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2229218.
3. Olga A. Kuchar, Swen Boehm, Thomas Naughton, Suhas Somnath, Ben Mintz, Jack Lange, Scott Atchley, Rohit Srivastava, and Patrick Widener. INTERSECT Architecture Specification: System-of-systems Architecture (Version 0.9). Technical Report, ORNL/TM-2023/3168, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2023. DOI 10.2172/2333813.
4. Michael Brim and Christian Engelmann. INTERSECT Architecture Specification: Microservice Architecture (Version 0.5). Technical Report, ORNL/TM-2022/2715, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1902805.
5. Christian Engelmann and Suhas Somnath. INTERSECT Architecture Specification: Use Case Design Patterns (Version 0.5). Technical Report, ORNL/TM-2022/2681, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1896984.
6. Olga A. Kuchar, Swen Boehm, Thomas Naughton, Suhas Somnath, Ben Mintz, Jack Lange, Scott Atchley, Rohit Srivastava, and Patrick Widener. INTERSECT Architecture Specification: System-of-systems Architecture (Version 0.5). Technical Report, ORNL/TM-2022/2717, Oak Ridge National Laboratory, Oak Ridge, TN, USA, September 1, 2022. DOI 10.2172/1968700.

Talks and Lectures

1. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT). Invited talk at the Hartree Centre, Science and Technology Facilities Council, Daresbury, UK, October 4, 2023.
2. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT) Architecture. Invited talk at the 20th Smoky Mountains Computational Sciences & Engineering Conference (SMC), Knoxville, TN, USA, August 21-23, 2023.
3. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT) Architecture. Seminar at the Leibniz Rechenzentrum (LRZ), Garching, Germany, July 10, 2023.
4. Christian Engelmann. The Interconnected Science Ecosystem (INTERSECT) Architecture. Invited talk at the 1st Ecosystems for Smart Autonomous Interconnected Labs (E-SAIL) Workshop, held in conjunction with the 38th ISC High Performance (ISC) 2023, Hamburg, Germany, May 25, 2023.
5. Ben Mintz, Christian Engelmann, Elke Arenholz, and Ryan Coffee. Enabling Self-Driven Experiments for Science through an Interconnected Science Ecosystem (INTERSECT). Panel at the 17th Smoky Mountains Computational Sciences & Engineering Conference (SMC), October 20, 2021.

Symbols: Abstract, Publication, Presentation, BibTeX Citation