//// ******************************************************************************* * Copyright (c) 2019 Contributors to the Eclipse Foundation * * See the NOTICE file(s) distributed with this work for additional * information regarding copyright ownership. * * This program and the accompanying materials are made available under the * terms of the Eclipse Public License v. 2.0 which is available at * http://www.eclipse.org/legal/epl-2.0. * * SPDX-License-Identifier: EPL-2.0 ******************************************************************************* //// openKonsequenz - Architecture of the module 'Contact Base Data' =============================================================== :Author: Frank Dietrich :Email: frank.dietrich@pta.de :Date: 2019-12-17 :Revision: 1 :icons: :source-highlighter: highlightjs :highlightjs-theme: solarized_dark :imagesdir: ../img :iconsdir: ../img/icons This documentation is based on the ARC42-Template (v7.0): == Introduction and Goals === Requirements Overview Many user modules of an openKONSEQUENZ installion need contact datas for their daily business. Furthermore they have to fulfil the regulatory requirement of the General Data Protection Regulation (GDPR). This core module 'Contact Base Data' provides a central component for managing contact datas including the crucial functionality of GDPR. The full requirements of the module 'Contact Base Data' (in German: Modul 'Kontaktstammdaten') is described in the document * "Anforderungsspezifikation Modul Kontaktstammdaten" version 1.2 / 07-11-2019. === Quality Goals The module 'Contact Base Data' represents a core module that is based on the architecture platform of openKONSEQUENZ. The main quality goals of the platform are: * *Flexibility* The reference platform shall grant that different systems and modules from different vendors/developers can interact and interoperate, and may be exchanged or recombined. * *Availability* All platform modules that are running on the platform can only be as available as the platform same for user modules that are based on platform modules. * *Maintainability* (and testability as part of maintainability) The platform and its platform modules shall be used longer than 15 years. * *Integration performance* New implemented functionality of oK own modules and external modules shall be included fast / automatically. * *Security* The platform and its modules need to underly security-by-design The main quality goals of the core module Contact Base Data are: * *Functionality* The core module must fulfil the functional requirements mentioned in the section before * *Ergonomics* The web interface must be realized according to oK-GUI-Styleguide. * *Good documentation* (i.e. code and architecture documentation) makes code changes easier and automatic tests facilitate rigorous verification. * *Modifiability* (and testability as part of modifiability) * *Integration performance* The core module must be easy integratable in different production environments. The following documents contain the quality goals in detail: * Architecture Committee Handbook v1.6.0 from 10-07-2019 * Quality Committee Handbook v2.0.1 from 15-10-2018 The architecture is based on the AC-Handbook. The quality demands are described in the QC-Handbook. Both specifications were fully complied with in the project, so that a high quality is given. The code quality regarding static code analysis and unit test code coverage on the backend and fronend sides are ensured by the use of sonarqube. The rule set and the qualtity gate are defined by the default, the so called "sonar way". The module 'Contact Base Data' is part of the Eclipse project 'Eclipse openK Core Modules'. This project bases on the Eclipse Public Licence 2.0. === Stakeholders .Stakeholders [options="header,footer"] |========================================================= |Role/Name|Contact|Expectations |Product Owner (represents the Distribution System Operators)|Gordon Pickfort, Rainer Fuhrmann|The software must fulfil their functional and nonfunctional requirements. |Module Developer|Michel Alessandrini, Jonas Tewolde, Frank Dietrich|All relevant business and technical information must be available for implementing the software. |External Reviewer (represents the AC/QC)|n.n.|The software and the documentation is realized according to the Quality and Architecture Handbook of openKONSEQUENZ. |External Reviewer (represents the Eclipse-Requirements)|n.n.|The software is licensed under the EPL 2.0. It must be validated that all requirements are fulfilled. |System Integrator|n.n.|A documentation for the integration of the module in the DSO specific environments must be available. |========================================================= == Architecture Constraints The main architecture constraints are: * *Public License* The module must be available under the “Eclipse Public License 2.0”. * *Standardization* The module must use the reference platform. * *Availability* The source code of the module must be accessible to any interested person/company. Therefore the project is published under the following repositories: * https://git.eclipse.org/r/openk-usermodules/org.eclipse.openk-usermodules.contactBaseData.backend * https://git.eclipse.org/r/openk-usermodules/org.eclipse.openk-usermodules.contactBaseData.frontend === Technical Constraints The following technical constraints are given: .Technical Contraints [options="header,footer"] |======================================================== |Component|Constraints |Base components of the reference platform a|* Application Server Tomcat * JPA EclipseLink * Database PostgreSQL |Enterprise service bus a|* ESB Talend * Communication via RESTful Webservices |Programming language frontend a|* Angular * Bootstrap * jQuery * REST/JSON Interfaces |GUI design a|* According to oK-GUI-Styleguide |Java QA environment a| * Sonarqube 5.6.6 |Programming language a|* Backend: Java 1.8 * Frontend: Angular 7+ (Javascript, Typescript, HTML5, CSS3) |IDE a|* Not restricted (Eclipse, IntelliJ, Microsoft Developer Studio, Microsoft Visual Code ...) |Build system a|* Backend: Maven * Frontend: NodeJS + Angular/cli |Libraries, frameworks, components a|* Used Libraries/Frameworks have to be compatible to the Eclipse Public License |Architecture Documentation a|* According ARC42-Template |======================================================== === Technical Dependencies ==== Modules The following modules are required to use the 'Contact Base Data': .Modules [options="header,footer"] |========================================================= |Name of the module|Purpose|Status of the module |'Auth&Auth'|Authentification and Authorization|available |========================================================= ==== Libraries The following libraries are used: .Libraries [options="header,footer"] |========================================================= |Name of the library|Version|Artefact-id|Usage|License|Tier |org.springframework.boot.spring-boot-starter-parent|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.springframework.boot.spring-boot-starter-data-jpa|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.springframework.boot.spring-boot-starter-oauth2-client|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.springframework.boot.spring-boot-starter-security|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.springframework.boot.spring-boot-starter-web|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.flywaydb.flyway-core|6.0.8 a| ||Apache License 2.0|Backend |org.springframework.cloud.spring-cloud-starter-openfeign|2.2.0.RELEASE a| ||Apache License 2.0|Backend |org.springframework.cloud.spring-cloud-starter-netflix-ribbon|2.2.0.RELEASE a| ||Apache License 2.0|Backend |org.keycloak.keycloak-core|3.4.2_Final a| ||Apache License 2.0|Backend |org.postgresql.postgresql|42.2.8 a| ||New BSD License|Backend |org.projectlombok.lombok|1.18.10 a| ||MIT|Backend |org.mapstruct.mapstruct-processor|1.2.0.Final a| ||Apache License 2.0|Backend |io.jsonwebtoken.jjwt|0.9.1 a| ||Apache License 2.0|Backend |io.springfox.springfox-swagger2|2.9.2 a| ||Apache License 2.0|Backend |io.springfox.springfox-swagger-ui|2.9.2 a| ||Apache License 2.0|Backend |org.springframework.boot.spring-boot-starter-test|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.springframework.security.spring-security-test|5.2.1.RELEASE a| ||Apache License 2.0|Backend |org.powermock.powermock-reflect|2.0.0 a| ||Apache License 2.0|Backend |com.h2database.h2|1.4.200 a| ||EPL|Backend |org.springframework.cloud.spring-cloud-dependencies|Hoxton.RELEASE a| ||Apache License 2.0|Backend |org.springframework.boot.spring-boot-maven-plugin|2.2.1.RELEASE a| ||Apache License 2.0|Backend |org.jacoco.jacoco-maven-plugin|0.7.9 a| ||EPL 2.0|Backend |org.sonarsource.scanner.maven.sonar-maven-plugin|3.2 a| ||LGPL 3.0|Backend |org.asciidoctor.asciidoctor-maven-plugin|1.5.3 a| ||Apache License 2.0|Backend |org.jruby.jruby-complete|9.0.0.0 a| ||EPL 2.0|Backend |org.asciidoctor.asciidoctorj-pdf|1.5.0-alpha.11 a| ||Apache 2.0|Backend |org.asciidoctor.asciidoctorj|1.5.4 a| ||Apache 2.0|Backend |org.asciidoctor.asciidoctorj-pdf|1.5.0-alpha.11 a| ||Apache 2.0|Backend |org.asciidoctor.asciidoctorj-diagram|1.5.4.1 a| ||Apache 2.0|Backend |Angular Font Awesome|3.1.2 a| ||MIT License|Frontend |@auth0/angular-jwt|3.0.1 a| ||MIT License|Frontend |font-awesome|4.7.0 a| ||MIT License|Frontend |@ngrx/core|1.2.0 a| ||MIT License|Frontend |@ngrx/effects|8-2-0 a| ||MIT License|Frontend |@ngrx/store|8.3.0 a| ||MIT License|Frontend |@ngrx/store-devtools|8.2.0 a| ||MIT License|Frontend |@ngx-translate/core|11.0.1 a| ||MIT License|Frontend |@ngx-translate/http-loader|4.0.0 a| ||MIT License|Frontend |ag-grid-angular|21.2.1 a| ||MIT License|Frontend |ag-grid-community|21.2.1 a| ||MIT License|Frontend |angular2-notifications|2.0.0 a| ||MIT License|Frontend |bootstrap|4.4.1 a| ||MIT License|Frontend |jquery|3.4.1 a| ||MIT License|Frontend |classlist.js|1.1.20150312 a| ||MIT License|Frontend |core-js|3.2.1 a| ||MIT License|Frontend |moment|2.24.0 a| ||MIT License|Frontend |ng2-popover|0.0.14 a| ||MIT License|Frontend |ngrx-forms|5.2.1 a| ||MIT License|Frontend |npm-install-peers|1.2.1 a| ||MIT License|Frontend |reselect|4.0.0 a| ||MIT License|Frontend |rxjs|6.5.3 a| ||MIT License|Frontend |rxjs-compat|6.5.4 a| ||MIT License|Frontend |ts-helpers|1.1.2 a| ||MIT License|Frontend |tslib|1.10.0 a| ||MIT License|Frontend |web-animations-js|2.3.2 a| ||MIT License|Frontend |zone.js|0.10.1 a| ||MIT License|Frontend |@swimlane/ngx-datatable|15.0.2 a| ||MIT License|Frontend |puppeteer|2.0.0 a| ||MIT License|Frontend |ngx-toastr|11.2.1 a| ||MIT License|Frontend |popper.js|1.16.0 a| ||MIT License|Frontend |@ng-bootstrap|5.1.5 a| ||MIT License|Frontend |========================================================= == System Scope and Context === Business Context The core module 'Contact Base Data' communicates via Restful Webservices with the follwowing modules: * *Core Module 'Auth & Auth'* The 'Contact Base Data' can only be used by authorized users. Therefore, it is essential to invoke the module 'Auth & Auth' for authorization and authentication purposes. === Technical Context The following aspects have to be taken into account for external communication of the module 'Contact Base Data': * RESTful web services are used as interface-technology. * Each external interface (interfaces between modules or external systems) has to be documented. * Dependencies of modules to services realized by other modules have to be specified and documented explicitly. The interfaces of the module 'Contact Base Data' are described in the interface documentation. === Solution Strategy The module 'Grid Failure Information' bases on a small microservice architecture, including an asynchronous messaging system. == Building Block View === Whitebox Overall System The module 'grid failure information' contains several components: . *SIT-Web-FE* - This component (SPA with Angular) provides two HTML pages: a table and a map with failure information. This component is in the DMZ and can be called up from the Internet. . *SIT-Web-Guard* - The SIT-Web-Guard (Java Spring Cloud) is a Zuul proxy (API gateway) that only forwards explicitly configured services. For the failure information tool, only certain services are available on the Internet that do not require authentication. Since the SIT-BE is protected by Spring Security, the SIT-Web-Guard procures guest authentication for forwarding. . *SIT-FE* - This component (SPA with Angular) provides the user interface for the failure information application. The SIT-FE receives its authentication when it is called from the PortalFE in the form of a JWT. . *SIT_BE* - The SIT-BE (Java Spring Boot Microservice) provides all CRUD services in the form of ReST services that the two frontends require. The SIT-BE is the only component that has access to the database. Every call is authorized against the PortalBE. This microservice also includes the JobManager subcomponent. . *JobManager* - The various imports and exports are controlled via the JobManager. The JobManager knows all available import and export jobs and can control them via the internal message bus (RabbitMQ internal). Because the communication between the JobManager and the jobs takes place via the message bus, the JM does not need to know their configuration and URLs. For this, the JM must ensure synchronization (incl. timeout behavior) for asynchronous message communication. . *Import- and export jobs* - All jobs provide a uniform (MessageBus) interface to start a job, return the result of a job, or to provide information about the respective job (name, version, timeout, status, etc.). The job manager cyclically requests all configured jobs to send current status information to it. The interfaces of the individual jobs "outside" can be very different (file system, Internet, message queue, etc.) .Architecture of the grid failure information tool [options="header,footer"] image::architectureSIT.png[] ==== contactBaseDataFE This component implements the presentation logic for the *contact-base-data*-module using the *Angular*-TypeScript framework. The Frontend is a so called *Single Page Application* (SPA) because it behaves like a single HTML-page. ==== contact-base-data.jar (backend tier) This component implements the business functionality of the contact base data. And it provides services, that the contactBaseDataFE can use the functions in the frontend. The "spring boot/spring cloud" framework is used to implement this application. ==== ContactBaseDataDev-DB (Database tier) This component stores the data of the contact base data. It provides an interface to the contact-base-data.jar to create or change data in the database. The ContactBaseDataDev-DB runs on a Postgres DBMS. (The decision to use the Postgres DBMS was made by the openKONSEQUENZ architecture committee) === Level 2 ==== ContactBaseDataFE (frontend tier) The frontend component implements the concept of a single-page application (SPA). The framework used is Angular5. It divides the contactBaseDataFE into three layers: . *Components* - The components (pages, lists, dialogs, common comp.) represent the presentation layer and the control layer. A component contains the control logic (.ts-file), an HTML-fragment as presentation description (.html-file) and a style definition (.css-file). . *Services* - The service component communicates with the interfaces of the backend via HTTP requests by using the model component. . *Model* - The model corresponds to the view-model of the backend tier. .Frontend tier [options="header,footer"] [plantuml] ---- node contactBaseData_Frontend { component Model node Components { component "Pages" component Lists component "Common Components" } component Services Components --> Services Components --> Model Services --> Model } node "Contact Base Data Backend (simplified)" { component RestService component ViewModel_API__DTO } Services .. RestService Model .. ViewModel_API__DTO RestService --> ViewModel_API__DTO ---- ==== contact-base-data.jar (backend tier) The backend tier contains five components which can be summarized in three layers: . *Presentation layer* - Represented by .. REST-Srv .. View model/DTO . *Controller layer* - Represented by .. Controller .. Service . *Model layer* - Represented by .. Repository .. Model .Backend tier [options="header,footer"] [plantuml] ---- node "Contact Base Data Backend" { component Model component RestService component ViewModel_DTO component Controller component Service component Repository RestService --> ViewModel_DTO RestService --> Controller Controller --> Service Service --> Repository Repository --> Model } node DBMS { component ContactBaseDataDB } Repository --> ContactBaseDataDB ---- ==== ContactBaseData-DB (database tier) The ContactBaseData-DB is realized as a relational database system. .Database tier [options="header,footer"] [plantuml] ---- node DBMS { component ContactBaseDataDB } ---- ==== Program Configuration == Runtime view === Login / authentication There is no login page, since the openK-Portal-Application is responsible for authentication and the whole SSO (single sign on) process. Therefore the application has to be started by providing a valid authentication token. This token is a JWT (JSON Web Token). .contactBaseData application is called by the *portal* application. The User is already logged in [plantuml] .... actor User participant PortalFrontend participant PortalBackend participant ContactBaseDataFrontend entity ContactBaseDataStorage participant ContactBaseDataBackend User->PortalFrontend: Start ContactBaseData(JWT) PortalFrontend->ContactBaseDataFrontend: nav. to frontend-URL with JWT ContactBaseDataFrontend->ContactBaseDataStorage: Extract JWT and store token in session ... some delay ... ContactBaseDataFrontend->ContactBaseDataBackend: Call any secured service with JWT group Call secured service ContactBaseDataBackend->PortalBackend: "/checkAut(JWT)" group Authorization succeeded ContactBaseDataBackend->ContactBaseDataBackend: run service ContactBaseDataBackend->ContactBaseDataFrontend: return service result end group Authorization failed ContactBaseDataBackend->ContactBaseDataFrontend: return HTTP Code 401 end end .... === Deployment of the application components ==== Deployment of the frontend TODO: ==== Deployment of the backend TODO: ==== Deployment of the database The component "Flyway" is used to make to distribute structural or content related changes to the database. The database is built out of the scripts in the directory "db/migrations". Every sql script contains the complete db script for the contact base data database (in different versions). The highest version number indicates the currently valid script. ==== Configuration of the system ===== DB based configuration TODO: ===== Configuration of the contact base data backend The backend service is configured in the * .yaml files, which are located in the JAR file. This yml-file can be divided into different configuration profiles. When starting the backend-service one has the possibility to specify the active profile * *spring.datasource* configuration section for the database connection * *flyway.enabled* If enabled=true then the database migrations will automatically performed when starting the application (this parameter should normally be set to "false" * *server.max-http-header-size* Maximum size for the http-headers * *jwt.tokenHeader* Name of the http-header which carries the authentication-token. (should be "Authorization") * *jwt.useStaticJwt* If set to "true" then the backend will use *jwt.staticJwt* as Authorization-token. (This won't work for calls to other modules like the Auth'n'Auth-Modul, because the token will be out of date) * *authNAuthService.ribbon.listOfServers* Here one can configure the base url to the Auth'n'Auth-Service === CI- and CD-Components ==== GIT-Repository Backend: https://git.eclipse.org/c/openk-usermodules/org.eclipse.openk-usermodules.contactBaseData.backend.git/ Frontend: https://git.eclipse.org/c/openk-usermodules/org.eclipse.openk-usermodules.contactBaseData.frontend.git/ === Continuous deployment The continuous deployment is realized on two platforms: * the development platform (Dev-Environment) * the quality platform (Q-Environment) The automatic deployment on both of the environments is directly linked to the branches on the GIT-repositories: . "SNAPSHOT" or "DEVELOP" . "MASTER" or "TRUNC" The running development is exclusively made on the Snapshot-Branch. Every time a developer checks in (pushes) code to the repository, an automatic build starts on the hudson ci-server. If the Snapshot-build is successful, then the result of that build is directly deployed on the Dev-environment. At the end of a scrum sprint or when a big user story is realized, all the code changes are merged from the *Snapshot*-Branch to the *Trunc*. This automatically triggers the build and the deployment on the Q-environment. == Design decisions All architecture decisions are based on the Architecture Committee Handbook. There are no deviations. == Risks and Technical Debts (Currently there aren't any known issues) <<< == Glossary .Abbreviations and glossary terms [options="header,footer"] |======================================================== |Short|Long|German|Description |AC|Architecture Committee|Architektur-Komittee|Gives framework and constraints according to architecture for oK projects. |CNCU|Central Network Control Unit|| |DAO|Data Access Objects|| |DTO|Data Transfer Object|| |DSO|Distribution System Operator|Verteilnetz-betreiber (VNB)|Manages the distribution network for energy, gas or water. |EPL|Eclipse Public License||Underlying license model for Eclipse projects like contact-base-data@openK |ESB|Enterprise Service Bus||Central instance to exchange data to overcome point-to-point connections. |oK|openKONSEQUENZ|openKONSEQUENZ|Name of the consortium of DSOs |QC|Quality Committee|Qualitätskomitee|Gives framework and constraints according to quality for oK projects. |SCADA|Supervisory Control and Data Acquisition|Netzleitsystem|System, that allows DSOs view/control actual parameters of their power grid. |========================================================