# KubeRocketCI Documentation > Explore KubeRocketCI Documentation for CI/CD, operator and user guides, quick start, and API. Learn how to enhance your DevOps practices with KubeRocketCI. This file contains all documentation content in a single document following the llmstxt.org standard. ## About the KubeRocketCI Platform # About the KubeRocketCI Platform **KubeRocketCI (previously known as EPAM Delivery Platform)** is an **open-source** cloud-agnostic SaaS/PaaS solution for software development, licensed under **Apache License 2.0**. It provides a pre-defined set of CI/CD patterns and tools, which allow a user to start product development quickly with established **code review**, **release**, **versioning**, **branching**, **build** processes. These processes include static code analysis, security checks, linters, validators, dynamic feature environments provisioning. Platform consolidates the top Open-Source CI/CD tools by running them on Kubernetes/OpenShift, enabling web/app development in isolated (on-prem) or cloud environments. KubeRocketCI, which is also called **"The Rocket"**, is a platform that allows shortening the time that is passed before an active development can start from several months to several hours. The platform consists of the following blocks: - The platform is based on managed infrastructure and container orchestration - Security covering authentication, authorization, and SSO for platform services - Development and testing toolset - Well-established engineering process and EPAM practices (EngX) reflected in CI/CD pipelines and delivery analytics - A set of pre-configured pipelines for different types of applications (polyglot microservices) - Observability stack ## Features - Deployed and configured CI/CD toolset ([Tekton](https://tekton.dev/), [ArgoCD](https://argoproj.github.io/cd/), [Nexus Repository Manager](https://help.sonatype.com/en/sonatype-nexus-repository.html), [SonarQube](https://www.sonarsource.com/), [DefectDojo](https://www.defectdojo.org/), [Dependency-Track](https://dependencytrack.org/)). - [GitHub](https://about.gitlab.com/features/)(by default), [GitLab](https://about.gitlab.com/features/), [Bitbucket](https://www.atlassian.com/software/bitbucket/features/pipelines) or [Gerrit](https://gerrit-review.googlesource.com/Documentation/intro-quick.html). - [Tekton](./operator-guide/install-tekton.md) is a pipeline orchestrator. - [CI pipelines](./user-guide/index.md) for polyglot applications: |Language|Framework|Build Tool|Application|Library|Autotest| |:-|:-:|:-:|:-:|:-:|:-:| |Java|Java 17, Java 21, Java 25|Gradle, Maven|:white_check_mark:|:white_check_mark:|:white_check_mark:| |Python|Python 3.8, FastAPI, Flask|Python|:white_check_mark:|:white_check_mark:|| |C#|.Net 3.1, .Net 6.0|.Net|:white_check_mark:|:white_check_mark:|| |C|None|Make,CMake|:white_check_mark:||| |C++|None|Make,CMake|:white_check_mark:||| |Go|Beego, Gin, Operator SDK|Go|:white_check_mark:||| |JavaScript|React, Vue, Angular, Express, Next.js, Antora|NPM, PNPM|:white_check_mark:|:white_check_mark:|| |HCL|Terraform|Terraform||:white_check_mark:|| |Helm|Helm, Pipeline|Helm||:white_check_mark:|| |Groovy|Codenarc|Codenarc||:white_check_mark:|| |Rego|OPA|OPA||:white_check_mark:|| |Container|Docker|Kaniko||:white_check_mark:|| :::note As of KubeRocketCI version 3.12, Java 8 and Java 11 frameworks have been deprecated. ::: - [Portal UI](./user-guide/index.md) as a single entry point - [Deployment Flows](./user-guide/add-cd-pipeline.md) for Microservice Deployment - Kubernetes native approach ([CRD, CR](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)) to declare CI/CD pipelines ## What's Inside KubeRocketCI KubeRocketCI offers a comprehensive solution for the entire software delivery lifecycle, from development to deploying production environments. It's designed to be suitable for all aspects of delivery, ensuring a seamless and efficient process. Platform modules are represented on a diagram below (more [on the Reference Architecture](./developer-guide/reference-architecture.md) page) ![KubeRocketCI Modules](./assets/kuberocketci-components.png "KubeRocketCI Modules") The architecture of KubeRocketCI is designed to support a broad range of delivery aspects effectively: 1. **Infrastructure as a Service (IaaS)**: The platform leverages managed infrastructure services, providing a robust foundation for container orchestration and other platform services. Central to our approach to IaaS is the adoption of the **Infrastructure as Code (IaC)** methodology. This methodology ensures that infrastructure is defined and managed through code, enabling the automation of infrastructure provisioning and management. 2. **GitOps Approach**: Embracing GitOps principles, KubeRocketCI ensures an efficient and reliable way to manage deployments and configurations using Git as the single source of truth. This approach is applied not only for deploying **business applications** but also for the robust **management of the platform itself**, utilizing an [add-ons methodology](https://github.com/epam/edp-cluster-add-ons). By treating both platform components and business applications as code, KubeRocketCI facilitates automated, transparent, and collaborative processes, significantly enhancing deployment velocity and platform governance. 3. **Container Orchestration and Centralized Services**: At its core, the platform utilizes Kubernetes/OpenShift for orchestrating containerized applications, coupled with centralized services to manage the lifecycle of these applications. 4. **Security**: Comprehensive security measures are in place, covering authentication, authorization, and Single Sign-On (SSO) for platform services, ensuring the safeguarding of your applications and data. On top of the primary architectural pillars, KubeRocketCI integrates several key components to enhance the CI/CD process: **CI/CD Components**: These components support various CI/CD functionalities, like artifacts storage and distribution (via Nexus or Artifactory), static code analysis (Sonar), and more. **Artifacts Management**: Representing the deliverables within the pipelines, artifacts can range from frontend and backend applications to functional and non-functional autotests, and workloads for 3rd party components. **Development and Production Deployment Flows**: Deployment Flows in KubeRocketCI maintain consistency across development and production, encapsulating artifacts with specific versions to ensure quality through the Software Development Life Cycle (SDLC). **Pipelines**: Covering the end-to-end CI/CD process, pipelines in KubeRocketCI automate the integration and delivery workflows, connecting the above components seamlessly. ### Technology Stack KubeRocketCI's technology stack is adaptable to most popular public clouds such as AWS, Azure, and GCP, with support for private/hybrid clouds based on OpenStack. The containerization layer is powered by Kubernetes or OpenShift, offering flexibility and choice in deployment options. Explore the KubeRocketCI technology stack diagram ![KubeRocketCI technology stack](./assets/kuberocketci-tech-stack.png "KubeRocketCI technology stack") The platform provides two main options for Kubernetes: - Managed Kubernetes in Public Clouds to avoid installation and management of Kubernetes cluster, and get all benefits of scaling, reliability of this solution; - OpenShift is a Platform as a Service on the top of Kubernetes from Red Hat. OpenShift is the default option for on-premise installation and it can be considered whether the _solution built on the top of KubeRocketCI_ should be **cloud-agnostic** or require **enterprise support**; There is **no limitation** to run KubeRocketCI on vanilla Kubernetes. ## Related Articles - [Quick Start](./quick-start/platform-installation.md) - [Basic Concepts](basic-concepts.md) - [Glossary](glossary.md) - [Supported Versions and Compatibility](supported-versions.md) --- ## Basic Concepts # Basic Concepts Consult [KubeRocketCI Glossary](./glossary.md) section for definitions mentioned on this page. The below table contains a list of capabilities provided by the platform. |Features|Description| |-|-| |**Cloud Agnostic**|Platform runs on any Kubernetes cluster, including those provided by Public Cloud Providers or deployed on-premises.| |**CI/CD for Microservices**|KubeRocketCI is designed to support CI/CD for [Microservices](https://microservices.io/) running as containerized applications inside a Kubernetes Cluster. It provides CI Pipelines for [different technology stacks](./about-platform.md#features).| |**Version Control System (VCS)**|The platform integrates with **GitHub**, **GitLab**, **Bitbucket**, and **Gerrit**.| |**Branching Strategy**|KubeRocketCI supports both [Trunk-based](https://trunkbaseddevelopment.com/) development and [GitHub/GitLab flow](https://docs.github.com/en/get-started/using-github/github-flow). It creates two Pipelines per Project branch: *Code Review* and *Build*.| |**Repository Structure**|By default, the platform provides a separate Git repository for each Project (also Codebase). It does not support *Monorepo* out of the box, but can be customized to align with the *Monorepo* approach.| |**Artifacts Versioning**|The platform supports two approaches for Artifacts versioning: - `default` (BRANCH-[DATETIME])- `SEMVER` (MAJOR.MINOR.PATCH-BUILD_ID), which follows [SemVer](https://semver.org/). You can also create custom versioning by implementing the `get-version` CI step.| |**Application Library**|KubeRocketCI offers baseline codebase templates for Microservices and Libraries, using the *create* strategy when onboarding new Projects.| |**CI Pipelines**|The platform provides CI Pipelines for [different technology stacks](./about-platform.md#features).| |**CD Pipelines**|KubeRocketCI allows you to design CD Pipelines (in Portal UI) for Microservices and defines the logic for artifact flow (promotion) from lower to higher environments. Artifact promotion can be performed automatically (*Autotests*), manually (*User Approval*), or using a combination of both.| |**Autotests**|KubeRocketCI provides a CI pipeline for autotests implemented in Java. These autotests can be used as *Quality Gates* in CD Pipelines.| |**Custom Pipeline Library**|You can extend KubeRocketCI pipelines by introducing a Custom Pipeline Library.| |**Dynamic Environments**|KubeRocketCI's CD Pipelines can create and destroy environments based on user requests, allowing for flexible and dynamic deployment of applications.| |**GitOps Approach**| The platform adopts a GitOps approach for CI/CD, using Git repositories as the single source of truth for infrastructure and application configuration. This practice enhances collaboration, simplifies the deployment process, and improves the auditability of changes.| |**Monitoring and Observability**| The platform includes an observability stack that enables logging, monitoring, and tracing of applications and infrastructure. This helps in identifying issues early and maintaining the health and performance of the deployed applications.| |**Security Covering Authentication, Authorization, and SSO for Platform Services**| KubeRocketCI implements robust security mechanisms, including authentication, authorization, and Single Sign-On (SSO) capabilities for all platform services. This ensures that access to the platform is secure and controlled, aligning with best practices for cloud-native application development.| |**Static Application Security Testing (SAST)**| KubeRocketCI integrates tools that perform [static analysis](operator-guide/devsecops/overview.md) of the source code to identify potential vulnerabilities early in the development process. This analysis is performed without executing the code and can pinpoint issues such as input validation errors, insecure dependencies, and other common vulnerabilities.| |**Container Registries**| KubeRocketCI demonstrates adaptability and interoperability with leading [container registries](user-guide/manage-container-registries.md), empowering developers to select the most suitable tools and services based on their specific project requirements. The integration process is designed to be as simple as possible, making it easy for developers to connect their projects with the desired registries. The incorporation of these registries is pivotal in upholding the security, effectiveness, and dependability of the CI/CD pipeline.| --- ## Compliance # Compliance The integrity of your deployments is our paramount commitment. We are devoted to strengthening our Kubernetes platform to comply with the most stringent security standards. Trust is the bedrock of our relationships, and we manifest this commitment by undergoing rigorous third-party audits to ensure compliance. We pledge unwavering support as you manage and deploy solutions within your environment, emphasizing security and reliability. Examine our compliance with various frameworks, laws, and regulations to understand our dedication to upholding robust security standards for the solutions you manage and deploy. [![KubeRocketCI Badge](assets/star-level1-badge.svg)](https://cloudsecurityalliance.org/star/registry/epam/services/epam-delivery-platform/) --- ## Annotations and Labels # Annotations and Labels KubeRocketCI uses labels to interact with various resources in a Kubernetes cluster. This guide details the resources, annotations, and labels used by the platform to streamline operations, enhance monitoring, and enforce governance. ## Labels The table below contains all the labels used in KubeRocketCI: | Label Key | Target Resources | Possible Values | Description | | :---------------------------------- |:---------------------------------| :----------------------------------------------------------------------------------| :-------------------------------------------------------------------------- | | app.edp.epam.com/secret-type | Secrets | `jira`, `nexus`, `sonar`, `defectdojo`, `dependency-track`,`repository` | Identifies the type of the secret. | | app.edp.epam.com/integration-secret | Secrets | `true` | Indicates if the secret is used for integration. | | app.edp.epam.com/integration | Codebase | `codemie` | Indicates that the codebase should be integrated with Codemie. When applied, the Codemie operator reconciles the codebase and creates a Codemie Application. | | app.edp.epam.com/codebase | PipelineRun, CodebaseImageStream | `` | Identifies the codebase associated with the PipelineRun and CodebaseImageStream. | | app.edp.epam.com/codebasebranch | PipelineRun, CodebaseImageStream | `` | Identifies the codebase branch associated with the PipelineRun and CodebaseImageStream. | | app.edp.epam.com/pipeline | PipelineRun, TaskRun | `` | Used by the KubeRocketCI to display autotests' status (on Deploy environment) | | app.edp.epam.com/triggertemplate | Pipeline, PipelineRun | `` | Allows to pull default parameters from the TriggerTemplate resource. | | app.edp.epam.com/pipelinetype | Pipeline, PipelineRun, TaskRun | `autotestRunner`, `build`, `review`, `deploy`, `clean` | Identifies the Pipeline type. | | app.edp.epam.com/parentPipelineRun | PipelineRun | `` | Used by the KubeRocketCI to display autotests' status (on Deploy environment) | | app.edp.epam.com/stage | PipelineRun, TaskRun | `` | Used by the KubeRocketCI to display autotests' status (on Deploy environment) | | app.edp.epam.com/branch | PipelineRun | `` | Identifies the branch associated with the PipelineRun. | | app.edp.epam.com/codebaseType | Codebase | `system`,`application` | Identifies the type of the codebase. | | app.edp.epam.com/systemType | Codebase | `gitops` | Identifies system repositories. | | app.edp.epam.com/gitServer | Ingress, EventListener, Trigger | `` | Identifies resources associated with the Git Server. On a Trigger, registers it with the Git Server's EventListener via its labelSelector. | | app.edp.epam.com/cdpipeline | PipelineRun, TaskRun | `` | Identifies Deployment Flow associated with the Pipeline Run. | | app.edp.epam.com/cdstage | PipelineRun, TaskRun | `` | Identifies Environment associated with the Pipeline Run. | | app.edp.epam.com/edit-protection | Codebase, CodebaseBranch, CDPipeline, Stage | `delete-update`, `delete`, `update` | When applied to a resource, protects a resource from deletion and/or modification. | ### Labels Usage in Secrets The table below shows what labels are used by specific secrets: | Secret Name | Labels | | :------------------- | :------------------------------------------------------------------------------------------- | | ci-argocd | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=argocd | | ci-defectdojo | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=defectdojo | | ci-dependency-track | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=dependency-track | | ci-jira | app.edp.epam.com/secret-type=jira | | ci-nexus | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=nexus | | ci-sonarqube | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=sonar | | gerrit-ciuser-sshkey | app.edp.epam.com/secret-type=repository | | kaniko-docker-config | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=registry | | regcred | app.edp.epam.com/integration-secret=true app.edp.epam.com/secret-type=registry | ### Labels Usage in Tekton Pipeline Runs The table below displays what labels are used in specific Tekton pipelines: | PipelineRun | Labels | | :----------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | review-pipeline | app.edp.epam.com/codebase: `` app.edp.epam.com/codebasebranch: ``-`` app.edp.epam.com/pipelinetype: `review` | | build-pipeline | app.edp.epam.com/codebase: `` app.edp.epam.com/codebasebranch: ``-`` app.edp.epam.com/pipelinetype: `build` | | autotest-runner-pipeline | app.edp.epam.com/pipeline: `` app.edp.epam.com/pipelinetype: `autotestRunner` app.edp.epam.com/stage: `` | | autotest-pipeline | app.edp.epam.com/branch: `` app.edp.epam.com/codebase: `` app.edp.epam.com/parentPipelineRun: ``-`` app.edp.epam.com/pipeline: `` app.edp.epam.com/stage: `` | | deploy | app.edp.epam.com/cdpipeline: `` app.edp.epam.com/cdstage: `` app.edp.epam.com/pipelinetype: `deploy` | | clean | app.edp.epam.com/cdpipeline: `` app.edp.epam.com/cdstage: `` app.edp.epam.com/pipelinetype: `clean` | ### Pipeline Usage Example To demonstrate label usage in the Tekton pipelines, find below some KubeRocketCI resource examples: ```yaml title="Codebase specification" metadata: ... name: demo ... spec: ... defaultBranch: main type: application framework: react lang: javascript ... ``` ```yaml title="CD Pipeline specification" spec: ... applications: - demo inputDockerStreams: - demo-main name: mypipe ... ``` ```yaml title="Stage specification" spec: ... cdPipeline: mypipe name: dev namespace: krci-delivery-ms-delivery-dev-mypipe-dev qualityGates: - autotestName: autotests branchName: master qualityGateType: autotests stepName: autotest ... ``` The table below shows all the pipelines associated with the `demo` codebase: | Pipeline Name | Type | Labels | | :--------------------------------- | :----- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | gerrit-npm-react-app-review | Review | app.edp.epam.com/codebase: demo app.edp.epam.com/codebasebranch: demo-main app.edp.epam.com/pipelinetype: review | | gerrit-npm-react-app-build-default | Build | app.edp.epam.com/codebase: demo app.edp.epam.com/codebasebranch: demo-main app.edp.epam.com/pipelinetype: build | | autotest-runner | Deploy | app.edp.epam.com/pipeline: mypipe app.edp.epam.com/pipelinetype: autotestRunner app.edp.epam.com/stage: dev | | autotests-gradle, autotests-maven | Deploy | app.edp.epam.com/branch: master app.edp.epam.com/codebase: autotests app.edp.epam.com/parentPipelineRun: mypipe-dev-hash app.edp.epam.com/pipeline: mypipe app.edp.epam.com/stage: dev | | deploy | Deploy | app.edp.epam.com/cdpipeline: deployapp.edp.epam.com/cdstage: deploy-dev app.edp.epam.com/pipelinetype: deploy | | clean | Clean | app.edp.epam.com/cdpipeline: deployapp.edp.epam.com/cdstage: deploy-dev app.edp.epam.com/pipelinetype: clean | The list of all the tasks associated with the `demo` codebase is presented below: | Task Name | Labels | | :----------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | init-autotest | app.edp.epam.com/pipeline: mypipe app.edp.epam.com/pipelinetype: autotestRunner app.edp.epam.com/stage: dev | | run-autotest | app.edp.epam.com/branch: master app.edp.epam.com/codebase: autotests app.edp.epam.com/parentPipelineRun: mypipe-dev-hash app.edp.epam.com/pipeline: mypipe app.edp.epam.com/stage: dev | | wait-for-autotests | app.edp.epam.com/pipeline: mypipe app.edp.epam.com/pipelinetype: autotestRunner app.edp.epam.com/stage: dev | | promote-images | app.edp.epam.com/pipeline: mypipe app.edp.epam.com/pipelinetype: autotestRunner app.edp.epam.com/stage: dev | --- ## Verification of KubeRocketCI Artifacts # Verification of KubeRocketCI Artifacts This documentation outlines platform SLSA integration and guides verifying image authenticity and provenance. Supply Chain Levels of [Software Assurance (SLSA)](https://slsa.dev/) is a framework for assessing and enhancing software supply chain security. Software Supply Chain Security is a critical aspect of modern software development and deployment. Supply Chain Levels of Software Assurance (SLSA) provides a framework for assessing and enhancing the security of your software supply chain. ## Prerequisites Ensure you have installed [rekor-cli](https://docs.sigstore.dev/logging/installation/) and [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) on your environment before proceeding. *** ## Release Assets The table below represents a list of KubeRocketCI components with corresponding images that are signed and pushed to DockerHub: | Asset | Description | |-------------------------|-------------------------------| | [codebase-operator](https://github.com/epam/edp-codebase-operator/tree/master) | [Docker Image](https://hub.docker.com/r/epamedp/cd-pipeline-operator) | | [edp-headlamp](https://github.com/epam/edp-headlamp) | [Docker Image](https://hub.docker.com/r/epamedp/edp-headlamp) | | [edp-tekton](https://github.com/epam/edp-tekton) | [Docker Image](https://hub.docker.com/r/epamedp/edp-tekton) | | [cd-pipeline-operator](https://github.com/epam/edp-cd-pipeline-operator) | [Docker Image](https://hub.docker.com/r/epamedp/cd-pipeline-operator) | | [gerrit-operator](https://github.com/epam/edp-gerrit-operator) | [Docker Image](https://hub.docker.com/r/epamedp/gerrit-operator) | | [edp-gerrit](https://github.com/epam/edp-gerrit) | [Docker Image](https://hub.docker.com/r/epamedp/edp-gerrit) | *** ## Verify Container Images KubeRocketCI's container images are signed using cosign with the [cosign.pub](https://raw.githubusercontent.com/epam/edp-install/master/cosign.pub) key for signing and transparency. You can verify a container image's signature by executing the `cosign verify` command. To confirm the authenticity of the image, run the `cosign verify` command. See the example below: ```bash cosign verify --key https://raw.githubusercontent.com/epam/edp-install/master/cosign.pub epamedp/codebase-operator:2.20.0 | jq . ``` Verification for [epamedp/codebase-operator:2.20.0](https://hub.docker.com/layers/epamedp/codebase-operator/2.20.0/images/sha256-36585a13b5b5ff5a15138e9d16cc74eb3aac4560b77be15161d3b3db25b89e1d?context=repo): ```bash Verification for index.docker.io/epamedp/codebase-operator:2.20.0 The following checks were performed on each of these signatures: - The cosign claims were validated - The claims were present in the transparency log - The signatures were integrated into the transparency log when the certificate was valid - The signatures were verified against the specified public key [ { "critical": { "identity": { "docker-reference": "index.docker.io/epamedp/codebase-operator" }, "image": { "docker-manifest-digest": "sha256:36585a13b5b5ff5a15138e9d16cc74eb3aac4560b77be15161d3b3db25b89e1d" }, "type": "cosign container image signature" }, "optional": null } ] ``` ## Verify Container Image With SLSA Attestations An [SLSA](https://slsa.dev/) Level 3 provenance is verified using. The following command will verify the signature of an attestation and how it was issued. It will contain the payloadType, payload, and signature. Run the `cosign verify-attestation` command using the [cosign.pub](https://raw.githubusercontent.com/epam/edp-install/master/cosign.pub): ```bash cosign verify-attestation --key https://raw.githubusercontent.com/epam/edp-install/master/cosign.pub --type slsaprovenance epamedp/codebase-operator:2.20.0 | jq . ``` Verification for [epamedp/codebase-operator:2.20.0](https://hub.docker.com/layers/epamedp/codebase-operator/2.20.0/images/sha256-36585a13b5b5ff5a15138e9d16cc74eb3aac4560b77be15161d3b3db25b89e1d?context=repo): ```bash Verification for epamedp/codebase-operator:2.20.0 The following checks were performed on each of these signatures: - The cosign claims were validated - The claims were present in the transparency log - The signatures were integrated into the transparency log when the certificate was valid - The signatures were verified against the specified public key { "payloadType": "application/vnd.in-toto+json", "payload": "eyJfdHlwZSI6Imh0dHBzOi8vaW4tdG90by5pby9TdGF0ZW1lbnQvdjAuMSIsInByZWRpY2F0ZVR5cGUiOiJodHRwczovL3Nsc2EuZGV2L3Byb3ZlbmFuY2UvdjAuMiIsInN1YmplY3QiOlt7Im5hbWUiOiJpbmRleC5kb2NrZXIuaW8vZXBhbWVkcC9jb2RlYmFzZS1vcGVyYXRvciIsImRpZ2VzdCI6eyJzaGEyNTYiOiIzNjU4NWExM2I1YjVmZjVhMTUxMzhlOWQxNmNjNzRlYjNhYWM0NTYwYjc3YmUxNTE2MWQzYjNkYjI1Yjg5ZTFkIn19XSwicHJlZGljYXRlIjp7ImJ1aWxkZXIiOnsiaWQiOiJodHRwczovL3Rla3Rvbi5kZXYvY2hhaW5zL3YyIn0sImJ1aWxkVHlwZSI6InRla3Rvbi5kZXYvdjFiZXRhMS9UYXNrUnVuIiwiaW52b2NhdGlvbiI6eyJjb25maWdTb3VyY2UiOnt9LCJwYXJhbWV0ZXJzIjp7IkJVSUxERVJfSU1BR0UiOiJnY3IuaW8va2FuaWtvLXByb2plY3QvZXhlY3V0b3I6djEuMTIuMS1kZWJ1ZyIsIkNPTlRFWFQiOiIuLyIsIkRPQ0tFUkZJTEUiOiJEb2NrZXJmaWxlIiwiSU1BR0UiOiJlcGFtZWRwL2NvZGViYXNlLW9wZXJhdG9yOjIuMjAuMCIsIklNQUdFX1RBUiI6ImNvZGViYXNlLW9wZXJhdG9yXzIuMjAuMCJ9LCJlbnZpcm9ubWVudCI6eyJhbm5vdGF0aW9ucyI6eyJtZXRhLmhlbG0uc2gvcmVsZWFzZS1uYW1lIjoiZWRwLWN1c3RvbS1waXBlbGluZXMiLCJtZXRhLmhlbG0uc2gvcmVsZWFzZS1uYW1lc3BhY2UiOiJlZHAtZGVsaXZlcnkiLCJwaXBlbGluZS50ZWt0b24uZGV2L2FmZmluaXR5LWFzc2lzdGFudCI6ImFmZmluaXR5LWFzc2lzdGFudC1iZjRkNzRkMWM0IiwicGlwZWxpbmUudGVrdG9uLmRldi9yZWxlYXNlIjoiMjI5OWIxNSIsInRla3Rvbi5kZXYvY2F0ZWdvcmllcyI6IkltYWdlIEJ1aWxkIiwidGVrdG9uLmRldi9kaXNwbGF5TmFtZSI6IkJ1aWxkIGFuZCB1cGxvYWQgY29udGFpbmVyIGltYWdlIHVzaW5nIEthbmlrbyIsInRla3Rvbi5kZXYvcGlwZWxpbmVzLm1pblZlcnNpb24iOiIwLjE3LjAiLCJ0ZWt0b24uZGV2L3BsYXRmb3JtcyI6ImxpbnV4L2FtZDY0IiwidGVrdG9uLmRldi90YWdzIjoiaW1hZ2UtYnVpbGQifSwibGFiZWxzIjp7ImFwcC5rdWJlcm5ldGVzLmlvL21hbmFnZWQtYnkiOiJIZWxtIiwiYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbiI6IjAuOC4wIiwiaGVsbS5zaC9jaGFydCI6ImVkcC1jdXN0b20tcGlwZWxpbmVzLTAuOC4wIiwiazhzbGVucy1lZGl0LXJlc291cmNlLXZlcnNpb24iOiJ2MSIsInRla3Rvbi5kZXYvbWVtYmVyT2YiOiJ0YXNrcyIsInRla3Rvbi5kZXYvcGlwZWxpbmUiOiJnZXJyaXQtb3BlcmF0b3JzLWFwcC1yZWxlYXNlLWVkcCIsInRla3Rvbi5kZXYvcGlwZWxpbmVSdW4iOiJlZHAtY29kZWJhc2Utb3BlcmF0b3ItcmVsZWFzZSIsInRla3Rvbi5kZXYvcGlwZWxpbmVUYXNrIjoia2FuaWtvLWJ1aWxkIiwidGVrdG9uLmRldi90YXNrIjoia2FuaWtvLXJlbGVhc2UifX19LCJidWlsZENvbmZpZyI6eyJzdGVwcyI6W3siZW50cnlQb2ludCI6Ii9rYW5pa28vZXhlY3V0b3IgXFxcbiAgLS1kb2NrZXJmaWxlPS93b3Jrc3BhY2Uvc291cmNlL0RvY2tlcmZpbGUgXFxcbiAgLS1jb250ZXh0PS93b3Jrc3BhY2Uvc291cmNlLy4vIFxcXG4gIC0tZGVzdGluYXRpb249ZXBhbWVkcC9jb2RlYmFzZS1vcGVyYXRvcjoyLjIwLjAgXFxcbiAgLS1kaWdlc3QtZmlsZT0vdGVrdG9uL3Jlc3VsdHMvSU1BR0VfRElHRVNUIFxcXG4gIC0tdGFyLXBhdGg9Y29kZWJhc2Utb3BlcmF0b3JfMi4yMC4wLnRhciBcXFxuIiwiYXJndW1lbnRzIjpudWxsLCJlbnZpcm9ubWVudCI6eyJjb250YWluZXIiOiJidWlsZC1hbmQtcHVzaCIsImltYWdlIjoib2NpOi8vZ2NyLmlvL2thbmlrby1wcm9qZWN0L2V4ZWN1dG9yQHNoYTI1NjphN2VhOWY2OWQ3N2Q3ZTdhMGVhODIxZjE1MDY5YmU0NTQyMGE1MzZmODFhYjU3ODdhOTg4NjU5ZTQ4YzI1Mzc3In0sImFubm90YXRpb25zIjpudWxsfSx7ImVudHJ5UG9pbnQiOiJzZXQgLWVcbmltYWdlPVwiZXBhbWVkcC9jb2RlYmFzZS1vcGVyYXRvcjoyLjIwLjBcIlxuZWNobyAtbiBcIiR7aW1hZ2V9XCIgfCB0ZWUgXCIvdGVrdG9uL3Jlc3VsdHMvSU1BR0VfVVJMXCJcbiIsImFyZ3VtZW50cyI6bnVsbCwiZW52aXJvbm1lbnQiOnsiY29udGFpbmVyIjoid3JpdGUtdXJsIiwiaW1hZ2UiOiJvY2k6Ly9kb2NrZXIuaW8vbGlicmFyeS9hbHBpbmVAc2hhMjU2OjcxNDRmN2JhYjNkNGMyNjQ4ZDdlNTk0MDlmMTVlYzUyYTE4MDA2YTEyOGM3MzNmY2ZmMjBkM2E0YTU0YmE0NGEifSwiYW5ub3RhdGlvbnMiOm51bGx9XX0sIm1ldGFkYXRhIjp7ImJ1aWxkU3RhcnRlZE9uIjoiMjAyMy0xMS0wM1QxMzozMDo1NVoiLCJidWlsZEZpbmlzaGVkT24iOiIyMDIzLTExLTAzVDEzOjMxOjE1WiIsImNvbXBsZXRlbmVzcyI6eyJwYXJhbWV0ZXJzIjpmYWxzZSwiZW52aXJvbm1lbnQiOmZhbHNlLCJtYXRlcmlhbHMiOmZhbHNlfSwicmVwcm9kdWNpYmxlIjpmYWxzZX0sIm1hdGVyaWFscyI6W3sidXJpIjoib2NpOi8vZ2NyLmlvL2thbmlrby1wcm9qZWN0L2V4ZWN1dG9yIiwiZGlnZXN0Ijp7InNoYTI1NiI6ImE3ZWE5ZjY5ZDc3ZDdlN2EwZWE4MjFmMTUwNjliZTQ1NDIwYTUzNmY4MWFiNTc4N2E5ODg2NTllNDhjMjUzNzcifX0seyJ1cmkiOiJvY2k6Ly9kb2NrZXIuaW8vbGlicmFyeS9hbHBpbmUiLCJkaWdlc3QiOnsic2hhMjU2IjoiNzE0NGY3YmFiM2Q0YzI2NDhkN2U1OTQwOWYxNWVjNTJhMTgwMDZhMTI4YzczM2ZjZmYyMGQzYTRhNTRiYTQ0YSJ9fV19fQ==", "signatures": [ { "keyid": "SHA256:7E2nAQnycq4vfPlzmLZGzpK/Vr6oXKqqGokDyrBSLck", "sig": "MEUCIAZLrA/wTkqmnCZXh85R9Y/Ue5f8wuGgjLMYdoFw9GRLAiEA/sE598EX5fppqbry+xvE+aap8+qHPioOin8t6Ttzx3k=" } ] } ``` For more details about attestation, please refer to the official [cosign documentation](https://docs.sigstore.dev/cosign/verifying/verify/) page. ## Verify Release Pipeline Within each release component, you will discover a [Rekor UUID](https://search.sigstore.dev/?uuid=24296fb24b8ad77a671b24f6b83f79e46fe5214cde46ed045ceba35d640a7e017cbc5524e90329ff), which serves to validate the flow of the release pipeline. Execute the following command to obtain comprehensive information about the release pipeline of [codebase-operator](https://github.com/epam/edp-codebase-operator/releases/tag/v2.20.0) with UUID: ```24296fb24b8ad77a671b24f6b83f79e46fe5214cde46ed045ceba35d640a7e017cbc5524e90329ff```: ```bash rekor-cli get --uuid 24296fb24b8ad77a671b24f6b83f79e46fe5214cde46ed045ceba35d640a7e017cbc5524e90329ff --format json | jq -r .Attestation | jq . ``` The result: ```bash { "_type": "https://in-toto.io/Statement/v0.1", "predicateType": "https://slsa.dev/provenance/v0.2", "subject": [ { "name": "index.docker.io/epamedp/codebase-operator", "digest": { "sha256": "36585a13b5b5ff5a15138e9d16cc74eb3aac4560b77be15161d3b3db25b89e1d" } } ], "predicate": { "builder": { "id": "https://tekton.dev/chains/v2" }, "buildType": "tekton.dev/v1beta1/TaskRun", "invocation": { "configSource": {}, "parameters": { "BUILDER_IMAGE": "gcr.io/kaniko-project/executor:v1.12.1-debug", "CONTEXT": "./", "DOCKERFILE": "Dockerfile", "IMAGE": "epamedp/codebase-operator:2.20.0", "IMAGE_TAR": "codebase-operator_2.20.0" }, "environment": { "annotations": { ... }, "labels": { ... } } }, "buildConfig": { "steps": [ { "entryPoint": "/kaniko/executor \\\n --dockerfile=/workspace/source/Dockerfile \\\n --context=/workspace/source/./ \\\n --destination=epamedp/codebase-operator:2.20.0 \\\n --digest-file=/tekton/results/IMAGE_DIGEST \\\n --tar-path=codebase-operator_2.20.0.tar \\\n", "arguments": null, "environment": { "container": "build-and-push", "image": "oci://gcr.io/kaniko-project/executor@sha256:a7ea9f69d77d7e7a0ea821f15069be45420a536f81ab5787a988659e48c25377" }, "annotations": null }, { "entryPoint": "set -e\nimage=\"epamedp/codebase-operator:2.20.0\"\necho -n \"${image}\" | tee \"/tekton/results/IMAGE_URL\"\n", "arguments": null, "environment": { "container": "write-url", "image": "oci://docker.io/library/alpine@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a" }, "annotations": null } ] }, "metadata": { ... }, "materials": [ { "uri": "oci://gcr.io/kaniko-project/executor", "digest": { "sha256": "a7ea9f69d77d7e7a0ea821f15069be45420a536f81ab5787a988659e48c25377" } }, { "uri": "oci://docker.io/library/alpine", "digest": { "sha256": "7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a" } } ] } } ``` By signing all our artifacts, we assure you that they are trustworthy. This guide is indispensable for developers and administrators to enhance their software's reliability and meet modern security standards. The adoption of SLSA will bring you confidence while using the platform. ## Related Articles * [In-Toto Attestations](https://docs.sigstore.dev/cosign/verifying/verify/) --- ## Quality Control # Quality Control In KubeRocketCI, we guarantee the quality of the product not only by using the most advanced tools and best practices but also by covering the whole product functionality with our dedicated automated tests. ## Autotest Coverage Scheme Autotests are significant part of our verification flow. Continuous improvement of the verification mechanisms quality is performed to provide users with the most stable version of the platform. The autotest coverage status is presented on the scheme below: ![Autotest coverage status](../assets/developer-guide/autotests-coverage.png "Autotest coverage status") ## Release Testing In our testing flow, each release is verified by the following tests: | Test Group | Description | What's Covered | |-|:-------------------------------------------------------------------|:-| | API Tests | Tekton Bitbucket, Gerrit, GitHub, and GitLab API long regression | Codebase provisioning, reviewing and building pipelines, adding new branches, deploying applications (in a custom namespace), Jira integration, and rechecking for review pipeline. | | UI Tests | Tekton Bitbucket, Gerrit, GitHub, and GitLab UI long regression | Codebase provisioning, reviewing and building pipelines, adding new branches, deploying applications (in a custom namespace), Jira integration, and rechecking for review pipeline. | | Short Tests | Tekton Bitbucket, Gerrit , GitHub, and GitLab API short regression | Codebase provisioning, reviewing and building pipelines, deploying applications (in a custom namespace), rechecking for review pipeline | | Smoke | Tekton Gerrit Smoke | Codebase provisioning, reviewing and building pipelines, deploying applications. | ## Related Articles * [KubeRocketCI Project Rules. Working Process](edp-workflow.md) * [Operator Development](local-development.md) --- ## KubeRocketCI Deployment on AWS # KubeRocketCI Deployment on AWS This document describes the KubeRocketCI deployment architecture on AWS. It utilizes various AWS services such as Amazon Elastic Kubernetes Service (EKS), Amazon EC2, Amazon Route 53, and others to build and deploy software in a repeatable, automated way. ## Overview The KubeRocketCI deployment architecture consists of two AWS accounts: Shared and Explorer. The *Shared* account hosts shared services, while the *Explorer* account runs the development team workload and KubeRocketCI services. Both accounts have an AWS EKS cluster deployed in multiple Availability Zones (AZs). The EKS cluster runs the KubeRocketCI Services, development team workload, and shared services in the case of the Shared account. ![KubeRocketCI Deployment Diagram on AWS](../assets/developer-guide/architecture/aws-deploymen-diagram.png) ## Key Components 1. **AWS Elastic Kubernetes Service (EKS)**: A managed Kubernetes service used to run the KubeRocketCI Services, development team workload, and shared services. EKS provides easy deployment and management of Kubernetes clusters. 2. **Amazon EC2**: Instances running within private subnets that serve as nodes for the EKS cluster. Autoscaling Groups are used to deploy these instances, allowing for scalability based on demand. 3. **Amazon Route 53**: A DNS web service manages external and internal DNS records for the KubeRocketCI deployment. It enables easy access to resources using user-friendly domain names. 4. **AWS Application Load Balancer (ALB)**: Used for managing ingress traffic into the KubeRocketCI deployment. Depending on requirements, ALBs can be configured as internal or external load balancers. 5. **AWS WAF**: Web Application Firewall service used to protect external ALBs from common web exploits by filtering malicious requests. 6. **AWS Certificate Manager (ACM)**: A service that provisions manages, and deploys SSL/TLS certificates for use with AWS services. ACM is used to manage SSL certificates for secure communication within the KubeRocketCI deployment. 7. **AWS Elastic Container Registry (ECR)**: A fully-managed Docker container registry that stores and manages Docker images. ECR provides a secure and scalable solution for storing container images used in the KubeRocketCI deployment. 8. **AWS Systems Manager Parameter Store**: Used to securely store and manage secrets required by various components of the KubeRocketCI deployment. Parameter Store protects sensitive information such as API keys, database credentials, and other secrets. ## High Availability and Fault Tolerance The EKS cluster is deployed across multiple AZs to ensure high availability and fault tolerance. This allows for automatic failover in case of an AZ outage or instance failure. Autoscaling Groups automatically adjust the number of EC2 instances based on demand, ensuring scalability while maintaining availability. ## Design Considerations ### Reliability - Using multiple AZs ensures high availability and fault tolerance for the EKS cluster. - Autoscaling Groups enable automatic scaling of EC2 instances based on demand, providing reliability during peak loads. - Multiple NAT gateways are deployed in each AZ to ensure reliable outbound internet connectivity. ### Performance Efficiency - Utilizing AWS EKS allows for efficient management of Kubernetes clusters without the need for manual configuration or maintenance. - Spot instances can be utilized alongside on-demand instances within the EKS cluster to optimize costs while maintaining performance requirements. - Amazon Route 53 enables efficient DNS resolution by managing external and internal DNS records. ### Security - External ALBs are protected using AWS WAF, which filters out malicious traffic and protects against common web exploits. - ACM is used to provision SSL/TLS certificates, ensuring secure communication within the KubeRocketCI deployment. - Secrets required by various components are securely stored and managed using the AWS Systems Manager Parameter Store. ### Cost Optimization - Utilizing spot and on-demand instances within the EKS cluster can significantly reduce costs while maintaining performance requirements. - Autoscaling Groups allow for automatic scaling of EC2 instances based on demand, ensuring optimal resource utilization and cost efficiency. ## Conclusion The KubeRocketCI deployment architecture on AWS follows best practices and patterns from the Well-Architected Framework. By leveraging AWS services such as EKS, EC2, Route 53, ALB, WAF, ACM, and Parameter Store, the KubeRocketCI provides a robust and scalable CI/CD system that enables developers to deploy and manage infrastructure and applications quickly. The architecture ensures high availability, fault tolerance, reliability, performance efficiency, security, and cost optimization for the KubeRocketCI deployment. --- ## AWS Infrastructure Cost Estimation # AWS Infrastructure Cost Estimation Effective planning and budgeting are essential for developing applications in cloud computing, with a key part being accurate infrastructure cost estimation. This not only helps in keeping within budget but also enables informed decision-making and resource optimization for project viability. This guide aims to offer an in-depth look at the factors affecting AWS infrastructure costs for KubeRocketCI and includes analytics and tools for cost estimation. ## Platform Components and Approximate Costs This section contains tables outlining the key components of our AWS infrastructure, including a brief description of each component's role, its purpose within our infrastructure, and an estimate of its monthly cost. :::note The costs mentioned below are estimates. For the most accurate and up-to-date pricing, please refer to the [AWS official documentation](https://aws.amazon.com/pricing/?aws-products-pricing.sort-by=item.additionalFields.productNameLowercase&aws-products-pricing.sort-order=asc&awsf.Free%20Tier%20Type=*all&awsf.tech-category=*all). ::: The table below outlines key AWS infrastructure components for KubeRocketCI, detailing each component's role, purpose, and estimated monthly cost: | Component | Description | Purpose Within Infrastructure | |-----------------------------------------------|-----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------| | Application Load Balancer (ALB) | Distributes incoming application traffic across multiple targets. | Ensures high availability and fault tolerance for our applications. | | Virtual Private Cloud (VPC) | Provides an isolated section of the AWS cloud where resources can be launched. | Segregates our infrastructure for enhanced security and management. | | 3x Network Address Translation (NAT) Gateways | Enables instances in a private subnet to connect to the internet or other AWS services. | Provides internet access to EC2 instances without exposing them to the public internet. | | Elastic Container Registry (ECR) | A fully managed container registry. | Stores, manages, and deploys container images. | | Elastic Kubernetes Service (EKS) | A managed Kubernetes service. | Simplifies running Kubernetes applications on AWS. | | Elastic Block Store (EBS) | Provides persistent block storage volumes for use with EC2 instances. | Offers highly available and durable storage for our applications. | | Elastic Compute Cloud (EC2) | Provides scalable computing capacity. | Hosts our applications, supporting varied compute workloads. | The table below presents an itemized estimate of monthly costs for KubeRocketCI's AWS infrastructure components, including ALB, VPC, EC2, and more: | Component | Approximate Monthly Cost | |---------------------------------------------------------------------------------------------------------|--------------------------------| | Application Load Balancer (ALB) | **$30.00** | | Virtual Private Cloud (VPC) - 3x Network Address Translation Gateways - 3x Public IPv4 Address | **$113.88** **$10.95** | | Elastic Container Registry (ECR) | **$5.00** | | Elastic Kubernetes Service (EKS) - 1x EKS Clusters | **$73.00** | | Elastic Block Store (EBS) | **$14.28** | | Elastic Compute Cloud (EC2) - 2x c5.2xlarge (Spot) - 2x c5.2xlarge (On-Demand) | **$219.11** **$576.00** | ## AWS Pricing Calculator To further assist in your planning and budgeting efforts, we have pre-configured the AWS Pricing Calculator with inputs matching our infrastructure setup. This tool allows you to explore and adjust the cost estimation based on your specific needs, giving you a personalized overview of potential expenses. Access the AWS Pricing Calculator with our pre-configured setup here: [AWS Pricing Calculator](https://calculator.aws/#/estimate?id=42ed1a892c891ebcd905734b437f722122983f61) ## Related Articles * [KubeRocketCI Deployment on AWS](aws-deployment-diagram.md) * [KubeRocketCI Reference Architecture on AWS](aws-reference-architecture.md) --- ## KubeRocketCI Reference Architecture on AWS # KubeRocketCI Reference Architecture on AWS The reference architecture of the KubeRocketCI on AWS is designed to provide a robust and scalable CI/CD system for developing and deploying software in a repeatable and automated manner. The architecture leverages AWS Managed Services to enable developers to quickly deploy and manage infrastructure and applications. KubeRocketCI recommends to follow the best practices and patterns from the Well-Architected Framework, the AWS Architecture Center, and EKS Best Practices Guide. ## Architecture Details The AWS Cloud comprises three accounts: Production, Shared, and Development. :::note AWS Account management is out of scope for this document. ::: Each account serves specific purposes: * The **Production** account is used to host production workloads. The Production account serves as the final destination for deploying business applications. It maintains a separate ECR registry to store Docker images for production-level applications. The environment is designed to be highly resilient and scalable, leveraging the KubeRocketCI's CI/CD pipeline to ensure consistent and automated deployments. With proper access control and separation from development environments, the Production account provides a stable and secure environment for running mission-critical applications. * The **Development** account is dedicated to development workload and lower environments. This account hosts the KubeRocketCI itself, running on AWS EKS. It provides developers an isolated environment to build, test, and deploy their applications in lower environments, ensuring separation from production workloads. Developers can connect to the AWS Cloud using a VPN, enforcing secure access. * The **Shared** holds shared services that are accessible to all accounts within the organization. These services include SonarQube, Nexus, and Keycloak, which are deployed in Kubernetes Clusters managed by AWS Elastic Kubernetes Service (EKS). The shared services leverage AWS RDS, AWS EFS, and AWS ALB/NLB. The deployment of the shared services is automated using Kubernetes cluster-addons approach with GitOps and Argo CD. ![KubeRocketCI Reference Architecture on AWS](../assets/developer-guide/architecture/aws-reference-architecture.png) ## Infrastructure as Code Infrastructure as Code (IaC) is a key principle in the KubeRocketCI architecture. Terraform is the IaC tool to provision and manage all services in each account. AWS S3 and AWS DynamoDB serve as the backend for Terraform state, ensuring consistency and reliability in the deployment process. This approach enables the architecture to be version-controlled and allows for easy replication and reproducibility of environments. ## Container Registry The architecture utilizes AWS Elastic Container Registry (ECR) as a Docker Registry for container image management. ECR offers a secure, scalable, and reliable solution for storing and managing container images. It integrates seamlessly with other AWS services and provides a highly available and durable storage solution for containers in the CI/CD pipeline. ## IAM Roles for Service Accounts (IRSA) The KubeRocketCI implements IAM Roles for Service Accounts (IRSA) to provide secure access to AWS services from Kubernetes Clusters. This feature enables fine-grained access control with individual Kubernetes pods assuming specific IAM roles for authenticated access to AWS resources. IRSA eliminates the need for managing and distributing access keys within the cluster, significantly enhancing security and reducing operational complexity. ## SSL Certificates The architecture uses the AWS Certificate Manager (ACM) to secure communication between services to provide SSL certificates. ACM eliminates the need to manually manage SSL/TLS certificates, automating the renewal and deployment process. The KubeRocketCI ensures secure and encrypted traffic within its environment by leveraging ACM. ## AWS WAF The architecture's external Application Load Balancer (ALB) endpoint is protected by the AWS Web Application Firewall (WAF). WAF protects against common web exploits and ensures the security and availability of the applications hosted within the KubeRocketCI. It offers regular rule updates and easy integration with other AWS services. ## Parameter Store and Secrets Manager The architecture leverages the AWS Systems Manager Parameter Store and Secrets Manager to securely store and manage all secrets and parameters utilized within the EKS clusters—parameter Store stores general configuration information, such as database connection strings and API keys. In contrast, Secrets Manager securely stores sensitive information, such as passwords and access tokens. By centralizing secrets management, the architecture ensures proper access control and reduces the risk of unauthorized access. ## Observability and Monitoring For observability and monitoring, the KubeRocketCI leverages a suite of AWS Managed Services designed to provide comprehensive insights into the performance and health of applications and infrastructure: **AWS CloudWatch** is utilized for monitoring and observability, offering detailed insights into application and infrastructure performance. It enables real-time monitoring of logs, metrics, and events, facilitating proactive issue resolution and performance optimization. **AWS OpenSearch Service** (successor to Amazon Elasticsearch Service) provides powerful search and analytics capabilities. It allows for the analysis of log data and metrics, supporting enhanced application monitoring and user experience optimization. **AWS Managed Grafana** offers a scalable, secure, and fully managed Grafana service, enabling developers to create and share dashboards for visualizing real-time data. **AWS Prometheus Service**, a managed Prometheus-compatible monitoring service, is used for monitoring Kubernetes and container environments. It supports powerful queries and provides detailed insights into container and microservices architectures. ## Summary The reference architecture of the KubeRocketCI on AWS provides a comprehensive and scalable environment for building and deploying software applications. With a strong focus on automation, security, and best practices, this architecture enables developers to leverage the full potential of AWS services while following industry-standard DevOps practices. --- ## Working With Documentation # Working With Documentation This section outlines the necessary steps to start developing the KubeRocketCI documentation using Docusaurus. Docusaurus is a modern static website generator that simplifies creating, deploying, and maintaining open source project websites. :::note For more details on Docusaurus, please refer to the Docusaurus [official website](https://docusaurus.io/). ::: Adjusting the documentation to use npm instead of Yarn for working with Docusaurus involves minor changes. Below is the revised version of the documentation section, now specifying npm commands: ## Prerequisites Before you begin, ensure you have the following installed: - [Node.js](https://nodejs.org/en/) (version >= 18.0) - [npm](https://www.npmjs.com/) (usually comes with Node.js) ## Local Development 1. Fork the [KubeRocketCI documentation repository](https://github.com/KubeRocketCI/docs) to your personal account. 2. Clone the forked documentation repository to your local machine. 3. Install the dependencies: ```bash npm install ``` 4. Start a local development server: ```bash npm start ``` This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. To quit the session, press `ctrl + c` or `cmd + c`. 5. Navigate to `localhost:3000` (or the URL provided in the terminal after running the `npm start` command) in your web browser to view the documentation site. 6. Make the necessary changes to the documentation. The files are located in the `docs` directory. Changes will automatically refresh in the browser. 7. Commit your changes and create a pull request. ## Build To build the static content of the site into the `build` directory, run: ```bash npm run build ``` The content generated in the `build` directory can be served using any static content hosting service. ## Deployment To deploy the documentation site, you can use the following command, which builds the site and pushes it to the `gh-pages` branch: Using SSH: ```bash USE_SSH=true npm run deploy ``` Not using SSH: ```bash GIT_USER= npm run deploy ``` Ensure you replace `` with your actual GitHub username if you are not using SSH. --- ## KubeRocketCI Contribution # KubeRocketCI Contribution This page contains the details on the project rules and working process for KubeRocketCI team and contributors. Explore the main points about working with GitHub, following the main commit flow, as well as the details about commit types and message below. ## Project Rules Before starting the development, please check the project rules: 1. It is highly recommended to become familiar with the GitHub flow. For details, please refer to the [GitHub official documentation](https://docs.github.com/en/get-started/using-github/github-flow) and pay attention to the main points: a. Creating pull requests in GitHub. b. Resolution of Merge Conflict. c. Comments resolution. d. One GitHub task should have one Pull Request (PR) if it doesn't change multiple operators. If there are many changes within one PR, amend the commit. 2. Only the Assignee is responsible for the PR merger and Jira task status. 3. Every PR should be merged in a timely manner. 4. Log time to Jira ticket. ## Working Process With KubeRocketCI, the main workflow is based on the getting a Jira task and creating a Pull Request according to the rules described below. :::info Workflow Get Jira task → implement, verify by yourself the results → create Pull Request (PR) → send for review → resolve comments/add changes, ask colleagues for the final review → track the PR merge → verify by yourself the results → change the status in the Jira ticket to CODE COMPLETE or RESOLVED → share necessary links with a QA specialist in the QA Verification channel → QA specialist closes the Jira task after his verification → Jira task should be CLOSED. ::: ### Commit Flow 1. Get a task in the Jira/GitHub dashboard. Please be aware of the following points: a. Every task has a reporter who can provide more details in case something is not clear. b. The responsible person for the task and code implementation is the assignee who tracks the following: - Actual Jira task status. - Time logging. - Add comments, attach necessary files. - In comments, add link that refers to the merged PR (optional, if not related to many repositories). - Code review and the final merge. - MS Teams chats - ping other colleagues, answer questions, etc. - Verification by a QA specialist. - Bug fixing. c. Pay attention to the task Status that differs in different entities, the workflow will help to see the whole task processing: ![View Jira workflow](../assets/developer-guide/jira-workflow.png "View Jira workflow") d. There are several entities that are used on the KubeRocketCI project: Story, Improvement, Task, Bug. a. Every task has a reporter who can provide more details in case something is not clear. b. The responsible person for the task and code implementation is the assignee who tracks the following: - Actual GitHub task status. - Add comments, attach necessary files. - In comments, add link that refers to the merged PR (optional, if not related to many repositories). - Code review and the final merge. - MS Teams chats - ping other colleagues, answer questions, etc. - Verification by a QA specialist. - Bug fixing. c. If the task is created on your own, make sure it is populated completely. See an example below: ![GitHub issue](../assets/developer-guide/github-task.png "GitHub issue") 2. Implement feature, improvement, fix and check the results on your own. If it is impossible to check the results of your work before the merge, verify all later. 3. When committing, use the pattern: commit type: Commit message (#GitHub ticket number). a. commit type: `feat`: (new feature for the user, not a new feature for build script) `fix`: (bug fix for the user, not a fix to a build script) `docs`: (changes to the documentation) `style`: (formatting, missing semicolons, etc; no production code change) `refactor`: (refactoring production code, eg. renaming a variable) `test`: (adding missing tests, refactoring tests; no production code change) `chore`: (updating grunt tasks etc; no production code change) `!`: (added to other commit types to mark breaking changes) For example: feat!: Add ingress links column into Applications table on stage page (#77) BREAKING CHANGE: Ingress links column has been added into the Applications table on the stage details page b. Commit message: * brief, for example: `fix: Remove secretKey duplication from registry secrets (#63)` or * descriptive, for example: ```bash feat: Provide the ability to configure hadolint check (#88) * Add configuration files .hadolint.yaml and .hadolint.yml to stash ``` :::note It is mandatory to start a commit message from a capital letter. ::: c. GitHub tickets are typically identified using a number preceded by the **#** sign and enclosed in parentheses. :::note Make sure there is a descriptive commit message for a breaking change Pull Request. For example: ```bash feat!: Add ingress links column into Applications table on stage page (#77) BREAKING CHANGE: Ingress links column has been added into the Applications table on the stage details page ``` ::: 4. Create a Pull Request, for details, please refer to the [Code Review Process](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/about-pull-request-reviews): ![GitHub issue](../assets/developer-guide/merge_request.png "GitHub issue") :::caution If a Pull Request contains both new functionality and breaking changes, make sure the functionality description is placed before the breaking changes. For example: ```bash feat!: Update CRD version to the v1beta1 * Implement Developers group creation process * Align group permissions BREAKING CHANGES: Update CRD version to the v1beta1 ``` ::: ## Related Articles * [Conventional Commits](https://www.conventionalcommits.org/) * [Karma](http://karma-runner.github.io/1.0/dev/git-commit-msg.html) --- ## KubeRocketCI Developer Guide: Architecture, Components, and Contribution Workflow # KubeRocketCI Developer Guide: Architecture, Components, and Contribution Workflow The KubeRocketCI Developer Guide serves as a comprehensive technical resource specifically designed for developers. It offers detailed insights into expanding the functionalities of KubeRocketCI. This section focuses on explaining the development approach and fundamental architectural blueprints that form the basis of the platform's ecosystem. Within these pages, you'll find architectural diagrams, component schemas, and deployment strategies essential for grasping the structural elements of the platform. These technical illustrations serve as references, providing a detailed understanding of component interactions and deployment methodologies. Understanding the architecture of the KubeRocketCI and integrating third-party solutions into its established framework enables the creation of efficient, scalable, and customizable solutions within the KubeRocketCI. The KubeRocketCI platform consists of several key components. These components are interconnected, and their relationships are illustrated in the diagram below: ![KubeRocketCI Components Diagram](../assets/developer-guide/architecture/components.svg) Each component represents one of the following: - A Helm chart with a Docker image. - A standalone Helm chart. - A standalone Docker image. All components are regularly updated and released as part of the platform's development cycle. Some components, such as keycloak-operator, nexus-operator, sonar-operator, and tekton-custom-task, are provided separately through the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. These components are optional and can be installed independently of the main platform. ## Release Channels As a publicly available product, the KubeRocketCI relies on various channels to share information, gather feedback, and distribute new releases effectively. This section outlines the diverse channels through which users can engage with our platform and stay informed about the latest developments and enhancements. ### Marketplaces Our product is presented on AWS and Civo marketplaces. It's essential to ensure that the product information on these platforms is up-to-date and accurately reflects the latest version of our software: - [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-u7xcz6pvwwwoa) - [CIVO Marketplace](https://www.civo.com/marketplace/edp) ### OperatorHub Our product operators are showcased on OperatorHub, enabling seamless integration and management capabilities: - [Keycloak Operator](https://operatorhub.io/operator/edp-keycloak-operator) - [Nexus Operator](https://operatorhub.io/operator/nexus-operator) - [Sonar Operator](https://operatorhub.io/operator/sonar-operator) ## GitHub Repositories Our platform components, optional enhancements, add-ons, and deployment resources are hosted on GitHub repositories. Explore the following repositories to access the source code of components. ### Platform Components Each platform component is available in its corresponding GitHub project: - [CD Pipeline Operator](https://github.com/epam/edp-cd-pipeline-operator/releases) - [Codebase Operator](https://github.com/epam/edp-codebase-operator/releases) - [EDP Gerrit](https://github.com/epam/edp-gerrit) - [EDP Install](https://github.com/epam/edp-install/releases) - [Gerrit Operator](https://github.com/epam/edp-gerrit-operator/releases) - [KubeRocketCI Portal](https://github.com/epam/edp-headlamp/releases) - [Tekton](https://github.com/epam/edp-tekton/releases) - [GitFusion](https://github.com/KubeRocketCI/gitfusion) - [KubeRocketCI Cache](https://github.com/KubeRocketCI/krci-cache) - [Tekton Custom Tasks](https://github.com/KubeRocketCI/tekton-custom-task) ### Optional Components These optional components enhance the platform's installation and configuration experience: - [Keycloak Operator](https://github.com/epam/edp-keycloak-operator/releases) - [Nexus Operator](https://github.com/epam/edp-nexus-operator/releases) - [Sonar Operator](https://github.com/epam/edp-sonar-operator/releases) ### Add-ons Repository The Add-ons repository offers a streamlined deployment process. It includes additional components that can be integrated separately into the cluster: - [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) ### Platform Test Data Access test data from the 'Create' onboarding strategy: - [EPMD-EDP Project](https://github.com/epmd-edp) ### Helm Charts Helm chart artifacts are available in repository: - [Helm Charts](https://github.com/epam/edp-helm-charts) ## DockerHub Our DockerHub repository hosts Docker images for various platform components: - [CD Pipeline Operator](https://hub.docker.com/r/epamedp/cd-pipeline-operator/) - [Codebase Operator](https://hub.docker.com/r/epamedp/codebase-operator/) - [EDP Gerrit](https://hub.docker.com/r/epamedp/edp-gerrit) - [Gerrit Operator](https://hub.docker.com/r/epamedp/gerrit-operator) - [Keycloak Operator](https://hub.docker.com/r/epamedp/keycloak-operator/) - [KubeRocketCI Portal](https://hub.docker.com/r/epamedp/edp-headlamp/) - [Nexus Operator](https://hub.docker.com/r/epamedp/nexus-operator/) - [Sonar Operator](https://hub.docker.com/r/epamedp/sonar-operator/) - [Tekton](https://hub.docker.com/r/epamedp/edp-tekton/) - [Tekton Cache](https://hub.docker.com/r/epamedp/tekton-cache/) - [Tekton Custom Task](https://hub.docker.com/r/epamedp/tekton-custom-task/) - [GitFusion](https://hub.docker.com/r/epamedp/gitfusion) ## Social Media To maintain an active presence on social media channels and share valuable content about our software releases, we continuously publish materials across the following media: - [SolutionsHub](https://solutionshub.epam.com/solution/kuberocketci) - [Medium](https://medium.com/kuberocketci) - [YouTube](https://www.youtube.com/@theplatformteam) ## Related Guides * [Kubernetes Deployment](./kubernetes-deployment.md) --- ## Kubernetes Deployment # Kubernetes Deployment This section provides a comprehensive overview of the KubeRocketCI deployment approach on a Kubernetes cluster. KubeRocketCI is designed and functions based on a set of key guiding principles: * **Operator Pattern Approach:** [Approach](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) is used for deployment and configuration, ensuring that the platform aligns with Kubernetes native methodologies (see schema below). * **Loosely Coupling:** KubeRocketCI comprises several loosely coupled operators responsible for different platform parts. These operators can be deployed independently, enabling the most straightforward platform customization and delivery approach. ![Kubernetes Operator](../assets/developer-guide/architecture/operator-pattern-approach.png) The following deployment diagram illustrates the platform's core components, which provide the minimum functional capabilities required for the platform operation: build, push, deploy, and run applications. The platform relies on several mandatory dependencies: * **Ingress:** An ingress controller responsible for routing traffic to the platform. * **Tekton Stack:** Includes Tekton pipelines, triggers, dashboard, chains, etc. * **ArgoCD:** Responsible for GitOps deployment. ![KubeRocketCI Deployment Diagram](../assets/developer-guide/architecture/deployment-diagram.png) * **Codebase Operator:** Responsible for managing git repositories, versioning, and branching. It also implements Jira integration controller. * **CD Pipeline Operator:** Manages Continuous Delivery (CD) pipelines and CD stages (which is an abstraction of Kubernetes Namespace). Operator acts as the bridge between the artifact and deployment tools, like Argo CD. It defines the CD pipeline structure, artifacts promotion logic and triggers the pipeline execution. * **Tekton Pipelines:** Manages Tekton pipelines and processes events (EventListener, Interceptor) from Version Control Systems. The pipelines are integrated with external tools like SonarQube, Nexus, etc. * **KubeRocketCI:** This is the User Interface (UI) component, built on top of Headlamp. **Business applications** are deployed on the platform using the CD Pipeline Operator and Argo CD. By default, the CD Pipeline Operator uses Argo CD as a deployment tool. However, it can be replaced with any other tool, like FluxCD, Spinnaker, etc. The target environment for the application deployment is a Kubernetes cluster where KubeRocketCI is deployed, but it can be any other Kubernetes cluster. --- ## Operator Development # Operator Development This page is intended for developers with the aim to share details on how to set up the local environment and start coding in Go language for KubeRocketCI. ## Prerequisites * [Git](https://github.com/git-guides/install-git) is installed; * One of our [repositories](https://github.com/epam/edp-install/blob/master/README.md#edp-repositories-description) where you would like to contribute is cloned locally; * [Docker](https://docs.docker.com/engine/install/) is installed; * [Kubectl](https://kubernetes.io/docs/setup/) is set up; * Local Kubernetes cluster ([Kind](https://kind.sigs.k8s.io/) is recommended) is installed; * [Helm](https://helm.sh/docs/intro/install/) is installed; * Any IDE ([GoLand](https://www.jetbrains.com/go/) is used here as an example) is installed; * [GoLang](https://go.dev/dl/) stable version is installed. :::note Make sure GOPATH and GOROOT environment variables are added in PATH. ::: ## Environment Setup Set up your environment by following the steps below. ### Set Up Your IDE We recommend using GoLand and enabling the Kubernetes plugin. Before installing plugins, make sure to save your work because IDE may require restarting. #### Set Up Your Operator To set up the cloned operator, follow the three steps below: 1. Configure Go Build Option. Open folder in GoLand, click the ![add_config_button](../assets/developer-guide/add_config_button.png "add_config_button") button and select the `Go Build` option: ![Add configuration](../assets/developer-guide/add_configuration.png "Add configuration") 2. Fill in the variables in Configuration tab: * In the `Files` field, indicate the path to the main.go file; * In the `Working directory` field, indicate the path to the operator; * In the `Environment field`, specify the namespace to watch by setting `WATCH_NAMESPACE` variable. It should equal `default` but it can be any other if required by the cluster specifications. * In the `Environment field`, also specify the platform type by setting `PLATFORM_TYPE`. It should equal either `kubernetes` or `openshift`. ![Build config](../assets/developer-guide/build_config.png "Build config") 3. Check cluster connectivity and variables. Local development implies working within local Kubernetes clusters. [Kind](https://kind.sigs.k8s.io/) (Kubernetes in Docker) is recommended so set this or another environment first before running code. ## Pre-commit Activities Before making commit and sending pull request, take care of precautionary measures to avoid crashing some other parts of the code. ### Testing and Linting Testing and linting must be used before every single commit with no exceptions. The instructions for the commands below are written [here](https://github.com/epam/edp-keycloak-operator/blob/master/Makefile). It is mandatory to run test and lint to make sure the code passes the tests and meets acceptance criteria. Most operators are covered by tests so just run them by issuing the commands "make test" and "make lint": ```bash make test ``` The command "make test" should give the output similar to the following: ![Tests directory for one of the operators](../assets/developer-guide/make_test.png) ```bash make lint ``` The command "make lint" should give the output similar to the following: ![Tests directory for one of the operators](../assets/developer-guide/make_lint.png) #### Observe Auto-Generated Docs, API and Manifests The commands below are especially essential when making changes to API. The code is unsatisfactory if these commands fail. * Generate documentation in the .MD file format so the developer can read it: ```bash make api-docs ``` The command "make api-docs" should give the output similar to the following: !["make api-docs" command with the file contents](../assets/developer-guide/api-docs.png) * There are also manifests within the operator that generate zz_generated.deepcopy.go file in /api/v1 directory. This file is necessary for the platform to work but it's time-consuming to fill it by yourself so there is a mechanism that does it automatically. Update it using the following command and check if it looks properly: ```bash make generate ``` The command "make generate" should give the output similar to the following: !["make generate" command](../assets/developer-guide/make_generate.png) * Refresh custom resource definitions for Kubernetes, thus allowing the cluster to know what resources it deals with. ```bash make manifests ``` The command "make manifests" should give the output similar to the following: !["make manifests" command](../assets/developer-guide/make_manifests.png) At the end of the procedure, you can push your code confidently to your branch and create a pull request. That's it, you're all set! Good luck in coding! ## Related Articles * [KubeRocketCI Project Rules. Working Process](./edp-workflow.md) --- ## Reference Architecture # Reference Architecture The KubeRocketCI Reference Architecture serves as a blueprint for software delivery, outlining the best practices, tools, and technologies leveraged by the platform to ensure efficient and high-quality software development. It provides a comprehensive guide to navigate the complexities of software delivery, from code to deployment. KubeRocketCI operates on Kubernetes, a leading open-source system for automating deployment, scaling, and management of containerized applications. It consolidates a variety of open-source tools, ensuring a flexible and adaptable system that can seamlessly run on any public cloud or on-premises infrastructure. This versatility allows for a wide range of deployment options, **catering to diverse business needs** and **operational requirements**. ## Key Principles The KubeRocketCI is built on a set of key principles that guide its design and functionality: * **Managed Infrastructure and Container Orchestration:** KubeRocketCI is based on a platform that leverages managed infrastructure and container orchestration, primarily through Kubernetes or OpenShift. * **Security:** KubeRocketCI places a high emphasis on security, covering aspects such as authentication, authorization, and Single Sign-On (SSO) for platform services. * **Development and Testing Toolset:** KubeRocketCI provides a comprehensive set of tools for development and testing, ensuring a robust and reliable software delivery process. * **Well-Established Engineering Process:** KubeRocketCI reflects EPAM’s well-established engineering practices (EngX) in its CI/CD pipelines and delivery analytics. * **Open-Source and Cloud-Agnostic:** As an open-source, cloud-agnostic solution, KubeRocketCI can be run on any preferred Kubernetes or OpenShift clusters. * **DevSecOps Excellence:** KubeRocketCI empowers DevSecOps by making security a mandatory quality gate. * **Automated Testing:** KubeRocketCI ensures seamless and predictable regression testing through automated test analysis. ## Architecture Overview KubeRocketCI encompasses a comprehensive CI/CD ecosystem integrating essential tools such as the **Tekton** and **Argo CD**, augmented by additional functionalities. Within this robust framework, KubeRocketCI seamlessly integrates **SonarQube** for continuous code quality assessment, enabling thorough analysis and ensuring adherence to coding standards. Additionally, incorporating **Static Application Security Testing (SAST)** toolset fortifies platform's security posture by proactively identifying vulnerabilities within the codebase. KubeRocketCI leverages dedicated artifact storage solutions to manage and version application artifacts securely, ensuring streamlined deployment processes and traceability throughout the software development lifecycle. See the reference architecture diagram below: ![KubeRocketCI Reference Architecture](../assets/developer-guide/architecture/reference-architecture.png) 1. Developers access the platform by authenticating with their corporate credentials. The platform utilizes OpenID Connect (OIDC) for authentication and authorization across all tools and Kubernetes clusters. Using OIDC, KubeRocketCI establishes a unified and secure authentication mechanism, ensuring seamless access control and user authentication for all integrated tools. This standardized approach upholds strict security protocols, ensuring consistency in authentication and authorization policies across the platform ecosystem. To integrate existing Identity Providers (IdPs), Keycloak serves as an identity broker on the platform. KubeRocketCI offers the keycloak-operator to streamline Keycloak integration within the platform. 2. Developers engage with the platform via the KubeRocketCI, an intuitive interface offering a comprehensive overview of the platform’s capabilities. This centralized hub facilitates seamless navigation and access to various platform tools and components. Within the KubeRocketCI, developers can generate new components (codebases). The platform integrates with version control systems, optimizing source code management, fostering collaboration, and streamlining code review processes. To create new codebases, developers utilize Application Templates, ensuring a standardized approach to application development. The platform accommodates a range of application templates such as Java, Node.js, .NET, Python, and more. Additionally, developers can design custom templates via the KubeRocketCI Marketplace to cater to their specific requirements. 3. Tekton is a potent, adaptable, and cloud-native framework designed for crafting CI/CD systems. It offers a collection of shared, reusable components that empower developers to construct, test, and deploy applications across various cloud providers or on-premises systems. As a foundational element within the KubeRocketCI CI/CD ecosystem, Tekton seamlessly integrates with other tools and services, providing a robust and adaptable framework for constructing CI/CD pipelines. Tekton Pipelines allow developers to efficiently build, test, and deploy applications, while Tekton Triggers initiate pipelines based on specific events. 4. The codebase operator is a crucial part of the platform ecosystem. It manages codebases, their creation, deletion, and scaffolding, as well as their associated resources. It provides versioning, branching, and release capabilities and enables seamless integration with Git servers and Jira. 5. The platform has various cloud-agnostic tools that offer different functionalities, such as artifact storage, static security analysis, and code quality assessment. These tools are accessible through pipelines and codebase controllers. Additionally, the platform supports integration with managed services from cloud providers to deliver its core functionality—for instance, AWS Parameter Store stores secrets and AWS ECR - container images. AzureDevops Artifacts is an option to store artifacts leveraging Azure Cloud capabilities. SonarCloud, a cloud-based version of SonarQube, can be integrated to conduct static code analysis. 6. The CD Pipeline Operator oversees CD pipelines and their related resources. It offers a collection of shared, reusable components for constructing CD pipelines. Integrated with Tekton and Argo CD, the CD Pipeline Operator harnesses their capabilities, ensuring a robust and dependable software delivery process. With a Kubernetes API interface, the CD Pipeline Operator facilitates the management of CD pipelines. It enables artifact promotion logic and the triggering of CD pipelines based on specific events, further enhancing the efficiency and adaptability of the software delivery workflow. 7. Argo CD is a pivotal deployment tool adopted within the platform, embracing the GitOps delivery approach. It serves as the foundation for deploying both operational and business workloads. KubeRocketCI recommends running a dedicated Argo CD instance to manage operational workloads, employing the [Kubernetes add-ons approach](https://github.com/epam/edp-cluster-add-ons) for streamlined management. 8. Production workloads operate in isolation within dedicated Kubernetes clusters to uphold stringent standards and ensure the utmost security and resource allocation. This approach guarantees the highest isolation and operational integrity levels for critical production systems, aligning with industry best practices. KubeRocketCI strongly recommends utilizing a pull model for production deployment. In this model, production deployment is initiated by the Argo CD instance explicitly deployed for the production environment. ## Technology Stack The Platform is meticulously engineered to uphold best practices in workload distribution across various environments, including development, testing (manual/automation), user acceptance (UAT), staging, and production. While lower environments like development and testing may feasibly share clusters for workload efficiency, KubeRocketCI strongly advocates and enforces the necessity of **segregating production workloads into dedicated clusters**. This segregation ensures the highest isolation, security, and resource allocation levels for mission-critical production systems, adhering to industry standards and ensuring optimal operational integrity. KubeRocketCI harnesses the robust capabilities of Kubernetes in conjunction with a suite of powerful tools tailored for monitoring, logging, and tracing. It integrates the **Prometheus** stack within ecosystem, leveraging its metrics collection, storage, and querying capabilities to enable comprehensive monitoring of system performance and health. KubeRocketCI runs **OpenSearch** for centralized logging, enabling efficient log aggregation, analysis, and management across the platform. Incorporating **OpenTelemetry** enables standardized and seamless observability data collection, facilitating deep insights into platform behavior and performance. Additionally, it allows for connection with external aggregators and tools that support the OpenTelemetry protocol (OTLP). ![Platform and Tools](../assets/developer-guide/architecture/reference-architecture-tools.png) KubeRocketCI integrates with GitLab, GitHub, Bitbucket, and Gerrit for version control. These systems are foundational components enabling efficient source code management, collaboration, and code review processes. Platform ensures robust security measures by leveraging OpenID Connect (OIDC) for authentication and authorization across all platform tools and Kubernetes clusters. By employing OIDC, KubeRocketCI establishes a unified and secure authentication mechanism, enabling seamless access control and user authentication for all tools integrated into the platform. This standardized approach ensures stringent security protocols, maintaining authentication consistency and authorization policies across the platform ecosystem. ## Related Articles * [Reference CI/CD Pipeline](./reference-cicd-pipeline.md) --- ## Reference CI/CD Pipeline # Reference CI/CD Pipeline This document provides an in-depth overview of the Continuous Integration and Continuous Delivery (CI/CD) pipeline reference architecture implemented within the KubeRocketCI. The pipeline is designed to facilitate efficient and automated software deployment across diverse environments, leveraging a suite of tools and methodologies for enhanced reliability, scalability, and security. ## CI/CD Pipeline Architecture The CI/CD pipeline within KubeRocketCI orchestrates the software delivery process, encompassing several sequential stages to ensure robustness and reliability. ![KubeRocketCI CI/CD Pipeline](../assets/developer-guide/architecture/reference-cicd-pipeline.png) The CI/CD Pipeline follows a modular and scalable architecture that leverages various tools to ensure the reliability and efficiency of the software delivery process. The architecture can be divided into stages, each responsible for specific tasks. Explore the key components involved in the pipeline and their functionalities: 1. **Source Code:** The pipeline starts with the source code, representing the application's codebase. Developers commit their changes to the source code repository, triggering the pipeline. 2. **Validate Commit Message:** The commit message validation component checks the format and content of the commit message. It ensures the commit message follows the correct format and includes a valid Tracking Issue key. It helps maintain a standardized commit message format throughout the application development. 3. **Build:** The Build component compiles the source code, runs unit tests, and generates the application artifact. It consumes the artifact from the Artifact Repository (Nexus), ensuring consistent and reliable builds. 4. **SAST with SonarQube:** The Static Analysis Security Testing (SAST) component utilizes SonarQube to analyze the source code for potential security vulnerabilities, code smells, and quality issues. This step helps identify and address security or code quality issues early in development. 5. **SCA:** The Software Composition Analysis (SCA) component performs dependency analysis using cdxgen, Dependency-Track, Semgrep, and DefectDojo. It checks for known vulnerabilities or license compliance issues in the application's dependencies. By identifying and resolving these issues, it ensures the security and stability of the software. 6. **Publish:** The Publish component publishes the application artifact to the Artifact Repository. It posts Docker images to the Docker Registry and stores binary artifacts in the Nexus Repository. This process ensures that the artifacts are securely stored and easily accessed for future deployments. 7. **Deploy:** The Deploy component uses Argo CD or Tekton to deploy applications to target environments in Kubernetes, leveraging Helm charts to ensure seamless deployment. Deploy to Test/Quality Assurance/Performance Environments: The final stages of the pipeline involve deploying the application to different environments for testing and quality assurance purposes. The results of the tests are consolidated and reported to the Report Portal, facilitating efficient test reporting and analysis. The overall architecture of the CI/CD Pipeline ensures a streamlined and automated software delivery process, from source code to deployment. It provides developers with the necessary tools and processes to ensure their applications' quality, security, and scalability. Furthermore, Tekton Chains enhances supply chain security by signing and generating in-toto metadata that verifies the integrity of artifacts and the CI/CD Pipeline. :::note The tools mentioned in this document are just examples and can be replaced with other tools that offer similar functionality. For instance, instead of **Harbor** for the Docker Registry, it is possible to use **AWS ECR**. Consider using **Azure Artifacts** or **JFrog Artifactory** instead of **Nexus** for the artifact repository. Instead of setting up an self-managed instance of **SonarQube**, leverage *SonarCloud*, the cloud-based version of SonarQube, as an alternative. The CI/CD Pipeline architecture is flexible and adaptable, allowing the use of different tools based on specific project requirements and platform preferences. ::: --- ## Telemetry # Telemetry The codebase operator for the KubeRocketCI gathers anonymous data through telemetry. This data provides a clear picture of how the platform is being used and empowers the development team to make informed decisions and strategic enhancements to meet evolving operational needs. The anonymous data collected also plays an essential role in adopting a Software Development Life Cycle (SDLC) process strategically. ## Telemetry Data The codebase-operator collects the following data: - The version of the platform - The number of codebases created and their parameters: language (for example, Java, NodeJS, etc.), framework (for example, FastAPI, Flask, etc.), build tool (for example, Maven, Gradle, etc.), strategy (for example, Clone, Create, Import), and Type (for example, library, application, etc.) - The number of CD pipelines created and their parameters: deployment type (for example, Auto, Manual), and the number of stages - The number of Git providers connected to the platform and their types (for example GitHub, GitLab, Bitbucket, or Gerrit) - Where Jira is enabled or not - The type of the Docker registry connected to the platform (for example, Docker Hub, Harbor, ECR) ```go package telemetry type CodebaseMetrics struct { Lang string `json:"lang"` Framework string `json:"framework"` BuildTool string `json:"buildTool"` Strategy string `json:"strategy"` Type string `json:"type"` Versioning string `json:"versioning"` } type CdPipelineMetrics struct { DeploymentType string `json:"deploymentType"` NumberOfStages int `json:"numberOfStages"` } type PlatformMetrics struct { CodebaseMetrics []CodebaseMetrics `json:"codebaseMetrics"` CdPipelineMetrics []CdPipelineMetrics `json:"cdPipelineMetrics"` GitProviders []string `json:"gitProviders"` JiraEnabled bool `json:"jiraEnabled"` RegistryType string `json:"registryType"` Version string `json:"version"` } ``` You can verify the code which collects the data in the [codebase-operator](https://github.com/epam/edp-codebase-operator/tree/master/pkg/telemetry) repository. ## Collecting Timeline The codebase-operator collects the data every 24 hours and sends it to the KubeRocketCI Telemetry Service. The first data points are collected 24 hours after the codebase-operator is deployed to allow users to opt-out of telemetry. ## Disabling Telemetry The codebase-operator collects telemetry data by default. To disable telemetry, set the `TELEMETRY_ENABLED` environment variable to `false` in the codebase-operator's deployment configuration. To achieve this, run the following command: ```bash helm upgrade --install codebase-operator codebase-operator/codebase-operator --set "telemetryEnabled=false" ``` --- ## Platform Glossary # Glossary To help you navigate the platform more effectively, familiarize yourself with the definitions and context of the most useful platform terms presented in the table below. | Terms | Details | |---|---| | **Platform Component** - an item used in the CI/CD process |**Portal UI** - component that facilitates the management, setup, and control of business entities.| | |**Artifactory** - component that serves as a repository for all binary artifacts. _**NOTE**: Nexus is a possible implementation of a repository._| | |**CI/CD Server** - component that executes pipelines responsible for building, testing, and deploying code. _**NOTE**: Tekton is a possible implementation of a CI/CD server._| | |**Code Review tool** - component that facilitates collaboration and review of code changes. _**NOTE**: GitHub/GitLab is a possible implementation of a code review tool._| | |**Identity Server** - an authentication server that provides a centralized way to verify requests for all applications. _**NOTE**: Keycloak is a possible implementation of an identity server._| | |**Security Realm Tenant** - a realm in the identity server (e.g., Keycloak) where user accounts and access permissions are managed. The realm is unique to the identity server instance.| | |**Static Code Analyzer** - component that continuously inspects code quality before changes are merged into the master branch. _**NOTE**: SonarQube is a possible implementation of a static code analyzer._| | |**VCS (Version Control System)** - a repository that tracks all changes made by developers. _**NOTE**: GitHub and GitLab are possible implementations of a version control system._| | **KubeRocketCI Business Entity** - a part of the CI/CD process (the integration, delivery, and deployment of any codebase changes) |**Application** - a codebase type that is built as the binary artifact and deployable unit with the code that is stored in VCS. As a result, the application becomes a container and can be deployed in an Environment.| | |**Autotests** - a codebase type that inspects a product (e.g. an application set) on a Deployment Environment. Autotests are not deployed to any container and launched from the respective code repository.| | |**Deployment (Continuous Delivery Pipeline)** - business entity that describes the whole delivery process of the selected application set via the respective Environments. The main idea of the CD pipeline is to promote the application version between the Environments by applying the sequential verification (i.e. the second Environment will be available if the verification on the first Environment is successfully completed). _**NOTE**: The Deployment can include the essential set of applications with its specific Environments as well._| | |**Environment** - a logical entity that facilitates the inspection of application sets. Each Environment is linked to a distinct Kubernetes namespace dedicated to containerized workloads, ensuring a one-to-one mapping between the Environment and the namespace. This structure allows for the sequential promotion of applications from one Environment to the next.| | |**Project (also codebase)** - business entity that possesses a code.| | |**Project Branch** - business entity that represents a specific version in a Git branch. Every Project branch has a Codebase Docker Stream entity.| | |**Codebase Image Stream** - serves as a key component in managing the deployment lifecycle of an application version. It facilitates the identification and progression of application versions that are ready for deployment. Within the Deployment, each Environment leverages Codebase Image Streams (CBIS) to manage application versions effectively. For instance, an application named 'application1' associated with a master branch would be initially tagged in the CBIS using the format: `[codebase name]-[branch name]`, e.g., 'application1-main'. As the application version progresses through the Deployment, reaching readiness for deployment in a specific Environment, the CBIS naming convention evolves to indicate its Deployment and target Environment, following the structure: `[deployment name]-[environment name]-[project name]-verified`. This approach ensures a clear and systematic tracking of application versions from their initial identification through to their readiness for deployment in various Environments within the deploy pipeline.| | |**Git Server** - a custom resource that is responsible for integration with Version Control System (VCS), whether it is GitHub, GitLab or Gerrit.| | |**Infrastructure** - a Project type that is used to define and manage the underlying infrastructure of projects using the _Infrastructure as Code (IaC)_ approach, ensuring consistency and reproducibility.| | |**Library** - a Project type that is built as the binary artifact, i.e. it`s stored in the Artifactory and can be uploaded by other applications, autotests or libraries.| | |**Quality Gate** - a business entity that defines the minimum criteria that must be met after testing to allow an application to be promoted to the next Environment. Each Environment enforces its own quality gate, which may require either manual approval from a QA specialist or a successful run of automated tests (autotests).| | |**Quality Gate Type** - this value defines trigger type that promotes artifacts (images) to the next Environment in a deployment. There are manual and automatic types of quality gates. The manual type means that the promoting process should be confirmed in Tekton. The automatic type promotes the images automatically in case there are no errors in the Allure Report. _**NOTE**: If any of the test types is not passed, the deploy pipeline will fail._| | |**Trigger Type** - a value that defines a trigger type used for the CD pipeline triggering. There are manual and automatic types of triggering. The manual type means that the CD pipeline should be triggered manually. The automatic type triggers the CD pipeline automatically as soon as the Codebase Docker Stream was changed.| | |**Automated Tests** - different types of automated tests that can be run on the Deployment for a specific Environment.| | |**Build Pipeline** - a Tekton pipeline that builds a corresponding Project branch in the Project.| | |**Build Stage** - a stage that takes place after the code has been submitted/merged to the repository of the main branch (_the pull request from the feature branch is merged to the main one, the Patch set is submitted in GitHub/GitLab_).| | |**Code Review Pipeline** - a Tekton pipeline that inspects the code candidate in the Code Review tool, triggered on Pull Request created/updated event.| | |**Code Review Stage** - a stage where code is reviewed before it goes to the main branch repository of the version control system (_the commit to the feature branch is pushed, the Patch set is created in GitHub/GitLab_).| | |**Deploy Pipeline** - a Tekton pipeline that is responsible for the CD Pipeline Stage deployment with the full set of applications and autotests.| | |**Environment** - a part of the Continuous Delivery where artifacts are being deployed to.| | |**CI Pipelines** - an orchestrator for stages that is responsible for the common technical events, e.g. initialization, in Tekton pipeline.| | |**Environment** - a namespace where the built and packed into an image applications are deployed for further testing. It is possible to deploy several applications to several Environments (Team and Integration environments) within one Deployment.| | |**Integration Environment** - an environment type that is always deployed as soon as the new application version is built in order to launch the integration test and promote images to the next Environments. The Integration Environment can be triggered manually or in case a new image appears in the Container registry.| |**OpenShift / Kubernetes (K8S)**|**ConfigMap** - a resource that stores configuration data and processes the strings that do not contain sensitive information.| | |**Container** - is a lightweight, standalone, and executable package.| | |**Container Registry** - a store for the Container that is created for the application after the Build pipeline performance.| | |**OpenShift Web Console** - a web console that enables to view, manage, and change OpenShift / K8S resources using browser.| | |**Operator Framework** - a deployable unit in OpenShift that is responsible for one or a set of resources and performs its life circle (adding, displaying, and provisioning).| | |**Path** - a route component that helps to find a specified path (e.g. /api) at once and skip the other.| | |**Pod** - the smallest deployable unit of the large microservice application that is responsible for the application launch. The pod is presented as the one launched container. When the container is collected, it will be kept in Container Registry and then saved as Pod in the OpenShift project. _**NOTE**: The Deployment Config is responsible for the Pod push, restart, and stop processes._| | |**PV (Persistent Volume)** - a cluster resource that captures the details of the storage implementation and has an independent lifecycle of any individual pod.| | |**PVC (Persistent Volume Claim)** - a user request for storage that can request specific size and access mode. PV resources are consumed by PVCs.| | |**Route** - a resource in OpenShift that allows getting the external access to the pushed application.| | |**Secret** - an object that stores and manages all the sensitive information (e.g. passwords, tokens, and SSH keys).| | |**Service** - an external connection point with Pod that is responsible for the network. A specific Service is connected to a specific Pod using labels and redirects all the requests to Pod as well.| | |**Site** - a route component (link name) that is created from the indicated application name and applies automatically the project name and a wildcard DNS record.| --- ## KubeRocketCI Documentation # KubeRocketCI Documentation **KubeRocketCI (previously known as EPAM Delivery Platform)** is an **open-source** cloud-agnostic SaaS/PaaS solution for software development, licensed under **Apache License 2.0**. It provides a pre-defined set of CI/CD patterns and tools, which allow a user to start product development quickly with established **code review**, **release**, **versioning**, **branching**, **build** processes. ## Why KubeRocketCI? KubeRocketCI, also called **"The Rocket"**, shortens the time from project initiation to active development from several months to just hours. The platform delivers: - **Integrated CI/CD Tools**: Pre-configured toolset including Tekton, ArgoCD, Nexus, SonarQube, and more - **Multi-Language Support**: CI pipelines for Java, Python, C#, Go, JavaScript and others - **Infrastructure Agnostic**: Runs on Kubernetes/OpenShift in any cloud or on-premises - **Security Focus**: Built-in authentication, authorization, and SSO for all platform services - **Production-Ready**: Includes observability, monitoring, and established engineering practices ## Documentation Sections ### For New Users - [Quick Start](./quick-start/platform-installation.md) - Get up and running quickly - [About the Platform](./about-platform.md) - Overview of KubeRocketCI - [Basic Concepts](./basic-concepts.md) - Key terminology and concepts ### For Developers - [User Guide](./user-guide/index.md) - Working with the KubeRocketCI portal and CI/CD flow - [Developer Guide](./developer-guide/index.md) - Detailed information for platform developers - [Use Cases](./use-cases/index.md) - Real-world implementation examples ### For Administrators - [Operator Guide](./operator-guide/index.md) - Installation, configuration, and administration - [API Reference](./api/cd-pipeline.md) - Programmatic interface details - [Supported Versions](./supported-versions.md) - Compatibility information ### Additional Resources - [Glossary](./glossary.md) - Terms and definitions - [Compliance](./compliance.md) - Security and compliance information - [FAQ](/faq/general-questions) - Frequently asked questions ## Getting Started The fastest way to explore KubeRocketCI is to follow our [Quick Start Guide](./quick-start/platform-installation.md), which will guide you through the installation process and basic platform usage. For a deeper understanding of the platform architecture, components, and capabilities, visit the [About the Platform](./about-platform.md) page. --- ## Install via Add-Ons # Install via Add-Ons This page provides an overview of Cluster Add-Ons for KubeRocketCI, including their purpose, benefits, and usage. ## What Are Add-Ons KubeRocketCI Add-Ons is essentially a Kubernetes-based deployment pattern that allows users to easily install additional components for the platform using Argo CD applications. Add-Ons have been introduced into KubeRocketCI starting from version 3.4.0. They empower users to seamlessly integrate the platform with various additional components, such as SonarQube, Nexus, Keycloak, Jira, and more. This eliminates the need for manual installations, as described in the [Install KubeRocketCI](../operator-guide/install-kuberocketci.md) page. In a nutshell, Add-Ons are separate Helm charts that can be installed with just one click using the Argo CD tool. ## Add-Ons Repository Structure All the Add-Ons for KubeRocketCI are stored in our [GitHub repository](https://github.com/epam/edp-cluster-add-ons) following the GitOps approach. In addition to default Helm and Git files, the repository contains custom resources called Applications for Argo CD and application source code. The repository adheres to the GitOps approach, allowing for easy rollback of changes when necessary. The structure of the repository is as follows:
View: edp-cluster-add-ons repo structure ```bash edp-cluster-add-ons ├── argo-cd │ ├── Chart.yaml │ ├── README.md │ ├── templates │ │ ├── appProjectCore.yaml │ │ ├── appProjectKRCI.yaml │ │ ├── ... │ └── values.yaml ├── ... ├── clusters │ ├── core │ │ ├── addons │ │ │ ├── atlantis │ │ │ │ ├── Chart.yaml │ │ │ │ ├── README.md │ │ │ │ ├── templates │ │ │ │ └── values.yaml │ │ │ ├── aws-efs-csi-driver │ │ │ │ ├── Chart.yaml │ │ │ │ ├── README.md │ │ │ │ ├── templates │ │ │ │ └── values.yaml │ │ │ ├── ... │ │ ├── apps │ │ │ ├── templates │ │ │ │ ├── atlantis.yaml │ │ │ │ ├── aws-efs-csi-driver.yaml │ │ │ │ ├── ... │ │ │ ├── Chart.yaml │ │ │ ├── README.md │ │ │ └── values.yaml │ │ └── bootstrap-addons.yaml │ └── prod ├── ... ```
- **argo-cd** - The directory containing the Helm chart for deploying the main Argo CD instance and prepared project templates. - **clusters** - A directory for organizing configurations and Helm charts specific to individual clusters. Each subdirectory corresponds to a particular cluster environment (e.g., core, prod, or dev). - **addons** - The directory containing subdirectories with Helm charts for applications to install, such as Nexus, SonarQube, Keycloak, etc. See the full add-ons list [below](#available-add-ons-list). - **apps** - Contains application `templates` directory used to create Argo CD application. The deployment of these applications is managed by modifying the `apps/values.yaml` file, where you can enable or disable specific applications. - **bootstrap-addons.yaml** - The manifest file defining the app of apps application responsible for deploying all enabled application to the cluster. This manifest is applied manually. ## Enable KubeRocketCI Add-Ons To enable Add-Ons, it is necessary to have the configured Argo CD, and connect and synchronize the forked repository. To do this, follow the guidelines below: 1. Create a fork of the [Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) in your personal Git account. 2. Align **repoUrl** in [/clusters/core/apps/values.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml#L14), [/clusters/core/bootstrap-addons.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/bootstrap-addons.yaml#L16) and **repoSource** in the [/argo-cd/templates/appProjectCore.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/argo-cd/templates/appProjectCore.yaml#L36), [/argo-cd/templates/appProjectCore.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/argo-cd/templates/appProjectKRCI.yaml#L55) files of the repository, specify the SSH URL of your fork. For GitHub, replace `kuberocketci` with your ``. 3. Clone a forked add-ons repository to your local machine and install pre-configured **Argo CD** Helm chart from `/argo-cd` folder using the command below: ```bash helm install argocd argo-cd -n argocd --create-namespace ``` 4. (Optional) If you don't have an ingress controller created in your Kubernetes cluster, enable port-forwarding for the Argo CD service using the command below: ```bash kubectl port-forward -n argocd service/argo-cd-argocd-server 65080:80 ``` This will forward the Argo CD service to `http://localhost:65080` on your local machine. Open this URL in your browser to access the Argo CD interface. 5. As soon as Helm chart is deployed, open your Argo CD endpoint and navigate to **Settings** -> **Repositories**. Connect your repository where you have the values.yaml files changed by clicking the **+ CONNECT REPO** button: ![Connect the forked repository](../assets/operator-guide/addons-overview/argo_cd_connect_repo.png "Connect the forked repository") 6. In the appeared window, fill in the following fields and click the **CONNECT** button: - Choose your connection method - `VIA SSH` - Name - `addons-demo` (optional) - Project - select project - Repository URL - enter the `SSH URL` of your forked repository (ssh://git@github.com:22/``/edp-cluster-add-ons.git) ![Repository parameters](../assets/operator-guide/addons-overview/argo_cd_repo_fields.png "Repository parameters") 7. As soon as the repository is connected, the new item will appear in the repository list: ![Connected repository](../assets/operator-guide/addons-overview/argo_cd_connected_repo.png "Connected repository") 8. Apply the add-ons management application manifest and open **Application** tab: ```bash kubectl apply -f edp-cluster-add-ons/clusters/core/bootstrap-addons.yaml -n argocd ``` ![Applications list](../assets/operator-guide/addons-overview/argo_cd_application_list.png "Applications list") 9. Click the **addon-core** application to open its details: ![Add-ons management menu](../assets/operator-guide/addons-overview/argo_cd_app_of_apps.png "Add-ons management menu") ## Install Add-Ons Now that Add-Ons are enabled in Argo CD, they can be installed by following the steps below: 1. Choose the Add-On to install. 2. Enable the Add-Ons chosen in the [/clusters/core/apps/values.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml) file. You can also re-define Add-On parameters. Refer to the example below: ```bash sonar: createNamespace: true enable: true sonar-operator: createNamespace: true enable: true ``` 3. Navigate to the Argo CD Add-Ons application. On the chosen Add-On, click the **⋮** button and then **Details**: ![Open the Add-On](../assets/operator-guide/addons-overview/argo_cd_sync_schonen_app.png "Open Add-Ons") 4. To install the Add-On, click the **⋮** button -> **Sync**: ![Install Add-Ons](../assets/operator-guide/addons-overview/argocd_synchronize.png "Install Add-Ons") 5. Once the Add-On is installed, the *Sync OK* message will appear in the Add-On status bar: ![Sync OK message](../assets/operator-guide/addons-overview/argo_cd_sync_ok.png "Sync OK message") 6. Open the application details by clicking on the little square with an arrow underneath the Add-On name: ![Open details](../assets/operator-guide/addons-overview/argo_cd_open_details_sonar.png "Open details") 7. By default in all addons application `auto sync is not enabled` to install application click **Sync** button. ![Open details](../assets/operator-guide/addons-overview/acgo_cd_sync_sonar_app.png "Open details") 8. Track application resources and status in the **App details** menu: ![Application details](../assets/operator-guide/addons-overview/argo_cd_application_status.png "Application details") Argo CD provides excellent observability and monitoring capabilities for its resources, which is particularly beneficial when utilizing KubeRocketCI Add-Ons. ## Available Add-Ons List The list of the available Add-Ons: :::info Consult repository [README.md](https://github.com/epam/edp-cluster-add-ons?tab=readme-ov-file#available-add-ons) for the most up-to-date information. ::: |Name|Description|Default| |:-|:-|:-:| |Argo CD|A GitOps continuous delivery tool that helps automate the deployment, configuration, and lifecycle management of applications in Kubernetes clusters.|false| |AWS EFS CSI Driver|A Container Storage Interface (CSI) driver that enables the dynamic provisioning of Amazon Elastic File System (EFS) volumes in Kubernetes clusters.|false| |Cert Manager|A native Kubernetes certificate management controller that automates the issuance and renewal of TLS certificates.|false| |Capsule|A multi-tenancy solution for Kubernetes clusters that provides each tenant with a dedicated namespace where they can freely run their workloads with resource quotas.|false| |Capsule tenant|A specific tenant within the Capsule multi-tenancy solution. Each tenant has its own isolated environment within the Kubernetes cluster.|false| |DefectDojo|A security vulnerability management tool that allows tracking and managing security findings in applications.|false| |Dependency-Track|A Software Composition Analysis (SCA) platform that helps identify and manage open-source dependencies and their associated vulnerabilities.|false| |KubeRocketCI(EDP)|An internal platform created by EPAM to enhance software delivery processes using DevOps principles and tools.|false| |Extensions OIDC|KubeRocketCI Helm chart to provision OIDC clients for different Add-Ons using EDP Keycloak Operator.|false| |External Secrets|A Kubernetes Operator that fetches secrets from external secret management systems and injects them as Kubernetes Secrets.|false| |Fluent Bit|A lightweight and efficient log processor and forwarder that collects and routes logs from various sources in Kubernetes clusters.|false| |Harbor|A cloud-native container image registry that provides support for vulnerability scanning, policy-based image replication, and more.|false| |Ingress nginx|An Ingress controller that provides external access to services running within a Kubernetes cluster using Nginx as the underlying server.|false| |Jaeger Operator|An operator for deploying and managing Jaeger, an end-to-end distributed tracing system, in Kubernetes clusters.|false| |Keycloak|An open-source Identity and Access Management (IAM) solution that enables authentication, authorization, and user management in Kubernetes clusters.|false| |Keycloak PostgreSQL|A PostgreSQL database operator that simplifies the deployment and management of PostgreSQL instances in Kubernetes clusters.|false| |krci-audit|An append-only admission audit trail (capture, store, and read API) recording who created, updated, or deleted KubeRocketCI resources.|false| |MinIO Operator|An operator that simplifies the deployment and management of MinIO, a high-performance object storage server compatible with Amazon S3, in Kubernetes clusters.|false| |Nexus|Serves as a repository manager, enabling the proxying, aggregation, and management of dependencies, as well as the storage of artifacts.|false| |Nexus Operator|A tool designed to streamline the handling of Nexus resources and configurations, facilitating the proxying, collection, and management of dependencies within a repository manager.|false| |OpenSearch|A community-driven, open-source search and analytics engine that provides scalable and distributed search capabilities for Kubernetes clusters.|false| |OpenTelemetry Operator|An operator for automating the deployment and management of OpenTelemetry, a set of observability tools for capturing, analyzing, and exporting telemetry data.|false| |PostgreSQL Operator|An operator for running and managing PostgreSQL databases in Kubernetes clusters with high availability and scalability.|false| |Report Portal|An powered test automation dashboard that allows you to analyze test results, identify issues, and track testing progress.|false| |Prometheus Operator|An operator that simplifies the deployment and management of Prometheus, a monitoring and alerting toolkit, in Kubernetes clusters.|false| |Redis Operator|An operator for managing Redis, an in-memory data structure store, in Kubernetes clusters, providing high availability and horizontal scalability.|false| |Sonar|A comprehensive open-source platform dedicated to the continuous evaluation of code quality, automatically scrutinizing code to identify bugs, code smells, and security vulnerabilities.|false| |Sonar Operator|An operator that simplifies the management and configuration of SonarQube.|false| |StorageClass|A Kubernetes resource that provides a way to define different classes of storage with different performance characteristics for persistent volumes.|false| |Tekton|A flexible and cloud-native framework for building, testing, and deploying applications using Kubernetes-native workflows.|false| |Tekton Monitoring|Prometheus ServiceMonitors and Grafana dashboards for the Tekton stack, covering pipeline throughput, task duration, queueing, and control plane health.|false| |Tekton-cache||false| |Vault|An open-source secrets management solution that provides secure storage, encryption, and access control for sensitive data in Kubernetes clusters.|false| |Velero|An open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.|false| --- ## Install Capsule # Install Capsule This article explains how the KubeRocketCI leverages Capsule capabilities to enable isolation for both the [core](../install-kuberocketci.md) platform components and the business applications [workload](../../user-guide/add-cd-pipeline.md). KubeRocketCI uses Capsule to ensure resource isolation. It is essential to define constraints through the Capsule tenant approach. This approach serves two primary objectives: limiting the resources allocated to the KubeRocketCI components and regulating the resources utilized by each deployed environment. To ensure isolation for the core platform components, create the `krci` namespace under the Capsule tenant. Use the template provided in this instruction to create the Capsule tenant specifically for the core components. ## Integration The following diagram outlines the general steps involved in configuring Capsule for seamless integration with KubeRocketCI. This process ensures efficient resource isolation, allowing for the deployment and management of both KubeRocketCI core platform components and business application workloads. Follow the sequential flow of the diagram to successfully integrate Capsule with the KubeRocketCI platform: ```mermaid graph LR; A(Capsule Installation) --> B(Tenant Configuration) --> C(Impersonation) --> D(Create Namespace)--> E(Deploy KubeRocketCI) ``` 1. **Capsule Installation** - This initial step involves setting up Capsule on your environment. 2. **Tenant Configuration** - Once Capsule is installed, the next critical step is configuring the Capsule tenant. This involves defining specific parameters and constraints to regulate the allocation of resources to KubeRocketCI components. 3. **Impersonation** - Impersonation plays a role in managing user identities and permissions within the Capsule environment. This step ensures secure and controlled access to resources. 4. **Create Namespace** - The creation of a dedicated namespace under Capsule is crucial for isolating and managing the core components of the platform. This step establishes the environment where KubeRocketCI will be deployed (`krci` by default). 5. **Deploy KubeRocketCI** - The final step involves deploying the platform within the configured Capsule environment. ## Installation The installation procedure consists of two steps: 1. Define the namespace where KubeRocketCI will be installed in Capsule values: ```yaml title="values.yaml" manager: options: # -- Boolean, enforces the Tenant owner, during Namespace creation, to name it using the selected Tenant name as prefix, separated by a dash forceTenantPrefix: true # enable capsule for KubeRocketCI tenant and cd-pipeline-operator # NOTE: Capsule cannot manage cluster system namespaces capsuleUserGroups: - capsule.clastix.io # enable for cd-pipeline-operator https://github.com/epam/edp-cd-pipeline-operator/blob/release/2.17/deploy-templates/values.yaml#L10 - system:serviceaccounts:krci # krci - Namespace where KubeRocketCI will be installed. - masters # Kubernetes group that manages the KubeRocketCI tenant. # uncomment if Argo CD manage main KubeRocketCI tenant #- system:serviceaccounts:argocd # uncomment if edp-oidc-admins group manage main KubeRocketCI tenant #- edp-oidc-admins ``` 2. To install the Capsule tool, use the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) approach. For more details, please refer to the [Capsule](https://capsule.clastix.io/docs/general) official page. ## Configuration After installing Capsule, the next crucial step is configuration. Follow the guidelines provided below to configure Capsule and ensure seamless integration with your system: 1. Specify the Capsule tenant configuration: ```yaml title="krci-tenant-example.yaml" apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: krci-tenant spec: ... owners: - clusterRoles: - admin - capsule-namespace-manager kind: Group name: system:masters # # Uncomment if Argo CD manage KubeRocketCI tenant #- clusterRoles: # - admin # - capsule-namespace-manager # kind: ServiceAccount # name: system:serviceaccount:argocd:argocd-application-controller # # Uncomment if edp-oidc-admins group manage KubeRocketCI tenant #- clusterRoles: # - admin # - capsule-namespace-manager # kind: Group # name: edp-oidc-admins ... ``` `system:masters` Group that manages the KubeRocketCI tenant. Must be declared in the above configuration of the `capsuleUserGroups`. The following example includes additional specifications that can be used to manage the KubeRocketCI tenant: :::note Tenant configuration example ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: krci-tenant spec: ingressOptions: allowWildcardHostnames: false allowedHostnames: allowedRegex: ^.*example.com$ # DNSwildcard for KubeRocketCI usage hostnameCollisionScope: Tenant limitRanges: items: - limits: # The default limits apply to each container unless otherwise specified by default - default: cpu: 768m memory: 768Mi # The default requests apply to each container unless otherwise specified by default defaultRequest: cpu: 256m memory: 512Mi type: Container - limits: # In case Tekton pipelines need to use volume workspaces. If use emptydir please set it to 0. - max: storage: 3Gi min: storage: 3Gi type: PersistentVolumeClaim # Since EDKubeRocketCIP uses one namespace, the namespace quota is set to 1 namespaceOptions: quota: 1 networkPolicies: items: - ingress: - from: - namespaceSelector: matchLabels: capsule.clastix.io/tenant: krci-tenant - podSelector: {} - ipBlock: cidr: 172.32.0.0/16 podSelector: {} policyTypes: - Ingress # Default KubeRocketCI admins group to make admin users tenant owners owners: - clusterRoles: - admin - capsule-namespace-manager kind: Group name: system:masters # # Uncomment if Argo CD manage main KubeRocketCI tenant #- clusterRoles: # - admin # - capsule-namespace-manager # kind: ServiceAccount # name: system:serviceaccount:argocd:argocd-application-controller # # Uncomment if edp-oidc-admins group manage main KubeRocketCI tenant #- clusterRoles: # - admin # - capsule-namespace-manager # kind: Group # name: sandbox-oidc-cluster-admins resourceQuotas: items: # The maximum CPU and Memory capacity for the KubeRocketCI tenant - hard: limits.cpu: '2' limits.memory: 2Gi # The maximum number of pods that can be deployed within a namespace - hard: pods: '15' scope: Tenant serviceOptions: # Enable the capabilities to create ClusterIP service types only allowedServices: externalName: false loadBalancer: false nodePort: false ``` ::: 2. To install the Capsule tenant, use the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) approach. 3. [Create namespace](https://capsule.clastix.io/docs/general/tutorial/#create-namespaces) for the KubeRocketCI tenant: 1. Log in as a `system:masters` member and create a namespace for platform deployment under the Capsule tenant using the following command: ```bash kubectl create namespace krci ``` 2. To ensure that the `krci` namespace is successfully created under the Capsule tenant, you can verify the status of the Capsule tenant using the following command: ```bash kubectl get tenant krci-tenant -o yaml ``` Expected output: ```yaml ... status: namespaces: - krci size: 1 state: Active ``` 4. Enable Capsule in the values.yaml file to make it possible to manage deployable environments: ```yaml title="values.yaml" cd-pipeline-operator: tenancyEngine: "capsule" ``` By following these instructions, Capsule policies are utilized to manage the KubeRocketCI core components and deployable environments. By adhering to these guidelines, you should successfully install KubeRocketCI with Capsule enabled as the `tenancyEngine`. ## Related Articles * [Install KubeRocketCI With Values File](../install-kuberocketci.md) * [Cluster Add-Ons Overview](../add-ons-overview.md) --- ## Custom Certificates # Custom Certificates Familiarize yourself with the detailed instructions on adding certificates to KubeRocketCI resources as well as with the respective setup for Keycloak. KubeRocketCI components that support custom certificates can be found in the table below: | Helm Chart | Sub Resources | | - | - | | gerrit-operator | edp-gerrit | | sonar-operator | sonar-operator, edp-sonar | | keycloak-operator | keycloak-operator | | nexus-operator | oauth2-proxy | | edp-install | oauth2-proxy | | edp-headlamp | edp-headlamp | ## Prerequisites - The certificate in the `*.crt` format is used - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. - [Java](https://www.oracle.com/in/java/technologies/downloads/) with the `keytool` command inside - [jq](https://stedolan.github.io/jq/) is installed. ## Enable the SPI Truststore of Keycloak To import custom certificates to Keycloak, follow the steps below: 1. Generate the `cacerts` local keystore and import the certificate there using the keytool tool: ```bash keytool -importcert -file CA.crt \ -alias CA.crt -keystore ./cacerts \ -storepass changeit -trustcacerts \ -noprompt ``` 2. Create the `custom-keycloak-keystore` keystore secret from the `cacerts` file in the `security` namespace: ```bash kubectl -n security create secret generic custom-keycloak-keystore \ --from-file=./cacerts ``` 3. Create the `spi-truststore-data` SPI truststore secret in the `security` namespace: ```bash kubectl -n security create secret generic spi-truststore-data \ --from-literal=KC_SPI_TRUSTSTORE_FILE_FILE=/opt/keycloak/spi-certs/cacerts \ --from-literal=KC_SPI_TRUSTSTORE_FILE_PASSWORD=changeit ``` 4. Update the Keycloak `values.yaml` file from the [Install Keycloak](../auth/keycloak.md) page.
values.yaml ```yaml ... extraVolumeMounts: | ... # Use the Keycloak truststore for SPI connection over HTTPS/TLS - name: spi-certificates mountPath: /opt/keycloak/spi-certs readOnly: true ... extraVolumes: | ... # Use the Keycloak truststore for SPI connection over HTTPS/TLS - name: spi-certificates secret: secretName: custom-keycloak-keystore defaultMode: 420 ... ... extraEnvFrom: | - secretRef: name: spi-truststore-data ... ```
## Enable Custom Certificates in KubeRocketCI Components Creating custom certificates is a necessary but not sufficient condition for applying, therefore, certificates should be enabled as well. 1. Create the `custom-ca-certificates` secret in the platform namespace (e.g `krci`). ```bash kubectl -n krci create secret generic custom-ca-certificates \ --from-file=CA.crt ``` 2. Add the certificate by mounting the `custom-ca-certificates` secret to the operator pod as a volume. Example of specifying custom certificates for the `keycloak-operator`: ```yaml ... keycloak-operator: enabled: true # -- Additional volumes to be added to the pod extraVolumes: - name: custom-ca secret: defaultMode: 420 secretName: custom-ca-certificates # -- Additional volumeMounts to be added to the container extraVolumeMounts: - name: custom-ca mountPath: /etc/ssl/certs/CA.crt readOnly: true subPath: CA.crt ... ``` :::warning Before moving ahead, be aware that starting from version 3.3.0, our development team has officially deprecated the Jenkins deploy scenario. This means that as of version 3.3.0 and in all subsequent versions (3.3.x and above), the Jenkins deploy scenario is no longer supported. For users running versions 3.3.x and below, the Jenkins deploy scenario remains available. However, we encourage you to plan for the transition to a supported deployment method to ensure continued compatibility and access to the latest features and enhancements. To perform migration, please familiarize yourself with the [Migrate CI Pipelines From Jenkins to Tekton](../upgrade/migrate-ci-pipelines-from-jenkins-to-tekton.md). For those who still use EDP v3.3.x and below, the information below remains valid and applicable. ::: 3. For Sonar, Jenkins and Gerrit, change the flag in the `caCerts.enabled` field to `true`. Also, change the name of the secret in the `caCerts.secret` field to `custom-ca-certificates`. Example of specifying custom certificates for `Gerrit` via the `gerrit-operator` helm chart values: ```yaml ... gerrit-operator: enabled: true gerrit: caCerts: # -- Flag for enabling additional CA certificates enabled: true # -- Change init CA certificates container image image: adoptopenjdk/openjdk11:alpine # -- Name of the secret containing additional CA certificates secret: custom-ca-certificates ... ``` ## Integrate Custom Certificates Into Jenkins Agents This section describes how to add custom certificates to Jenkins agents to use them from Java applications. :::info For example, `curl` doesn't use keystore files specified in this part of the documentation. ::: Jenkins agents keep keystore files in two places: - `/etc/ssl/certs/java` folder with the `cacerts` file - `/opt/java/openjdk/lib/security` folder with the `blocked.certs`, `cacerts`, `default.policy` and `public_suffix_list.dat` files. 1. Copy the files in `/etc/ssl/certs/java` and `/opt/java/openjdk/lib/security` directories from Jenkins agent pod to the local `tmp` folder. There is a `copy_certs.sh` script below that can manage this. It copies the files in `/etc/ssl/certs/java` and `/opt/java/openjdk/lib/security` directories from Jenkins agent pod to the local `tmp` folder and imports the custom certificate into the keystore files, after which it creates the `jenkins-agent-opt-java-openjdk-lib-security-cacerts` and `jenkins-agent-etc-ssl-certs-java-cacerts` secrets from updated keystore files in KubeRocketCI namespace. Also, the `jenkins-agent-opt-java-openjdk-lib-security-cacerts` secret contains three additional files: `blocked.certs`, `default.policy` and `public_suffix_list.dat` which managed by the `copy_certs.sh` script as well. Expand the drop-down button below to see the contents of the `copy_certs.sh` script.
copy_certs.sh ```bash # Fill in the variables `ns` and `ca_file` ns="krci" ca_file="/tmp/CA.crt" images=$(kubectl get -n "${ns}" cm jenkins-slaves -ojson | jq -r ".data[]" | grep image\> | sed 's/\s*<.*>\(.*\)<.*>/\1/') image=$(for i in ${images[@]}; do echo $i; done | grep maven-java8) pod_name=$(echo "${image}" | tr '.:/' '-') overrides="{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"name\":\"${pod_name}\", \"namespace\": \"${ns}\"}, \"spec\":{\"containers\":[{\"name\":\"${pod_name}\",\"image\":\"${image}\", \"command\":[\"sh\",\"-c\",\"while true;do sleep 30;done;\"]}]}}" kubectl run -n "${ns}" "${pod_name}" --image "${image}" --overrides="${overrides}" kubectl wait --for=condition=ready pod "${pod_name}" -n "${ns}" cacerts_location=$(kubectl exec -n "${ns}" "${pod_name}" \ -- find / -name cacerts -exec ls -la "{}" \; 2>/dev/null | grep -v ^l | awk '{print $9}') for cacerts in ${cacerts_location[@]}; do echo $(dirname "${cacerts}"); kubectl exec -n "${ns}" "${pod_name}" -- ls $(dirname "${cacerts}"); done for cacerts in ${cacerts_location[@]}; do \ echo $(dirname "${cacerts}"); \ mkdir -p "/tmp$(dirname "${cacerts}")"; \ from_files=''; \ for file in $(kubectl exec -n "${ns}" "${pod_name}" -- ls $(dirname "${cacerts}")); do \ kubectl exec -n "${ns}" "${pod_name}" -- cat "$(dirname "${cacerts}")/${file}" > "/tmp$(dirname "${cacerts}")/${file}"; \ from_files="${from_files} --from-file=/tmp$(dirname "${cacerts}")/${file}" done ; \ keytool -import -storepass changeit -alias kubernetes -file ${ca_file} -noprompt -keystore "/tmp${cacerts}"; \ kubectl -n "${ns}" create secret generic "jenkins-agent${cacerts//\//-}" $from_files \ done kubectl delete -n "${ns}" pod "${pod_name}" --force --grace-period=0 ```
Before using the `copy_certs.sh` script, keep in mind the following: - assign actual values to the variables `ns` and `ca_file` - the script collects all the images from the `jenkins-slaves` ConfigMap and uses the image of the `maven-java8` agent as the base image of the temporary pod to get the keystore files; - custom certificate is imported using the `keytool` application - the `jenkins-agent-opt-java-openjdk-lib-security-cacerts` and `jenkins-agent-etc-ssl-certs-java-cacerts` secrets will be created in the `krci` namespace. 2. Run the `copy_certs.sh` script from the previous point after the requirements are met. 3. Update manually the `jenkins-slaves` ConfigMap. Add this block with the mount of secrets to the `` block of each Jenkins agent: ```xml ... /etc/ssl/certs/java jenkins-agent-etc-ssl-certs-java-cacerts /opt/java/openjdk/lib/security jenkins-agent-opt-java-openjdk-lib-security-cacerts ... ``` As an example, the template of `gradle-java11-template` is shown below: ```xml ... /etc/ssl/certs/java jenkins-agent-etc-ssl-certs-java-cacerts /opt/java/openjdk/lib/security jenkins-agent-opt-java-openjdk-lib-security-cacerts ... ``` 4. Reload the Jenkins pod: ```bash ns="krci" kubectl rollout restart -n "${ns}" deployment/jenkins ``` ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install Keycloak](../auth/keycloak.md) --- ## Advanced Installation Overview: Third-Party Tools # Advanced Installation Overview: Third-Party Tools This page provides a concise overview of the advanced components available in KubeRocketCI. While these third-party tools are not obligatory, they greatly enhance the capabilities of the platform, enabling the creation of a robust CI/CD environment. ## Third-Party Components Find below the list of the key components used by KubeRocketCI: |Component|Requirement Level|Description| |:-|:-:|:-| |[Tekton](../install-tekton.md)|Mandatory|A cloud-native solution for building CI/CD pipelines.| |[Argo CD](../install-argocd.md)|Mandatory|A declarative, GitOps continuous delivery tool for Kubernetes.| |[NGINX Ingress Controller](../install-ingress-nginx.md)[^1]| Mandatory|Manages external access to HTTP services in a Kubernetes cluster.| |[Keycloak](../auth/keycloak.md)|Optional|An open-source identity and access management solution.| |[DefectDojo](../devsecops/defectdojo.md)|Optional|An open-source application vulnerability management tool.| |[ReportPortal](../project-management-and-reporting/install-reportportal.md)|Optional|An AI-powered test automation dashboard for analyzing test results.| |[Capsule](capsule.md)[^2]|Optional|A lightweight tool for managing Kubernetes namespaces.| |[External Secrets](../secrets-management/install-external-secrets-operator.md)|Optional|Manages external secret storage like AWS Secrets Manager, integrating with Kubernetes.| |[Nexus](../artifacts-management/nexus-sonatype.md)|Optional|A repository manager supporting various formats including Docker, npm, and Helm charts.| |[Harbor](../artifacts-management/harbor-installation.md)|Optional|An open-source trusted cloud native registry project that stores, signs, and scans content.| Although we have dedicated instructions for all of these third-party tools, we recommend installing them via [add-ons approach](../add-ons-overview.md). [^1]: OpenShift cluster uses Routes to provide access to pods from external resources. [^2]: These tools need to be installed in advance before deploying KubeRocketCI. --- ## Install Harbor # Install Harbor One of the storage options for application images within KubeRocketCI is Harbor, utilized during the application build process. Inspect the prerequisites and the main steps to perform for enabling [Harbor](https://goharbor.io/docs/2.8.0/) on the platform. ## Prerequisites - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation To install Harbor with Helm, follow the steps below: 1. Create a namespace for Harbor: ```bash kubectl create namespace harbor ``` 2. Create a secret for administrator user and registry: 1. Manually ```bash kubectl create secret generic harbor \ --from-literal=HARBOR_ADMIN_PASSWORD= \ --from-literal=REGISTRY_HTPASSWD= \ --from-literal=REGISTRY_PASSWD= \ --from-literal=secretKey= \ --namespace harbor ``` 2. Using `External Secret Operator` (ESO)
Create Secret data with ESO ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: harbor namespace: harbor spec: refreshInterval: 1h secretStoreRef: kind: SecretStore name: aws-parameterstore data: - secretKey: HARBOR_ADMIN_PASSWORD remoteRef: conversionStrategy: Default decodingStrategy: None key: /control-plane/deploy-secrets property: harbor.HARBOR_ADMIN_PASSWORD - secretKey: secretKey remoteRef: conversionStrategy: Default decodingStrategy: None key: /control-plane/deploy-secrets property: harbor.secretKey - secretKey: REGISTRY_HTPASSWD remoteRef: conversionStrategy: Default decodingStrategy: None key: /control-plane/deploy-secrets property: harbor.REGISTRY_HTPASSWD - secretKey: REGISTRY_PASSWD remoteRef: conversionStrategy: Default decodingStrategy: None key: /control-plane/deploy-secrets property: harbor.REGISTRY_PASSWD ```
:::note The `HARBOR_ADMIN_PASSWORD` is the initial password of Harbor admin. The `secretKey` is the secret key that is used for encryption. Must be 16 characters long. The `REGISTRY_PASSWD` is Harbor registry password. The `REGISTRY_HTPASSWD` is login and password in htpasswd string format. This value is the string in the password file generated by the `htpasswd` command where the username is `harbor_registry_user` and the encryption type is `bcrypt`. See the example below: ```bash htpasswd -bBc passwordfile harbor_registry_user harbor_registry_password ``` The username must be `harbor_registry_user`. The password must be the value from `REGISTRY_PASSWD`. ::: 3. Add the Helm Harbor Charts for the local client. ```bash helm repo add harbor https://helm.goharbor.io ``` 4. Check the parameters in the Harbor installation chart. For details, please refer to the [values.yaml](https://github.com/goharbor/harbor-helm/blob/master/values.yaml) file. 5. Install Harbor in the _harbor_ namespace with the Helm tool. ```bash helm install harbor harbor/harbor --version 1.17.2 \ --namespace harbor \ --values values.yaml ``` See the details on the parameters below: :::note Example values.yaml ```yaml # we use Harbor secret to consolidate all the Harbor secrets existingSecretAdminPassword: harbor existingSecretAdminPasswordKey: HARBOR_ADMIN_PASSWORD existingSecretSecretKey: harbor core: # The XSRF key. Will be generated automatically if it isn't specified xsrfKey: "" jobservice: # Secret is used when job service communicates with other components. # If a secret key is not specified, Helm will generate one. # Must be a string of 16 chars. secret: "" registry: # Secret is used to secure the upload state from client # and registry storage backend. # If a secret key is not specified, Helm will generate one. # Must be a string of 16 chars. secret: "" credentials: username: harbor_registry_user existingSecret: harbor fullnameOverride: harbor # If Harbor is deployed behind the proxy, set it as the URL of proxy externalURL: https://core.harbor.domain ipFamily: ipv6: enabled: false expose: tls: enabled: false ingress: hosts: core: core.harbor.domain notary: notary.harbor.domain updateStrategy: type: Recreate persistence: persistentVolumeClaim: registry: size: 30Gi jobservice: jobLog: size: 1Gi database: size: 2Gi redis: size: 1Gi trivy: size: 5Gi database: internal: # The initial superuser password for internal database password: "changeit" ``` ::: 6. To check if the installation is successful, run the command below: ```bash helm status -n harbor ``` You can also check ingress endpoints to get Harbor endpoint to enter Harbor UI: ```bash kubectl describe ingress -n harbor ``` ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Integrate Harbor With KubeRocketCI Pipelines](../artifacts-management/harbor-integration.md) --- ## Integrate Harbor With Tekton Pipelines # Integrate Harbor With Tekton Pipelines [Harbor](https://goharbor.io/) is an advanced, open-source cloud-native registry that securely manages container images and Helm charts, and also supports a wide range of OCI artifacts. It enforces policies and role-based access control, while ensuring all managed content, including container images and other artifacts, are scanned and certified free from vulnerabilities. For those looking to integrate Harbor into their workflow, the documentation provides comprehensive steps on [creating projects](https://goharbor.io/docs/2.0.0/working-with-projects/create-projects/) and establishing robot accounts. These accounts facilitate automated and secure interactions with the registry, crucial for CI pipelines and enhancing overall security and efficiency in artifact management. ## Overview Harbor integration with Tekton enables the centralized storage of container images within the cluster, eliminating the need for external services. By leveraging Harbor as the container registry, users can manage and store their automation results and reports in one place. ## Integration Procedure The integration process involves two steps: 1. Creating a project to store application images. 2. Creating two service accounts (robot account) with different permissions to push (read/write) and pull (read-only) project images. ### Create New Project The process of creating new projects is the following: 1. Log in to the Harbor console using your credentials. 2. Navigate to the **Projects** menu, click the **New Project** button: ![Harbor Console](../../assets/operator-guide/artifacts-management/harbor-console-projects.png "Projects menu") 3. On the **New Project** menu, enter a project name that matches your platform namespace in the **Project Name** field. Keep other fields as default and click **OK** to continue: ![New Project](../../assets/operator-guide/artifacts-management/harbor-new-project.png "New Project menu") ### Set Up Robot Account To facilitate seamless interaction between KubeRocketCI and a Harbor project, it is crucial to establish a robot account. This process involves: 1. Navigate to your newly created project, select **Robot Accounts** menu and choose **New Robot Account**: ![New Project](../../assets/operator-guide/artifacts-management/harbor-robot-accounts-menu.png "Create Robot Account menu") 2. In the pop-up window, fill in the fields as follows: - **Name** - `edp-push` - **Expiration time** - set the value which is aligned with your organization policy - **Description** - `read/write permissions` - **Permissions** - `Pull Repository` and `Push Repository`. To proceed, click the **ADD** button: ![New Project](../../assets/operator-guide/artifacts-management/harbor-create-robot-account.png "Robot Accounts menu") 3. In the appeared window, copy the robot account credentials or click the **Export to file** button to save the secret and account name locally: ![New Project](../../assets/operator-guide/artifacts-management/harbor-new-credentials-of-robot-account.png "New credentials for Robot Account") 4. Provision the **kaniko-docker-config** secrets using kubectl, KubeRocketCI portal or with the externalSecrets operator: The `auth` string can be generated by this command: ```bash echo -n "robot\$edp-project+edp:secret" | base64 ``` Navigate to **Portal** -> **Configuration** -> **Artifacts storage** -> **Registry**. Update or click **+ Add registry** fill in the required fields and click `Save`. ![Registry update manual secret](../../assets/operator-guide/artifacts-management/regcred-secret.png "Registry update manual secret") ```yaml apiVersion: v1 kind: Secret metadata: name: kaniko-docker-config namespace: krci labels: app.edp.epam.com/secret-type: registry app.edp.epam.com/integration-secret: "true" type: kubernetes.io/dockerconfigjson stringData: .dockerconfigjson: | { "auths" : { "harbor-registry.com": { "username":"registry-username", "password":"registry-password", "auth": "secret-string" } } } ``` ```json "kaniko-docker-config": {"auths" : "harbor-registry.com": { "username":"registry-username", "password":"registry-password", "auth": "secret-string" } } ``` Navigate to **Portal** -> **Configuration** -> **Artifacts storage** -> **Registry**. Here, you will observe the `Managed by ExternalSecret` message: ![Registry managed by external secret operator](../../assets/operator-guide/artifacts-management/kaniko-secret.png "Registry managed by external secret operator") :::note More details of External Secrets Operator Integration can be found in the [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) page. ::: 5. Repeat steps 2-3 with values below: - **Name** - `edp-pull` - **Expiration time** - set the value which is aligned with your organization policy - **Description** - `read-only permissions` - **Permissions** - `Pull Repository`. 6. Provision the **regcred** secrets using kubectl, KubeRocketCI portal or with the externalSecrets operator: The `auth` string can be generated by this command: ```bash echo -n "robot\$edp-project+edp-push:secret" | base64 ``` Navigate to **Portal** -> **Configuration** -> **Artifacts storage** -> **Registry**. Update or click **+ Add registry** fill in the required fields and click `Save`. ![Registry update manual secret](../../assets/operator-guide/artifacts-management/regcred-externalsecret.png "Registry update manual secret") ```yaml apiVersion: v1 kind: Secret metadata: name: regcred namespace: krci labels: app.edp.epam.com/secret-type: registry app.edp.epam.com/integration-secret: "true" type: kubernetes.io/dockerconfigjson stringData: .dockerconfigjson: | { "auths" : { "harbor-registry.com": { "username":"registry-username", "password":"registry-password", "auth": "secret-string" } } } ``` ```json "regcred": {"auths" : "harbor-registry.com": { "username":"registry-username", "password":"registry-password", "auth": "secret-string" } } ``` Navigate to **Portal** -> **Configuration** -> **Artifacts storage** -> **Registry**. Here, you will observe the `Managed by ExternalSecret` message: ![Registry managed by external secret operator](../../assets/operator-guide/artifacts-management/kaniko-externalsecret.png "Registry managed by external secret operator") :::note More details of External Secrets Operator Integration can be found in the [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) page. ::: 7. In the [values.yaml](https://github.com/epam/edp-install/blob/master/deploy-templates/values.yaml) file for the **edp-install** Helm chart, set the following values for the specified fields: If the `kaniko-docker-config` secret has been created manually: ```yaml title="values.yaml" ... kaniko: existingDockerConfig: "kaniko-docker-config" global: dockerRegistry: url: harbor-registry.com type: "harbor" ... ``` If the `kaniko-docker-config` secret has been created via External Secrets Operator: ```yaml title="values.yaml" ... kaniko: existingDockerConfig: "kaniko-docker-config" externalSecrets: enabled: true global: dockerRegistry: url: harbor-registry.com type: "harbor" ... ``` 8. (Optional) If you've already deployed the edp-install Helm chart, you can update it using the following command: ```bash helm update --install krci epamedp/edp-install \ --values values.yaml \ --namespace krci ``` As a result, application images built in UI Portal will be stored in Harbor project and will be deployed from the harbor registry. Harbor projects can be added and retained with a retention policy generated through the script in [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/harbor-ha/hack/harbor). ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install Harbor](../artifacts-management/harbor-installation.md) * [Adjust Jira Integration](../project-management-and-reporting/jira-integration.md) * [Custom SonarQube Integration](../code-quality/sonarqube.md) --- ## Harbor OIDC Configuration # Harbor OIDC Configuration This guide outlines the steps for integrating [OIDC (OpenID Connect) authorization](https://openid.net/connect/) with Harbor, facilitating Single Sign-On (SSO) capabilities. By adopting OIDC, Harbor gains a unified authentication mechanism, simplifying user access management and enhancing security. This approach centralizes user permissions and access control, streamlining administration through a singular configuration interface. ## Prerequisites Before the beginning, ensure your installation meets the following criteria: - [Keycloak](../auth/keycloak.md) is installed - [KubeRocketCI](../install-kuberocketci.md) is installed. ## Configure Keycloak This section details the procedure to configure Harbor with Keycloak for secure authentication using the following steps: 1. Generate the Keycloak Client Secret: Start by creating a secret named `keycloak-client-harbor-secret` for Keycloak. This can be accomplished either through the commands provided below or by employing the External Secrets Operator. The secret acts as a secure token for establishing communication between Keycloak and Harbor. Generate the secret using OpenSSL and store it in a Kubernetes secret within the platform namespace (e.g, `krci`): ```bash KEYCLOAK_CLIENT_HARBOR_SECRET=$(openssl rand -base64 32 | head -c 32) kubectl -n krci create secret generic keycloak-client-harbor-secret \ --from-literal=cookie-secret=${KEYCLOAK_CLIENT_HARBOR_SECRET} ``` 2. Deploy the KeycloakClient Custom Resource: Proceed to create the `KeycloakClient` custom resource by applying the `harborkeycloakclient.yaml` configuration in the appropriate namespace. This step involves utilizing the previously generated `keycloak-client-harbor-secret`. Upon successful deployment, Keycloak will register Harbor as a client, with the Kubernetes secret serving as the client password. Apply the `harborkeycloakclient.yaml` to establish the resource: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakClient metadata: name: harbor spec: advancedProtocolMappers: true clientId: harbor directAccess: true public: false secret: keycloak-client-harbor-secret defaultClientScopes: - profile - email - roles targetRealm: control-plane webUrl: protocolMappers: - name: roles protocol: openid-connect protocolMapper: oidc-usermodel-realm-role-mapper config: access.token.claim: true claim.name: roles id.token.claim: true userinfo.token.claim: true multivalued: true ``` Ensure to replace `` with your actual Harbor URL. This configuration links Harbor to Keycloak, enabling OIDC-based Single Sign-On (SSO) for a streamlined authentication process. ## Configure Harbor The next stage is to configure Harbor. Proceed with following the steps below: 1. Log in to Harbor UI with an account that has Harbor system administrator privileges. To get the administrator password, execute the command below: ```bash kubectl get secret harbor -n harbor -o jsonpath='{.data.HARBOR_ADMIN_PASSWORD}' | base64 --decode ``` 2. Navigate to **Administration** -> **Configuration** -> **Authentication**. Configure OIDC using the parameters below: ```ini auth_mode: oidc_auth oidc_name: keycloak oidc_endpoint: /auth/realms/control-plane oidc_client_id: harbor oidc_client_secret: oidc_groups_claim: roles oidc_admin_group: administrator oidc_scope: openid,email,profile,roles verify_certificate: true oidc_auto_onboard: true oidc_user_claim: preferred_username ``` ![Harbor Authentication Configuration](../../assets/operator-guide/oidc-harbor-cfg-ui.png "Harbor Authentication Configuration") As a result, users will be prompted to authenticate themselves when logging in to Harbor UI. ## Related Articles * [EKS OIDC With Keycloak](../auth/configure-keycloak-oidc-eks.md) * [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) * [Integrate Harbor With KubeRocketCI Pipelines](harbor-integration.md) --- ## Nexus Image Registry # Nexus Image Registry This documentation guide provides comprehensive instructions for integrating **Nexus Repository Manager** as a docker image registry with KubeRocketCI. ## Installation To install Nexus in environment, it's recommended to use the resources provided in the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) repository. This approach involves installing both the Nexus repository manager and the [nexus-operator](https://github.com/epam/edp-nexus-operator). Leveraging the Cluster Add-Ons simplifies the deployment and management process, providing a streamlined method to integrate Nexus into infrastructure. 1. Navigate to the forked cluster Add-Ons repository and align the [nexus values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/nexus/values.yaml) and [nexus-operator values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/nexus-operator/values.yaml) files. Follow the instructions to deploy Nexus, ensuring it's correctly configured to serve as artifact repository. 2. Update the configuration to enable **nexus** and **nexus-operator** in [Add-Ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml) repository: ```yaml title="clusters/core/apps/values.yaml" nexus: createNamespace: true enable: true nexus-operator: createNamespace: true enable: true ``` 3. Sync resources and wait till the all **nexus** and **nexus-operator** resources are created: ![Nexus ArgoCD applications](../../assets/operator-guide/artifacts-management/nexus-registry-addons-deploy.png "Nexus ArgoCD applications") 4. **Sign in** into **Nexus Repository Manager** using the default credentials **Username**: `admin` and **Password**: `admin123` and update the default password for better security: ![Nexus ArgoCD applications](../../assets/operator-guide/artifacts-management/nexus-registry-change-password.png "Nexus ArgoCD applications") ## Configuration With [Add-Ons approach](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core) `nexus-operator` create the necessary roles, the **Service Account** `ci.user`, blob stores, repository, scripts, cleanup policies automatically. Below is a comprehensive guide on how to create all resources manually. ## Configuration Nexus Repository Manager using nexus-operator 1. Create a Kubernetes Secret that the **nexus-operator** will use to connect to the **Nexus Repository Manager** and create all resources: ```yaml apiVersion: v1 kind: Secret metadata: name: nexus-admin-password namespace: nexus type: Opaque data: user: YWRtaW4= # base64-encoded value of "admin" password: cGFzcw== # base64-encoded value of "pass" ``` ```json "nexus-admin-password": { "user": "XXXXXXX", "password": "XXXXXXX" } ``` 2. Create a secret using `manifests` or with `External Secrets Operator` to enable the **edp-nexus-operator** to connect to the **Nexus Repository Manager** for provisioning: ```yaml apiVersion: v1 kind: Secret metadata: name: ci-nexus namespace: nexus type: Opaque stringData: password: ``` ```json "ci-nexus": { "password": "XXXXXXX" } ``` ## Configuration Nexus Repository Manager without nexus-operator Without using the **nexus-operator**, manual updates are required for the Helm chart values of the **Nexus Repository Manager**, as well as the creation of blob stores, repositories, roles, and the **Service Account** `ci.user`. 1. Update **Nexus Repository Manager** helm chart values: ```yaml nexus-repository-manager: nexus: docker: enabled: true registries: - host: nexus-ci-container.example.com port: 5000 ``` 2. Create blob store. Open the Nexus UI and navigate to **Server administration and configuration** -> **Repository** -> **Blob Stores** and click **Create Blob Store**, fill the following fields and click **Save** button: - **Type**: `File` - **Name**: `krci-container` ![Nexus create blob store](../../assets/operator-guide/artifacts-management/nexus-registry-create-blobstore.png "Nexus create blob store") 3. Create repository. Open the Nexus UI and navigate to **Server administration and configuration** -> **Repository** -> **Repositories**, click **+ Create repository**, select **docker (hosted)**, fill the following fields and click **Save** button: - **Name**: `A unique identifier for this repository` - **HTTP**: `5000` - **Enable Docker V1 API**: `check` - **Blob store name**: `krci-container` - **Strict Content Type Validation**: `check` ![Nexus create registry](../../assets/operator-guide/artifacts-management/nexus-registry-create-repository.png "Nexus create registry") 4. Open the Nexus UI and navigate to **Server administration and configuration** -> **Security** -> **Roles**. Click the **Create Role** button, fill the following fields and click **Save** button: - **Type**: `Nexus role` - **Role ID**: `krci-admin` - **Role Name**: `krci-admin` - **Role Descriptions**: `Read and write access to all repos and scripts` - **Applied Privileges**: `nx-apikey-all`, `nx-repository-view-*-*-add`, `nx-repository-view-*-*-browse`,`nx-repository-view-*-*-edit`, `nx-repository-view-*-*-read`, `nx-script-*-add`, `nx-script-*-delete`, `nx-script-*-read`, `nx-script-*-run` ![Roles settings](../../assets/operator-guide/artifacts-management/nexus-registry-create-roles.png "Roles settings") 5. Create repository Service Account `ci.user`. Open the Nexus UI and navigate to **Server administration and configuration** -> **Security** -> **User**. Click the `Create local user` button to create a new user: ![Nexus user settings](../../assets/operator-guide/artifacts-management/nexus-settings-user.png "Nexus user settings") 6. Type the `ci.user` username, fill the following fields and click the **Create local user** button to create the ***Service Account**, fill the following fields and click **Save** button: - **ID**: `ci.user` - **First name**: `ci.user` - **Last name**: `CI` - **Email**: `ci.user@krci.com` - **Password**: `` - **Status**: `Active` - **Roles**: `krci-admin` ![Nexus create user](../../assets/operator-guide/artifacts-management/nexus-registry-create-user.png "Nexus create user") ## Integration Nexus Repository Manager with KubeRocketCI 1. Open **KubeRocketCI** -> **Configuration** -> **Artifacts storage** -> **Registry**. Click **+ Add registry** fill in the fields below, and then click **SAVE** button: - **Registry Provider**: `Nexus` - **Registry Endpoint**: `nexus-ci-container.example.com` - **Repository**: `krci` # The name of folder in nexus repository - **Push account User**: `ci.user` - **Push account Password/Token**: `` - **Use the Push Accounts credentials**: `check` ![Kuberocketci add nexus registry](../../assets/operator-guide/artifacts-management/nexus-registry-kuberocketci-integration.png "Kuberocketci add nexus registry") 2. After build application open **Nexus Repository Manager** -> **Browse server contents** -> **Browse** and open container registry: ![Nexus create user](../../assets/operator-guide/artifacts-management/nexus-registry-browse-overview.png "Nexus create user") ## Related Articles * [Install via Add-Ons](../add-ons-overview.md) * [Manage Container Registries](../../user-guide/manage-container-registries.md) * [Nexus Sonatype Integration](nexus-sonatype.md) * [Change Container Registry](../../user-guide/change-container-registry.md) --- ## Sonatype Nexus Repository OSS Integration # Sonatype Nexus Repository OSS Integration This documentation guide provides comprehensive instructions for integrating Sonatype Nexus Repository OSS (Nexus) with KubeRocketCI. ## Prerequisites Before proceeding, ensure the following prerequisites are met: - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation To install Nexus in environment, it's recommended to use the resources provided in the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) repository. This approach involves installing both the Nexus repository manager and the [nexus-operator](https://github.com/epam/edp-nexus-operator). Leveraging the Cluster Add-Ons simplifies the deployment and management process, providing a streamlined method to integrate Nexus into infrastructure: 1. **Nexus Repository Manager**: First, navigate to the forked cluster Add-Ons repository and align the [nexus values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/nexus/values.yaml) and [nexus-operator values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/nexus-operator/values.yaml) files. Follow the instructions to deploy Nexus, ensuring it's correctly configured to serve as artifact repository. 2. Update the configuration to enable **nexus** and **nexus-operator** in [Add-Ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml) repository: ```yaml title="clusters/core/apps/values.yaml" nexus: createNamespace: true enable: true nexus-operator: createNamespace: true enable: true ``` 3. Sync resources and wait till the all **nexus** and **nexus-operator** resources is created: ![Nexus ArgoCD applications](../../assets/operator-guide/artifacts-management/nexus-artifact-addons-deploy.png "Nexus ArgoCD applications") 4. **Sign In** into **Nexus Repository Manager** using the default credentials **Username**: `admin` and **Password**: `admin123` and update the default password for better security: ![Nexus change password](../../assets/operator-guide/artifacts-management/nexus-artifact-change-password.png "Nexus change password") ## Configuration With [Add-ons approach](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core) `nexus-operator` create the necessary roles, **Service Account** `ci.user`, blob stores, repository, scripts, cleanup policies automatically. KubeRocketCI organizes artifacts within Nexus according to the following hierarchy: ```bash ├── krci-dotnet-group │ ├── krci-dotnet-proxy │ ├── krci-dotnet-releases │ ├── krci-dotnet-snapshots │ └── krci-dotnet-hosted krci-maven-group │ ├── krci-maven-proxy │ ├── krci-maven-releases │ ├── krci-maven-snapshots krci-npm-group │ ├── krci-npm-proxy │ ├── krci-npm-releases │ ├── krci-npm-snapshots │ └── krci-npm-hosted krci-python-group │ ├── krci-python-proxy │ ├── krci-python-releases │ ├── krci-python-snapshots krci-yum-group ├── krci-yum-releases ├── krci-yum-snapshots └── krci-container-hosted ``` Below is a comprehensive guide on how to create all the resources manually. ### Configure Nexus Repository Manager With nexus-operator 1. Create a Kubernetes Secret that the **nexus-operator** will use to connect to the **Nexus Repository Manager** and create all resources: ```yaml apiVersion: v1 kind: Secret metadata: name: nexus-admin-password namespace: nexus type: Opaque data: user: YWRtaW4= # base64-encoded value of "admin" password: cGFzcw== # base64-encoded value of "pass" ``` ```json "nexus-admin-password": { "user": "XXXXXXX", "password": "XXXXXXX" } ``` 2. Create a secret using `manifests` or with `External Secrets Operator` to enable the **edp-nexus-operator** to connect to the **Nexus Repository Manager** for provisioning: ```yaml apiVersion: v1 kind: Secret metadata: name: ci-nexus namespace: nexus type: Opaque stringData: password: ``` ```json "ci-nexus": { "password": "XXXXXXX" } ``` ### Configure Nexus Repository Manager Without nexus-operator Without using the **nexus-operator**, manual updates are required for the Helm chart values of the **Nexus Repository Manager**, as well as the creation of blob stores, repositories, roles, and the **Service Account** `ci.user`: 1. Create blob store. Open the Nexus UI and navigate to **Server administration and configuration** -> **Repository** -> **Blob Stores** and click **Create Blob Store**, fill the following fields and click **Save** button: - **Type**: `File` - **Name**: `krci-maven` - **Type**: `File` - **Name**: `krci-npm` - **Type**: `File` - **Name**: `krci-dotnet` - **Type**: `File` - **Name**: `krci-python` - **Type**: `File` - **Name**: `krci-yum` 2. KubeRocketCI operates using a specific repository naming convention, where all repository names adhere to predefined standards. To align with this, navigate to **Server administration and configuration** -> **Repository** -> **Repositories** within Nexus. Here, repository creation is limited to those that match the required programming language, ensuring consistency and compliance with the KubeRocketCI framework. ![Nexus repository list](../../assets/operator-guide/artifacts-management/nexus-repository.png "Nexus repository list") a) Click Create a repository by selecting `maven2(proxy)` fill the following fields and click **Create repository**: - **Name** `krci-maven-proxy` - **Remote storage URL** `https://repo1.maven.org/maven2/` - **Blob store** `krci-maven` - **HTTP request settings** `check` - **Use-agent customization** `user-agent-suffix` - **Connection retries** `10` - **Connection timeout** `1440` b) Click Create a repository by selecting `maven2(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-maven-snapshots` - **Version policy** `snapshots` - **Blob store** `krci-maven` c) Click Create a repository by selecting `maven2(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-maven-releases` - **Version policy** `release` - **Blob store** `krci-maven` d) Click Create a repository by selecting `maven2(group)` fill the following fields and click **Create repository**: - **Name** `krci-maven-group` - **Version policy** `release` - **Member repositories** `krci-maven-proxy`, `krci-maven-snapshots`, `krci-maven-releases` a) Click Create a repository by selecting `npm(proxy)` fill the following fields **Create repository**: - **Name** `krci-npm-proxy` - **Remote storage URL** `https://registry.npmjs.org` - **Blob store** `krci-npm` - **HTTP request settings** `check` - **Use-agent customization** `user-agent-suffix` - **Connection retries** `10` - **Connection timeout** `1440` b) Click Create a repository by selecting `npm(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-npm-snapshots` - **Blob store** `krci-npm` c) Click Create a repository by selecting `npm(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-npm-releases` - **Blob store** `krci-npm` d) Click Create a repository by selecting `npm(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-npm-hosted` - **Blob store** `krci-npm` e) Click Create a repository by selecting `npm(group)` fill the following fields and click **Create repository**: - **Name** `krci-npm-group` - **Member repositories** `krci-npm-proxy`, `krci-npm-snapshots`, `krci-npm-releases`, `krci-npm-hosted` a) Click Create a repository by selecting `nuget(proxy)` fill the following fields and click **Create repository**: - **Name** `krci-dotnet-proxy` - **Protocol version NuGet V3** `check` - **Remote storage URL** `https://api.nuget.org/v3/index.json` - **Blob store** `krci-dotnet` - **HTTP request settings** `check` - **Use-agent customization** `user-agent-suffix` - **Connection retries** `10` - **Connection timeout** `1440` b) Click Create a repository by selecting `nuget(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-dotnet-snapshots` - **Blob store** `krci-dotnet` c) Click Create a repository by selecting `nuget(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-dotnet-releases` - **Blob store** `krci-dotnet` d) Click Create a repository by selecting `nuget(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-dotnet-hosted` - **Blob store** `krci-dotnet` e) Click Create a repository by selecting `nuget(group)` fill the following fields and click **Create repository**: - **Name** `krci-dotnet-group` - **Member repositories** `krci-dotnet-proxy`, `krci-dotnet-snapshots`, `krci-dotnet-releases`, `krci-dotnet-hosted` a) Click Create a repository by selecting `pypi(proxy)` fill the following fields and click **Create repository**: - **Name** `krci-python-proxy` - **Remote storage URL** `https://pypi.org` - **Blob store** `krci-python` - **HTTP request settings** `check` - **Use-agent customization** `user-agent-suffix` - **Connection retries** `10` - **Connection timeout** `1440` b) Click Create a repository by selecting `pypi(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-python-snapshots` - **Blob store** `krci-python` c) Click Create a repository by selecting `pypi(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-python-releases` - **Blob store** `krci-python` d) Click Create a repository by selecting `pypi(group)` fill the following fields and click **Create repository**: - **Name** `krci-python-group` - **Member repositories** `krci-python-proxy`, `krci-python-snapshots`, `krci-python-releases` a) Click Create a repository by selecting `yum(proxy)` fill the following fields and click **Create repository**: - **Name** `krci-yum-proxy` - **Remote storage URL** `http://fedora.ip-connect.info/linux/development/rawhide/Everything/x86_64/os/` - **Blob store** `krci-yum` - **HTTP request settings** `check` - **Use-agent customization** `user-agent-suffix` - **Connection retries** `10` - **Connection timeout** `1440` b) Click Create a repository by selecting `yum(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-yum-snapshots` - **Repo data Depth** `3` - **Blob store** `krci-yum` c) Click Create a repository by selecting `yum(hosted)` fill the following fields and click **Create repository**: - **Name** `krci-yum-releases` - **Repo data Depth** `3` - **Blob store** `krci-yum` 3. Open the Nexus UI and navigate to **Server administration and configuration** -> **Security** -> **Roles**. Click the **Create Role** button, fill the following fields and click **Save** button: - **Type**: `Nexus role` - **Role ID**: `krci-admin` - **Role Name**: `krci-admin` - **Role Descriptions**: `Read and write access to all repos and scripts` - **Applied Privileges**: `nx-apikey-all`, `nx-repository-view-*-*-add`, `nx-repository-view-*-*-browse`,`nx-repository-view-*-*-edit`, `nx-repository-view-*-*-read`, `nx-script-*-add`, `nx-script-*-delete`, `nx-script-*-read`, `nx-script-*-run` ![Roles settings](../../assets/operator-guide/artifacts-management/nexus-artifact-create-roles.png "Roles settings") 4. Create repository Service Account `ci.user`. Open the Nexus UI and navigate to **Server administration and configuration** -> **Security** -> **User**. Click the `Create local user` button to create a new user: ![Nexus user settings](../../assets/operator-guide/artifacts-management/nexus-settings-user.png "Nexus user settings") 5. Type the `ci.user` username, fill the following fields and click the **Create local user** button to create the ***Service Account**, fill the following fields and click **Save** button:: - **ID**: `ci.user` - **First name**: `ci.user` - **Last name**: `CI` - **Email**: `ci.user@krci.com` - **Password**: `` - **Status**: `Active` - **Roles**: `krci-admin` ![Nexus create user](../../assets/operator-guide/artifacts-management/nexus-artifact-create-user.png "Nexus create user") ## Integrate Nexus Repository Manager With KubeRocketCI For provision secret using manifest, KubeRocketCI portal or with the externalSecrets operator: Go to **Portal** -> **Configuration** -> **Artifacts storage** -> **Nexus**. Update or click **+ Add integration**. Fill the following fields and click the `Save` button: - **Quick link URL** `https://nexus.example.com` - **URL** `http://nexus.nexus:8081` - **User** `nexus-user-id` - **Password** `nexus-user-password` ![Nexus update manual secret](../../assets/operator-guide/artifacts-management/nexus-secret-password.png "Nexus update manual secret") ```yaml apiVersion: v1 kind: Secret metadata: name: ci-nexus namespace: krci labels: app.edp.epam.com/secret-type: nexus app.edp.epam.com/integration-secret: "true" type: Opaque stringData: url: https://nexus.example.com username: password: ``` ```json "ci-nexus": { "url": "https://nexus.example.com", "username": "XXXXXXX", "password": "XXXXXXX" }, ``` Go to **KubeRocketCI Portal** -> **Configuration** -> **Artifacts storage** -> **Nexus** and see `Managed by External Secret` message: ![Nexus managed by external secret operator](../../assets/operator-guide/artifacts-management/nexus-externalsecret-password.png "Nexus managed by external secret operator") More detail of External Secrets Operator Integration can found on [the following page](../secrets-management/external-secrets-operator-integration.md) ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install External Secrets Operator](../secrets-management/install-external-secrets-operator.md) * [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) * [Cluster Add-Ons Overview](../add-ons-overview.md) --- ## Package Registry # Package Registry This page describes the supported package registry providers and provides detailed instruction on how to adjust configurations to work properly with these package registry providers. ## Supported Package Registry Providers Currently, KubeRocketCI support the following package registries: - Sonatype Nexus Repository OSS - GitLab - GitHub - Azure DevOps The table below displays the supported registries and the languages they correspond to: |Language|Framework|Build Tool|Proxy Registry|Snapshots/Releases Registry| |------------|:-------------------------------------------------------:|:------------------:|:-----------------------------------------:|:-----------------------------------------:| | Java|Java 17Java 21Java 25|MavenGradle| NexusGitlabGitHubAzure DevOps | NexusGitlabGitHubAzure DevOps| | Python|Python 3.11FastAPIFlask|Python|NexusGitlabAzure DevOps|NexusGitlabAzure DevOps| | C#|.Net 3.1.Net 6.0|.Net|No proxy is used for this language.|NexusGitlabGitHubAzure DevOps| | JavaScript | ReactVueAngularExpressNext.jsAntora |NPMPNPM| NexusGitlabGitHubAzure DevOps | NexusGitlabGitHubAzure DevOps | ## Proxy Package Registry Configuration By default, KubeRocketCI uses Nexus as the proxy registry for storing and caching application dependencies. This setting is fixed and cannot be modified. ## Snapshot/Release Package Registry Configuration The edp-tekton Helm Chart allows to override the default settings for package registries through `tekton.configs` part of its [values.yaml](https://github.com/epam/edp-tekton/blob/release/0.12/charts/pipelines-library/values.yaml#L81) file. To provide necessary credentials for accessing the package registries, the user should create the `package-registries-auth-secret` secret and set the `tekton.packageRegistriesSecret.enabled` value to `true` to mount the secret into the pipeline. To replace the default name of the secret, the user should set the `tekton.packageRegistriesSecret.name` parameter to the desired value: ```yaml tekton: packageRegistriesSecret: enabled: true name: "package-registries-auth-secret" configs: # Build Tool: Maven mavenConfigMap: "custom-maven-settings" # Build Tool: NPM npmConfigMap: "custom-npm-settings" # Build Tool: Python pythonConfigMap: "custom-python-settings" # Build Tool: .Net nugetConfigMap: "custom-nuget-settings" ``` ### Customizing Maven Settings A new custom configuration map should contain the `settings.xml` file, which overrides the default Maven configuration. For example, the following configuration map contains the `settings.xml` file with the following settings: - **Proxy registry**: Nexus. - **Snapshots/Releases registry**: Azure DevOps registry. - **Authentication**: - `CI_USERNAME` and `CI_PASSWORD` - these environment variables are used for authentication to Nexus. - `CI_AZURE_DEVOPS_USERNAME` and `CI_AZURE_DEVOPS_PASSWORD` - these environment variables are used for authentication to Azure DevOps registry. - **Secrets**: - `CI_AZURE_DEVOPS_USERNAME` and `CI_AZURE_DEVOPS_PASSWORD` - these environment variables are taken from the `package-registries-auth-secret` secret. :::note Example: custom-maven-settings.yaml ```yaml apiVersion: v1 kind: ConfigMap metadata: name: new-custom-maven-settings data: settings.xml: | /workspace/source/cache org.sonarsource.scanner.maven nexus ${env.CI_USERNAME} ${env.CI_PASSWORD} azure-devops-registry ${env.CI_AZURE_DEVOPS_USERNAME} ${env.CI_AZURE_DEVOPS_PASSWORD} nexus * http://nexus.nexus:8081/repository/krci-maven-group sonar true ${env.SONAR_TOKEN} ${env.SONAR_HOST_URL} azure-devops-registry azure-devops-registry::default::https://pkgs.dev.azure.com///_packaging//maven/v1 azure-devops-registry::default::https://pkgs.dev.azure.com///_packaging//maven/v1 azure-devops-registry ``` ::: ### Customizing NPM Settings A new custom configuration map should contain the `.npmrc-ci`, `.npmrc-publish-snapshots` and `.npmrc-publish-releases` files, which override the default npm configuration. For example, the following configuration map contains the `.npmrc-ci`, `.npmrc-publish-snapshots` and `.npmrc-publish-releases` files with the following settings: - **Proxy registry**: Nexus. - **Snapshots/Releases registry**: Azure DevOps registry. - **Authentication**: - 'upBase64' (`"${CI_USERNAME}:${CI_PASSWORD}"` string in base64) - this environment variable is used for authentication to Nexus. - `CI_AZURE_DEVOPS_USERNAME` and `CI_AZURE_DEVOPS_PASSWORD_IN_BASE64` - these environment variables are used for authentication to Azure DevOps registry. - **Secrets**: - `CI_AZURE_DEVOPS_USERNAME` and `CI_AZURE_DEVOPS_PASSWORD_IN_BASE64` - these environment variables are taken from the `package-registries-auth-secret` secret. :::note Example: custom-npm-settings.yaml ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-npm-settings data: .npmrc-ci: | registry=${NEXUS_HOST_URL}/repository/krci-npm-group _auth=${upBase64} cache=${NPM_CACHE_DIR} .npmrc-publish-snapshots: | registry=https://pkgs.dev.azure.com///_packaging//npm/registry username=${CI_AZURE_DEVOPS_USERNAME} _password=${CI_AZURE_DEVOPS_PASSWORD_IN_BASE64} email=${CI_AZURE_DEVOPS_USERNAME} cache=${NPM_CACHE_DIR} .npmrc-publish-releases: | registry=https://pkgs.dev.azure.com///_packaging//npm/registry _auth=${upBase64} cache=${NPM_CACHE_DIR} ``` ::: ### Customizing Python Configurations For customizing python settings, a new configuration map can be prepared to replace the default pythonConfigMap. This custom configuration map should contain the PIP_TRUSTED_HOST, PIP_INDEX, PIP_INDEX_URL, REPOSITORY_URL_SNAPSHOTS and REPOSITORY_URL_RELEASES environment variables, which overrides the default python configuration. For example, the following configuration map contains the following settings: - **Proxy registry**: Azure DevOps registry. - **Snapshots/Releases registry**: Azure DevOps registry. - **Authentication**: - 'upBase64' (`"${CI_USERNAME}:${CI_PASSWORD}"` string in base64) - this environment variable is used for authentication to Azure DevOps registry. :::note Example: custom-python-settings.yaml ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-python-settings data: PIP_INDEX_PATH: "///_packaging//pypi" PIP_INDEX_URL_PATH: "///_packaging//pypi/simple" REPOSITORY_SNAPSHOTS_PATH: "///_packaging//pypi/upload" REPOSITORY_RELEASES_PATH: "///_packaging//pypi/upload" ``` ::: ### Customizing NuGet Settings A new custom configuration map should contain the `nuget.config` file, which overrides the default NuGet configuration. For example, the following configuration map contains the `nuget.config` file with the following settings: - **Snapshots/Releases registry**: Azure DevOps registry. - **Authentication**: - `CI_AZURE_DEVOPS_USERNAME` and `CI_AZURE_DEVOPS_PASSWORD` - these environment variables are used for authentication to Azure DevOps registry. - **Secrets**: - `CI_AZURE_DEVOPS_USERNAME` and `CI_AZURE_DEVOPS_PASSWORD` - these environment variables are taken from the `package-registries-auth-secret` secret. :::note Example: custom-nuget-settings.yaml ```yaml apiVersion: v1 kind: ConfigMap metadata: name: custom-nuget-settings data: nuget.config: | ``` ::: ## Related Articles * [Manage Container Registries](../../user-guide/manage-container-registries.md) * [Nexus Sonatype Integration](nexus-sonatype.md) * [Manage Git Servers](../../user-guide/git-server-overview.md) --- ## Keycloak as OIDC Provider for AWS EKS # EKS OIDC With Keycloak Configure Keycloak as the OIDC identity provider for an AWS EKS cluster so users can log in to `kubectl` with their existing Keycloak/SSO credentials instead of static IAM users. This guide walks through creating the Keycloak client and groups, associating Keycloak as an EKS OIDC provider (via Terraform or the AWS Console), and mapping Keycloak groups to Kubernetes RBAC roles — following the KubeRocketCI add-ons approach. For provisioning the underlying Keycloak realm/client/group resources through the Keycloak operator's Custom Resources, see [Provision Keycloak Resources for EKS OIDC](eks-oidc-integration.md). ## Prerequisites ### Required 1. The [kubelogin](https://github.com/int128/kubelogin) plugin, installed and configured to connect OIDC with a cluster. For Windows, it is recommended to download the kubelogin as a binary and add it to your PATH. Use the [latest kubelogin release](https://github.com/int128/kubelogin/releases). 2. A running [Keycloak instance](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/keycloak). 3. The [Keycloak operator](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/keycloak-operator) is deployed. 4. The Keycloak Realm's OIDC discovery URL and jwks_uri endpoints are publicly accessible.
Optional prerequisites - A recent Terraform 1.x release, if associating the EKS OIDC provider via Terraform instead of the AWS Console. - [EDP Cluster Add-ons](../add-ons-overview.md) Solution is applied. - [External Secrets Operator](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/external-secrets), if managing Keycloak credentials as synced secrets rather than manually.
## Solution Overview By the end of this guide, Keycloak groups will be mapped to Kubernetes RBAC roles, letting users authenticate to the EKS cluster with `kubectl` using their Keycloak credentials instead of static IAM users. This architecture encompasses three primary resource types: AWS (EKS), Keycloak, and Kubernetes. Within this setup, the Keycloak resources, once established, remain static, facilitating the assignment of claims based on user group memberships. This stability contrasts with the dynamic nature of other resources, which may be created, modified, or deleted as necessary. Of particular importance within the Kubernetes ecosystem are the `RoleBindings` and `ClusterRoles/Roles`. These elements define a permissions framework, where Roles specify the permissions available, and RoleBindings serve to associate those Roles with specific Keycloak groups. This association ensures that members of a group are granted only the permissions that are pertinent to their role. ![EKS Keycloak OIDC](../../assets/operator-guide/oidc_eks.drawio.png) ## Keycloak Configuration The initial step involves setting up the Keycloak operator (configure connections to the Keycloak instance) and along creation its associated resources, including realms, clients, roles, and groups. 1. Create user with [necessary](keycloak.md#configuration) privileges to Keycloak: Store user credentials in AWS Parameter Store: ```json title="AWS Parameter Store" { "keycloak": { "username": "", "password": "" } } ``` Deactivate the External Secret Operator within the primary [kuberocketci-rbac](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/kuberocketci-rbac/values.yaml#L27) chart: ```yaml title="values.yaml" # Configure components of the External Secrets Operator (ESO). eso: # -- Install components of the ESO. enabled: false ``` Create secret with user credentials: ```bash kubectl create secret generic keycloak \ --from-literal=username= \ --from-literal=password= ``` 2. Begin by installing the [**kuberocketci-rbac**](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) add-on. This can be accomplished through the use of the add-ons method, as detailed in the [addons approach](../add-ons-overview.md). Utilize the following values in the `values.yaml` file: ```yaml title="values.yaml" keycloakUrl: "https://example.com" # -- This block enable the creation of Keycloak operator resources for the # EKS OIDC configuration, such as client, client scope, and realm groups. kubernetes: enabled: true ``` 3. Activate the [**kuberocketci-rbac**](https://github.com/epam/edp-cluster-add-ons/blob/main/chart/values.yaml#L58) add-on within the application settings: ```yaml title="values.yaml" kuberocketci-rbac: createNamespace: true enable: true ``` This add-on sets up a broker realm to manage traffic redirection between external Identity Providers (IdP) and internal clients. Additionally, it creates a shared realm that encompasses all clients, including EKS, Sonar, Nexus, and Portal. The [KubeRocketCI RBAC add-on](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) creates Keycloak groups that are used in the KubeRocketCI platform to manage access to resources. For more details refer to the [KubeRocketCI Groups](platform-auth-model.md#groups) documentation. ## AWS Configuration Below are the guidelines for configuring identity provider in Kubernetes cluster: For integrating OpenID Connect (OIDC) with Amazon EKS through terraform, it's essential to update the EKS module within your terraform repository. The relevant repository can be found at KubeRocketCI's [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform/tree/master/eks). Adjust the module by incorporating the following configuration in the `eks/template.tfvars` file: ```json title="eks/template.tfvars" # OIDC Identity provider configuration cluster_identity_providers = { keycloak = { client_id = "eks" issuer_url = "https://example.com/auth/realms/shared" groups_claim = "groups" username_claim = "preferred_username" username_prefix = "-" } } ``` This configuration snippet specifies the Keycloak as the OIDC Identity Provider for your EKS cluster. It includes the client ID (`eks`), the issuer URL (pointing to the Keycloak realm), and the claim used for groups (`groups`). This setup ensures that authentication and authorization mechanisms for accessing the EKS cluster are correctly configured to use Keycloak as the identity provider. `username_claim` and `username_prefix` control how the authenticated user shows up as a Kubernetes username (in RBAC, `kubectl` output, and audit logs). Leaving them unset makes Kubernetes fall back to the `sub` claim (an opaque Keycloak user ID) prefixed with the issuer URL — see the [Access Validation](#access-validation) section below. Setting `username_claim` to `preferred_username` (or `email`) yields a human-readable username instead. Prefer `preferred_username` over `email`: Kubernetes' OIDC authenticator rejects tokens with `email_verified: false`, which is common for users brokered from a SAML upstream IdP (e.g. Azure AD) since SAML has no `email_verified` equivalent. `username_prefix = "-"` disables prefixing entirely; RBAC in this setup is unaffected either way, since group membership (not username) drives authorization. The objective is to configure an Identity Provider within your Kubernetes cluster. The process involves several steps within the AWS Management Console: 1. Begin by opening the AWS Management Console. 2. Navigate to the Elastic Kubernetes Service (EKS) section. 3. Select your specific **Cluster name**. 4. Go to the **Access** tab, then find and select the **OIDC identity providers** section. 5. Click on **Associate identity provider**. When associating the identity provider, ensure you input the following details: ```text Issuer URL: https://example.com/auth/realms/shared Client ID: eks Groups Claim: groups Username Claim: preferred_username Username Prefix: - ``` ## Kubeconfig Template for kubeconfig: ```yaml apiVersion: v1 preferences: {} kind: Config clusters: - cluster: server: https://.eks.amazonaws.com certificate-authority-data: name: eks contexts: - context: cluster: eks user: name: eks current-context: eks users: - name: user: exec: apiVersion: client.authentication.k8s.io/v1beta1 command: kubectl args: - oidc-login - get-token - -v1 - --oidc-issuer-url=https:///auth/realms/shared - --oidc-client-id=eks - --oidc-client-secret= ``` Flag `-v1` can be used for debug, in a common case it's not needed and can be deleted. To find the client secret: 1. Open Keycloak 2. Choose **Shared realm** 3. Find **eks** keycloak client 4. Open Credentials tab 5. Copy Secret ## Access Validation To validate access to the Kubernetes cluster by using the default **cluster-admin** role. Assign the user the **oidc-cluster-admins** Keycloak group. To add a user to a Keycloak group, follow these steps: 1. Open Keycloak 2. Choose **Shared** realm 3. Open user screen with search field 4. Find a user and open the configuration 5. Open Groups tab 6. In Available Groups, choose an **oidc-cluster-admins** group 7. Click the **Join** button 8. The group should appear in the User's Group Membership list As a result, the required access mapping is implemented using the following resources: | Keycloak Group Name | Kubernetes ClusterRole | Kubernetes ClusterRoleBinding | |----------------------|-------------------------|-------------------------------| | [oidc-cluster-admins](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/kuberocketci-rbac/templates/kubernetes/keycloak-realmgroups-cluster-admins.yaml) | cluster-admin (built-in) | [cluster-admin](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/kuberocketci-rbac/templates/kubernetes/clusterrolebinding-admin.yaml) | In this configuration, the Keycloak **oidc-cluster-admins** group is mapped to the Kubernetes **cluster-admin** role. This setup grants members of the **oidc-cluster-admins** group the necessary permissions to perform administrator management in the Kubernetes cluster. You can further customize access by associating different Keycloak groups with specific Kubernetes roles. KubeRocketCI follows the same approach for managing access to its resources. For more information, refer to the [KubeRocketCI Groups](platform-auth-model.md#groups) documentation. Follow the steps below to test the configuration: - Run kubectl command, it is important to specify the correct kubeconfig: ```bash KUBECONFIG= kubectl get ingresses -n ``` - After the first run and redirection to the Keycloak login page, log in using credentials (login:password) or using SSO Provider. In case of the successful login, you will receive the following notification that can be closed: ![OIDC Successful Login](../../assets/operator-guide/oidc-success-login.png) - As the result, a respective response from the Kubernetes will appear in the console in case a user is configured correctly and is a member of the correct group and Roles/RoleBindings. - If something is not set up correctly, the following output error will be displayed: ```bash Error from server (Forbidden): ingresses.networking.k8s.io is forbidden: User "" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "" ``` :::note The `User` field reflects whatever `username_claim`/`username_prefix` are configured in [AWS Configuration](#aws-configuration) above. Without `username_claim` set, it instead shows `https:///auth/realms/shared#` (the `sub` claim prefixed with the issuer URL) — harder to attribute to a real user in `kubectl` output and audit logs. ::: ## Session Update To update the session, clear cache. The default location for the login cache: ```bash rm -rf ~/.kube/cache ``` ## Access Cluster via Lens To access the Kubernetes cluster via [Lens](https://k8slens.dev/), follow the steps below to configure it: - Add a new kubeconfig to the location where Lens has access. The default location of the kubeconfig is **~/.kube/config** but it can be changed by navigating to **File** -> **Preferences** -> **Kubernetes** -> **Kubeconfig Syncs**. - (Optional) Using Windows, it is recommended to reboot the system after adding a new kubeconfig. - Authenticate on the Keycloak login page to be able to access the cluster. :::note Lens does not add namespaces of the project automatically, so it is necessary to add them manually, simply go to **Settings** -> **Namespaces** and add the namespaces of a project. ::: ## Changing the Lifespan of an Access Token By default, the Keycloak token has a lifespan of 5 minutes. To modify this duration refer to the guidelines outlined in this [document](ui-portal-oidc.md#changing-the-lifespan-of-an-access-token). For application-level OIDC via OAuth2-Proxy rather than cluster `kubectl` access, see [Tekton Dashboard Authentication](./oauth2-proxy.md). Using Microsoft Entra instead of Keycloak? See [AWS EKS OIDC With Microsoft Entra](../microsoft-entra/aws-eks-portal-authentication.md). ## Related Articles * [Headlamp OIDC Configuration](ui-portal-oidc.md) * [Provision Keycloak Resources for EKS OIDC](eks-oidc-integration.md) --- ## Provision Keycloak Resources for EKS OIDC # AWS EKS OIDC Integration This page shows how to provision the Keycloak realm, client, group, and user resources needed for EKS OIDC by installing the [edp-keycloak-operator](https://github.com/epam/edp-keycloak-operator) via Helm and applying its Custom Resources — `Keycloak`, `KeycloakRealm`, `KeycloakRealmGroup`, `KeycloakClientScope`, `KeycloakClient`, and `KeycloakRealmUser`. It covers only the Keycloak-side resource provisioning; for the end-to-end setup of EKS cluster access — associating Keycloak as the EKS OIDC provider and mapping groups to Kubernetes RBAC roles — see [Keycloak as OIDC Provider for AWS EKS](./configure-keycloak-oidc-eks.md). ## Prerequisites - [EKS Configuration](./configure-keycloak-oidc-eks.md) is performed - A recent Helm 3.x release is installed - [Keycloak](../../operator-guide/auth/keycloak.md) is installed ## Install Keycloak Operator :::info Alternately, the edp-keycloak-operator can be installed using a GitOps approach via the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) repository. For detailed installation instructions, please refer to the [Install via Add-ons](../add-ons-overview.md) guide. ::: To install the Keycloak operator, follow the steps below: 1. Add the `epamedp` Helm chart to a local client: ```bash helm repo add epamedp https://epam.github.io/edp-helm-charts/stable helm repo update ``` 2. Install the Keycloak operator: ```bash helm install keycloak-operator epamedp/keycloak-operator --namespace security --set name=keycloak-operator ``` ## Connect Keycloak Operator to Keycloak :::info It is also possible to install Keycloak resources using the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/eks) repository. For details, please refer to the [Install via Add-Ons](../add-ons-overview.md) page. ::: The next stage after installing Keycloak is to integrate it with the Keycloak operator. It can be implemented with the following steps: 1. Create the **keycloak** secret that contains username and password defined on the [configuration step](../../operator-guide/auth/keycloak.md#configuration): ```bash kubectl -n security create secret generic keycloak \ --from-literal=username= \ --from-literal=password= ``` 2. Create the Keycloak Custom Resource with the Keycloak instance URL and the secret created in the previous step: ```yaml apiVersion: v1.edp.epam.com/v1 kind: Keycloak metadata: name: main namespace: security spec: secret: keycloak # Secret name url: https://keycloak.example.com # Keycloak URL ``` 3. Create the KeycloakRealm Custom Resource: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakRealm metadata: name: control-plane namespace: security spec: realmName: control-plane keycloakOwner: main ``` 4. Create the `KeycloakRealmGroup` Custom Resource for both administrators and developers: - administrators: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakRealmGroup metadata: name: administrators namespace: security spec: realm: control-plane name: eks-oidc-administrator ``` - developers: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakRealmGroup metadata: name: developers namespace: security spec: realm: control-plane name: eks-oidc-developers ``` 5. Create the `KeycloakClientScope` Custom Resource: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakClientScope metadata: name: groups-keycloak-eks namespace: security spec: name: groups realm: control-plane description: "Group Membership" protocol: openid-connect protocolMappers: - name: groups protocol: openid-connect protocolMapper: "oidc-group-membership-mapper" config: "access.token.claim": "true" "claim.name": "groups" "full.path": "false" "id.token.claim": "true" "userinfo.token.claim": "true" ``` 6. Create the `KeycloakClient` Custom Resource: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakClient metadata: name: eks namespace: security spec: advancedProtocolMappers: true clientId: eks directAccess: true public: false defaultClientScopes: - groups targetRealm: control-plane webUrl: "http://localhost:8000" ``` 7. Create the KeycloakRealmUser Custom Resource for both administrator and developer roles: - administrator: ``` yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakRealmUser metadata: name: keycloakrealmuser-admin namespace: security spec: realm: control-plane username: "administrator" firstName: "John" lastName: "Snow" email: "administrator@example.com" enabled: true emailVerified: true password: "12345678" keepResource: true requiredUserActions: - UPDATE_PASSWORD groups: - eks-oidc-administrator ``` - developer: ``` yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakRealmUser metadata: name: keycloakrealmuser-developer namespace: security spec: realm: control-plane username: "developers" firstName: "John" lastName: "Snow" email: "developers@example.com" enabled: true emailVerified: true password: "12345678" keepResource: true requiredUserActions: - UPDATE_PASSWORD groups: - eks-oidc-developers ``` 8. To connect the created Keycloak resources with permissions, it is necessary to bind the created Keycloak groups to Kubernetes roles, e.g., assigning the Keycloak group `administrators` the Kubernetes Cluster role `cluster-admin`. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: oidc-cluster-admins subjects: - kind: Group apiGroup: rbac.authorization.k8s.io name: administrators roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin ``` 9. As a result, Keycloak is integrated with the AWS Elastic Kubernetes Service. This integration allows users to easily log in to the EKS cluster using their kubeconfig files and `kubelogin`, while managing permissions through Keycloak. This seamless integration enhances the user experience and streamlines the management of access control within the KubeRocketCI platform. ## Related Articles * [Keycloak Installation](keycloak.md) * [EKS OIDC With Keycloak](configure-keycloak-oidc-eks.md) --- ## Install Keycloak # Install Keycloak Inspect the prerequisites and the main steps to perform for installing Keycloak. :::info The installation process below is given for a Kubernetes cluster. The steps that differ for an OpenShift cluster are indicated in the warnings blocks. ::: ## Prerequisites - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. :::info Out team recommends a Keycloakx helm chart from the [codecentric](https://github.com/codecentric/helm-charts/tree/master/charts/keycloakx) repository, but other repositories can be used as well (e.g. [Bitnami](https://github.com/bitnami/charts/tree/master/bitnami/keycloak/)). Before installing Keycloak, it is necessary to install a [PostgreSQL database](https://www.postgresql.org/download/). ::: :::info It is also possible to install Keycloak using the cluster add-ons. For details, please refer to the [Install via Add-Ons](../add-ons-overview.md) page. ::: ## PostgreSQL Installation To install PostgreSQL, follow the steps below: 1. Check that a security namespace is created. If not, run the following command to create it: ```bash kubectl create namespace security ``` :::warning On the OpenShift platform, apply the `SecurityContextConstraints` resource. Change the namespace in the `users` section if required.
View: keycloak-scc.yaml ```yaml allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: null apiVersion: security.openshift.io/v1 allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - min: 999 max: 65543 groups: [] kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: keycloak priority: 1 readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsRange uidRangeMin: 1 uidRangeMax: 65543 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: - system:serviceaccount:security:keycloakx volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
View: postgresql-keycloak-scc.yaml ```yaml allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: null apiVersion: security.openshift.io/v1 allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - min: 999 max: 65543 groups: [] kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: postgresql-keycloak priority: 1 readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsRange uidRangeMin: 1 uidRangeMax: 65543 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: - system:serviceaccount:security:default volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
::: 2. Create PostgreSQL admin secret: ```bash kubectl -n security create secret generic keycloak-postgresql \ --from-literal=password= \ --from-literal=postgres-password= ``` 3. Add a helm chart repository: ```bash helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update ``` 4. Install PostgreSQL v15.2.0 using [bitnami/postgresql](https://artifacthub.io/packages/helm/bitnami/postgresql) Helm chart v12.1.15: :::info The PostgreSQL can be deployed in production ready mode. For example, it may include multiple replicas, persistent storage, autoscaling, and monitoring. For details, please refer to the [official Chart documentation](https://github.com/bitnami/charts/tree/master/bitnami/postgresql). ::: ```bash helm install postgresql bitnami/postgresql \ --version 12.1.15 \ --values values.yaml \ --namespace security ``` Check out the _values.yaml_ file sample of the PostgreSQL customization:
View: values.yaml ```yaml # PostgreSQL read only replica parameters readReplicas: # Number of PostgreSQL read only replicas replicaCount: 1 image: tag: 15.2.0-debian-11-r0 global: postgresql: auth: username: admin existingSecret: keycloak-postgresql database: keycloak primary: persistence: enabled: true size: 3Gi ```
## Keycloak Installation To install Keycloak, follow the steps below: 1. Use `security` namespace from the PostgreSQL installation. 2. Add a chart repository: ```bash helm repo add codecentric https://codecentric.github.io/helm-charts helm repo update ``` 3. Create Keycloak admin secret: ```bash kubectl -n security create secret generic keycloak-admin-creds \ --from-literal=username= \ --from-literal=password= ``` 4. Install Keycloak 20.0.3 using [codecentric/keycloakx](https://artifacthub.io/packages/helm/codecentric/keycloakx) Helm chart: :::info Keycloak can be deployed in production ready mode. For example, it may include multiple replicas, autoscaling, and monitoring. For details, please refer to the [official Chart documentation](https://github.com/codecentric/helm-charts/tree/master/charts/keycloakx). ::: ```bash helm install keycloakx codecentric/keycloakx \ --version 2.3.0 \ --values values.yaml \ --namespace security ``` Check out the _values.yaml_ file sample of the Keycloak customization:
View: values.yaml ```yaml replicas: 1 # Deploy the latest version image: tag: "24.0.4" # The following parameter is unrecommended to expose. Exposed health checks lead to an unnecessary attack vector. health: enabled: false # The following parameter is unrecommended to expose. Exposed metrics lead to an unnecessary attack vector. metrics: enabled: false command: - "/opt/keycloak/bin/kc.sh" - "--verbose" - "start" extraEnv: | - name: KC_HOSTNAME value: "keycloak." - name: KC_SPI_HOSTNAME_DEFAULT_ADMIN value: "keycloak." - name: KC_HTTP_ENABLED value: "true" - name: KC_HOSTNAME_STRICT value: "false" - name: KC_HOSTNAME_STRICT_HTTPS value: "false" - name: KC_SPI_EVENTS_LISTENER_JBOSS_LOGGING_SUCCESS_LEVEL value: "info" - name: KEYCLOAK_ADMIN valueFrom: secretKeyRef: name: keycloak-admin-creds key: username - name: KEYCLOAK_ADMIN_PASSWORD valueFrom: secretKeyRef: name: keycloak-admin-creds key: password - name: JAVA_OPTS_APPEND value: >- -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0 -Djava.awt.headless=true -Djgroups.dns.query={{ include "keycloak.fullname" . }}-headless -Dkeycloak.connectionsHttpClient.default.expect-continue-enabled=true -Dkeycloak.connectionsHttpClient.default.reuse-connections=false - name: HTTP_ADDRESS_FORWARDING value: "true" - name: PROXY_ADDRESS_FORWARDING value: "true" # This block should be uncommented if you install Keycloak on Kubernetes ingress: enabled: true annotations: nginx.ingress.kubernetes.io/proxy-buffer-size: 256k # Defines the class of the Ingress Controller. # It allows you to choose which Ingress controller in cluster should be used to expose the Keycloak service to the outside world. ingressClassName: "nginx" # Exposes Keycloak paths according to the rules from documentation. # Ref: https://www.keycloak.org/server/reverseproxy#_exposed_path_recommendations console: enabled: false rules: - host: keycloak. paths: - path: '{{ tpl .Values.http.relativePath $ | trimSuffix "/" }}/' pathType: Prefix proxy: enabled: true mode: "edge" # This block should be uncommented if you set Keycloak to OpenShift and change the host field # route: # enabled: false # # Path for the Route # path: '/' # # Host name for the Route # host: "keycloak." # # TLS configuration # tls: # enabled: true resources: limits: memory: "2048Mi" requests: cpu: "50m" memory: "512Mi" # Check database readiness at startup dbchecker: enabled: true database: vendor: postgres existingSecret: keycloak-postgresql hostname: postgresql port: 5432 username: admin database: keycloak ```
## Configure Keycloak with the EDP Keycloak Operator {#configuration} For configuring Keycloak within your environment, it's recommended to utilize the [edp-keycloak-operator](https://github.com/epam/edp-keycloak-operator). This operator simplifies the integration process by automating the deployment and management of Keycloak instances. It ensures that Keycloak is correctly configured to serve as an identity provider, facilitating secure access management across your applications. By leveraging the edp-keycloak-operator, you can efficiently manage realms, users, and client configurations, making it an essential tool for robust access control in Kubernetes environments. To create a Keycloak user in the Master realm who can manage other realms, follow these steps: 1. Create the `edp_` user and set the password in the `Master` realm. :::note This user should be used by Keycloak Operator to access Keycloak. Please refer to the [Install KubeRocketCI](../install-kuberocketci.md) page for details. ::: 2. In the `Role Mapping` tab, assign the proper roles to the user: - Realm Roles: - create-realm, - offline_access, - uma_authorization - Client Roles `openshift-realm`: - impersonation, - manage-authorization, - manage-clients, - manage-users ![Role mappings](../../assets/operator-guide/keycloak-roles.png "Role mappings") :::note `openshift-realm` is the realm name used as the `broker` realm, which is utilized for integrating third-party Identity Providers. You can find more information about this integration in the [kuberocketci-rbac](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) repository of the KubeRocketCI platform. ::: ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install Harbor](../artifacts-management/harbor-installation.md) * [Install via Add-Ons](../add-ons-overview.md) --- ## KubeRocketCI CLI Keycloak Client # KubeRocketCI CLI Keycloak Client The [KubeRocketCI CLI](https://github.com/KubeRocketCI/cli) authenticates users through the browser (OAuth 2.0 authorization code flow with PKCE). This page describes how to register a **public** OpenID Connect client in Keycloak so operators can run `krci auth login` against your KubeRocketCI Portal. :::info Create the client in the **same realm** the Portal already uses. To confirm the realm, inspect the OIDC settings returned by the Portal, for example the endpoint `GET /rest/v1/config/oidc` on your Portal base URL (replace the host with yours): ```http GET https://portal.example.com/rest/v1/config/oidc ``` Create the client only in the **Platform realm** you identified above. Application clients belong there, not in Keycloak’s built-in administrative realms. ::: ## Configure via KeycloakClient (GitOps) If your cluster is managed with [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) and the **Keycloak Operator** is installed, you can provision the `krci-cli` client as a `KeycloakClient` custom resource instead of using the Admin Console. The reference manifest ships with the `kuberocketci-rbac` add-on: [`clusters/core/addons/kuberocketci-rbac/templates/broker/keycloak-client-krci-cli.yaml`](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/kuberocketci-rbac/templates/broker/keycloak-client-krci-cli.yaml) It creates a public OpenID Connect client in the Platform **broker** realm (`realmRef.kind: KeycloakRealm`). This matches the realm the Portal typically uses (confirm via `GET /rest/v1/config/oidc` on your Portal URL). The manifest configures: - **Client ID**: `krci-cli` - **Redirect URIs**: `http://127.0.0.1/*`, `http://localhost/*` - **Default scopes**: `profile`, `email`, `roles`, `groups` - **Audience mapper**: `kubernetes` (for clusters with an OIDC-enabled API server) :::note Other Keycloak clients in add-ons may reference `ClusterKeycloakRealm` (for example realm `main` / `shared`). The CLI client is intentionally created in the **broker** realm, alongside the Portal OIDC issuer. ::: Enable the add-on in your fork, sync the Argo CD application, then verify reconciliation: ```bash kubectl get keycloakclient krci-cli -n security ``` Expected status: `OK`. :::warning When the client is reconciled from Git by the Keycloak Operator, changes made in the Keycloak Admin Console may be overwritten. Update the manifest in your add-ons repository to persist configuration changes. ::: If add-ons are not used, or you manage a standalone Keycloak instance, follow the manual steps below in the same realm your Portal uses. ## Configure Manually in Keycloak Admin Console For a manual setup, use the Keycloak Admin Console to select the same realm as the Portal, create the public `krci-cli` client, review client scopes and token settings, add the Kubernetes audience mapper, and then verify login from your workstation. ### Select Realm In the Keycloak Admin Console, use the realm drop-down in the top-left corner and select the realm your Portal uses. ### Create Client The next step is to create a Keycloak client itself: 1. In the left sidebar, open **Clients**, then click **Create client**. 2. On **General settings**, set the fields as follows: | Field | Value | |-------|--------| | Client type | OpenID Connect | | Client ID | `krci-cli` | | Name | KubeRocketCI CLI (display only) | | Description | Public OIDC client for the KubeRocketCI CLI (PKCE) | | Always display in console | Off | 3. Click **Next**. 4. On **Capability config**, set the toggles: | Toggle | Value | Note | |--------|-------|------| | Client authentication | Off | Public client (no client secret) | | Authorization | Off | — | | Standard flow | On | Authorization code flow | | Direct access grants | Off | — | | Implicit flow | Off | — | | Service accounts roles | Off | — | | OAuth 2.0 Device Authorization Grant | Off | Browser-based login | | OIDC CIBA Grant | Off | — | 5. Click **Next**. 6. On **Login settings**, configure URLs: | Field | Value | |-------|--------| | Root URL | *(empty)* | | Home URL | *(empty)* | | Valid redirect URIs | `http://127.0.0.1/*` and `http://localhost/*` (add **both**) | | Valid post logout redirect URIs | *(empty)* | | Web origins | `+` (allow all origins configured for this client) | :::note The CLI binds a callback listener on `localhost` with a **random port** on each login, so wildcard redirect URIs are required. ::: 7. Click **Save**. You are taken to the client **Settings** page. The screenshots below illustrate **Access settings** (client ID, redirect URIs, web origins) and **Capability config** / **Login settings** on a typical Keycloak deployment. ![Keycloak client access settings — client ID, redirect URIs, and web origins](../../assets/operator-guide/krci-cli-client-for-keycloak/krci-cli-keycloak-client-access-settings.png) ![Keycloak client capability and login settings](../../assets/operator-guide/krci-cli-client-for-keycloak/krci-cli-keycloak-capability-login-settings.png) ### Enforce PKCE (S256) To enforce PKCE, do these steps: 1. Open the **Advanced** tab for the `krci-cli` client. 2. Under **Advanced settings**, set **Proof Key for Code Exchange Code Challenge Method** to **S256**. 3. Click **Save** for that section. This ensures Keycloak only accepts logins that send a PKCE code challenge, which matches how the CLI performs authorization. ### Configure Token Lifetimes On the client **Advanced** tab, you can leave **Access Token Lifespan** and session timeouts **inherited from the realm** (for example five to fifteen minutes for access tokens), or set client-specific values. At realm level (**Realm settings** → **Tokens**), confirm **OAuth 2.0 Refresh Token Rotation** matches your security policy. The CLI refreshes tokens using the standard OAuth2 token endpoint. ### Configure Client Scopes To configure client scopes (openid, profile, email, roles, groups), follow the steps below: 1. Open the **Client scopes** tab for `krci-cli`. 2. Under **Default** client scopes, ensure typical OpenID scopes are present, including **`openid`**, **`profile`**, **`email`**, and **`roles`** (exact names can vary slightly by Keycloak version). 3. If your Platform expects **group** membership in tokens, add a **groups** scope (or your realm’s equivalent, for example a custom **groups** / **groups-core** scope) as a **Default** scope: - Click **Add client scope** → choose the groups scope → **Add** → **Default**. If the realm has no suitable groups scope yet: 1. Go to **Client scopes** in the left sidebar → **Create client scope**. 2. Set **Name** (for example `groups`), **Type** Default, **Protocol** OpenID Connect, then save. 3. Open that scope → **Mappers** → **Configure a new mapper** → **Group Membership** with settings such as: - **Token Claim Name**: `groups` - **Full group path**: Off - **Add to ID token** / **Add to access token** / **Add to userinfo**: On (as required by your cluster and Portal) 4. Return to the `krci-cli` client → **Client scopes** → add the new scope as **Default**. ![Keycloak client scopes assigned to krci-cli](../../assets/operator-guide/krci-cli-client-for-keycloak/krci-cli-keycloak-client-scopes.png) ### Configure Audience Mapper for Kubernetes API The same ID token can be validated by the Kubernetes API server (when OIDC is enabled on the cluster). Add an **Audience** mapper on the **dedicated** client scope Keycloak creates for this client (name pattern `krci-cli-dedicated`): 1. On `krci-cli` → **Client scopes** tab, click the dedicated scope **`krci-cli-dedicated`**. 2. Open the **Mappers** tab → **Configure a new mapper** → **Audience**. 3. Configure the mapper so the issued token’s `aud` claim includes **both** the CLI client ID and the value your API server uses for **`--oidc-client-id`** (often `kubernetes`): | Field | Suggested value | |-------|------------------| | Name | `k8s-audience` (or any descriptive name) | | Included Client Audience | *(optional)* depends on whether you model audience as another client | | Included Custom Audience | `kubernetes` *(or the exact `--oidc-client-id` value on the API server)* | | Add to ID token | On | | Add to access token | On | | Add to lightweight access token | Off | 4. Save the mapper. :::warning If you also change the API server to expect `aud: kubernetes`, ensure the **Portal** client’s tokens carry a compatible audience as well, or Portal logins and API calls may fail. Align this step with your [EKS / cluster OIDC configuration](configure-keycloak-oidc-eks.md). ::: The following screenshots show the **Mappers** list on the dedicated scope and an example **Audience** mapper form. Field names and whether you use **Included Client Audience** vs **Included Custom Audience** depend on your Keycloak version and how cluster OIDC is configured. ![Dedicated client scope — mappers list](../../assets/operator-guide/krci-cli-client-for-keycloak/krci-cli-keycloak-dedicated-scope-mappers.png) ![Audience mapper configuration example](../../assets/operator-guide/krci-cli-client-for-keycloak/krci-cli-keycloak-audience-mapper.png) ## Verify Configuration To ensure everything is configured properly, carefully read the checklist below: 1. **Clients** → `krci-cli` → **Settings**: Client ID, **Standard flow** enabled, redirect URIs include `http://127.0.0.1/*` and `http://localhost/*`. 2. **Advanced**: PKCE code challenge method **S256**. 3. **Client scopes**: Default scopes include `openid`, profile-related scopes, email, roles, and your groups scope as required. 4. **Client scopes** → `krci-cli-dedicated` → **Mappers**: audience mapper present and aligned with cluster `--oidc-client-id`. ## Test From Workstation Run the command below in a terminal (replace the Portal URL with yours): ```bash krci auth login --portal-url https://portal.example.com ``` Expected behavior: 1. A browser window opens on the Keycloak login page. 2. After successful login (and consent, if enabled), the browser redirects to `http://localhost:/callback?code=...`. 3. The terminal prints a success message with the signed-in identity. Decode the ID token (for example with [jwt.io](https://www.jwt.io/) or your own tooling) and confirm claims similar to: ```json { "iss": "https://keycloak.example.com/realms/your-realm", "aud": ["krci-cli", "kubernetes"], "azp": "krci-cli", "email": "user@example.com", "groups": ["admin", "developers"] } ``` If `aud` does not include your Kubernetes OIDC client ID, revisit **Configure Audience Mapper for Kubernetes API**. If `groups` (or your chosen claim) is missing, revisit **Configure Client Scopes**. ## Related Articles * [Install Keycloak](keycloak.md) * [EKS OIDC With Keycloak](configure-keycloak-oidc-eks.md) * [OIDC Integration With EKS](eks-oidc-integration.md) * [Headlamp OIDC Configuration](ui-portal-oidc.md) --- ## Kubernetes Namespace Management # Kubernetes Namespace Management KubeRocketCI [deploys](../../user-guide/add-cd-pipeline.md) applications to namespaces. By default, the platform automatically creates these namespaces. This page provides information on an alternative approach to namespace management. ## Overview Namespaces are created by the platform when user creates CD Pipelines. The operator creates them according to the specific format: `--`. The [cd-pipeline-operator](https://github.com/epam/edp-cd-pipeline-operator) should have the permissions to automatically create namespaces when deploying applications and delete them when uninstalling applications. ## Disable Automatic Namespace Creation In certain scenarios, the automatic generation of namespaces might be restricted. For instance, project-specific security protocols might necessitate disabling this feature. This can be achieved by adjusting the `manageNamespace` parameter within the [values.yaml](https://github.com/epam/edp-cd-pipeline-operator/blob/release/2.20/deploy-templates/values.yaml#L96) file. By default, `manageNamespace` is enabled (true), but it can be toggled off (false). Consequently, users might encounter issues when attempting to deploy applications through the KubeRocketCI portal due to permission constraints: ![Namespace creation error](../../assets/operator-guide/failed_cd_pipeline_creation.png) The error highlighted above indicates the necessity for users to manually create a namespace following the `--` naming convention prior to stage creation. Moreover, the cd-pipeline-operator requires administrative rights to manage the namespace effectively. For instructions on manually creating a namespace, proceed with the steps outlined below: 1. Create the namespace by running the command below: ```bash kubectl create namespace -- ``` :::note The `--` format for namespaces is set by default but is not mandatory. You can set your custom namespace when creating an Environment. ::: 2. Create the administrator RoleBinding resource by applying the file below with the `kubectl apply -f grant_admin_permissions.yaml` command: :::note grant_admin_permissions.yaml ```yaml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: edp-cd-pipeline-operator-admin namespace: -- subjects: - kind: ServiceAccount name: edp-cd-pipeline-operator namespace: krci roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: admin ``` ::: 3. Restart the cd-pipeline-operator deployment, to speedup reconciliation. ## CD Pipeline Operator RBAC Model The `manageNamespace` parameter also defines the resources that will be created depending on the cluster deployed whether it is OpenShift or Kubernetes. This scheme displays the nesting of operator input parameters: ![CD Pipeline Operator Input Parameter Scheme](../../assets/operator-guide/manage-namespace.png) :::note When deploying application on the OpenShift cluster, the `registry-view` RoleBinding is created in the main namespace. ::: ## Related Articles * [KubeRocketCI Access Model](platform-auth-model.md) * [EKS OIDC With Keycloak](configure-keycloak-oidc-eks.md) --- ## Tekton Dashboard Authentication # Tekton Dashboard Authentication [OAuth2-Proxy](https://oauth2-proxy.github.io/oauth2-proxy/) is a versatile tool that serves as a reverse proxy, utilizing the OAuth 2.0 protocol with various providers like Google, GitHub, and Keycloak to provide both authentication and authorization. This guide instructs readers on how to protect their applications' endpoints using OAuth2-Proxy. By following these steps, users can enhance the security of their endpoints without modifying their current application code. In the context of KubeRocketCI, it has integration with the Keycloak OIDC provider, enabling it to connect with any component that lacks built-in authentication. ## Prerequisites - [Keycloak](keycloak.md) with OIDC authentication is installed. - [Keycloak operator](../add-ons-overview.md) is installed. ## Integration OAuth2-Proxy To streamline the installation of OAuth2-Proxy in your environment, it is advised to utilize the resources available in the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/oauth2-proxy) and their [applications](https://github.com/epam/edp-cluster-add-ons/blob/main/chart/values.yaml#L120). ## Enable OAuth2-Proxy on Tekton Dashboard The example below illustrates how to use OAuth2-Proxy in practice when using the Tekton dashboard: Edit the Tekton dashboard Ingress annotation by adding `auth-signin` and `auth-url` of oauth2-proxy by `kubectl` command: ```bash kubectl annotate ingress \ nginx.ingress.kubernetes.io/auth-signin='https:///oauth2/start?rd=https://$host$request_uri' \ nginx.ingress.kubernetes.io/auth-url='http://oauth2-proxy.oauth2-proxy.svc.cluster.local:80/oauth2/auth' ``` 1. Generate a cookie-secret for proxy with the following command: ```bash tekton_dashboard_cookie_secret=$(openssl rand -base64 32 | head -c 32) ``` 2. Create `tekton-dashboard-proxy-cookie-secret` in the krci namespace: ```bash kubectl -n krci create secret generic tekton-dashboard-proxy-cookie-secret \ --from-literal=cookie-secret=${tekton_dashboard_cookie_secret} ``` 3. Run `helm upgrade` to update edp-install release: ```bash helm upgrade --version --set 'edp-tekton.dashboard.openshift_proxy.enabled=true' edp-install --namespace krci ``` ## Related Articles * [Keycloak Installation](keycloak.md) * [Keycloak OIDC Installation](configure-keycloak-oidc-eks.md) * [Tekton Installation](../install-tekton.md) --- ## Authentication and Authorization: Overview # Authentication and Authorization: Overview KubeRocketCI uses [Keycloak](https://keycloak.org) as central access management solution, handling user permissions across various platform tools, including Kubernetes. Kubernetes clusters leverage Keycloak for RBAC implementation via [OIDC](https://kubernetes.io/docs/reference/access-authn-authz/authentication/). This consistent approach allows administrators to manage access uniformly. Within Kubernetes, [Custom Resources (CRs)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) are used to refine and enforce access controls by [configuring Keycloak](https://operatorhub.io/operator/edp-keycloak-operator), translating Kubernetes configurations into Keycloak access policies. ## Keycloak Components Overview Keycloak is a critical component in KubeRocketCI for managing authentication and authorization. It uses realms, roles and groups to control access across the platform. The diagram below illustrates the Keycloak resources within the KubeRocketCI system. It shows how tools and their Keycloak configurations are managed within the `shared` and `broker` realms. The `broker` realm serves as a central "authentication proxy" (broker), connecting to external Identity Providers (IdPs) such as Google, Facebook, and Active Directory (AD) through OIDC and/or SAML protocols. The `shared` realm consolidates platform tools such as Argo CD, Nexus Repository, SonarQube, Report-portal, Harbor, and others as OIDC (OpenID Connect) clients. Each component can have its own unique configurations, including Keycloak roles, groups, or client scopes. ```mermaid graph TD; style integrated_services fill:#FFF2CC,stroke:#333,stroke-width:1px style broker_realm fill:#FFF2CC,stroke:#333,stroke-width:1px style IdP fill:#E1D5E7,stroke:#333,stroke-width:2px,dashed subgraph integrated_services ["shared realm"] direction TB Report-Portal["Report-portal---Keycloak Client Keycloak Client Scope"] Sonar["Sonar---Keycloak ClientKeycloak Realm Role Batch"] Vault-kms["Vault-kms --- Keycloak Client"] Vault-okd["Vault-okd --- Keycloak Client"] ArgoCD["Argo CD --- Keycloak Client Keycloak Realm Group"] EKS["EKS --- Keycloak Client Keycloak Client Scope Keycloak Realm Groups"] Harbor["Harbor --- Keycloak Client"] Harbor-ha["Harbor-ha --- Keycloak Client"] Harbor-ha-okd["Harbor-ha-okd --- Keycloak Client"] AWX-operator["AWX Operator --- Keycloak Client"] Nexus["Nexus --- Keycloak Client"] OpenSearch["OpenSearch --- Keycloak Client Keycloak Client Scope"] Prometheus-operator["Prometheus-operator --- Keycloak Client (Grafana)"] Dependency-Track["Dependency-Track --- Keycloak Client"] DefectDojo["DefectDojo --- Keycloak Client"] Portal["KubeRocketCI portal --- KeycloakClient Keycloak Client Secret"] %% Define the new 4x4 grid layout Report-Portal ~~~ Nexus ~~~ EKS ~~~ Portal Sonar ~~~ Harbor ~~~ OpenSearch ~~~ ArgoCD Vault-kms ~~~ Harbor-ha ~~~ Prometheus-operator ~~~ AWX-operator Vault-okd ~~~ Harbor-ha-okd ~~~ Dependency-Track ~~~ DefectDojo end subgraph broker_realm ["broker realm"] shared["shared --- Keycloak Client"] end integrated_services --> shared["shared --- Keycloak Client"] broker_realm --> IdP["IdP (Google, Facebook, AD)"] ``` ## Keycloak Roles and Groups The diagram below provides an overview of the access control mechanisms for various tools integrated with Keycloak. It illustrates two primary methods of managing user access: [Keycloak roles](#roles) and [Keycloak groups](#groups). ```mermaid graph TD; style component_access_control fill:#F5F5F5,stroke:#333,stroke-width:1px style keycloak_roles fill:#F5F5F5,stroke:#333,stroke-width:1px style keycloak_groups fill:#F5F5F5,stroke:#333,stroke-width:1px style mock_object fill:#FFF2CC,stroke:#FFF2CC,stroke-width:1px style AWX-operator fill:#F8CECC,stroke:#333,stroke-width:1px style DefectDojo fill:#F8CECC,stroke:#333,stroke-width:1px style ReportPortal fill:#F8CECC,stroke:#333,stroke-width:1px style Vault fill:#F8CECC,stroke:#333,stroke-width:1px style Gerrit fill:#F8CECC,stroke:#333,stroke-width:1px style keycloak_roles_components fill:#FFF2CC,stroke:#333,stroke-width:1px style keycloak_groups_components fill:#FFF2CC,stroke:#333,stroke-width:1px subgraph keycloak_roles_components [" "] direction TB Dependency-Track["Dependency-Track --- administrator developer"] Harbor["Harbor --- administrator"] TektonDashboard["Tekton Dashboard --- administrator developer"] OpenSearch["OpenSearch --- administrator developer"] Grafana["Grafana --- administrator developer"] Sonar["Sonar --- sonar-administrators sonar-developers"] Nexus["Nexus --- administrator developer --- Specific roles are assigned through the CR NexusUser"] AWX-operator["AWX Operator --- Specific roles are assigned through the UI"] DefectDojo["DefectDojo --- Specific roles are assigned through the UI"] ReportPortal["Report-portal --- Specific roles are assigned through the UI"] Vault["Vault --- Role 'reader' by default"] Dependency-Track ~~~ OpenSearch ~~~ Sonar ~~~ DefectDojo Harbor ~~~ Grafana ~~~ ReportPortal ~~~ Vault TektonDashboard ~~~ Nexus ~~~ AWX-operator ~~~ mock_object[" "] end subgraph keycloak_groups_components [" "] direction TB ArgoCD["Argo CD --- ArgoCDAdmins ArgoCDDevelopers"] PortalUI["KubeRocketCI portal --- ${platform}-oidc-admins ${platform}-oidc-developers ${platform}-oidc-viewers"] Gerrit["Gerrit --- Specific roles are assigned through the CR GerritGroupMember"] ArgoCD ~~~ Gerrit PortalUI ~~~ Gerrit end keycloak_roles --> keycloak_roles_components keycloak_groups --> keycloak_groups_components component_access_control["Component Access Control"] --> keycloak_roles["Keycloak Roles"] component_access_control["Component Access Control"] --> keycloak_groups["Keycloak Groups"] ``` Components, highlighted in red, require additional configuration directly within the tool's UI. :::info `${platform}` is a placeholder for the platform name (namespace), where the KubeRocketCI is deployed. ::: For detailed instructions on setting up OIDC integration for various components, please refer to the following articles: 1. **AWX** - [Community Documentation](https://ansible.readthedocs.io/projects/awx/en/latest/) 2. **DefectDojo** - [DefectDojo Permissions](https://docs.defectdojo.com/en/customize_dojo/user_management/configure_sso/#keycloak) 3. **Nexus** - [Nexus authentication process](#nexus-repository-manager) 4. **Report Portal** - [Report Portal Keycloak Integration](../project-management-and-reporting/reportportal-keycloak.md) 5. **Vault** - [OIDC provider configuration](https://developer.hashicorp.com/vault/docs/auth/jwt/oidc-providers/keycloak) 6. **Tekton** - [Tekton Dashboard Authentication](oauth2-proxy.md) 7. **Gerrit** - [Gerrit Authentication](#gerrit) ### Roles The `shared` Keycloak realm defines two composite roles: - **Administrator Role**: - Designed for users who need administrative access to the tools used on the platform. - This composite role includes the `administrator` and `sonar-administrators` roles. - Users assigned the `administrator` composite role will inherit both roles. - **Developer Role**: - Designed for users who need access to the development tools used on the platform. - This composite role includes the `developer` and `sonar-developers` roles. - Users assigned the `developer` composite role will inherit both roles. These composite roles simplify the assignment of administrative and development permissions within the KubeRocketCI. The table below provides an overview of the `shared` realm roles and their types: | Realm Role Name | Regular Role | Composite role | | -------------------- | :----------------: | :----------------: | | administrator | | :white_check_mark: | | developer | | :white_check_mark: | | sonar-administrators | :white_check_mark: | | | sonar-developers | :white_check_mark: | | ### Groups KubeRocketCI uses the `shared` realm for group management. The [groups](https://github.com/epam/edp-install/tree/master/deploy-templates/templates/rbac) are designed to control access to various components such as Argo CD, the KubeRocketCI portal, and the EKS cluster. | Group Name | Purpose | | ----------------------------- | ------------------------------------------------------------------- | | `ArgoCDAdmins` | Administrator access to Argo CD instance | | `ArgoCD-${platform}-users` | Access to the Argo CD project mapped to the `${platform}` tenant | | `oidc-cluster-admins` | Full administrator (cluster-admin) access to the kubernetes cluster | | `${platform}-oidc-admins` | Administrator access to KubeRocketCI | | `${platform}-oidc-developers` | Developer access to KubeRocketCI | | `${platform}-oidc-viewers` | Read-only access to view resources in KubeRocketCI | These groups simplify the management of permissions and ensure that users have the appropriate level of access based on their roles and responsibilities. ## SonarQube OIDC Access to SonarQube is managed through configurations in Keycloak (Keycloak Operator) and SonarQube (Sonar Operator). This section provides a detailed walkthrough for configuring access control. ### Keycloak Configuration SonarQube access is managed using Keycloak roles within the `shared` realm: - **sonar-developers**: Grants developer access to SonarQube. - **sonar-administrators**: Grants administrative access to SonarQube. To grant access, the appropriate role must be assigned to the user in Keycloak. :::note Example A user who needs developer access to SonarQube should be assigned the `sonar-developers` role or the `developer` composite role in Keycloak. ::: ### SonarQube Configuration [KubeRocketCI Sonar Operator](https://github.com/epam/edp-sonar-operator) provides a [SonarQube Permission Template](https://docs.sonarsource.com/sonarqube-server/latest/instance-administration/system-functions/security/) to manage user access and permissions for SonarQube projects. The template is stored in the SonarQube [custom resource](https://github.com/epam/edp-sonar-operator/blob/master/deploy-templates/_crd_examples/permission_template.yaml) of the operator. An example of a custom resource is shown below. :::note Sonar Permission Template ```yaml apiVersion: v2.edp.epam.com/v1 kind: SonarPermissionTemplate metadata: name: edp-default spec: name: edp-default description: "KubeRocketCI permission templates (DO NOT REMOVE)" projectKeyPattern: ".+" default: true groupsPermissions: view-group: - user sonar-developers: - user - securityhotspotadmin - issueadmin - codeviewer sonar-administrators: - admin - user sonarRef: name: sonar ``` ::: The SonarQube Permission Template defines three groups: `view-group`, `sonar-administrators`, and `sonar-developers`: - **view-group**: Users who have read-only access to the project. They can view project's data and metrics but cannot modify or interact with it. - **sonar-administrators**: Users with full control over the SonarQube project. They can create, modify, delete projects, manage user access, and configure SonarQube settings. - **sonar-developers**: Users actively working on the SonarQube project. They have read and write access, can modify project data and metrics, and configure project-specific settings. These groups provide different levels of access based on the user's role and responsibilities. :::info If a user is not assigned to any group, they will be placed in the `sonar-users` group by default. This group does not have any permissions in the `edp-default` Permission Template. ::: ### Group Permissions The following table describes the permissions assigned to each group: | Group Name | Sonar Permissions | | -------------------- | -------------------------------------------------- | | view-group | user | | sonar-administrators | admin, user | | sonar-developers | codeviewer, issueadmin, securityhotspotadmin, user | | sonar-users | - | ## Nexus Repository Manager KubeRocketCI leverages oauth2-proxy to authenticate users in Nexus Repository Manager OSS. The user's access level is determined by the roles assigned in Keycloak. :::info Only users with either the `administrator` or `developer` role in Keycloak can access Nexus. ::: The platform manages two distinct roles: `edp-admin`, `edp-viewer`. :::note Example of edp-admin NexusRole and NexusUser ```yaml apiVersion: edp.epam.com/v1alpha1 kind: NexusRole metadata: name: edp-admin spec: id: edp-admin nexusRef: name: nexus kind: Nexus name: edp-admin description: Read and write access to all repos and scripts privileges: - nx-apikey-all - nx-repository-view-*-*-add - nx-repository-view-*-*-browse - nx-repository-view-*-*-edit - nx-repository-view-*-*-read - nx-script-*-add - nx-script-*-delete - nx-script-*-run - nx-search-read ``` ```yaml apiVersion: edp.epam.com/v1alpha1 kind: NexusUser metadata: name: ci-user namespace: nexus spec: email: ci.user@edp.com firstName: ci.user id: ci.user lastName: CI nexusRef: kind: Nexus name: nexus roles: - edp-admin secret: $ci-nexus:password status: active ``` ::: ## KubeRocketCI Portal and Kubernetes Cluster Both the KubeRocketCI Portal and the Kubernetes cluster use Keycloak groups for access control. Users must be added to the appropriate group in Keycloak to gain access. These groups are configured in the `broker` realm. :::warning For the Kubernetes cluster to manage access control correctly, Keycloak must be configured as an OpenID Connect (OIDC) provider. Ensure that your Keycloak setup is properly [configured for OIDC](./configure-keycloak-oidc-eks.md) before proceeding. ::: :::info The `broker` realm keeps Kubernetes OIDC client. ::: ### Keycloak Groups Each Keycloak group corresponds to a specific level of access within the Kubernetes cluster and the KubeRocketCI portal. The table below summarizes the groups and their associated permissions. | Group Name | View | Build | Deploy | Full Namespace Access | |-------------------------------|:------------------:|:------------------:|:------------------:|:---------------------:| | `${platform}-oidc-admins` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-developers` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | | `${platform}-oidc-viewers` | :white_check_mark: | :x: | :x: | :x: | #### View Permissions In the KubeRocketCI portal, the following **View** permissions are granted to users based on their group membership: | Group Name | View Projects | View Branches | View Pipelines | View Deployments | View Environments | View Widgets | View Security | View Observability | |-------------------------------|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:| | `${platform}-oidc-admins` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-developers` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-viewers` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | #### Permissions for the Configuration sections and Kubernetes resources Permissions for **Configuration** sections in the KubeRocketCI portal: | Group Name | View QuickLinks | Create QuickLinks | Edit QuickLinks | Delete QuickLinks | View GitOps | Create GitOps | Delete GitOps | View Clusters | Create Clusters | Edit Clusters | Delete Clusters | View GitServers | Create GitServers | Edit GitServers | Delete GitServers | View Integrations | Create Integrations | Edit Integrations | Delete Integrations | |-------------------------------|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:-------------------:|:------------------:|:-------------------:| | `${platform}-oidc-admins` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-developers` | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | | `${platform}-oidc-viewers` | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | Permissions for the most common Kubernetes resources: | Group Name | View Pods | Create Pods | Edit Pods | Delete Pods | View Deployments | Create Deployments | Edit Deployments | Delete Deployments | View Ingresses | Create Ingresses | Edit Ingresses | Delete Ingresses | View Services | Create Services | Edit Services | Delete Services | View Config Maps | Create Config Maps | Edit Config Maps | Delete Config Maps | View Cron Jobs | Create Cron Jobs | Edit Cron Jobs | Delete Cron Jobs | View Secrets | Create Secrets | Edit Secrets | Delete Secrets | View Roles | Create Roles | Edit Roles | Delete Roles | View Role Bindings | Create Role Bindings | Edit Role Bindings | Delete Role Bindings | |-------------------------------|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:--------------------:|:------------------:|:--------------------:| | `${platform}-oidc-admins` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-developers` | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | | `${platform}-oidc-viewers` | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | #### Build Permissions The following **Build** permissions are granted to users based on their group membership: | Group Name | Create Project | Edit Project | Delete Project | Create Branch | Edit Branch | Delete Branch | Build Project | |-------------------------------|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:| | `${platform}-oidc-admins` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-developers` | :x: | :x: | :x: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-viewers` | :x: | :x: | :x: | :x: | :x: | :x: | :x: | #### Deploy Permissions The following **Deploy** permissions are granted to users based on their group membership: | Group Name | Create PipelineRun | Create Deployment Flow | Edit Deployment Flow | Delete Deployment Flow | Create Environment | Edit Environment | Delete Environment | Clean Environment | Deploy Environment | Approve/Reject ApprovalTasks | |-------------------------------|:---------------------:|:----------------------:|:--------------------:|:----------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:----------------------------:| | `${platform}-oidc-admins` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-developers` | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | `${platform}-oidc-viewers` | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: | ### Cluster RBAC Resources The platform defines five RoleBindings that grant the necessary permissions to the corresponding Keycloak groups mentioned above. | RoleBinding Name | Role Name | Groups | | ---------------- | ---------------- | ---------------------------------------------------------- | | tenant-admin | cluster-admin | `${platform}-oidc-admins` | | tenant-developer | tenant-developer | `${platform}-oidc-developers` | | tenant-viewer | view | `${platform}-oidc-viewers` , `${platform}-oidc-developers` | Platform includes RBAC settings for the full cluster administration privileges. | Cluster Role Binding Name | Cluster Role Name | Group | | ------------------------- | ----------------- | --------------------- | | cluster-admin | cluster-admin | `oidc-cluster-admins` | :::note KubeRocketCI provides an [aggregated](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) ClusterRole `edp-aggregate-view-${platform}` with the permissions to view the KubeRocketCI [custom resources](../../api/overview.md). ::: :::info The `tenant-admin` RoleBinding is created in a namespace by the `cd-pipeline-operator`. The `tenant-admin` RoleBinding assigns the `admin` role to the `${platform}-oidc-admins` and `${platform}-oidc-developers` groups. ::: ### Grant User Access to the Created Namespaces To provide users with admin or developer privileges for project namespaces, they need to be added to the `${platform}-oidc-admins` and `${platform}-oidc-developers` groups in Keycloak. ## Argo CD In Argo CD, groups are specified when creating an AppProject to restrict access to deployed applications. To gain access to deployed applications within a project, users must be added to the corresponding Argo CD group in Keycloak. This ensures that only authorized users can access and modify applications within the project. :::info By default, only the `ArgoCDAdmins` group is automatically created in Keycloak. ::: ## Gerrit Users should authenticate to Gerrit using their Keycloak credentials. After logging into Gerrit, users are not automatically assigned to any groups. To add a user to a group, a `GerritGroupMember` custom resource must be created. This custom resource specifies the user's email address and the name of the group to which they should be added. Below is an example of a `GerritGroupMember` custom resource: :::note Gerrit Group Member ```yaml apiVersion: v2.edp.epam.com/v1 kind: GerritGroupMember metadata: name: user-admins spec: accountId: user@user.com groupId: Administrators ``` ::: Once the `GerritGroupMember` resource is created, the user will inherit the permissions and access levels associated with the specified group. ## Related Articles * [KubeRocketCI Portal Overview](../../user-guide/index.md) * [EKS OIDC With Keycloak](configure-keycloak-oidc-eks.md) * [Argo CD Integration](../cd/argocd-integration.md) * [Audit Trails Setup](../monitoring-and-observability/audit-trails-setup.md) --- ## Portal OIDC Configuration # Portal OIDC Configuration This page provides instructions for configuring the [OIDC authorization](https://openid.net/connect/) for the [KubeRocketCI portal](../../user-guide/index.md), enabling the use of Single Sign-On (SSO) for authorization in the Portal. This configuration allows for centralized control of user access and rights from a single configuration point. ## Prerequisites Ensure the following values are set first before starting the Portal OIDC configuration: 1. `realm_id` = **shared** 2. `client_id` = **eks** 3. `keycloak_client_key`= **keycloak_client_secret_key** (received from: `Shared realm` -> `clients` -> `eks` -> `Credentials` -> `Client secret`) :::note The values indicated above are the result of the Keycloak configuration as an OIDC identity provider. To obtain these values, please refer to the guidelines on the [Keycloak OIDC EKS Configuration](configure-keycloak-oidc-eks.md) page. ::: ## Configure Keycloak To proceed with the Keycloak configuration, perform the following: 1. Add the URL of the Headlamp to the `valid_redirect_uris` variable in [Keycloak](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/kuberocketci-rbac/templates/kubernetes/keycloak-client.yaml#L17): :::note keycloak_openid_client ```yaml redirectUris: - "https://portal./*" - "http://localhost:8000/*" ``` ::: Make sure to define the following Keycloak client values as indicated: ![Keycloak client configuration](../../assets/operator-guide/headlamp-oidc-keycloak-2.png "Keycloak client configuration") 2. Configure the Keycloak client key in Kubernetes using the Kubernetes secrets or the [External Secrets Operator](../secrets-management/external-secrets-operator-integration.md): ```yaml apiVersion: v1 kind: Secret metadata: name: keycloak-client-headlamp-secret namespace: krci type: Opaque stringData: clientSecret: ``` 3. Assign user to one or more groups in Keycloak. ## Integrate Portal With Kubernetes Portal is integrated in Kubernetes in three steps: 1. Update the [values.yaml](../install-kuberocketci.md) file by enabling OIDC: :::note values.yaml ```yaml edp-headlamp: config: oidc: enabled: true ``` ::: 2. Navigate to KubeRocketCI portal and log in by clicking the `Sign In` button: ![Portal login page](../../assets/operator-guide/headlamp-oidc-headlamp-1.png "Portal login page") 3. Upon logging in, ensure the namespace for KubeRocketCI is specified by clicking the **Manage Namespaces** button: ![Portal namespace settings](../../assets/operator-guide/krci-portal-ui.png "Portal namespace settings") As a result, it is possible to control access and rights from the Keycloak endpoint. ## Configure User Profile Picture To enhance account personalization, a profile picture can be configured in the Portal using Keycloak. This requires setting up the Keycloak client scope with the necessary user attributes. Follow the steps below to configure the user profile picture in Keycloak: 1. Open the Keycloak admin console and navigate to the **Broker** realm. In the left sidebar menu, select **Client scopes** and choose the **profile** scope: ![Keycloak profile scope](../../assets/operator-guide/keycloak-user-attributes/keycloak-profile-scope.png "Keycloak profile scope") 2. Navigate to the **Mappers** tab and ensure that the **picture** attribute is created. If the attribute is not present, click **Add mapper** button. In the dropdown menu, select **By configuration**: :::note If the **picture** user attribute is already present, skip to step 5. ::: ![Keycloak create mapper](../../assets/operator-guide/keycloak-user-attributes/keycloak-create-mapper.png "Keycloak create mapper") 3. In the **Configure a new mapper** dialog, choose the **User Attribute** type: ![Keycloak user attribute](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-attribute.png "Keycloak user attribute") 4. Fill in the following fields to configure the user attribute: - **Name**: `picture` - **User Attribute**: `picture` - **Token Claim Name**: `picture` - **Claim JSON Type**: `String` ![Keycloak user attribute configuration](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-attribute-configuration.png "Keycloak user attribute configuration") Click **Save** to apply the changes. 5. After configuring the user attribute, navigate to the **Users** tab in the left sidebar menu and select the user you want to set a profile picture for: ![Keycloak user profile](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-profile.png "Keycloak user profile") 6. In the user profile, click **Attributes** and add the **picture** attribute with the URL of the desired profile picture: :::note The profile picture URL should refer to an image accessible via a public URL. ::: ![Keycloak user picture](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-picture.png "Keycloak user picture") Click **Save** to apply the changes. 7. After configuring the user profile picture in Keycloak, the profile picture will be displayed in the KubeRocketCI portal: ![Portal user profile picture](../../assets/operator-guide/keycloak-user-attributes/portal-user-profile-picture.png "Portal user profile picture") ## Configure Default Namespace After logging into the Portal, it is necessary to manually configure account settings like **Default namespace** and **Allowed namespaces**. To simplify this process, the `default_namespace` user attribute can be added to the user profile in Keycloak. This will ensure that the account settings are automatically configured during login. Follow the steps below to configure the `default_namespace` user attribute in Keycloak: 1. Open the Keycloak admin console and navigate to the **Broker** realm. In the left sidebar menu, select **Client scopes** and choose the **profile** scope: ![Keycloak profile scope](../../assets/operator-guide/keycloak-user-attributes/keycloak-profile-scope.png "Keycloak profile scope") 2. Navigate to the **Mappers** tab and click the **Add mapper** button. In the dropdown menu, select **By configuration**: ![Keycloak create mapper](../../assets/operator-guide/keycloak-user-attributes/keycloak-create-mapper.png "Keycloak create mapper") 3. In the **Configure a new mapper** dialog, choose the **User Attribute** type: ![Keycloak user attribute](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-attribute.png "Keycloak user attribute") 4. Fill in the following fields to configure the user attribute: - **Name**: `default_namespace` - **User Attribute**: `default_namespace` - **Token Claim Name**: `default_namespace` - **Claim JSON Type**: `String` ![Keycloak user attribute configuration](../../assets/operator-guide/keycloak-user-attributes/keycloak-default-namespace-attribute.png "Keycloak user attribute configuration") Click **Save** to apply the changes. 5. After configuring the user attribute, navigate to the **Users** tab in the left sidebar menu and select the user you want to set a default namespace for: ![Keycloak user profile](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-profile.png "Keycloak user profile") 6. In the user profile, click **Attributes** and add the **default_namespace** attribute with the name of the desired default namespace: ![Keycloak user default namespace](../../assets/operator-guide/keycloak-user-attributes/keycloak-user-default-namespace.png "Keycloak user default namespace") Click **Save** to apply the changes. 7. After configuring the user attribute with the default namespace in Keycloak, the account settings, such as the **Default namespace** and **Allowed namespaces**, will be automatically configured during login: ![Portal user default namespace](../../assets/operator-guide/krci-portal-ui.png "Portal user default namespace") ## Changing the Lifespan of an Access Token Change the Access Token Lifespan: go to your Keycloak and select *Openshift realm* > *Realm settings* > *Tokens* > *Access Token Lifespan* > set a new value to the field and save this change. By default, "Access Token Lifespan" value is 5 minutes: ![Access Token Lifespan](../../assets/keycloak-access-token-lifespan.png "Access Token Lifespan") ## Related Articles * [EKS OIDC With Keycloak](configure-keycloak-oidc-eks.md) * [External Secrets Operator](../secrets-management/external-secrets-operator-integration.md) --- ## Install via AWS Marketplace # Install via AWS Marketplace This documentation provides detailed instructions on how to install the KubeRocketCI via the AWS Marketplace. To initiate the installation process, navigate to our dedicated [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-u7xcz6pvwwwoa#pdp-overview) page and commence the deployment of KubeRocketCI. :::warning KubeRocketCI is aligned with industry standards for storing and managing sensitive data, ensuring optimal security. However, the use of custom solutions introduces uncertainties, thus the responsibility for the safety of your data is totally covered by the platform administrator. ::: ## Prerequisites :::note - A basic understanding of AWS services and navigation is preferred to facilitate smoother setup and deployment processes. If you are new to AWS, please refer to the [AWS Documentation](https://docs.aws.amazon.com/index.html) for detailed information on the services and their usage. - Understanding of Kubernetes: Knowledge of Kubernetes concepts and architecture is recommended for effective management and operation of clusters. ::: Please ensure that you review the [Prerequisites](prerequisites.md) page before proceeding with the deployment of the product. To perform a minimal installation, please ensure that you meet the following requirements: - You have an available AWS Elastic Kubernetes Service (EKS) cluster for deployment. For detailed instructions on creating a new cluster, please refer to the [AWS EKS Cluster Creation Guide](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html). Additionally, you can consult our [EKS Deployment Guide](deploy-aws-eks.md) for step-by-step instructions tailored to your specific needs. - You have a domain name available and associated with the ingress object in the cluster. - You have cluster administrator access. - The [Tekton](install-tekton.md) resources are deployed. - You have access to the cluster via a Service Account token. ## Deploy KubeRocketCI Platform To deploy the platform, follow the steps below: 1. Deploy Tekton resources using the following commands: ```bash kubectl create ns tekton-pipelines kubectl create ns tekton-chains kubectl create ns tekton-pipelines-resolvers kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml ``` 2. Define the mandatory parameters you would like to use for installation using the following command: ```bash kubectl create ns krci helm install edp-install \ --namespace krci ./* \ --set global.dnsWildCard=example.com ``` 3. (Optional) Provide token to sign in to the Portal. Run the following command to create Service Account with cluster admin permissions: ```bash kubectl create serviceaccount edp-admin -n krci kubectl create clusterrolebinding edp-cluster-admin --clusterrole=cluster-admin --serviceaccount=krci:edp-admin kubectl apply -f - < # Argo CD Integration KubeRocketCI uses Argo CD as a [part of the Continuous Delivery/Continuous Deployment](../../user-guide/cd-pipeline-details.md) implementation. Argo CD follows the best GitOps practices, uses Kubernetes native approach for the Deployment Management, has rich UI and required RBAC capabilities. ## Argo CD Deployment Approach in KubeRocketCI Argo CD can be installed using [two different approaches](https://argo-cd.readthedocs.io/en/stable/operator-manual/installation): - Cluster-wide scope with the cluster-admin access - Namespaced scope with the single namespace access Both approaches can be deployed with High Availability (HA) or Non High Availability (non HA) installation manifests. KubeRocketCI uses the HA deployment with the cluster-admin permissions, to minimize cluster resources consumption by sharing single Argo CD instance across multiple KubeRocketCI Tenants. Please follow [the installation instructions](../install-argocd.md) to deploy Argo CD. ## Integration See a diagram below for the details: ![edp-argocd](../../assets/operator-guide/edp-argocd.png "Argo CD Diagram") - Argo CD is deployed in a separate `argocd` namespace. - Argo CD uses a `cluster-admin` role for managing cluster-scope resources. - The `control-plane` application is created using the App of Apps approach, and its code is managed by the `control-plane` members. - The `control-plane` is used to onboard new Argo CD Tenants (Argo CD Projects - AppProject). - The `KubeRocketCI Tenant Member` manages `Argo CD Applications` using `kind: Application` in the `krciTenant` namespace. The [App Of Apps approach](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/) is used to manage the `KubeRocketCI Tenants`. Inspect the [edp-grub](https://github.com/SergK/edp-grub) repository structure that is used to provide the KubeRocketCI Tenants for the Argo CD Projects: ```bash edp-grub ├── LICENSE ├── README.md ├── apps ### All Argo CD Applications are stored here │   ├── grub-argocd.yaml # Application that provisions Argo CD Resources - Argo Projects (KubeRocketCI Tenants) │   └── grub-keycloak.yaml # Application that provisions Keycloak Resources - Argo CD Groups (KubeRocketCI Tenants) ├── apps-configs │   └── grub │   ├── argocd ### Argo CD resources definition │   │   └── edp.yaml │   └── keycloak ### Keycloak resources definition │   └── edp.yaml ├── bootstrap │   └── root.yaml ### Root application in App of Apps, which provision Applications from /apps └── examples ### Examples └── tenant └── edp-petclinic.yaml ``` The Root Application must be created under the `control-plane` scope. ## Argo CD Configuration Now that Argo CD is integrated, it is time to configure it properly. To configure Argo CD for KubeRocketCI, follow the steps below: 1. Modify the `argocd-cmd-params-cm` ConfigMap in the `argocd` namespace and add the `application.namespaces` parameter to the `data` subsection: ```bash kubectl patch configmap argocd-cmd-params-cm -n argocd --type merge -p '{"data":{"application.namespaces":"krci"}}' ``` ```yaml data: application.namespaces: krci ``` 2. Add a [credential template](https://argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/#private-repositories) for GitHub, GitLab, Bitbucket, or Gerrit integrations. The credential template must be created for each Git Server. Generate an SSH key pair and add a public key to GitLab, GitHub, or Bitbucket account. :::warning Use an additional GitHub/GitLab User to access a repository. For example: - GitHub, add a User to a repository with a "Read" role. - GitLab, add a User to a repository with a "Guest" role. ::: ```bash ssh-keygen -t ed25519 -C "email@example.com" -f argocd ``` Copy SSH private key to Argo CD namespace. ```bash KRCI_NAMESPACE="krci" VCS_HOST="" ACCOUNT_NAME="" URL="ssh://git@${VCS_HOST}:22/${ACCOUNT_NAME}" kubectl create secret generic ${KRCI_NAMESPACE} -n argocd \ --from-file=sshPrivateKey=argocd \ --from-literal=url="${URL}" kubectl label --overwrite secret ${KRCI_NAMESPACE} -n argocd "argocd.argoproj.io/secret-type=repo-creds" ``` Copy existing SSH private key for Gerrit to Argo CD namespace. ```bash KRCI_NAMESPACE="krci" GERRIT_PORT=$(kubectl get gerrit gerrit -n ${KRCI_NAMESPACE} -o jsonpath='{.spec.sshPort}') GERRIT_ARGOCD_SSH_KEY_NAME="gerrit-ciuser-sshkey" GERRIT_URL=$(echo "ssh://edp-ci@gerrit.${KRCI_NAMESPACE}:${GERRIT_PORT}" | base64) kubectl get secret ${GERRIT_ARGOCD_SSH_KEY_NAME} -n ${KRCI_NAMESPACE} -o json | jq 'del(.data.username,.metadata.annotations,.metadata.creationTimestamp,.metadata.labels,.metadata.resourceVersion,.metadata.uid,.metadata.ownerReferences)' | jq '.metadata.namespace = "argocd"' | jq --arg name "${KRCI_NAMESPACE}" '.metadata.name = $name' | jq --arg url "${GERRIT_URL}" '.data.url = $url' | jq '.data.sshPrivateKey = .data.id_rsa' | jq 'del(.data.id_rsa,.data."id_rsa.pub")' | kubectl apply -f - kubectl label --overwrite secret ${KRCI_NAMESPACE} -n argocd "argocd.argoproj.io/secret-type=repo-creds" ``` 3. Add [SSH Known hosts](https://argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/#unknown-ssh-hosts) for GitHub, GitLab, Bitbucket, or Gerrit integration. Add GitHub, GitLab, or Bitbucket host to Argo CD config map with known hosts. ```bash KRCI_NAMESPACE="krci" VCS_HOST="" KNOWN_HOSTS_FILE="/tmp/ssh_known_hosts" ARGOCD_KNOWN_HOSTS_NAME="argocd-ssh-known-hosts-cm" rm -f ${KNOWN_HOSTS_FILE} kubectl get cm ${ARGOCD_KNOWN_HOSTS_NAME} -n argocd -o jsonpath='{.data.ssh_known_hosts}' > ${KNOWN_HOSTS_FILE} ssh-keyscan ${VCS_HOST} >> ${KNOWN_HOSTS_FILE} kubectl create configmap ${ARGOCD_KNOWN_HOSTS_NAME} -n argocd --from-file ${KNOWN_HOSTS_FILE} -o yaml --dry-run=client | kubectl apply -f - ``` Add Gerrit host to Argo CD config map with known hosts ```bash KRCI_NAMESPACE="krci" KNOWN_HOSTS_FILE="/tmp/ssh_known_hosts" ARGOCD_KNOWN_HOSTS_NAME="argocd-ssh-known-hosts-cm" GERRIT_PORT=$(kubectl get gerrit gerrit -n ${KRCI_NAMESPACE} -o jsonpath='{.spec.sshPort}') rm -f ${KNOWN_HOSTS_FILE} kubectl get cm ${ARGOCD_KNOWN_HOSTS_NAME} -n argocd -o jsonpath='{.data.ssh_known_hosts}' > ${KNOWN_HOSTS_FILE} kubectl exec -it deployment/gerrit -n ${KRCI_NAMESPACE} -- ssh-keyscan -p ${GERRIT_PORT} gerrit.${KRCI_NAMESPACE} >> ${KNOWN_HOSTS_FILE} kubectl create configmap ${ARGOCD_KNOWN_HOSTS_NAME} -n argocd --from-file ${KNOWN_HOSTS_FILE} -o yaml --dry-run=client | kubectl apply -f - ``` 4. Create an Argo CD Project (Tenant), for example, with the `krci` name: ```yaml title="AppProject" apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: krci namespace: argocd # Finalizer that ensures that project is not deleted until it is not referenced by any application finalizers: - resources-finalizer.argocd.argoproj.io spec: destinations: # by default krci work with 'krci-*' namespace - namespace: 'krci-*' # allow to deploy to specific server (local in our case) name: in-cluster # Deny all cluster-scoped resources from being created, except for Namespace clusterResourceWhitelist: - group: '' kind: Namespace # Allow all namespaced-scoped resources to be created, except for ResourceQuota, LimitRange, NetworkPolicy namespaceResourceBlacklist: - group: '' kind: ResourceQuota - group: '' kind: LimitRange - group: '' kind: NetworkPolicy # we are ok to create any resources inside namespace namespaceResourceWhitelist: - group: '*' kind: '*' # enable access only for specific git server. The example below 'krci' - it is namespace where KubeRocketCI deployed sourceRepos: - ssh://git@github.com/* # enable capability to deploy objects from namespaces sourceNamespaces: - krci ``` 5. Check that your new **Repository**, **Known Hosts**, and **AppProject** are added to the Argo CD UI. 6. Generate Argo CD project token for deploy integration: ```bash URL= TOKEN=$(argocd proj role create-token krci developer -i argocd-ci -t) cat < Generate Argo CD project token via UI, follow the steps below: 1. Open your Argo CD endpoint. 2. In the Argo CD main menu, select **Settings**: ![Argo CD settings](../../assets/operator-guide/cd/argo-cd-open-settings.png "Argo CD settings") 3. On the **Settings** page, select **Projects**: ![Argo CD projects](../../assets/operator-guide/cd/argo-cd-open-projects.png "Argo CD projects") 4. On the **Projects** page, select the previously created Argo CD project: ![Select krci project](../../assets/operator-guide/cd/argo-cd-project-krci.png "Select krci project") 5. On the project details page, select the **Roles** tab: ![Roles tab](../../assets/operator-guide/cd/argo-cd-project-roles-tab.png "Roles tab") 6. On the **Roles** tab, select the **developer** role: ![Developer role](../../assets/operator-guide/cd/argo-cd-project-select-role.png "Developer role") 7. On the appeared window, scroll down to the **JWT Tokens** section, specify the required fields and click **Create**: - **Token ID**: Token name. - **Expires in**: Defines the period for which the token is considered valid. Must be in the "[0-9]+[s,m,h,d]" format. For example, "12h", "7d". ![Create JWT token](../../assets/operator-guide/cd/argo-cd-project-create-token.png "Create JWT token") 8. Confirm the token creation: ![Confirm token creation](../../assets/operator-guide/cd/argo-cd-create-token-confirmation.png "Confirm token creation") 9. Copy the data of the newly created token: ![Copy token data](../../assets/operator-guide/cd/argo-cd-project-copy-token.png "Copy token data") 10. Navigate to **KubeRocketCI portal** -> **Configuration** -> **Deployment** -> **Argo CD** and click **+ Add integration**: ![Argo CD configuration page](../../assets/operator-guide/cd/argo-cd-integration-portal.png "Argo CD configuration page") 11. In the integration window, specify the required fields and click **Save**: - **Quick Link URL**: Enter the URL of your Argo CD instance (e.g., https://argocd.example.com). A quick link will be added to the Overview section for quick access to Argo CD from the KubeRocketCI portal. - **URL**: Enter the URL of your Argo CD instance (e.g., https://argocd.example.com). - **Token**: Paste the JWT token data copied earlier. ![Specify required fields](../../assets/operator-guide/cd/argo-cd-project-integration-portal.png "Specify required fields") 12. Verify that the integration status is green: ![Verify integration status](../../assets/operator-guide/cd/argo-cd-project-token-added.png "Verify integration status") Once Argo CD is successfully integrated, KubeRocketCI user can utilize Argo CD to deploy [CD pipelines](../../user-guide/add-cd-pipeline.md). ## Check Argo CD Integration (Optional) This section provides the information on how to test the integration with Argo CD and is not mandatory to be followed. 1. Add an Argo CD application:
View: argocd-values.yaml ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: demo spec: project: krci destination: namespace: krci-demo server: https://kubernetes.default.svc source: helm: parameters: - name: image.tag value: master-0.1.0-1 - name: image.repository value: image-repo path: deploy-templates # github/gitlab example ssh://git@github.com//.git # gerrit example ssh://@gerrit.krci:30007/.git repoURL: ssh://git@github.com/krci/demo.git targetRevision: master syncPolicy: syncOptions: - CreateNamespace=true automated: selfHeal: true prune: true ```
2. Check that your new Application is added to the Argo CD UI under the `krci` Project scope. ## Deploy Argo CD Application to Remote Cluster (Optional) KubeRocketCI also supports deploying Argo CD applications to a remote cluster. To deploy applications to remote clusters, follow the steps below: 1. Create `ServiceAccount` `ClusterRoleBinding` and `Secret` for that `ServiceAccount`. 2. Receive the bearer token: ```bash BEAR_TOKEN=$(kubectl get secret -o jsonpath='{.data.token}' | base64 --decode) ``` 3. Create ArgoCD secret for remote cluster in the ArgoCD namespace (Argo CD only discovers cluster secrets in its own control-plane namespace): ```yaml title="manifest" apiVersion: v1 kind: Secret metadata: name: krci-remote-cluster namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: # Remote cluster name. Must exactly match the cluster name used in the # Environment (Stage) and in the 'available_clusters' parameter of the # krci-config ConfigMap: the generated Argo CD Applications reference # the destination cluster by this name. name: "krci-remote-cluster" # Cluster endpoint URL server: "https://xxxxxxxxxxxxxxxxxxxx.sk1.eu-central-1.eks.amazonaws.com" # Remote cluster config config: | { "bearerToken": "", "tlsClientConfig": { "insecure": false, "caData": "" } } ``` 4. Update an Argo CD Project (KubeRocketCI Tenant), with the `krci` name. Without this step, deployments to the remote cluster fail with `InvalidSpecError: application destination server 'krci-remote-cluster' and namespace 'krci-...' do not match any of the allowed destinations`: ```yaml title="AppProject" apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: krci spec: destinations: # Add block that allow deploy in remote cluster # by default, krci works with 'krci-*' namespaces - namespace: 'krci-*' # allow to deploy to specific server (remote in our case) name: krci-remote-cluster ``` 5. Add a remote cluster in the KubeRocketCI portal. Please refer to the [Add Cluster](../../user-guide/add-cluster.md) page for details. :::note When a cluster is added through the KubeRocketCI portal with the Bearer credentials type, the cd-pipeline-operator generates an equivalent Argo CD cluster secret in the platform namespace after the connectivity check succeeds. Its name is the cluster Secret name with the `-argocd-cluster` suffix appended: a cluster added as `dev2` is stored as Secret `dev2-cluster` and produces `dev2-cluster-argocd-cluster`. You can copy that secret into the ArgoCD namespace instead of composing the manifest above manually. ::: ## Keycloak Integration (Optional) :::note To proceed with the steps below, you need the [edp-keycloak-operator](https://github.com/epam/edp-keycloak-operator) to be deployed. ::: To provide Argo CD with the Keycloak SSO authorization mechanism, follow the guidelines below: 1. Create secret `keycloak-client-argocd-secret`. ```bash kubectl create secret generic keycloak-client-argocd-secret \ --from-literal=clientSecret=$(openssl rand -base64 32) \ --namespace=argocd ``` 2. Update the `argocd-cm` ConfigMap: ```bash kubectl patch configmap argocd-cm -n argocd --patch "$(cat </auth/realms/edp clientID: argocd-tenant clientSecret: $keycloak-client-argocd-secret:clientSecret requestedScopes: - openid - profile - email - groups EOF )" ``` ```yaml data: oidc.config: url: "https://argocd.<.Values.global.dnsWildCard>" application.instanceLabelKey: argocd.argoproj.io/instance-edp oidc.config: | name: Keycloak issuer: https://<.Values.global.keycloakEndpoint>/auth/realms/edp clientID: argocd-tenant clientSecret: $keycloak-client-argocd-secret:clientSecret requestedScopes: - openid - profile - email - groups ``` 3. Create a Keycloak Group: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakRealmGroup metadata: name: argocd-krci-users spec: name: ArgoCD-krci-users realm: main ``` 4. Create a Keycloak Client: ```yaml apiVersion: v1.edp.epam.com/v1 kind: KeycloakClient metadata: name: argocd namespace: argocd spec: advancedProtocolMappers: true attributes: post.logout.redirect.uris: + clientId: argocd-tenant defaultClientScopes: - groups realmRef: kind: ClusterKeycloakRealm name: main secret: keycloak-client-argocd-secret webUrl: "https://argocd.<.Values.global.dnsWildCard>" ``` 5. In Keycloak, add users to the `ArgoCD-krci-users` Keycloak Group. 6. Update spec in project: ```yaml title="AppProject" spec: description: CD pipelines for krci roles: - name: developer description: Users for krci tenant policies: - p, proj:krci:developer, applications, create, krci/*, allow - p, proj:krci:developer, applications, delete, krci/*, allow - p, proj:krci:developer, applications, get, krci/*, allow - p, proj:krci:developer, applications, override, krci/*, allow - p, proj:krci:developer, applications, sync, krci/*, allow - p, proj:krci:developer, applications, update, krci/*, allow - p, proj:krci:developer, repositories, create, krci/*, allow - p, proj:krci:developer, repositories, delete, krci/*, allow - p, proj:krci:developer, repositories, update, krci/*, allow - p, proj:krci:developer, repositories, get, krci/*, allow groups: # Keycloak Group name - ArgoCD-krci-users ``` 7. Then restart the deployment: ```bash kubectl -n argocd rollout restart deployment argo-argocd-server ``` ## Related Articles * [Install Argo CD](../install-argocd.md) --- ## Customize Deploy Pipeline # Customize Deploy Pipeline When deploying applications into environments, it's important to automate both `pre-deployment` and `post-deployment` steps. `Pre-deployment` procedures include essential tasks such as deploying databases, configuring specific software, and preparing the environment. Additionally, `post-deployment` procedures, such as testing, configuring, and removing old information from the environment, are crucial for ensuring the smooth operation of the deployed application. To facilitate these processes, the custom deployment feature is implemented in KubeRocketCI. This page provides comprehensive guidelines on how to adjust the deployment logic to cater your needs. ## Deploy Custom Pipeline Overall, the custom pipeline creation involves the following steps: ```mermaid graph LR; A(Create TriggerTemplate resource) --> B(Create Pipeline resource) --> C(Deploy custom environment) ``` 1. **Create TriggerTemplate resource** - On this step, create the `TriggerTemplate` custom resource that will appear as an option in the in the environment stage creation menu. 2. **Create Pipeline** - On this step, create custom resource called `Pipeline` that complements the trigger template. This resource contains all the tasks to perform within the custom pipeline. 3. **Integration** - On this step, you simply select your custom pipeline logic when creating a stage for your environment. To customize your deployment pipeline, follow the steps below: 1. Create the `TriggerTemplate` custom resource by adding the following label: ```bash labels: app.edp.epam.com/pipelinetype: deploy ``` :::note Please refer to the `TriggerTemplate` [example](https://github.com/epam/edp-tekton/blob/release/0.12/charts/pipelines-library/templates/triggers/cd/deploy.yaml) for more details. Remember to set your pipeline name in the `spec.resourcetemplates.spec.pipelineRef.name` [parameter](https://github.com/epam/edp-tekton/blob/release/0.12/charts/pipelines-library/templates/triggers/cd/deploy.yaml#L35). ::: 2. Create the custom pipeline with your custom logic. Refer to the custom pipeline [example](https://github.com/epam/edp-tekton/blob/release/0.12/charts/pipelines-library/templates/triggers/cd/deploy.yaml) for more details. 3. Apply the created manifest files in the platform namespace (e.g., `krci`). 4. On the `Pipeline Configuration` step on the KubeRocketCI portal, select the added trigger template in the `Deploy Pipeline Template` section: ![Select trigger template](../../assets/operator-guide/select_trigger_template.png "Select trigger template") 5. (Optional) In case you need to implement custom deployment in a remote cluster, do the following: - Connect the KubeRocketCI platform with the remote cluster if it is not integrated yet. Please refer to the [Add Cluster](../../user-guide/add-cluster.md) page for more details - Mount the secret to the `run-quality-gate` resource by changing the [volumes](https://github.com/epam/edp-tekton/blob/release/0.16/charts/pipelines-library/templates/tasks/cd/run-quality-gate.yaml#L9) and [volumeMounts](https://github.com/epam/edp-tekton/blob/release/0.16/charts/pipelines-library/templates/tasks/cd/run-quality-gate.yaml#L40) sections - Switch the [context](https://github.com/epam/edp-tekton/blob/release/0.16/charts/pipelines-library/templates/tasks/cd/run-quality-gate.yaml#L33) by specifying the appropriate kube config file of the `run-quality-gate` resource - On the `Create stage` step on the KubeRocketCI portal, select the appropriate cluster in the `cluster` section: ![Select cluster](../../assets/user-guide/select-cluster.png "Select cluster") ## Related Articles * [Add Cluster](../../user-guide/add-cluster.md) * [Deploy Application With Custom Build Tool/Framework](../../use-cases/tekton-custom-pipelines.md) * [Add Environment](../../user-guide/add-cd-pipeline.md) * [Customize Environment Cleanup](./customize-environment-deletion.md) --- ## Customize Environment Cleanup # Customize Environment Cleanup KubeRocketCI provides capabilities to fully operate the CI/CD flow, starting from building applications to deploying its new release. The operation flow is designed to be seamless for most of the users. However, there may be cases where more specialized actions are necessary. These might include custom procedures for managing the deletion of application resources. ## Create Custom Clean Pipeline Delete pipeline is a typical Tekton pipeline but with its own instructions. The custom delete pipeline creation involves the following steps: ```mermaid graph LR; A(Create TriggerTemplate resource) --> B(Create Pipeline resource) --> C(Create Task resource) --> D(Apply changes to environment) ``` Clean pipeline consists of **Trigger Template**, **Pipeline**, and **Task** custom resources. To create a custom clean pipeline, follow the steps below: 1. Create and apply the **TriggerTemplate** custom resource. Refer to the example [TriggerTemplate](https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/templates/triggers/cd/clean.yaml). Ensure to set the appropriate pipeline type in the labels: ```bash ... labels: app.edp.epam.com/pipelinetype: clean ... ``` ```bash ... spec: taskRunTemplate: serviceAccountName: tekton pipelineRef: name: clean-custom ... ``` 2. Create and apply the **Pipeline** custom resource. Refer to the example [Pipeline](https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/templates/pipelines/cd/clean.yaml). Additionally, set the concrete task to run: ```bash tasks: - name: clean taskRef: kind: Task name: clean-custom ``` 3. Create and apply the **Task** custom resource. Refer to the example [Task](https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/templates/tasks/cd/clean.yaml). ## Enable Clean Pipeline As soon as all the three custom resources are correctly configured and applied, you can select the newly created pipeline in the KubeRocketCI portal. ### New Environments To apply the newly created clean pipeline to your environments, follow the steps below: 1. In the deployment flow details page, click the **Create environment** button: ![Create environment button"](../../assets/operator-guide/create-environment-button.png "Create environment button") 2. In the **Create environment** window, select the custom clean pipeline: ![Select custom clean pipeline](../../assets/operator-guide/custom-clean-pipeline1.png "Select custom clean pipeline") ### Existing Environments If you need to change the delete pipeline in an already existing environment, follow the steps below: 1. Navigate to the environment details page and click the **Edit** button: ![Edit environment button](../../assets/operator-guide/edit-environment-button.png "Edit environment button") 2. In the **Edit environment** window, select the custom clean pipeline: ![Select custom clean pipeline](../../assets/operator-guide/custom-clean-pipeline2.png "Select custom clean pipeline") As soon as the clean pipeline is created and selected within an environment, you can trigger it by clicking the **Clean** button. ## Related Articles * [Customize Deploy Pipeline](../../operator-guide/cd/customize-deploy-pipeline.md) * [Add Deployment Flow](../../user-guide/add-cd-pipeline.md) * [Manage Deployment Flows](../../user-guide/manage-environments.md) --- ## Deploy Application In Remote Cluster via IRSA # Deploy Application In Remote Cluster via IRSA KubeRocketCI enables secure deployment of applications to remote AWS EKS clusters using IAM Roles for Service Accounts (IRSA). This guide explains how to configure cross-account access between Kubernetes clusters, allowing applications to be deployed from one AWS account to another without storing long-term credentials. IRSA provides temporary, scoped credentials by allowing Kubernetes service accounts to assume specific IAM roles with precisely defined permissions, enhancing both security and operational efficiency when working with multi-account AWS environments. ## Prerequisites Before implementing IRSA for cross-account deployments, the following requirements must be met: - Configured two [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/getting-started.html): Account A (where KubeRocketCI is deployed) and Account B (the target account for application deployment) - Administrative access to both AWS accounts - Running [EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) in Account A with configured [OIDC Identity Provider](../auth/configure-keycloak-oidc-eks.md) authentication - Running [EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) in Account B as the target cluster for application deployment - Installed and configured [Argo CD](../install-argocd.md). ## Architecture and Working Principles KubeRocketCI uses IRSA to establish secure connections between AWS accounts without storing long-term credentials. The architecture follows these key principles: 1. **Service Account Federation** - Kubernetes service accounts in Account A are federated with AWS IAM through OIDC 2. **Role Chaining** - IAM roles in Account A assume corresponding roles in Account B 3. **Temporary Credentials** - All access is provided through short-lived, automatically rotated tokens The cross-account deployment process involves two main components: **CD Pipeline Operator Flow:** - The [cd-pipeline-operator](https://github.com/epam/edp-cd-pipeline-operator) in Account A uses a service account linked to the `AWSIRSA_{cluster_name}_CDPipelineOperator` IAM role - When new environments or clusters are configured in KubeRocketCI, the `AWSIRSA_{cluster_name}_CDPipelineOperator` role in Account A assumes the `AWSIRSA_{cluster_name}_CDPipelineAgent` role in Account B - The agent role creates the necessary Kubernetes resources (namespaces, service accounts, secrets) in the target cluster **Argo CD Flow:** - Argo CD controllers in Account A use service accounts linked to the `AWSIRSA_{cluster_name}_ArgoCDMaster` IAM role - During deployment, this role assumes the `AWSIRSA_{cluster_name}_ArgoCDAgentAccess` role in Account B - The agent role accesses the target EKS cluster to deploy and manage applications The diagram below illustrates the IRSA access model for cross-account deployments: ```mermaid flowchart TD subgraph subGraph0["AWS Account A"] SA["cd-pipeline-operator"] ROLE_A["IAM Role CDPipelineOperator"] end subgraph subGraph1["AWS Account B"] ROLE_B["IAM Role CDPipelineAgent"] RESOURCES["Kubernetes resources"] end subgraph subGraph2["CD Pipeline Operator Flow"] subGraph0 subGraph1 end subgraph subGraph3["AWS Account A"] ARGOCD["Argo CD Controllers"] ROLE_MASTER["IAM Role ArgoCDMaster"] end subgraph subGraph4["AWS Account B"] ROLE_AGENT["IAM Role ArgoCDAgentAccess"] APP_RESOURCES["Applications & Resources"] end subgraph subGraph5["Argo CD Flow"] subGraph3 subGraph4 end SA -- IRSA --> ROLE_A ROLE_A -- AssumeRole --> ROLE_B ROLE_B -- Create/Configure --> RESOURCES ARGOCD -- IRSA --> ROLE_MASTER ROLE_MASTER -- AssumeRole --> ROLE_AGENT ROLE_AGENT -- Deploy/Manage --> APP_RESOURCES ``` ## Create IAM Roles To enable cross-account deployments using IRSA, it is necessary to create specific IAM roles in both AWS accounts. There are two approaches to create the required IAM roles: - **Using the AWS Management Console**: This approach is based on creating roles through the AWS web-based interface. - **Using Terraform & AWS Management Console**: This hybrid approach is based on using the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform) repository to create the required IAM roles in Account A, while using the AWS Management Console to create the roles in Account B. ### Using AWS Management Console To create the required IAM roles in Account A using the AWS Management Console, follow the steps below: 1. Log in to the AWS Management Console for Account A. 2. Navigate to the **IAM** service and select **Roles**. 3. Create the `AWSIRSA_{cluster_name}_CDPipelineOperator` IAM role with the following settings:
Trust Policy ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks..amazonaws.com/id/:sub": [ "system:serviceaccount:krci:edp-cd-pipeline-operator" ], "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com" } } } ] } ```
Permissions Policy ```json { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Resource": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineAgent" } ] } ```
4. Create the `AWSIRSA_{cluster_name}_ArgoCDMaster` IAM role with the following settings:
Trust Policy ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringLike": { "oidc.eks..amazonaws.com/id/:sub": [ "system:serviceaccount:argocd:argocd-application-controller", "system:serviceaccount:argocd:argocd-applicationset-controller", "system:serviceaccount:argocd:argocd-server" ], "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com" } } } ] } ```
Permissions Policy ```json { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Resource": [ "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDAgentAccess" ] } ] } ```
To create the required IAM roles in Account B using the AWS Management Console, follow the steps below: 1. Log in to the AWS Management Console for Account B. 2. Navigate to the **IAM** service and select **Roles**. 3. Create the `AWSIRSA_{cluster_name}_CDPipelineAgent` IAM role with the following settings:
Trust Policy ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineOperator" }, "Action": "sts:AssumeRole" } ] } ```
4. Create the `AWSIRSA_{cluster_name}_ArgoCDAgentAccess` IAM role with the following settings:
Trust Policy ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" }, "Action": "sts:AssumeRole" } ] } ```
### Using Terraform and AWS Management Console In case of using the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform) repository for creating and managing AWS EKS clusters, the required IAM roles in Account A can be created using Terraform, while the roles in Account B can be created using the AWS Management Console. To create the required IAM roles in Account A using Terraform, follow the steps below: 1. Clone the forked [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform) repository. 2. Navigate to the `eks` directory. 3. Update the `terraform.tfvars` configuration file by specifying the following variables: ```hcl title="terraform.tfvars" create_cd_pipeline_operator_irsa = true create_argocd_irsa = true cd_pipeline_operator_agent_role_arn = "arn:aws:iam:::role/AWSIRSA__CDPipelineAgent" argocd_agent_role_arn = "arn:aws:iam:::role/AWSIRSA__ArgoCDAgentAccess" ``` 4. Run the following commands to create the IAM roles: :::note It is highly recommended to run the `terraform plan` command before applying the changes to check the resources that will be created or modified. ::: ```bash terraform init terraform apply -var-file=./template.tfvars ``` To create the required IAM roles in Account B using the AWS Management Console, follow the steps below: 1. Log in to the AWS Management Console for Account B. 2. Navigate to the **IAM** service and select **Roles**. 3. Create the `AWSIRSA_{cluster_name}_CDPipelineAgent` IAM role with the following settings:
Trust Policy ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineOperator" }, "Action": "sts:AssumeRole" } ] } ```
4. Create the `AWSIRSA_{cluster_name}_ArgoCDAgentAccess` IAM role with the following settings:
Trust Policy ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" }, "Action": "sts:AssumeRole" } ] } ```
## Annotate Service Accounts To enable the cross-account access for the cd-pipeline-operator and Argo CD, it is necessary to annotate the Kubernetes service accounts in the AWS Account A with the corresponding IAM role ARNs. This allows the service accounts to assume the roles and access resources in the target AWS Account B. ### CD Pipeline Operator Service Account On the AWS Account A, where the KubeRocketCI is deployed, annotate the `edp-cd-pipeline-operator` service account with the `AWSIRSA_{cluster_name}_CDPipelineOperator` IAM role ARN. There are several ways to annotate the service account: Update the `cd-pipeline-operator` configuration in [edp-install](https://github.com/epam/edp-install/blob/master/deploy-templates/values.yaml) repository to automatically add annotations to service account: ```yaml title="deploy-templates/values.yaml" cd-pipeline-operator: serviceAccount: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineOperator" ``` ```bash title="ServiceAccount: edp-cd-pipeline-operator" kubectl patch serviceaccount edp-cd-pipeline-operator -n krci \ -p '{"metadata": {"annotations": {"eks.amazonaws.com/role-arn": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineOperator"}}}' ``` After applying annotations to service accounts, it is necessary to restart the corresponding deployments to ensure new pods are created with the updated IAM roles configuration. Use the following command: ```bash kubectl rollout restart deployment cd-pipeline-operator -n krci ``` ```yaml title="ServiceAccount: edp-cd-pipeline-operator" apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineOperator" name: edp-cd-pipeline-operator namespace: krci ``` After applying annotations to service accounts, it is necessary to restart the corresponding deployments to ensure new pods are created with the updated IAM roles configuration. Use the following command: ```bash kubectl rollout restart deployment cd-pipeline-operator -n krci ``` ### Argo CD Service Accounts On the AWS Account A, where the Argo CD is deployed, annotate the `argocd-application-controller`, `argocd-applicationset-controller`, and `argocd-server` service accounts with the `AWSIRSA_{cluster_name}_ArgoCDMaster` IAM role ARN. There are several ways to annotate the service accounts: Update the Argo CD configuration in [Add-Ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/argo-cd/values.yaml) repository to automatically add annotations to service accounts: ```yaml title="clusters/core/addons/argo-cd/values.yaml" argo-cd: controller: serviceAccount: # -- Annotations applied to created service account annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" server: serviceAccount: # -- Annotations applied to created service account annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" applicationSet: serviceAccount: # -- Annotations applied to created service account annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" ``` ```bash title="ServiceAccount: argocd-application-controller" kubectl patch serviceaccount argocd-application-controller -n argocd \ -p '{"metadata": {"annotations": {"eks.amazonaws.com/role-arn": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster"}}}' ``` ```bash title="ServiceAccount: argocd-applicationset-controller" kubectl patch serviceaccount argocd-applicationset-controller -n argocd \ -p '{"metadata": {"annotations": {"eks.amazonaws.com/role-arn": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster"}}}' ``` ```bash title="ServiceAccount: argocd-server" kubectl patch serviceaccount argocd-server -n argocd \ -p '{"metadata": {"annotations": {"eks.amazonaws.com/role-arn": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster"}}}' ``` After applying annotations to service accounts, it is necessary to restart the corresponding workloads to ensure new pods are created with the updated IAM roles configuration. To do this, use the following commands: ```bash kubectl delete pod -l app.kubernetes.io/name=argocd-application-controller -n argocd kubectl delete pod -l app.kubernetes.io/name=argocd-applicationset-controller -n argocd kubectl delete pod -l app.kubernetes.io/name=argocd-server -n argocd ``` ```yaml title="ServiceAccount: argocd service accounts" apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" name: argocd-application-controller namespace: argocd --- apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" name: argocd-applicationset-controller namespace: argocd --- apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" name: argocd-server namespace: argocd ``` After applying annotations to service accounts, it is necessary to restart the corresponding workloads to ensure new pods are created with the updated IAM roles configuration. To do this, use the following commands: ```bash kubectl delete pod -l app.kubernetes.io/name=argocd-application-controller -n argocd kubectl delete pod -l app.kubernetes.io/name=argocd-applicationset-controller -n argocd kubectl delete pod -l app.kubernetes.io/name=argocd-server -n argocd ``` ## Update Argo CD AppProject :::note API server endpoint can be found in the EKS cluster details in the AWS Management Console. ::: To allow Argo CD to deploy applications in the target cluster in Account B, it is necessary to update the **AppProject** configuration to include the target cluster as a destination. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: krci namespace: argocd spec: destinations: - namespace: krci-* server: https://.gr7..eks.amazonaws.com ``` ## Configure IAM Roles Access to EKS Cluster To allow the IAM roles created in Account B to access the EKS cluster, it is necessary to configure the access permissions for the roles in the cluster. This can be done by creating the [Access Entry](https://docs.aws.amazon.com/eks/latest/userguide/eks-access-entries.html) objects for the `AWSIRSA_{cluster_name}_CDPipelineAgent` and `AWSIRSA_{cluster_name}_ArgoCDAgentAccess` IAM roles. To configure the access permissions for the IAM roles for the EKS cluster in Account B, follow the steps below: ### CD Pipeline Operator Access Entry To create the Access Entry for the `AWSIRSA_{cluster_name}_CDPipelineAgent` IAM role, follow the steps below: 1. Log in to the AWS Management Console for Account B. 2. Navigate to the **EKS** service and select the target cluster. 3. In the **Access** tab, click on **Create access entry**. ![Create Access Entry](../../assets/operator-guide/deploy-application-in-remote-cluster-via-irsa/create-access-entry.png) 4. On the **Configure IAM access entry** page, specify the following settings: - **IAM principal ARN**: `arn:aws:iam:::role/AWSIRSA_{cluster_name}_CDPipelineAgent` - **Type**: `Standard` - **Kubernetes groups**: `cd-pipeline-operator` (required if Capsule Tenant is used) ![Configure IAM Access Entry](../../assets/operator-guide/deploy-application-in-remote-cluster-via-irsa/configure-iam-access-entry.png) Click **Next** to proceed. 5. On the **Add access policy** page, select the **AmazonEKSClusterAdminPolicy** policy and click **Add policy** button. :::note If Capsule Tenant is used, it is not necessary to add the `AmazonEKSClusterAdminPolicy` policy, as the access will be managed by Capsule. ::: ![Add Access Policy](../../assets/operator-guide/deploy-application-in-remote-cluster-via-irsa/add-access-policy.png) Click **Next** to proceed. 6. On the **Review and create** page, review the settings and click **Create access entry** to create the Access Entry. ### Argo CD Access Entry To create the Access Entry for the `AWSIRSA_{cluster_name}_ArgoCDAgentAccess` IAM role, follow the steps below: 1. Log in to the AWS Management Console for Account B. 2. Navigate to the **EKS** service and select the target cluster. 3. In the **Access** tab, click on **Create access entry**. ![Create Access Entry](../../assets/operator-guide/deploy-application-in-remote-cluster-via-irsa/create-access-entry.png) 4. On the **Configure IAM access entry** page, specify the following settings: - **IAM principal ARN**: `arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDAgentAccess` - **Type**: `Standard` ![Configure IAM Access Entry](../../assets/operator-guide/deploy-application-in-remote-cluster-via-irsa/configure-iam-argo-cd-access-entry.png) Click **Next** to proceed. 5. On the **Add access policy** page, select the **AmazonEKSClusterAdminPolicy** policy and click **Add policy** button. ![Add Access Policy](../../assets/operator-guide/deploy-application-in-remote-cluster-via-irsa/add-access-policy.png) Click **Next** to proceed. 6. On the **Review and create** page, review the settings and click **Create access entry** to create the Access Entry. ## Configure Capsule Tenant (Optional) If [Capsule](https://projectcapsule.dev/) is used for multi-tenancy in the EKS cluster in Account B, it is necessary to configure the Capsule Tenant for the `cd-pipeline-operator` group. This allows the `AWSIRSA_{cluster_name}_ArgoCDAgentAccess` IAM role to have the necessary permissions to manage resources within the Capsule Tenant. To configure the Capsule Tenant for the `cd-pipeline-operator` group, align the following Capsule resources in the EKS cluster in Account B:
Capsule Configuration ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: CapsuleConfiguration metadata: name: default spec: userGroups: - capsule.clastix.io - cd-pipeline-operator ```
Capsule Tenant ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: krci spec: owners: - clusterRoles: - admin - capsule-namespace-deleter kind: Group name: cd-pipeline-operator ```
## Next Steps After completing the IRSA configuration on both AWS accounts, it is possible to deploy applications to the remote EKS cluster in Account B using KubeRocketCI. To proceed with the adding the remote cluster and deploying applications, refer to the [Add Cluster](../../user-guide/add-cluster.md) guide. ## Related Articles * [Argo CD Integration](argocd-integration.md) * [Add Cluster](../../user-guide/add-cluster.md) * [Add Application](../../user-guide/add-application.md) * [Add CD Pipeline](../../user-guide/add-cd-pipeline.md) --- ## Deploy Application In Remote Cluster via Token # Deploy Application In Remote Cluster via Token This section enables the creation of the necessary resources to configure access to a remote cluster using a static token. ## Using Kubernetes RBAC To configure application deployment in remote cluster using Kubernetes RBAC, follow the steps below: 1. Create a ServiceAccount: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: krci-deployer namespace: ``` 2. Bind the ServiceAccount to the cluster-admin role using ClusterRoleBinding: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: krci-deployer-cluster-admin-binding subjects: - kind: ServiceAccount name: krci-deployer namespace: roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io ``` 3. Create a Secret with a token for the ServiceAccount: ```yaml apiVersion: v1 kind: Secret metadata: name: krci-deployer namespace: annotations: kubernetes.io/service-account.name: krci-deployer type: kubernetes.io/service-account-token ``` ## Using Capsule To configure application deployment in remote cluster using the Capsule tool, follow the steps below: 1. Create a ServiceAccount for deployment: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: krci-deployer namespace: ``` 2. Create a Secret with a token for the ServiceAccount: ```yaml apiVersion: v1 kind: Secret metadata: name: krci-deployer namespace: annotations: kubernetes.io/service-account.name: krci-deployer type: kubernetes.io/service-account-token ``` 3. Configure Capsule to recognize the ServiceAccount’s group: ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: CapsuleConfiguration metadata: name: default spec: userGroups: - capsule.clastix.io - system:serviceaccounts: ``` 4. Create a Tenant and assign ownership to the ServiceAccount through Capsule: ```yaml apiVersion: capsule.clastix.io/v1beta2 kind: Tenant metadata: name: krci-deployer spec: owners: - clusterRoles: - admin - capsule-namespace-deleter kind: ServiceAccount name: system:serviceaccount::krci-deployer ``` ## Get Kubernetes Token To obtain the token generated by following this documentation, run the command below: ```bash kubectl get secret -n krci-deployer -o jsonpath='{.data.token}' | base64 -d ``` ## Next Steps By completing all the steps in this documentation, you will obtain a token for integration with the Kubernetes cluster. The next step is to configure the cluster integration with KubeRocketCI, as described in the documentation [Add Cluster](../../user-guide/add-cluster.md). ## Related Articles * [Argo CD Integration](argocd-integration.md) * [Add Cluster](../../user-guide/add-cluster.md) * [Add Application](../../user-guide/add-application.md) * [Add CD Pipeline](../../user-guide/add-cd-pipeline.md) --- ## Deploy RPM Packages # Deploy RPM Packages KubeRocketCI can use two types of deployment packages: Helm chart and RPM packages. While the Helm chart is the default deployment package in KubeRocketCI, using RPM packages is beneficial for specific Linux distributions, such as Oracle, Fedora, openSUSE, and others where these packages are supported and widespread. To learn more about RPM packages, refer to the official [documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/rpm_packaging_guide/getting-started-with-rpm-packaging#introduction-to-rpm-packaging_getting-started-with-rpm-packaging). In KubeRocketCI, RPM support allows to collect applications, store them as Nexus artifacts, and deploy them using the [Ansible](https://docs.ansible.com/ansible/latest/index.html) tool. KubeRocketCI is support two approach to work with RPM packages: - **Default approach**: This approach involves configuring deployments using a GitOps repository and Kubernetes secrets. - **AWX approach**: This approach involves deploying applications using the [AWX](https://ansible.readthedocs.io/projects/awx/en/latest/) tool. ### Features of Each Approach #### Built-in Pipeline - **Full GitOps Configuration**: All resources, including roles, inventories, hosts, tasks, and instance keys, are fully managed via GitOps. - **Quick Start**: Enables a fast setup process without the need for third-party utilities. - **Scalable**: Easily scales to accommodate an increasing number of tasks. - **Customizable Task Execution**: Offers flexible configuration of task execution sequences. - **No UI**: Operates entirely without a graphical user interface, relying on code-based configurations. #### AWX Operator - **User-Friendly Interface**: Features a convenient UI for managing configurations and tasks. - **Partial Code-Based Configurations**: Retrieves some configurations (such as Ansible tasks) directly from the repository. - **UI-Based Configuration**: All configurations are performed within the graphical user interface. ## Default Approach The first method to delivery RPM packages is using KubeRocketCI RPM deploy pipeline. ### Integration Flow Default approach involves the following steps: ```mermaid graph LR; A(Onboard GitOps repository) --> B(Create secret for GitOps repository) --> C(Create secret for Ansible host's) --> D(Create application) --> E(Select deploy pipeline) ``` Here is a breakdown of the scheme above: 1. **Onboard GitOps repository**: GitOps repository is used to store pre-deploy information (such as dependencies) and Ansible configuration. 2. **Create secret for GitOps repository**: A secret providing access to the GitOps repository. 3. **Create secret for Ansible host's**: This is a secret used by Ansible to manage the instances where the application will be deployed. 4. **Create application**: [Add application](../../user-guide/add-application.md) to KubeRocketCI as an RPM package. 5. **Select deploy pipeline**: Choose a specific deploy pipeline when creating/editing an environment to deploy RPM packages. By completing these steps, will be able to add, build, and deploy RPM applications. To set up RPM integration, follow the steps below: 1. Open the KubeRocketCI portal. Navigate to **Configuration** -> **Deployment** -> **GitOps**. 2. Specify the GitOps repository in the `username/repository_name` format. 3. Create and apply the secret for the GitOps repository by running the command below. Specify SSH key path, Git account, and repository name: ```bash kubectl apply -f - < data: id_rsa: $(cat /path/to/repo_id_rsa | base64 | tr -d '\n') url: $(echo -n "git@github.com:edp-robot/ansible-gitops.git" | base64 | tr -d '\n') type: Opaque EOF ``` 4. Create and apply the secret for the hosts by running the command below. Don't forget to specify SSH key path: ```bash kubectl apply -f - < data: id_rsa: $(cat /path/to/instance_id_rsa | base64 | tr -d '\n') type: Opaque EOF ``` 5. When [creating codebases](../../user-guide/add-application.md), in the **Deployment option** field, select the **rpm-package** option: ![Create RPM application](../../assets/operator-guide/rpm/rpm.png "Create RPM application") 6. When [creating](../../user-guide/manage-environments.md#add-a-new-environment) or [editing](../../user-guide/manage-environments.md#edit-environment) environments, in the **Deploy pipeline template** field, select **deploy-ansible**: :::note The **pre-deploy.yml** file will take affect only if the **Values override** option is enabled when deploying application. ::: ![Create Ansible environment](../../assets/operator-guide/rpm/ansible-deploy-ansible.png "Create Ansible environment") ### GitOps Repository Structure Below is a structure of a GitOps repository designed for deploying RPM packages:
View: repository structure ```bash nexus.repo inventory.ini pre-deploy.yml ├── web │ ├── dev │ │ ├── nano │ │ │ ├── 01_playbook.yml │ │ │ └── 02_playbook.yml │ │ └── atop │ │ ├── 01_playbook.yml │ │ └── 02_playbook.yml │ ├── qa │ │ ├── nano │ │ │ ├── 01_playbook.yml │ │ │ └── 02_playbook.yml │ │ └── atop │ │ ├── 01_playbook.yml │ │ └── 02_playbook.yml │ └── prod │ ├── nano │ │ ├── 01_playbook.yml │ │ └── 02_playbook.yml │ └── atop │ ├── 01_playbook.yml │ └── 02_playbook.yml └── db ├── dev │ ├── nano │ │ ├── 01_playbook.yml │ │ └── 02_playbook.yml │ └── atop │ ├── 01_playbook.yml │ ├── 02_playbook.yml │ └── 03_playbook.yml ├── qa │ ├── nano │ │ ├── 01_playbook.yml │ │ ├── 02_playbook.yml │ └── atop │ ├── 01_playbook.yml │ ├── 02_playbook.yml │ └── 03_playbook.yml └── prod ├── mysql │ ├── 01_playbook.yml │ └── 02_playbook.yml └── cc ├── 01_playbook.yml ├── 02_playbook.yml └── 03_playbook.yml ```
The main configuration files description: - **nexus.repo**: Contains the configuration for connecting to Nexus, to pull artifacts to the instance. - **inventory.ini**: Contains groups with instances, user credentials, and additional settings for connecting to the instances. - **pre-deploy.yml**: Contains general playbooks that contain a set of tasks that must be done before starting work, such as copying and connecting Nexus config, installing additional packages depending on the group. The rest of the files are example of deployment configurations. The KubeRocketCI deploy pipeline uses naming pattern to comparisons deployment flows/environment with file structure in GitOps repository. ```bash ///01_.yml ``` For example, the **web/qa/nano/01_copy-file.yml** file related to: - Application: **nano** - Deployment Flow: **web** - Environment: **qa** The file name must begin with its serial number (e.g., 01, 02, 03, ...), followed by an underscore. ## AWX Approach The second approach to manage RPM packages is AWX-based. It features user interface and considered more simple. ### AWX Integration Flow Default approach involves the following steps: ```mermaid graph LR; A(Onboard Ansible Git repository) --> B(Install AWX) --> C(Configure AWX)--> D(Create application) --> E(Select deploy pipeline) ``` Here is a breakdown of the scheme above: 1. **Onboard Ansible Git repository**: Create an Ansible Git repository that contains playbooks that will be used in deploy pipelines. 2. **Install AWX**: Set Up AWX application. We recommend using [cluster add-ons](../add-ons-overview.md) to install it. 3. **Configure AWX**: Once AWX tool is installed, open its UI to create AWX Project, Inventory, Templates, etc. 4. **Create application**: [Add application](../../user-guide/add-application.md) to KubeRocketCI as an RPM package. 5. **Select deploy pipeline**: Choose a specific deploy pipeline when creating/editing an environment to deploy RPM packages. ### AWX Integration AWX Integration implies creating a Git repository, installing AWX and configuring it via UI. ### AWX Integration Prerequisites Ensure that External Secrets Operator is [installed](../secrets-management/install-external-secrets-operator.md) and [configured](../secrets-management/external-secrets-operator-integration.md) properly. ### AWX Integration Procedure To set up RPM integration using AWX tool, follow the steps below. Completing these steps enables the addition, building, and deployment of RPM applications: 1. Onboard Ansible Git repository where will store Ansible configuration files. Below is a structure of an Ansible Git repository designed for deploying RPM packages:
View: repository structure ```bash package-install.yaml └── roles ├── pre-deploy │ ├── tasks │ │ └── role.yaml │ ├── files │ │ └── nexus.repo ├── package-install │ └── tasks │ └── role.yaml └── test-app-dev-web └── tasks └── role.yaml ```
Here is a breakdown of the scheme above: - **package-install.yaml**: This is the main file that refers to playbooks located in the **roles** directory. - **roles**: This directory contains all the Ansible roles that will be executed in deploy pipeline. - **pre-deploy**: This is the first role the **package-install.yaml** file refers to. It connects to Nexus storage to interact with the application. Need to specify the required parameters in the **\/roles/pre-deploy/files/nexus.repo** file. - **package-install**: This role contains the playbooks that install all the dependencies to AWS EC2 instances. - **test-app-dev-web**: This role contains the application playbooks, named according to the **\-\-\** convention. It will be executed only if the **Values override** parameter is set to **true** in the Deployment Flow during the component deployment process. 2. Install AWX via [cluster add-ons](https://github.com/epam/edp-cluster-add-ons): a. Create private repository based on the [cluster add-ons](https://github.com/epam/edp-cluster-add-ons) fork. b. In the AWX Operator **[values.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/awx-operator/values.yaml)** file, specify hostname, secretName. OIDC mechanism is optional. c. In the Argo CD application **[values.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml)** file, enable the AWX operator: ```bash awx-operator: createNamespace: true enable: true ``` d. Synchronize state in Argo CD. Refer to the [README.md](https://github.com/epam/edp-cluster-add-ons/blob/main/README.md) file for more details. 3. Configure AWX via UI: a. Once AWX operator is deployed, navigate to its UI using the hostname specified earlier: ![AWX main menu](../../assets/operator-guide/rpm/awx-main-menu.png "AWX main menu") b. (Optional) Configure OIDC. Navigate to **Settings** -> **Generic OIDC settings** and specify the required fields: - **OIDC Key**: Enter **awx** - **OIDC Secret**: This is the client secret data - **URL**: Specify the Keycloak realm URL. ![Generic OIDC settings](../../assets/operator-guide/rpm/awx-oidc-settings.png "Generic OIDC settings") c. Navigate to **Resources** -> **Credentials** and create two credentials: **For Git repository**: These credentials of the **Source Control** type must contain the SSH key with access to the Ansible Git repo: ![Git credentials](../../assets/operator-guide/rpm/awx-git-credentials.png "Git credentials") **For AWS instances**: These credentials with access to the **Hosts** must contain the SSH key: ![AWS EC2 credentials](../../assets/operator-guide/rpm/awx-ec2-credentials.png "AWS EC2 credentials") d. Navigate to **Resources** -> **Project** and create a new project. The `Source Control URL` specifies the repository containing the [Ansible project](#awx-integration-flow). ![Create AWX project](../../assets/operator-guide/rpm/awx-create-project.png "Create AWX project") e. Navigate to **Resources** -> **Inventories** and create an inventory using the **Add inventory** option: ![AWX inventory](../../assets/operator-guide/rpm/awx-inventory.png "AWX inventory") f. Navigate to **Resources** -> **Hosts** and add hosts' IP addresses: ![AWX hosts](../../assets/operator-guide/rpm/awx-hosts.png "AWX hosts") g. Navigate to **Resources** -> **Templates**. Create job template called **package-install** as follows: :::note Don't forget to select the **Prompt on launch** checkbox for **inventory**, **limit**, and **variables** fields. ::: ![AWX template](../../assets/operator-guide/rpm/awx-template.png "AWX template") 4. Create application with the **rpm-package** deployment option. Refer to the [Add Application](../../user-guide/add-application.md) page for more details: ![Create RPM application](../../assets/operator-guide/rpm/rpm.png "Create RPM application") 5. When [creating environments](../../user-guide/manage-environments.md#add-a-new-environment), in the **Deploy pipeline template** field, select **deploy-ansible-awx**: ![Create Ansible environment](../../assets/operator-guide/rpm/ansible-deploy-ansible-awx.png "Create Ansible environment") 6. (Optional) When deploying application, enable the **Values override** option to apply configuration from the Ansible Git repository to be executed specifically for this component: ![Values override option](../../assets/operator-guide/rpm/ansible-values-override.png "Values override option") ## Related Articles * [Customize Deploy Pipeline](../../operator-guide/cd/customize-deploy-pipeline.md) * [Add Deployment Flow](../../user-guide/add-cd-pipeline.md) * [Manage Deployment Flows](../../user-guide/manage-environments.md) * [Add Application](../../user-guide/add-application.md) --- ## CI Pipeline for Terraform # CI Pipeline for Terraform KubeRocketCI ensures the implemented Terraform support by adding a separate component type called **Infrastructure**. The [Infrastructure](../../user-guide/components.md#types) codebase type allows to work with Terraform code that is processed by means of stages in the **Code-Review** and **Build** pipelines. ## Pipeline Stages for Terraform Under the hood, Infrastructure codebase type, namely Terraform, looks quite similar to other codebase types. The distinguishing characteristic of the `Infrastructure` codebase type is that there is a stage called **terraform-check** in both of **Code Review** and **Build** pipelines. This stage runs the pre-commit activities which in their turn run the following commands and tools: 1. [Terraform fmt](https://developer.hashicorp.com/terraform/cli/commands/fmt) - the first step of the stage is basically the `terraform fmt` command. The `terraform fmt` command automatically updates the formatting of Terraform configuration files to follow the standard conventions and make the code more readable and consistent. 2. [Lock provider versions](https://developer.hashicorp.com/terraform/tutorials/configuration-language/provider-versioning) - locks the versions of the Terraform providers used in the project. This ensures that the project uses specific versions of the providers and prevents unexpected changes from impacting the infrastructure due to newer provider versions. 3. [Terraform validate](https://developer.hashicorp.com/terraform/cli/commands/validate) - checks the syntax and validity of the Terraform configuration files. It scans the configuration files for all possible issues. 4. [Terraform docs](https://github.com/terraform-docs/terraform-docs) - generates human-readable documentation for the Terraform project. 5. [Tflint](https://github.com/terraform-linters/tflint) - additional validation step using the `tflint` linter to provide more in-depth checks in addition to what the `terraform validate` command does. 6. [Checkov](https://github.com/bridgecrewio/checkov) - runs the `checkov` command against the Terraform codebase to identify any security misconfigurations or compliance issues. 7. [Tfsec](https://github.com/aquasecurity/tfsec) - another security-focused validation step using the `tfsec` command. Tfsec is a security scanner for Terraform templates that detects potential security issues and insecure configurations in the Terraform code. :::note The commands and their attributes are displayed in the [**.pre-commit-config.yaml**](https://github.com/epmd-edp/hcl-terraform-terraform/blob/master/.pre-commit-config.yaml) file. ::: To learn more about how to operate with **Infrastructure** codebase type, read the [Add Infrastructure](../../user-guide/add-infrastructure.md) and [Manage Infrastructures](../../user-guide/infrastructure.md) pages. ## Related Articles * [User Guide Overview](../../user-guide/index.md) * [Add Infrastructure](../../user-guide/add-infrastructure.md) * [Manage Infrastructures](../../user-guide/infrastructure.md) --- ## Microsoft Teams Notification # Microsoft Teams Notification This section describes how to set up and add notification status to Tekton pipelines by sending pipeline status to the Microsoft Teams channel. ## Create Incoming WebHook To create a link to Incoming Webhook for the Microsoft Teams channel, follow the steps below: 1. Open the channel which will be receiving notifications and click the `•••` button from the upper-right corner. Select **Connectors** in the dropdown menu: ![Menu](../../assets/operator-guide/ms-notification.png "Microsoft Teams menu") 2. In the search field, type **Incoming Webhook** and click **Configure**: ![Connectors](../../assets/operator-guide/ms-notification-con.png "Connectors") 3. Provide a name and upload an image for the webhook if necessary. Click **Create**: ![Connectors setup](../../assets/operator-guide/ms-notification-settings.png "Connectors setup") 4. Copy and save the unique `WebHookURL` presented in the dialog. Click `Done`: ![WebhookURL](../../assets/operator-guide/ms-notification-settings-done.png "WebHookURL") 5. Create a secret with the `` within the krci namespace. ```yaml kubectl -n krci create secret generic microsoft-teams-webhook-url-secret \ --from-literal=url= ``` 6. Add the notification task to the pipeline and add the code below in `final-block` in the pipeline and save: ```yaml {{ include "send-to-microsoft-teams-build" . | nindent 4 }} ``` ## Customize Notification Message To make notification message informative, relevant text should be added to the message. Here are the steps to implement it: 1. Create a new pipeline with a unique name or modify your custom pipeline created before. 2. Add the task below in the `finally` block with a unique name. Edit the `params.message` value if necessary: :::note Task send-to-microsoft-teams ```yaml - name: microsoft-teams-pipeline-status-notification-failed params: - name: webhook-url-secret value: microsoft-teams-webhook-url-secret - name: webhook-url-secret-key value: url - name: message value: >- Build Failed project: $(params.CODEBASE_NAME) branch: $(params.git-source-revision) pipeline: $(context.pipelineRun.name) commit message: $(params.COMMIT_MESSAGE) taskRef: kind: Task name: send-to-microsoft-teams when: - input: $(tasks.status) operator: in values: - Failed - PipelineRunTimeout ``` ::: After customization, the following message is supposed to appear in the channel when failing pipelines: ![Notification example](../../assets/operator-guide/ms-notification-example.png "Notification example") ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install Tekton](../install-tekton.md) --- ## Tekton Long-Term Log Storage # Tekton Long-Term Log Storage Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller. This allows you to: - Provide custom Results metadata about your CI/CD workflows not available in the Tekton TaskRun/PipelineRun CRDs (for example: post-run actions). - Group related workloads together (e.g. bundle related TaskRuns and PipelineRuns into a single unit). - Make long-term result history independent of the Pipeline CRD controller, letting you free up etcd resources for Run execution. - Store logs produced by the TaskRuns/PipelineRuns so that completed Runs can be cleaned to save resources. ## Long-Term Log Access Workflow The following diagram illustrates the workflow for accessing long-term logs for pipelines in the KubeRocketCI Portal: ```mermaid sequenceDiagram actor U as User participant KR as KubeRocketCI Portal participant PC as Pipeline Controller participant RW as Result Watcher participant RA as Result API U->>KR: Trigger or create a PipelineRun/TaskRun KR->>PC: Start pipeline execution RW-->>PC: Watch PipelineRun/TaskRun Note over PC,RW: Wait for PipelineRun/TaskRun Completion RW->>RA: Update results database U->>KR: View pipeline logs KR-->>RA: Get long-term logs KR->>U: Return long-term logs ``` ## Install Tekton Results Tekton Results is deployed as part of the Tekton Pipelines installation. For installing Tekton Pipelines, we recommend using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/blob/4456631feb6510ff875b5b839534968c3846da7a/clusters/core/apps/values.yaml#L286C1-L289C30). Here's an example of how to configure values.yaml before installation: ```yaml tekton: createNamespace: true enable: true namespace: tekton-pipelines ``` ## Configuration :::note We use Postgres operator to connect to Tekton results, but you can use any other external databases supported. Please align configuration accordingly to Tekton result [documentation](https://github.com/tektoncd/results/blob/v0.20.0/docs/external-database.md). ::: To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below: 1. Storage sizes in the [results-pg.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results-pg.yaml) file define the allocated volumes for the Tekton Results PostgreSQL cluster: - **Database instance storage** - Path: `spec.instances.dataVolumeClaimSpec.resources.requests.storage` - Default: `2Gi` - Description: Specifies the size of the main PostgreSQL data volume. - **Backup repository storage** - Path: `spec.backups.pgbackrest.repos.volume.volumeClaimSpec.resources.requests.storage` - Default: `2Gi` - Description: Specifies the size of the pgBackRest backup repository volume. 2. Storage and retention settings in the [results.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results.yaml) file define how Tekton Results result records are retained in the database and how related storage is configured: - **Log storage size** - Path: `PersistentVolumeClaim.spec.resources.requests.storage` - Default: `5Gi` - Description: Specifies the allocated disk size for storing pipeline logs. - **Retention period** - Path: `ConfigMap.data.defaultRetention` in `tekton-results-config-results-retention-policy` - Default: `"30"` days - Description: Defines how long log files are retained before being automatically removed. - **Cleanup schedule** - Path: `ConfigMap.data.runAt` in `tekton-results-config-results-retention-policy` - Default: `"0 18 * * *0*"` - Description: Specifies the cron expression that determines when automated log cleanup runs (daily at 6:00 PM UTC). :::warning The retention policy agent removes only database records from PostgreSQL. It does not delete associated log files stored on persistent volumes, S3, or GCS backends. As a result, log data may remain on disk and consume storage even after the corresponding records are expired. ::: 3. To remove physical log files from PVCs (not just database records), use the settings in the [results-clean-old-logs-cronjob.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results-clean-old-logs-cronjob.yaml) file: - **Cleanup schedule** - Path: `spec.schedule` - Default: `"0 18 * * *"` - Description: Specifies the cron expression that determines when the CronJob runs (daily at 6:00 PM UTC). - **Physical log retention (days)** - Path: `spec.jobTemplate.spec.template.spec.containers.command` (the `-mtime +N` value in the `find` command) - Default: `30` days - Description: Physical log files in `/tekton-results/logs/` older than this many days are deleted from the PVC. The value is the number in `-mtime +N` (e.g. `+30` keeps files for 30 days). :::note In this example, the PGO (PostgreSQL Operator) is used for the Tekton Results database. ::: ## Database Performance Indexes Tekton Results creates its schema through GORM auto-migration, which adds no indexes beyond the primary keys. As the history grows, the queries behind the KubeRocketCI Portal pipeline run history fall back to sequential scans and the history views become progressively slower to load. Create the indexes below to keep those queries on index scans. :::note `CREATE INDEX CONCURRENTLY` builds the index without locking the table for writes, so these statements are safe to run against a live database. They cannot run inside a transaction block, which is why each statement is applied separately. ::: 1. Find the primary PostgreSQL pod: ```bash PG_POD=$(kubectl get pods -n tekton-pipelines \ -l postgres-operator.crunchydata.com/cluster=results,postgres-operator.crunchydata.com/role=master \ -o jsonpath='{.items[0].metadata.name}') ``` 2. Create the index covering list queries that filter by parent and type and sort by time: ```bash kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \ psql -U postgres -d results -c \ "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_records_parent_type_created ON records (parent, type, created_time DESC);" ``` 3. Create the index covering JSONB containment queries (`@>`) used for codebase and name filtering: ```bash kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \ psql -U postgres -d results -c \ "CREATE INDEX CONCURRENTLY IF NOT EXISTS results_annotations ON results USING GIN (annotations jsonb_path_ops);" ``` 4. Create the index covering summary annotation filtering, which the DORA metrics views rely on: ```bash kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \ psql -U postgres -d results -c \ "CREATE INDEX CONCURRENTLY IF NOT EXISTS record_summary_annotations ON results USING GIN (recordsummary_annotations jsonb_path_ops);" ``` 5. Verify that the indexes exist: ```bash kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \ psql -U postgres -d results -c "\di+" ``` 6. Confirm that queries use them. The plan should report an `Index Scan` or `Bitmap Index Scan` on `idx_records_parent_type_created` rather than a `Seq Scan`: ```bash kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \ psql -U postgres -d results -c " EXPLAIN ANALYZE SELECT * FROM records WHERE parent LIKE 'tekton-pipelines/results/%' AND type = 'tekton.dev/v1.PipelineRun' ORDER BY created_time DESC LIMIT 10; " ``` To remove the indexes, drop them by name: ```bash kubectl exec -it -n tekton-pipelines "$PG_POD" -c database -- \ psql -U postgres -d results -c " DROP INDEX IF EXISTS idx_records_parent_type_created; DROP INDEX IF EXISTS results_annotations; DROP INDEX IF EXISTS record_summary_annotations; " ``` ## Related Articles * [Add Application](../../user-guide/add-application.md) * [Install Tekton](../install-tekton.md) * [Install KubeRocketCI](../install-kuberocketci.md) --- ## Tekton Pipeline Monitoring with Prometheus and Grafana # Tekton Pipeline Monitoring with Prometheus and Grafana Tekton already exposes Prometheus metrics for pipeline runs, task runs, webhooks and its own control plane. This page explains how to collect them and read them in Grafana using the `tekton-monitoring` add-on, which ships the scrape configuration and the dashboards together so neither has to be assembled by hand. ## Prerequisites Ensure the following requirements are met before moving ahead: - The [tekton](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/tekton) add-on is deployed, so Tekton Pipelines, Triggers and Results are running. - The **prometheus-operator** custom resource definitions are installed, since the add-on creates `ServiceMonitor` resources. - [kube-prometheus-stack](https://prometheus-community.github.io/helm-charts) is installed with its Grafana sidecar enabled. The sidecar discovers dashboards in every namespace, so the add-on can ship them from `tekton-pipelines`. - The legacy integration is not in use. If Tekton metrics were configured before this add-on was introduced, follow [Migrate From the Legacy Integration](#migrate-from-the-legacy-integration) first. ## Set the Prometheus Release Label Prometheus picks up `ServiceMonitor` resources by release label: `kube-prometheus-stack` sets `serviceMonitorSelectorNilUsesHelmValues: true` by default, which limits Prometheus to monitors carrying its own release label. Set this value first, so the monitors are collected as soon as the add-on is deployed. Run `helm list -n monitoring` to get the release name of your `kube-prometheus-stack` installation, then set it in `clusters/core/addons/tekton-monitoring/values.yaml`: ```yaml values.yaml serviceMonitor: additionalLabels: release: kube-prometheus ``` If your Prometheus sets `serviceMonitorSelectorNilUsesHelmValues: false`, the label is not required and the value can be cleared. ## Enable the Add-On The add-on is disabled by default. Enable it in `clusters/core/apps/values.yaml`: ```yaml values.yaml tekton-monitoring: createNamespace: false enable: true namespace: tekton-pipelines ``` Commit the change and let Argo CD synchronize the `tekton-monitoring` application. Because the add-on deploys into the namespace owned by the `tekton` add-on, it never creates that namespace itself. Once the application is synchronized, the Tekton targets appear under **Status → Targets** in Prometheus, and the six dashboards appear in Grafana under titles starting with `Tekton /`. ## What the Add-On Deploys ### Metrics Collection Six `ServiceMonitor` resources point Prometheus at the endpoints Tekton already exposes: | ServiceMonitor | Component | What it provides | |:-|:-|:-| | `tekton-pipelines-controller` | Pipelines controller | Pipeline and task run counts, durations, throttling, reconciler and workqueue health | | `tekton-events-controller` | Events controller | CloudEvents emission | | `tekton-triggers-controller` | Triggers controller | Reconciliation of `EventListener` and `Trigger` resources | | `tekton-results-api` | Results API | gRPC server metrics | | `tekton-results-watcher` | Results watcher | Archival and pruning of completed runs | | `tekton-eventlisteners` | All EventListeners | Webhook volume, latency and triggered resources | The EventListener monitor is deliberately cluster-wide: KubeRocketCI creates an `EventListener` in each delivery namespace rather than in `tekton-pipelines`, and every one of them is labelled by the EventListener controller, so a single monitor covers them all. ### Dashboards Six dashboards are delivered as ConfigMaps and picked up by the Grafana sidecar: | Dashboard | Answers | |:-|:-| | Pipeline Overview | How many pipelines ran, how many failed, and how long they took | | Task Analytics | Which tasks are slowest and which fail most often | | Queue and Scheduling | Whether runs are waiting on quota, node capacity or resolution | | EventListener Traffic | Whether webhooks arrive and convert into pipeline runs | | Platform Health | Whether the Tekton control plane itself is healthy | | Resource Consumption | How much CPU and memory pipelines consume | ## Required Tekton Metrics Settings Two panels depend on metric labels that Tekton omits by default. The `tekton` add-on already sets both keys in its `config-observability` ConfigMap, so a standard installation needs no action. Re-apply them if that manifest is re-vendored from an upstream Tekton release, or if Tekton was installed from another source: | Key | Value | Without it | |:-|:-|:-| | `metrics.count.enable-reason` | `true` | The `reason` label is absent, so **Timeouts** reads `0` permanently and **Failure reasons** collapses into a single unnamed slice | | `metrics.running-pipelinerun.level` | `pipeline` | The running-pipelines gauge has no `pipeline` label, so **Running now** is empty once a pipeline filter is applied | Both keys belong in the `data` block of the ConfigMap. The `_example` block lists the same keys but is inert documentation, so values placed there have no effect. ## Reading the Dashboards Two conventions are worth knowing before drawing conclusions from a panel: - **Prefer p50 and p95 over the average.** A Tekton task that fails usually hangs until its timeout instead of failing fast, so a handful of failed runs pull the mean far above what a healthy run costs. A large gap between **Avg** and **p50** means the task hangs when it fails, not that it is slow. - **Counts cover completed runs.** A run is counted when it finishes, so a pipeline still in flight appears under **Running now** rather than in the range totals. ## Migrate From the Legacy Integration Earlier releases integrated Tekton metrics either by applying a single `ServiceMonitor` named `tekton-overview` by hand and importing a community Grafana dashboard, or by setting `edp-tekton.grafana.enabled` in the `edp-install` chart. Both approaches covered only the pipelines controller and a single dashboard. The add-on replaces them. :::warning Do not run the add-on and the legacy integration at the same time. The `tekton-overview` ServiceMonitor and the add-on's `tekton-pipelines-controller` ServiceMonitor select the **same** Kubernetes Service, so Prometheus scrapes it twice and every panel that sums across targets roughly doubles. ::: To migrate: 1. Set `edp-tekton.grafana.enabled` to `false`, or remove the value, and apply the change. 2. Delete a hand-applied ServiceMonitor if one exists: `kubectl delete servicemonitor tekton-overview -n tekton-pipelines`. 3. Remove any manually imported Tekton dashboard from Grafana to avoid duplicates. 4. Enable the add-on as described above. ## Troubleshooting | Symptom | Cause | |:-|:-| | All panels empty | The `release` label does not match the `kube-prometheus-stack` release name | | Targets missing in Prometheus | The prometheus-operator CRDs are absent, or the add-on has not synchronized | | Dashboards missing in Grafana | The Grafana sidecar is disabled or not searching all namespaces | | **Timeouts** always `0` | `metrics.count.enable-reason` is not enabled | | **Running now** empty when filtering by pipeline | `metrics.running-pipelinerun.level` is not set to `pipeline` | | Counts roughly doubled | The legacy integration is still enabled alongside the add-on | ## Related Articles * [Install Tekton](../install-tekton.md) * [Add-Ons Overview](../add-ons-overview.md) * [Install KubeRocketCI](../install-kuberocketci.md) --- ## Tekton Overview # Tekton Overview KubeRocketCI provides Continuous Integration based on [Tekton](https://tekton.dev/). Tekton is an open-source Kubernetes native framework for managing pipelines, allowing a user to compile, build and test applications. Tekton is an open-source Kubernetes native solution, designed for orchestrating complex pipelines that allows users to define `pipelines as code`. It enables users to compile, build, and test applications seamlessly. The [edp-tekton](https://github.com/epam/edp-tekton) GitHub repository provides all Tekton implementation logic on the platform. The Helm charts are used to deploy the resources inside the Kubernetes cluster. Tekton logic is decoupled into separate components: ![Edp-tekton components diagram](../../assets/operator-guide/ci/edp-tekton-components-diagram.png) The diagram above describes the following: - `Common-library` is the Helm chart of Library type which stores the common logic shareable across all Tekton pipelines. This library contains Helm templates that generate common Tekton resources. - `Pipelines-library` is the Helm chart of the Application type which stores the core logic for the KubeRocketCI pipelines. Tekton CRs like Pipelines, Tasks, EventListeners, Triggers, TriggerTemplates, and other resources are delivered with this chart. - `EDP Interceptor` is the custom `Tekton Interceptor` which enriches the payload from the VCSs events with the KubeRocketCI data from the `Codebase` Custom Resource (CR) specification. These data are used to define the Pipeline logic. - `Custom-pipelines` is the Helm chart of the Application type which implements custom logic running specifically for internal platform development, for example, CI and Release. It also demonstrates the customization flow on the platform. Inspect the schema below that describes the logic behind the Tekton functionality on the platform: ![Component view for the Tekton on KubeRocketCI](../../assets/operator-guide/ci/edp-tekton-component-view-diagram.png) The platform logic consists of the following: 1. The `EventListener` exposes a dedicated Pod that runs the sink logic and receives incoming events from the VCSs (Gerrit, GitHub, GitLab) through the Ingress. It contains triggers with filtering and routing rules for incoming requests. 2. Upon the Event Payload arrival, the `EventListener` runs triggers to process information or validate it via different interceptors. 3. The `EDP Interceptor` extracts information from the `codebases.v2.edp.epam.com` CR and injects the received data into top-level 'extensions' field of the Event Payload. The Interceptor consists of running Pod and Service. 4. The `Tekton Cel Interceptor` does simple transformations of the resulting data and prepares them for the Pipeline parameters substitution. 5. The `TriggerTemplate` creates a PipelineRun instance with the required parameters extracted from the Event Payload by Interceptors. These parameters are mandatory for Pipelines. 6. The `PipelineRun` has a mapping to the platform Tekton Pipelines using a template approach which reduces code duplication. Each Pipeline is designed for a specific VCS (Gerrit, GitLab, GitHub), technology stack (such as Java or Python), and type (code-review, build). 7. A `Pipeline` consists of separate Tekton Tasks. They are arranged in a specific order of execution in the Pipeline. 8. Each `Task` is executed as a Pod on the Kubernetes cluster. Also, Tasks can have a different number of steps that are executed as a Container in Pod. 9. The Kubernetes native approach allows the creation of PipelineRun either with the kubectl tool or using the [KubeRocketCI portal](../../user-guide/application.md). To learn more about Tekton Pipelines and their types in KubeRocketCI, you can read the [KubeRocketCI: Tekton Overview](../../user-guide/tekton-pipelines.md) page. The [Pipelines Overview](../../user-guide/pipelines.md) page describes how to operate with Tekton Pipelines in the KubeRocketCI portal. To create user-defined pipelines, refer to the following guides: - [Create and Use Custom Tekton Pipelines](../../use-cases/custom-pipelines-flow.md) - [Customize Deploy Pipeline](../cd/customize-deploy-pipeline.md) - [Customize Environment Cleanup](../cd/customize-environment-deletion.md) All of these guides have step-by-step video tutorials. ## Related Articles * [Install Tekton](../install-tekton.md) --- ## Publish SonarQube Reports to Pull Requests # Publish SonarQube Reports to Pull Requests Pull request decoration makes SonarQube publish its **Quality Gate status** and **analysis summary** directly onto a merge request, so reviewers see new issues, security hotspots, coverage, and duplications without leaving the Git provider. This guide enables the feature in KubeRocketCI end to end, using a self-hosted **GitLab** instance as the baseline example. The same steps apply to GitHub, Bitbucket, and Azure DevOps — only the DevOps Platform type changes. By the end, every review pipeline in KubeRocketCI adds a Quality Gate comment and a `SonarQube` commit status to the corresponding merge request. ## How SonarQube Pull Request Decoration Works KubeRocketCI review pipelines already run SonarQube analysis in **pull request mode** (the scanner receives the `sonar.pullrequest.key`, `sonar.pullrequest.branch`, and `sonar.pullrequest.base` parameters automatically). To turn that analysis into a merge request comment, SonarQube needs three things: 1. A **DevOps Platform integration** that tells SonarQube how to reach the Git provider and post back (URL + access token). 2. A **project binding** that maps the SonarQube project to its Git repository. 3. A reachable **Server Base URL** so the links in the decoration resolve to your SonarQube instance. ### Does SonarQube Community Edition Support Pull Request Decoration? Yes. SonarQube **Community** editions add branch and pull request analysis through the [sonarqube-community-branch-plugin](https://github.com/mc1arke/sonarqube-community-branch-plugin), which KubeRocketCI ships preinstalled. SonarQube Developer Edition and above provide the same capability natively, so no plugin is required there. ## Prerequisites Before proceeding, ensure the following prerequisites are in place: - SonarQube is integrated with KubeRocketCI. See [SonarQube Integration](sonarqube.md) for the base setup. - A codebase hosted in GitLab with review and build pipelines already running. - A GitLab **personal (or project/group) access token** with the `api` scope. SonarQube uses this token to post comments and commit statuses on merge requests. - Administrator access to the SonarQube UI (or an admin token for the API). ## Trust the Git Provider Certificate SonarQube posts decorations by calling the Git provider's API over HTTPS. If your GitLab (or other provider) serves a **self-signed certificate**, the SonarQube JVM must trust its CA — otherwise the integration fails on the TLS handshake with `Could not validate GitLab url`. The SonarQube Helm chart exposes a `caCerts` option that imports a certificate into the JVM truststore and wires it into both the web and compute-engine processes (the compute engine runs the decoration): ```yaml title="values.yaml" caCerts: enabled: true configMap: name: gitlab-ca # ConfigMap holding the provider CA certificate key: ca.crt path: ca.crt ``` :::note Skip this step when your Git provider uses a certificate signed by a public, already-trusted Certificate Authority. ::: ## Configure the DevOps Platform Integration Create a global GitLab configuration in SonarQube. Use the UI for a one-time manual setup, or the Web API to automate it. 1. Open the SonarQube UI and navigate to **Administration** -> **Configuration** -> **DevOps Platform Integrations**. Select the **GitLab** tab and click **Create configuration**: ![SonarQube create GitLab DevOps Platform configuration form](../../assets/operator-guide/code-quality/sonarqube-devops-integration-form.png "SonarQube create GitLab DevOps Platform configuration form") 2. Fill in the fields and click **Save configuration**: - **Configuration name** — a short identifier used when binding projects, for example `gitlab`. - **GitLab API URL** — the `/api/v4` endpoint of your instance, for example `https://gitlab.example.com/api/v4`. - **Personal Access Token** — the GitLab token with the `api` scope. 3. Once saved, the configuration reports **Quality Gate status reporting** and **Import repositories** as valid: ![SonarQube GitLab integration showing a valid configuration](../../assets/operator-guide/code-quality/sonarqube-devops-integration-gitlab.png "SonarQube GitLab integration showing a valid configuration") Create the global GitLab configuration with the [`alm_settings/create_gitlab`](https://next.sonarqube.com/sonarqube/web_api/api/alm_settings) endpoint. Authenticate with an admin token (note the trailing colon, which leaves the password empty): ```bash curl -u ":" -X POST "https://sonarqube.example.com/api/alm_settings/create_gitlab" \ --data-urlencode "key=gitlab" \ --data-urlencode "url=https://gitlab.example.com/api/v4" \ --data-urlencode "personalAccessToken=" ``` Verify the configuration can reach GitLab: ```bash curl -u ":" "https://sonarqube.example.com/api/alm_settings/validate?key=gitlab" ``` An empty response with HTTP `204` confirms the connection. A `Could not validate GitLab url` error indicates a connectivity or TLS problem — see [Troubleshooting](#troubleshooting). ## Bind the SonarQube Project to Its Repository Link each SonarQube project to the GitLab repository it should decorate. The project key matches the KubeRocketCI codebase name. 1. Open the project in SonarQube, click **Project Settings** (top-right), and select **DevOps Platform Integration** from the left menu of **General Settings**. 2. Choose the `gitlab` configuration created earlier, provide the repository's **GitLab project ID**, and save. :::note The GitLab project ID is shown on the repository's home page under the project name, and in GitLab under **Settings** -> **General**. ::: Bind the project with the [`alm_settings/set_gitlab_binding`](https://next.sonarqube.com/sonarqube/web_api/api/alm_settings) endpoint. The `repository` value is the numeric GitLab project ID: ```bash curl -u ":" -X POST "https://sonarqube.example.com/api/alm_settings/set_gitlab_binding" \ --data-urlencode "almSetting=gitlab" \ --data-urlencode "project=" \ --data-urlencode "repository=" \ --data-urlencode "monorepo=false" ``` ## Set the Server Base URL For the links inside a decoration to resolve back to SonarQube, set the server base URL to the address reachable from the Git provider and reviewers: ```bash curl -u ":" -X POST "https://sonarqube.example.com/api/settings/set" \ --data-urlencode "key=sonar.core.serverBaseURL" \ --data-urlencode "value=https://sonarqube.example.com" ``` The same value can be set in the UI under **Administration** -> **Configuration** -> **General** -> **Server base URL**. ## Verify Pull Request Decoration End to End With the integration in place, exercise the full flow on a real merge request: 1. In the GitLab repository, create a feature branch and open a **merge request** against the default branch. 2. KubeRocketCI triggers the **review pipeline**, which runs SonarQube analysis in pull request mode. 3. When the analysis completes, SonarQube posts a **Quality Gate** comment and a `SonarQube` commit status on the merge request: ![SonarQube Quality Gate report published on a GitLab merge request](../../assets/operator-guide/code-quality/sonarqube-pr-decoration-note.png "SonarQube Quality Gate report published on a GitLab merge request") The comment links to the full pull request analysis in SonarQube, and the commit status appears alongside the pipeline status, giving reviewers a Quality Gate signal before the merge request is merged. ## Troubleshooting | Symptom | Cause | Resolution | |---------|-------|------------| | `Could not validate GitLab url. Got an unexpected answer.` | SonarQube cannot complete the TLS handshake with a self-signed Git provider, or the URL/token is wrong. | Trust the provider CA as described in [Trust the Git Provider Certificate](#trust-the-git-provider-certificate). Confirm the API URL ends with `/api/v4` and the token has the `api` scope. | | Decoration links point to `localhost` or fail to open | `sonar.core.serverBaseURL` is unset. | Set the server base URL as shown above. | | No comment appears on the merge request | The analysis ran in **branch mode**, not pull request mode (most common), or the project is not bound. | Confirm pull request mode as described below, and check the project binding references the correct GitLab project ID. | ### Confirm the Analysis Ran in Pull Request Mode When both integrations are green but no comment appears, the scanner almost always submitted a **branch** analysis instead of a **pull request** analysis. Two quick checks: - In SonarQube, open the project's branch/pull request selector. A pull request analysis is listed under **Pull Requests** with the merge request number. If you only see entries named after commit SHAs under **Branches**, the run was a branch analysis. - In the generated `sonar-project.properties` (or the scanner log), a decorating run sets `sonar.pullrequest.key`, `sonar.pullrequest.branch`, and `sonar.pullrequest.base`, and **no** `sonar.branch.name`. Seeing `sonar.branch.name=` instead confirms the pull request parameters were not provided. To fix, make the review (merge request) pipeline pass the three pull request parameters and **omit** the branch parameter — `sonar.branch.name` and `sonar.pullrequest.*` are mutually exclusive: ```properties sonar.pullrequest.key= sonar.pullrequest.branch= sonar.pullrequest.base= ``` :::note In KubeRocketCI Tekton pipelines these map to the `sonarqube-general` task inputs `key-id`, `source-branch`, and `target-branch`. If the task is given only `branch`, it runs a branch analysis and no decoration is posted. ::: ## Related Articles * [SonarQube Integration](sonarqube.md) * [Manage Project Visibility](sonarqube-visibility.md) * [Sonarqube Project Properties for Application](../../user-guide/application-sonarqube-project-properties.md) * [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) --- ## Manage Project Visibility # Manage Project Visibility This manual offers detailed guidance on configuring permissions for SonarQube projects, ensuring that only authorized personnel can access and interact with the data on the SonarQube platform. It caters to both new and existing SonarQube projects, providing a clear pathway to refine visibility settings for enhanced security. By leveraging the OpenID Connect mechanism for login, users are automatically assigned to the `sonar-users` group, granting them access to all projects by default. This document outlines methods to alter these default permissions, enhancing the platform's security and data confidentiality. It is structured into two main sections: one focusing on [restricting access to new projects](#restrict-access-for-new-projects), and the other on [reconfiguring access settings for existing projects](#configure-access-for-existing-projects). ## Restrict Access for New Projects In its default configuration, SonarQube does not restrict access to newly created projects, making them accessible to all instance users. To modify this behavior and set new projects to private by default, follow these instructions: 1. Open the SonarQube UI in the browser. 2. Navigate to the `Administration` tab: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar_administration.png "SonarQube user settings") :::note Ensure you have admin rights to see the `Administration` section. ::: 3. Click the `Projects` button and select `Management`: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar_projects_management.png "SonarQube user settings") 4. On the project management page, click pencil icon at the top-right corner:: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar_pencil.png "SonarQube user settings") 5. Select **Private** and click **Change Default Visibility**: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar_private_visibility.png "SonarQube user settings") ## Configure Access for Existing Projects To make all the current projects private, follow the steps below: 1. In the **Projects** tab, enter the project you want to make private. 2. In the project page, click the **Project Settings** button and select **Permissions**: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar_project_permissions.png "SonarQube user settings") 3. In the project permissions page, select **Private**: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar_project_private_permissions.png "SonarQube user settings") 4. Repeat the procedure for all of the projects you want to make private. ## Related Articles * [SonarQube Integration](sonarqube.md) * [Nexus Sonatype Integration](../artifacts-management/nexus-sonatype.md) * [Integrate SonarQube](../../quick-start/integrate-sonarcloud.md) --- ## SonarQube Integration # SonarQube Integration This documentation offers detailed guidance on seamlessly integrating SonarQube into KubeRocketCI. It covers essential steps to ensure a robust connection between SonarQube's code quality and security analysis capabilities with the platform's continuous delivery ecosystem, facilitating enhanced code inspection and improvement practices within your development workflow. ## Prerequisites Before proceeding, ensure the following prerequisites are in place: - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation To install SonarQube in environment, it's recommended to use the resources provided in the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) repository. This approach involves installing both the sonarqube and the [sonar-operator](https://github.com/epam/edp-sonar-operator). Leveraging the Cluster Add-Ons simplifies the deployment and management process, providing a streamlined method to integrate SonarQube into infrastructure. 1. Navigate to the forked cluster Add-Ons repository and align the [sonar values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/sonar/values.yaml) and [sonar-operator values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/sonar-operator/values.yaml) files. Follow the instructions to deploy sonarqube, ensuring it's correctly configured to serve as artifact repository. 2. Update the configuration to enable **sonar** and **sonar-operator** in [Add-Ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml) repository: ```yaml title="clusters/core/apps/values.yaml" sonar: createNamespace: true enable: true sonar-operator: createNamespace: true enable: true ``` 3. Sync resources and wait till the all **sonar** and **sonar-operator** resources is created: ![sonarqube ArgoCD applications](../../assets/operator-guide/code-quality/sonarqube-addons-deploy.png "sonarqube ArgoCD applications") 4. **Sign In** into **sonarqube** using the default credentials **Username**: `admin` and **Password**: `admin` and update the default password for better security for that click on profile icon in the upper right corner, click on **Security**, enter a new password and click **Update** button: ![sonarqube change password](../../assets/operator-guide/code-quality/sonarqube-change-password.png "sonarqube change password") ### Applying GitOps approach installation changes To update SonarQube configurations: 1. Modify the relevant configuration files in the forked add-ons repository. 2. Commit and push the changes to the forker Git repository. 3. Synchronize the changes with ArgoCD. ## Configuration With Add-ons approach add predefined [SonarQube plugins](https://github.com/epam/edp-cluster-add-ons/blob/build/0.1.0-SNAPSHOT.236/clusters/core/addons/sonar/values.yaml#L26) and [sonar-operator](https://github.com/epam/edp-cluster-add-ons/tree/build/0.1.0-SNAPSHOT.236/clusters/core/addons/sonar-operator/templates/sonar) create the necessary groups, users, permission templates, quality profiles, quality gates automatically by deploy preconfigured custom resources. Manage these configurations centrally through a GitOps-based workflow. ## Configuration SonarQube with sonar-operator 1. Create a Kubernetes Secret that the **sonar-operator** will use to connect to the **sonarqube** and create all resources: ```yaml apiVersion: v1 kind: Secret metadata: name: sonar-admin-password namespace: sonar type: Opaque data: user: YWRtaW4= # base64-encoded value of "admin" password: cGFzcw== # base64-encoded value of "pass" ``` ```json "sonarqube": { "user": "XXXXXXX", "password": "XXXXXXX" } ``` 2. Create a secret using `manifests` or with `External Secrets Operator` to enable the **sonar-operator** to connect to the **sonarqube** for provisioning: ```yaml apiVersion: v1 kind: Secret metadata: name: ci-sonar namespace: sonar type: Opaque stringData: password: ``` ```json "sonarqube-ci-user": { "password": "XXXXXXX" } ``` ## SonarQube configuration overview with sonar-operator ### Plugin Management The SonarQube instance includes pre-installed plugins to meet common project needs. Additional plugins, if required, can be installed by updating the [SonarQube values.yaml](https://github.com/epam/edp-cluster-add-ons/blob/build/0.1.0-SNAPSHOT.236/clusters/core/addons/sonar/values.yaml#L26) file in the forked add-ons repository. ### Group Permissions Access control in SonarQube is organized through predefined groups: - **sonar-administrators:** System administrators with full access to manage the instance. - **sonar-developers:** Developers who can view and edit project settings (managed by the Sonar Operator). - **sonar-users:** Default group for all authenticated users with basic project viewing rights. - **view-group:** Grants browsing access for users requiring read-only permissions. ![sonarqube kuberocketci permission template](../../assets/operator-guide/code-quality/sonarqube-kuberocketci-permission-template.png "sonarqube kuberocketci permission template") ### Permission Templates Permissions in SonarQube are associated with projects through permission templates: - **edp-default:** The default template applied to all projects, ensuring consistent group-level access management across the platform. ### Quality Gates Quality Gates are used to ensure that code quality standards are met before further development or deployment proceeds: - **KubeRocketCI way (default):** KuberocketCI quality gates are defined in the SonarQube instance, specifically tailored for the KubeRocketCI project. - **Sonar way (built-in):** These are predefined quality gates available directly in SonarQube, not customized for any specific project but rather standardized for general use. ![sonarqube kuberocketci quality gate](../../assets/operator-guide/code-quality/sonarqube-kuberocketci-qualitygate.png "sonarqube kuberocketci quality gate") ### Quality Profiles Quality Profiles define the rules used during code analysis. The Sonar Operator manages quality profiles with the following default setup: - **KubeRocketCI way (default):** A Java quality profile based on KubeRocketCI best practices, offering tailored rule sets. ## Configuration SonarQube ci-user without sonar-operator To establish robust authentication and precise access control, generating a SonarQube token is essential. This token is a distinct identifier, enabling effortless integration between SonarQube and KubeRocketCI. To generate the SonarQube token, proceed with the following steps: 1. Open the SonarQube UI and navigate to **Administration** -> **Security** -> **User**. Create a new user or select an existing one. Click the `Options List` icon to create a token: ![SonarQube user settings](../../assets/operator-guide/code-quality/sonar-create-user.png "SonarQube user settings") 2. Type the `ci-user` username, define an expiration period, and click the `Generate` button to create the token: ![SonarQube create token](../../assets/operator-guide/code-quality/sonar-generate-token.png "SonarQube create token") 3. Click the `Copy` button to copy the generated ``: ![SonarQube token](../../assets/operator-guide/code-quality/sonar-copy-token.png "SonarQube token") ## Integrate SonarQube With KubeRocketCI For provision secret using manifest, KubeRocketCI portal or with the externalSecrets operator: Go to the **Portal** open **Configuration** -> **Code Quality** -> **SonarQube**. Update or click **+ Add integration** fill in the `URL` and `Token` fields and click the `Save` button: ![SonarQube update manual secret](../../assets/operator-guide/code-quality/sonar-secret-password.png "SonarQube update manual secret") ```yaml apiVersion: v1 kind: Secret metadata: name: ci-sonarqube namespace: krci labels: app.edp.epam.com/secret-type: sonar app.edp.epam.com/integration-secret: "true" type: Opaque stringData: url: https://sonarqube.example.com token: ``` ```json "ci-sonarqube": { "url": "https://sonarqube.example.com", "token": "XXXXXXXXXXXX" }, ``` Go to the **Portal** open **Configuration** -> **Code Quality** -> **SonarQube** and see the `Managed by External Secret` message: ![SonarQube managed by external secret operator](../../assets/operator-guide/code-quality/sonar-externalsecret-password.png "SonarQube managed by external secret operator") More details about External Secrets Operator integration can be found in the [External Secrets Operator Integration](https://epam.github.io/edp-install/operator-guide/code-quality/external-secrets-operator-integration/) page. ## Project Analysis Flow 1. SonarQube analysis is seamlessly integrated into KubeRocketCI pipelines, triggered during both review and build pipelines: - **Review Pipeline:** Analyzes only the changes made in a pull request. - **Build Pipeline:** Analyzes the main branch codebase to ensure overall project quality. 2. SonarQube projects are named based on the corresponding KubeRocketCI project names, making it easy to navigate analysis results in the SonarQube UI mode details how to configure repo [project-project.settings](../../user-guide/application-sonarqube-project-properties.md): ![SonarQube project](../../assets/operator-guide/code-quality/sonarqube-project-analysis.png "[SonarQube project") ## Related Articles * [Publish SonarQube Reports to Pull Requests](sonarqube-pr-decoration.md) * [Sonarqube Project Properties for Application](../../user-guide/application-sonarqube-project-properties.md) * [Install KubeRocketCI With Values File](../install-kuberocketci.md) * [Install External Secrets Operator](../secrets-management/install-external-secrets-operator.md) * [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) * [Cluster Add-Ons Overview](../add-ons-overview.md) --- ## Deploy AWS EKS Cluster # Deploy AWS EKS Cluster This instruction offers a comprehensive guide on deploying an Amazon Elastic Kubernetes Service (EKS) cluster, ensuring a scalable and secure Kubernetes environment on AWS. For those looking to optimize their EKS cluster configurations, it is highly recommended to consult the [AWS EKS Best Practices](https://aws.github.io/aws-eks-best-practices/) guide. This resource covers a wide range of topics crucial for the successful deployment and operation of your EKS clusters, including: - **Security**: Best practices for securing your EKS clusters, including IAM roles, network policies, and secrets management. - **Networking**: Guidance on setting up VPCs, subnets, and load balancers to ensure efficient and secure network traffic. - **Monitoring and Logging**: Strategies for implementing comprehensive monitoring and logging solutions using AWS CloudWatch and other tools to maintain visibility into cluster performance and operational health. - **Performance**: Tips for optimizing cluster performance through the proper selection of EC2 instances, efficient load balancing, and autoscaling configurations. - **Cost Optimization**: Techniques for managing and reducing costs associated with running EKS clusters, including instance selection and resource allocation strategies. By adhering to these best practices, developers and system administrators can ensure that their AWS EKS clusters are robust, secure, and cost-effective, facilitating a smooth and efficient CI/CD pipeline for software development. ## Prerequisites :::note Our approach to deploying the AWS EKS Cluster is based on the widely-used [terraform-aws-eks module](https://github.com/terraform-aws-modules/terraform-aws-eks) from the Terraform AWS Modules community. This module facilitates the creation of AWS Elastic Kubernetes Service (EKS) resources with best practices in mind. We encourage users to review the module's documentation to fully understand its capabilities and how it aligns with the requirements of your specific deployment scenario. ::: Before the EKS cluster deployment and configuration, make sure to check the prerequisites. Install the required tools listed below: - [Git](https://git-scm.com/book/en/v2) - [Terraform](https://www.terraform.io/) - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) - [tfenv](https://github.com/tfutils/tfenv) To check the correct tools installation, run the following commands: ```bash git --version terraform version aws --version tfenv --version ``` :::note Before initiating the deployment, it is crucial to consult the [AWS Service Limits](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) documentation. Please review and adjust these limits as necessary to ensure your deployment proceeds smoothly without hitting service constraints. ::: ## Terraform Backend This step covers the following topics: - Create S3 bucket with policy to store terraform states - Create DynamoDB to support state locking and consistency checking To create the required resources, follow the steps below: 1. Fork and clone git repo with project [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform), rename it in the correspondence with project name: ```bash git clone https://github.com/KubeRocketCI/terraform-aws-platform cd terraform-aws-platform/s3-backend ``` 2. Fill in the input variables for Terraform run in the `s3-backend/template.tfvars` file, refer to the [s3-backend/example.tfvars](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/s3-backend/example.tfvars) as an example: ```tf title="s3-backend/template.tfvars" region = "eu-central-1" tags = { "SysName" = "Terraform-Backend" "SysOwner" = "owner@example.com" "Environment" = "EKS-TEST-CLUSTER" } ``` Find the detailed description of the variables in the [s3-backend/variables.tf](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/s3-backend/variables.tf) file. 3. Initialize the backend and apply the changes: ```bash terraform init terraform apply -var-file=./template.tfvars ``` :::note "View: Terraform output example" ```bash Outputs: terraform_lock_table_dynamodb_id = "terraform_locks" terraform_states_s3_bucket_name = "terraform-states-012345678910" ``` ::: ## AWS IAM Roles This step covers the `KRCIDeployerRole` AWS IAM role creation procedure. To create the role, follow the steps below: 1. Navigate to the IAM module directory: ```bash cd ../iam ``` 2. Fill in the input variables for Terraform run in the `iam/template.tfvars` file. Find the detailed description of the variables in the [iam/variables.tf](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/iam/variables.tf) file: ```tf title="iam/template.tfvars" # -- e.g eu-central-1 region = "eu-central-1" # If you need to set role boundary iam_permissions_boundary_policy_arn = "arn:aws:iam::012345678910:policy/role_boundary" tags = { "SysName" = "KubeRocketCI" "Environment" = "core" "Project" = "my-proj" "ManagedBy" = "terraform" } ``` 3. Initialize and apply the changes: ```bash terraform init terraform apply -var-file=./template.tfvars ``` :::note "View: Terraform output example" ```bash Outputs: deployer_iam_role_arn = "arn:aws:iam::012345678910:role/KRCIDeployerRole" deployer_iam_role_name = "KRCIDeployerRole" ``` ::: ## AWS VPC Configuration (Optional) This step will cover the following topics: - Create the AWS VPC - Create the AWS VPC Subnets for instances and AWS ALB - Create the AWS VPC Routing To accomplish the tasks outlined above, follow these steps: 1. Navigate to the VPC module directory: ```bash cd ../vpc ``` 2. Set up the backend for store Terraform states remotely and support state locking and consistency checking via DynamoDB. Insert the missing fields in the file `vpc/versions.tf`: ```tf title="vpc/versions.tf" ... backend "s3" { bucket = "terraform-states-012345678910" key = "eu-central-1/test/vpc/terraform.tfstate" region = "eu-central-1" acl = "bucket-owner-full-control" dynamodb_table = "terraform_locks" encrypt = true role_arn = "arn:aws:iam:::role/KRCIDeployerRole" } ... ``` 3. Fill in the input variables for Terraform run in the `vpc/template.tfvars` file. Use the [vpc/example.tfvars](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/vpc/example.tfvars) as an example. Please find the detailed description of the variables in the [vpc/variables.tf](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/vpc/variables.tf) file: ```tf title="vpc/template.tfvars" region = "eu-central-1" role_arn = "arn:aws:iam::012345678910:role/KRCIDeployerRole" platform_name = "test" platform_cidr = "192.168.0.0/20" subnet_azs = ["eu-central-1a", "eu-central-1b", "eu-central-1c"] private_cidrs = ["192.168.0.0/22", "192.168.4.0/22", "192.168.8.0/22"] public_cidrs = ["192.168.12.0/24", "192.168.13.0/24", "192.168.14.0/24"] tags = { "SysName" = "Terraform-Backend" "SysOwner" = "owner@example.com" "Environment" = "EKS-TEST-CLUSTER" } ``` 4. Initialize the backend and apply the changes: ```bash terraform init terraform apply -var-file=./template.tfvars ``` :::note "View: Terraform output example" ```bash Outputs: private_subnets = [ "subnet-012345678910", "subnet-012345678910", "subnet-012345678910", ] public_subnets = [ "subnet-012345678910", "subnet-012345678910", "subnet-012345678910", ] vpc_id = "vpc-012345678910" ``` ::: ## Deploy and Pre-configure AWS EKS This step will cover the following topics: - Create the EKS Cluster - Create the AWS ASGs for the EKS Cluster - Create the AWS ALB - (Optional) Create the Kaniko AWS IAM Role - (Optional) Create the Atlantis AWS IAM Role To accomplish the tasks outlined above, follow these steps: 1. Navigate to the EKS module directory: ```bash cd ../eks ``` 2. Set up the backend for store Terraform states remotely and support state locking and consistency checking via DynamoDB. Insert the missing fields in the `eks/versions.tf` file: ```tf title="eks/versions.tf" ... backend "s3" { bucket = "terraform-states-012345678910" key = "eu-central-1/test/eks/terraform.tfstate" region = "eu-central-1" acl = "bucket-owner-full-control" dynamodb_table = "terraform_locks" encrypt = true role_arn = "arn:aws:iam:::role/KRCIDeployerRole" } ... ``` 3. Fill in the input variables for Terraform run in the `eks/template.tfvars` file. Use the [eks/example.tfvars](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/eks/example.tfvars) as an example. Please find the detailed description of the variables in the [eks/variables.tf](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/eks/variables.tf) file: ```tf title="eks/versions.tf" region = "eu-central-1" platform_name = "test" platform_domain_name = "example.com" role_arn = "arn:aws:iam::012345678910:role/KRCIDeployerRole" role_permissions_boundary_arn = "arn:aws:iam::012345678910:policy/eo_role_boundary" vpc_id = "vpc-012345678910" private_subnets_id = ["subnet-012345678910", "subnet-012345678910", "subnet-012345678910"] public_subnets_id = ["subnet-012345678910", "subnet-012345678910", "subnet-012345678910"] tags = { "SysName" = "Terraform-Backend" "SysOwner" = "owner@example.com" "Environment" = "EKS-TEST-CLUSTER" } ``` 4. (Optional) Create the Kaniko AWS IAM Role: If AWS Elastic Container Registry (ECR) will be used as the [Container Registry](../user-guide/manage-container-registries.md) in the KubeRocketCI platform, it is necessary to create a specific IAM role for Kaniko. To configure the Kaniko AWS IAM Role, set the `create_kaniko_iam_role` variable to `true` in the `eks/template.tfvars` configuration file: ```tf title="eks/template.tfvars" create_kaniko_iam_role = true ``` The Kaniko AWS IAM Role will be created as part of the EKS cluster provisioning process. Once the EKS cluster and the Kaniko AWS IAM Role are successfully created, you can use the resulting Amazon Resource Name (ARN) in the `edp-tekton.kaniko.roleArn` field within the `values.yaml` file during the [KubeRocketCI installation](install-kuberocketci.md). 5. (Optional) Create the Atlantis AWS IAM Role: If Atlantis will be used for automating Terraform workflows in Kubernetes environments, it is necessary to create a specific IAM role for Atlantis. To create the Atlantis AWS IAM Role, set the `create_atlantis_iam_role` variable to `true` in the `eks/template.tfvars` configuration file: ```tf title="eks/template.tfvars" create_atlantis_iam_role = true ``` The Atlantis AWS IAM Role will be created as part of the EKS cluster provisioning process. Once the EKS cluster and the Atlantis AWS IAM Role are successfully created, you can proceed with the [Atlantis installation](./infrastructure-providers/atlantis-installation.md) and use the resulting Amazon Resource Name (ARN) in the `atlantis.serviceAccount.annotations` field within the `values.yaml` file for the Atlantis Helm chart. 6. Initialize the backend and apply the changes: ```bash terraform init terraform apply -var-file=./template.tfvars ``` 7. Update local Kubernetes configuration. After the EKS cluster is successfully deployed, update the local Kubernetes configuration to interact with the cluster: ```bash aws eks update-kubeconfig --region --name ``` 8. Once AWS EKS Cluster is successfully deployed, you can navigate to our [KubeRocketCI addons](add-ons-overview.md) to install and manage cluster applications using the GitOps approach. ## Argo CD Configuration (Optional) This section covers configuring Argo CD for further integrating with an external EKS cluster. This integration necessitates creating two AWS IAM roles: - `AWSIRSA_Test_ArgoCDMaster` AWS IAM role - for setting up IRSA annotations for the Argo CD application - `EDPArgoCDClusterAdmin` AWS IAM role - for working with external EKS clusters and further adding them (please refer to the [Add Cluster](../user-guide/add-cluster.md) page for details). ### Argo CD IAM Role for Shared EKS Cluster This step covers the `AWSIRSA_Test_ArgoCDMaster` AWS IAM role creation procedure. To create the role, take the following steps: 1. Navigate to the Argo CD module directory: ```bash cd ../argo-cd ``` 2. Set up the backend for store Terraform states remotely and support state locking and consistency checking via DynamoDB. Insert the missing fields in the `argo-cd/versions.tf` file: ```tf title="argo-cd/versions.tf" ... backend "s3" { bucket = "terraform-states-012345678910" key = "eu-central-1/test/argo-cd/terraform.tfstate" region = "eu-central-1" acl = "bucket-owner-full-control" dynamodb_table = "terraform_locks" encrypt = true role_arn = "arn:aws:iam:::role/KRCIDeployerRole" } ... ``` 3. Fill in the input variables for Terraform run in the `argo-cd/template.tfvars` file. Refer to the [argo-cd/example.tfvars](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/argo-cd/example.tfvars) as an example. Please find the detailed description of the variables in the [argo-cd/variables.tf](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/argo-cd/variables.tf) file: ```tf title="argo-cd/template.tfvars" argocd_master_enabled = true argocd_master_role_name_list = [ "arn:aws:iam::012345678910:role/EDPArgoCDClusterAdmin", ] oidc_provider_arn = "arn:aws:iam::012345678910:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/9876543210" platform_name = "test" region = "eu-central-1" role_arn = "arn:aws:iam::012345678910:role/KRCIDeployerRole" role_permissions_boundary_arn = "arn:aws:iam::012345678910:policy/eo_role_boundary" tags = { "SysName" = "Terraform-Backend" "SysOwner" = "owner@example.com" "Environment" = "EKS-TEST-CLUSTER" } ``` 4. Initialize the backend and apply the changes: ```bash terraform init terraform apply ``` :::note "View: Terraform output example" ```bash Outputs: argocd_agent_role_iam_role_arn = "" argocd_irsa_iam_role_arn = "arn:aws:iam::012345678910:role/AWSIRSA_Test_ArgoCDMaster" ``` ::: 5. Once `AWSIRSA_Test_ArgoCDMaster` AWS IAM Role is successfully created, navigate to the [Cluster addons](add-ons-overview.md) to set up IRSA annotations for the Argo CD application: ```yaml title="add-ons/argo-cd/values.yaml" argo-cd: controller: serviceAccount: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::012345678910:role/AWSIRSA_Test_ArgoCDMaster" server: serviceAccount: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::012345678910:role/AWSIRSA_Test_ArgoCDMaster" ``` ### Argo CD IAM Role for External EKS Cluster This step covers the `EDPArgoCDClusterAdmin` AWS IAM role creation procedure. To create the role, take the following steps: 1. Navigate to the Argo CD module directory: ```bash cd ../argo-cd ``` 2. Set up the backend for store Terraform states remotely and support state locking and consistency checking via DynamoDB. Insert the missing fields in the `argo-cd/versions.tf` file: ```tf title="argo-cd/versions.tf" ... backend "s3" { bucket = "terraform-states-012345678910" key = "eu-central-1/test/argo-cd/terraform.tfstate" region = "eu-central-1" acl = "bucket-owner-full-control" dynamodb_table = "terraform_locks" encrypt = true role_arn = "arn:aws:iam:::role/KRCIDeployerRole" } ... ``` 3. Fill in the input variables for Terraform run in the `argo-cd/template.tfvars` file. Refer to the [argo-cd/example.tfvars](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/argo-cd/example.tfvars) as an example. Please find the detailed description of the variables in the [argo-cd/variables.tf](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/argo-cd/variables.tf) file: ```tf title="argo-cd/template.tfvars" argocd_agent_enabled = true argocd_agent_argocd_master_role_arn = "arn:aws:iam::012345678910:role/AWSIRSA_Test_ArgoCDMaster" platform_name = "test" region = "eu-central-1" role_arn = "arn:aws:iam::012345678910:role/KRCIDeployerRole" role_permissions_boundary_arn = "arn:aws:iam::012345678910:policy/eo_role_boundary" tags = { "SysName" = "Terraform-Backend" "SysOwner" = "owner@example.com" "Environment" = "EKS-TEST-CLUSTER" } ``` 4. Initialize the backend and apply the changes: ```bash terraform init terraform apply ``` :::note "View: Terraform output example" ```bash Outputs: argocd_agent_role_iam_role_arn = "arn:aws:iam::012345678910:role/EDPArgoCDClusterAdmin" argocd_irsa_iam_role_arn = "" ``` ::: ## Next Step After the AWS EKS cluster is installed and configured, you can proceed with installing prerequisites for KubeRocketCI, such as [Tekton](../operator-guide/install-tekton.md) and [Argo CD](install-argocd.md). ## Related Articles * [Cluster Add-Ons Overview](add-ons-overview.md) * [Install KubeRocketCI](install-kuberocketci.md) --- ## Install and Integrate DefectDojo # Integrate DefectDojo Install DefectDojo on Kubernetes with Helm and integrate it into KubeRocketCI pipelines for vulnerability management. Once installed, you can enable single sign-on by following the [DefectDojo Microsoft Entra OIDC SSO Setup](../microsoft-entra/defectdojo-oidc-authentication.md) guide. :::info It is also possible to install DefectDojo using the add-ons approach. For details, please refer to the [KubeRocketCI addons approach](https://github.com/epam/edp-cluster-add-ons). ::: :::note For details on how KubeRocketCI pipelines report findings to DefectDojo, including the data model, engagement naming, and scan types, refer to [Security Scanning Pipelines](./security-pipelines.md). ::: ## Prerequisites - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation :::info Please refer to the [DefectDojo Helm Chart](https://github.com/DefectDojo/django-DefectDojo/tree/master/helm/defectdojo) and [Deploy DefectDojo into the Kubernetes cluster](https://github.com/DefectDojo/django-DefectDojo/blob/dev/readme-docs/KUBERNETES.md) sections for details. ::: ## Installation DefectDojo Cluster Add-On Approach To install DefectDojo in environment, it's recommended to use the resources provided in the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons) repository. Leveraging the Cluster Add-Ons simplifies the deployment and management process, providing a streamlined method to integrate DefectDojo into infrastructure. 1. Navigate to the forked cluster Add-Ons repository and align the [defectdojo values.yaml](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/defectdojo/values.yaml) file. Follow the instructions to deploy DefectDojo. 2. Update the configuration to enable **DefectDojo** in [Add-Ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml) repository: ```yaml title="clusters/core/apps/values.yaml" defectdojo: createNamespace: true enable: true ``` 3. Sync resources and wait till the all **DefectDojo** resources is created: ![DefectDojo ArgoCD applications](../../assets/operator-guide/devsecops/defectdojo-addons-deploy.png "DefectDojo ArgoCD applications") ## Installation DefectDojo Helm Chart Approach 1. Check that a security namespace is created. If not, run the following command to create it: ```bash kubectl create namespace defectdojo ``` :::warning For the OpenShift users When using the OpenShift platform, install the `SecurityContextConstraints` resource. In case of using a custom namespace for `defectdojo`, change the namespace in the `users` section.
View: defectdojo-scc.yaml ```yaml allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: null apiVersion: security.openshift.io/v1 allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - min: 999 max: 65543 groups: [] kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: defectdojo priority: 1 readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsRange uidRangeMin: 1 uidRangeMax: 65543 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: - system:serviceaccount:defectdojo:defectdojo - system:serviceaccount:defectdojo:defectdojo-rabbitmq - system:serviceaccount:defectdojo:default volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
::: 2. Add a chart repository: ```bash helm repo add defectdojo 'https://raw.githubusercontent.com/DefectDojo/django-DefectDojo/helm-charts' helm repo update ``` 3. Create PostgreSQL admin secret: ```bash kubectl -n defectdojo create secret generic defectdojo-postgresql-specific \ --from-literal=postgresql-password= \ --from-literal=postgresql-postgres-password= ``` :::note The `postgresql_password` and `postgresql_postgres_password` passwords must be 16 characters long. ::: 4. Create Rabbitmq admin secret: ```bash kubectl -n defectdojo create secret generic defectdojo-rabbitmq-specific \ --from-literal=rabbitmq-password= \ --from-literal=rabbitmq-erlang-cookie= ``` :::note The `rabbitmq_password` password must be 10 characters long. The `rabbitmq_erlang_cookie` password must be 32 characters long. ::: 5. Create DefectDojo admin secret: ```bash kubectl -n defectdojo create secret generic defectdojo \ --from-literal=DD_ADMIN_PASSWORD= \ --from-literal=DD_SECRET_KEY= \ --from-literal=DD_CREDENTIAL_AES_256_KEY= \ --from-literal=METRICS_HTTP_AUTH_PASSWORD= ``` :::note The `dd_admin_password` password must be 22 characters long. The `dd_secret_key` password must be 128 characters long. The `dd_credential_aes_256_key` password must be 128 characters long. The `metric_http_auth_password` password must be 32 characters long. ::: 6. Install DefectDojo v.2.22.4 using [defectdojo/defectdojo](https://github.com/DefectDojo/django-DefectDojo/tree/master/helm/defectdojo) Helm chart v.1.6.69: ```bash helm upgrade --install \ defectdojo \ --version 1.6.69 \ defectdojo/defectdojo \ --namespace defectdojo \ --values values.yaml ``` Check out the *values.yaml* file sample of the DefectDojo customization:
View: values.yaml ```yaml tag: 2.22.4 fullnameOverride: defectdojo host: defectdojo. site_url: https://defectdojo. alternativeHosts: - defectdojo-django.defectdojo initializer: # should be false after initial installation was performed run: true django: ingress: enabled: true # change to 'false' for OpenShift activateTLS: false uwsgi: livenessProbe: # Enable liveness checks on uwsgi container. Those values are use on nginx readiness checks as well. # default value is 120, so in our case 20 is just fine initialDelaySeconds: 20 ```
7. For the OpenShift platform, install a Route:
View: defectdojo-route.yaml ```yaml kind: Route apiVersion: route.openshift.io/v1 metadata: name: defectdojo namespace: defectdojo spec: host: defectdojo. path: / tls: insecureEdgeTerminationPolicy: Redirect termination: edge to: kind: Service name: defectdojo-django port: targetPort: http wildcardPolicy: None ```
## Configuration DefectDojo ci-user To prepare DefectDojo for integration with KubeRocketCI, follow the steps below: 1. Create ci user in DefectDojo UI: - Login to DefectDojo UI using `admin` credentials: ```bash echo "DefectDojo admin password: $(kubectl \ get secret defectdojo \ --namespace=defectdojo \ --output jsonpath='{.data.DD_ADMIN_PASSWORD}' \ | base64 --decode)" ``` - Go to User section - Create new user Global role `Maintainer` permission: ![DefectDojo update manual secret](../../assets/operator-guide/devsecops/defectdojo-createuser.png "DefectDojo set user permission") 2. Get a token of the DefectDojo user: - Login to the DefectDojo UI using the credentials from previous steps. - Click on profile button select the `API v2 key`. - Copy the API key. ![DefectDojo api key](../../assets/operator-guide/devsecops/defectdojo-api-key.png "DefectDojo api-key") ## Integration DefectDojo with KubeRocketCI Provision the secret using `KubeRocketCI portal`, `Manifest` or with the `externalSecrets` operator: Go to the **Portal** open **Configuration** -> **SECURITY** -> **DEFECTDOJO**. Update or click **+ ADD INTEGRATION** fill fields `Quick Link URL`, `URL` and `Token` click the `save` button. ![DefectDojo update manual secret](../../assets/operator-guide/devsecops/defectdojo-token.png "DefectDojo update manual secret") ```yaml apiVersion: v1 kind: Secret metadata: name: ci-defectdojo namespace: krci labels: app.edp.epam.com/secret-type: defectdojo app.edp.epam.com/integration-secret: "true" stringData: url: https://defectdojo.example.com token: ``` Store defectdojo URL and Token in AWS Parameter Store with following format: ```json "ci-defectdojo": { "url": "https://defectdojo.example.com", "token": "XXXXXXXXXXXX" } ``` Go to the **Portal** open **Configuration** -> **SECURITY** -> **DefectDojo** and see the `Managed by External Secret` message. ![DefectDojo managed by external secret operator](../../assets/operator-guide/devsecops/defectdojo-external-secret.png "DefectDojo managed by external secret operator") More details about the External Secrets Operator integration procedure can be found in the [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) page. After following the instructions provided, you should be able to integrate your DefectDojo with the KubeRocketCI using one of the available scenarios. ## Related Articles * [Install External Secrets Operator](../secrets-management/install-external-secrets-operator.md) * [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) * [Install Harbor](../artifacts-management/harbor-installation.md) * [DefectDojo Microsoft Entra OIDC SSO Setup](../microsoft-entra/defectdojo-oidc-authentication.md) --- ## Integrate Dependency-Track # Integrate Dependency-Track This documentation guide provides comprehensive instructions for installing and integrating Dependency-Track with the KubeRocketCI. :::note For details on how KubeRocketCI pipelines generate SBOMs with cdxgen and upload them to Dependency-Track, refer to [Security Scanning Pipelines](./security-pipelines.md). ::: ## Prerequisites - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation To install Dependency-Track use [KubeRocketCI addons](../add-ons-overview.md) approach. ## Configuration 1. Open **Administration** -> **Access Management** -> **Teams**. Click **Create Team** -> **Automation** and click **Create**. 2. Click **+** in **Permissions** and add: ```bash BOM_UPLOAD PROJECT_CREATION_UPLOAD VIEW_PORTFOLIO ``` 3. Click **+** in **API keys** to create token: ![Dependency-Track settings](../../assets/operator-guide/devsecops/dependency-track-token.png "Dependency-Track settings") 4. Provision secrets using a manifest, Portal, or with the externalSecrets operator: Go to the **Portal** open **Configuration** -> **SECURITY** -> **DEPENDENCYTRACK**. Click **+ ADD INTEGRATION** fill fields `Quick Link URL`, `URL` and `Token` click the `save` button. ![Dependency-Track update manual secret](../../assets/operator-guide/devsecops/dependency-track-secret.png "Dependency-Track update manual secret") ```yaml apiVersion: v1 kind: Secret metadata: name: ci-dependency-track namespace: krci labels: app.edp.epam.com/secret-type: dependency-track app.edp.epam.com/integration-secret: "true" stringData: token: url: type: Opaque ``` Store Dependency-Track URL and Token in the AWS Parameter Store with the following format: ```yaml "ci-dependency-track": { "token": "XXXXXXXXXXXX", "url": "https://dependency-track.example.com" } ``` Go to the **KubeRocketCI Portal** -> **Configuration** -> **Security** -> **DependencyTrack** see the secret managed by the `Managed by External Secret`: ![Dependency-Track managed by external secret operator](../../assets/operator-guide/devsecops/defectdojo-external-secret.png "Dependency-Track managed by external secret operator") More detail on External Secrets Operator Integration can be found on [the following page](../secrets-management/external-secrets-operator-integration.md) After following the instructions provided, you should be able to integrate your Dependency-Track with the KubeRocketCI. ## Related Articles * [Install External Secrets Operator](../secrets-management/install-external-secrets-operator.md) * [External Secrets Operator Integration](../secrets-management/external-secrets-operator-integration.md) * [Cluster Add-Ons Overview](../add-ons-overview.md) * [Dependency-Track Microsoft Entra OIDC SSO](../microsoft-entra/dependency-track-authentication.md) --- ## Security and Compliance: Overview # Security and Compliance: Overview KubeRocketCI emphasizes the importance of incorporating security practices into the software development lifecycle through the DevSecOps approach. By integrating a diverse range of open-source and enterprise security tools tailored to specific functionalities, organizations can ensure efficient and secure software development. These tools, combined with fundamental DevSecOps principles such as collaboration, continuous security, and automation, contribute to the identification and remediation of vulnerabilities early in the process, minimizes risks, and fosters a security-first culture across the organization. ## Integrated Scanners The following tools are built into KubeRocketCI Tekton pipelines and run automatically as part of CI/CD workflows. For detailed information on when each scanner runs, how reports are produced, and how data flows into DefectDojo and Dependency-Track, refer to the [Security Scanning Pipelines](./security-pipelines.md) page. | Tool | Category | Runs In | Results Destination | |:----:|----------|---------|---------------------| | [SonarQube](https://www.sonarqube.org/) | SAST + Code Quality | Build and Review pipelines | [SonarQube server](../code-quality/sonarqube.md) | | [Semgrep](https://github.com/semgrep/semgrep) | SAST | Security-Scan pipeline | [DefectDojo](./defectdojo.md) | | [Gitleaks](https://github.com/gitleaks/gitleaks) | Secret Detection | Security-Scan pipeline | [DefectDojo](./defectdojo.md) | | [CycloneDX cdxgen](https://github.com/CycloneDX/cdxgen) | SCA / SBOM Generation | Security-Scan pipeline | [Dependency-Track](./dependency-track.md) | | [Trivy](https://github.com/aquasecurity/trivy) | Container Vulnerability Scan | Security-Scan, Image-Scan-Remote pipelines | [DefectDojo](./defectdojo.md) | | [Anchore Grype](https://github.com/anchore/grype) | Container Vulnerability Scan | Security-Scan pipeline | [DefectDojo](./defectdojo.md) | | [Hadolint](https://github.com/hadolint/hadolint) | Dockerfile Linting | Review pipelines | Pipeline log | | [Helm chart-testing](https://github.com/helm/chart-testing) | Helm Chart Linting | Review pipelines | Pipeline log | ## Required Integrations To enable the full security scanning stack, provision the following integration secrets. Each secret is optional — if missing, the corresponding scanning step is skipped gracefully, allowing incremental tool adoption. | Secret | Keys | Integration Guide | |:------:|------|:-----------------:| | `ci-sonarqube` | `token`, `url` | [SonarQube Integration](../code-quality/sonarqube.md) | | `ci-defectdojo` | `token`, `url` | [Integrate DefectDojo](./defectdojo.md) | | `ci-dependency-track` | `token`, `url` | [Integrate Dependency-Track](./dependency-track.md) | ## Supported Solutions Catalog Beyond the tools integrated into default pipelines, KubeRocketCI supports the integration of additional open-source and enterprise security tools for organizations with extended security requirements. The table below provides a comprehensive view of available options for each security aspect. | Functionality | Open-Source Tools (integrated in Pipelines) | Enterprise Tools (available for Integration) | |:--------------------------------------:|---------------------------------------------|--------------------------------------------------------| | Hardcoded Credentials Scanner | TruffleHog, GitLeaks, Git-secrets | GitGuardian, SpectralOps, Bridgecrew | | Static Application Security Testing | SonarQube, Semgrep CLI | Veracode, Checkmarx, Coverity | | Software Composition Analysis | OWASP Dependency-Check, cdxgen | Black Duck Hub, Mend, Snyk | | Container Security | Trivy, Grype, Clair | Aqua Security, Sysdig Secure, Snyk | | Infrastructure as Code Security | Checkov, Tfsec | Bridgecrew, Prisma Cloud, Snyk | | Dynamic Application Security Testing | OWASP Zed Attack Proxy | Fortify WebInspect, Rapid7 InsightAppSec, Checkmarx | | Continuous Monitoring and Logging | ELK Stack, OpenSearch, Loki | Splunk, Datadog | | Security Audits and Assessments | OpenVAS | Tenable Nessus, QualysGuard, BurpSuite Professional | | Vulnerability Management and Reporting | DefectDojo, OWASP Dependency-Track | Metasploit | ![Security tools in KubeRocketCI](../../assets/operator-guide/security-tools.png "Security tools in KubeRocketCI") ## Tool Descriptions ### Vulnerability Management [DefectDojo](https://www.defectdojo.com/) is a comprehensive vulnerability management and security orchestration platform facilitating the handling of uploaded security reports. Examine the prerequisites and fundamental instructions for [installing DefectDojo](./defectdojo.md) on Kubernetes or OpenShift platforms. [OWASP Dependency-Track](https://dependencytrack.org/) is an intelligent Software Composition Analysis (SCA) platform that provides a comprehensive solution for managing vulnerabilities in third-party and open-source components. See the [integration guide](./dependency-track.md) for setup instructions. ### Source Code Security [Semgrep CLI](https://github.com/semgrep/semgrep) is a versatile and user-friendly command-line interface for the Semgrep security scanner, enabling developers to perform Static Application Security Testing (SAST) for various programming languages. It focuses on detecting and preventing potential security vulnerabilities, code quality issues, and custom anti-patterns. [Gitleaks](https://github.com/gitleaks/gitleaks) is a versatile SAST tool used to scan Git repositories for hardcoded secrets, such as passwords and API keys, to prevent potential data leaks and unauthorized access. [TruffleHog](https://github.com/trufflesecurity/trufflehog) is an open-source tool designed for finding and identifying potentially sensitive and secret information in the source code and commit history of Git repositories. [Git-secrets](https://github.com/awslabs/git-secrets) is an open-source tool that helps prevent the accidental committing of secrets, sensitive information, and other types of confidential data into Git repositories. ### Container Security [Trivy](https://github.com/aquasecurity/trivy) is a simple and comprehensive vulnerability scanner for containers and other artifacts, providing insight into potential security issues across multiple ecosystems. Trivy can be seamlessly integrated into CI/CD pipelines or utilized as part of Harbor. [Grype](https://github.com/anchore/grype) is a fast and reliable vulnerability scanner for container images and filesystems, maintaining an up-to-date vulnerability database for efficient and accurate scanning. [Clair](https://github.com/quay/clair) is an open-source container security tool designed to help assess the security of container images and identify vulnerabilities within them. ### Software Composition Analysis [Cdxgen](https://github.com/CycloneDX/cdxgen) is a lightweight and efficient tool for generating Software Bill of Materials (SBOM) using CycloneDX, a standard format for managing component inventory. It helps organizations maintain an up-to-date record of all software components, their versions, and related vulnerabilities. [OWASP Dependency-Check](https://owasp.org/www-project-dependency-check/) is a software composition analysis tool that helps identify and report known security vulnerabilities in project dependencies. ### Infrastructure as Code Security [Tfsec](https://github.com/aquasecurity/tfsec) is an effective Infrastructure as Code (IaC) security scanner, tailored specifically for reviewing Terraform templates. It helps identify potential security issues related to misconfigurations and non-compliant practices. [Checkov](https://github.com/bridgecrewio/checkov) is a robust static code analysis tool designed for IaC security, supporting various IaC frameworks such as Terraform, CloudFormation, and Kubernetes. It assists in detecting and mitigating security and compliance misconfigurations. ### Dynamic Application Security Testing [OWASP Zed Attack Proxy (ZAP)](https://www.zaproxy.org/) is a security testing tool for finding vulnerabilities in web applications during the development and testing phases. ### Monitoring and Logging [ELK Stack](../monitoring-and-observability/kibana-ilm-rollover.md) (Fluent Bit, Elasticsearch, Kibana) is used in Kubernetes for log aggregation, supporting Logsight for Stage Verification and Incident Detection. [Loki](https://github.com/Neo23x0/Loki) is a log aggregation system designed for cloud-native environments. It is part of the CNCF and is often used alongside Prometheus for monitoring and observability. [OpenSearch](https://opensearch.org/) is the flexible, scalable, open-source way to build solutions for data-intensive applications. ### Security Audits [OpenVAS](https://openvas.org/) is an open-source network vulnerability scanner and security management tool designed to identify and assess security vulnerabilities in computer systems, networks, and applications. ## Related Articles * [Security Scanning Pipelines](./security-pipelines.md) * [Integrate DefectDojo](./defectdojo.md) * [Integrate Dependency-Track](./dependency-track.md) * [SonarQube Integration](../code-quality/sonarqube.md) --- ## Security Scanning Pipelines # Security Scanning Pipelines KubeRocketCI provides automated security scanning at three levels of the software delivery lifecycle: source code, dependencies, and container images. Scanners are embedded into Tekton pipelines and report findings to [DefectDojo](./defectdojo.md), [Dependency-Track](./dependency-track.md), and [SonarQube](../code-quality/sonarqube.md). | Tool | Category | Report Destination | DefectDojo Scan Type | |:----:|----------|--------------------|-----------------------| | [Semgrep](https://github.com/semgrep/semgrep) | SAST | DefectDojo | `Semgrep JSON Report` | | [Gitleaks](https://github.com/gitleaks/gitleaks) | Secret Detection | DefectDojo | `Gitleaks Scan` | | [CycloneDX cdxgen](https://github.com/CycloneDX/cdxgen) | SCA / SBOM | Dependency-Track | N/A (direct API upload) | | [SonarQube](https://www.sonarqube.org/) | SAST + Code Quality | SonarQube server | N/A | | [Trivy](https://github.com/aquasecurity/trivy) | Container Vulnerability Scan | DefectDojo | `Trivy Scan` | | [Anchore Grype](https://github.com/anchore/grype) | Container Vulnerability Scan | DefectDojo | `Anchore Grype` | | [Hadolint](https://github.com/hadolint/hadolint) | Dockerfile Linting | Pipeline log | N/A | | [Helm chart-testing](https://github.com/helm/chart-testing) | Helm Chart Linting | Pipeline log | N/A | ## Pipeline Types Security scanning is distributed across three pipeline types. Each serves a different stage of the development workflow. ### Build and Review Pipelines Build and review pipelines are language-specific and run for every codebase. They include [SonarQube](../code-quality/sonarqube.md) for code quality and security analysis, [Hadolint](https://github.com/hadolint/hadolint) for Dockerfile validation, and [Helm chart-testing](https://github.com/helm/chart-testing) for chart linting. **Build pipeline** — triggered on merge to a tracked branch: ```mermaid flowchart LR A[fetch-repo] --> B[get-version] --> C[get-cache] --> D[compile] --> E[test] E --> F[sonar] --> G[build] --> H[push] --> I[container-build] I --> J[save-cache] --> K[git-tag] --> L[update-cbis] L -. finally .-> M[report status] ``` **Review pipeline** — triggered on PR/MR creation or update: ```mermaid flowchart LR A[fetch-repo] --> B[get-cache] --> C[compile] --> D[test] D --> E[sonar] --> F[build] --> G[save-cache] A --> H[dockerfile-lint] --> I[dockerbuild-verify] A --> J[helm-lint] G -. finally .-> K[set review status] ``` :::info For details on SonarQube configuration and Quality Gate behavior, refer to the [SonarQube Integration](../code-quality/sonarqube.md) page. ::: ### Security-Scan Pipeline A dedicated pipeline that runs comprehensive security analysis independently from build/review. Available for all supported VCS providers (GitHub, GitLab, Gerrit, Bitbucket). Triggered via webhook or manually through the KubeRocketCI portal. After cloning the repository, two tasks execute **in parallel**: ```mermaid flowchart LR A[fetch-repo] --> B["security Semgrep + Gitleaks + cdxgen"] A --> C["image-scan-chart Trivy + Grype"] B --> D["DefectDojo + Dependency-Track"] C --> E[DefectDojo] ``` The pipeline returns two result URLs visible in the portal: - `SCAN_REPORT_URL` — DefectDojo link to code security findings (Semgrep + Gitleaks) - `IMAGE_SCAN_REPORT_URL` — DefectDojo link to container image findings (Trivy + Grype) ### Image-Scan-Remote Pipeline An on-demand pipeline for scanning pre-built container images from remote registries without source code. Accepts an array of fully-qualified image references: ```yaml params: - name: IMAGE_NAMES value: - "registry.example.com/myapp/backend:1.2.3" - "registry.example.com/myapp/frontend:1.2.3" - name: CODEBASE_NAME value: "myapp" ``` Uses Trivy only (no Grype) and uploads results to DefectDojo with per-image engagements. ### Scanner-to-Pipeline Matrix | Scanner | Build Pipeline | Review Pipeline | Security-Scan Pipeline | Image-Scan-Remote | |:-------:|:-:|:-:|:-:|:-:| | SonarQube | Yes | Yes (PR mode) | - | - | | Semgrep | - | - | Yes | - | | Gitleaks | - | - | Yes | - | | cdxgen (SBOM) | - | - | Yes | - | | Trivy | - | - | Yes | Yes | | Grype | - | - | Yes | - | | Hadolint | - | Yes | - | - | | Helm Lint | - | Yes | - | - | ## Scanner Reference This section describes each integrated scanner: what it detects, the command it runs, and where results are reported. ### Source Code Scanners #### Semgrep (SAST) Semgrep performs Static Application Security Testing across the entire source tree. It runs as the first step of the `security` task in the security-scan pipeline. **Command:** ```bash semgrep --jobs 1 --config=auto . --json \ --output semgrep-report.json --disable-version-check ``` Key behaviors: - Uses `--config=auto` which downloads community and recommended rules from the Semgrep registry, selecting rules based on detected languages and frameworks. - Runs single-threaded (`--jobs 1`) for stability in container environments. - Excludes `.docker/config.json` via a `.semgrepignore` file created at scan time. - Produces a JSON report consumed by the DefectDojo upload step. - Does **not** fail the pipeline on findings — results go to DefectDojo for triage. **DefectDojo mapping:** scan type = `Semgrep JSON Report`, engagement = `code-security-{branch}`. #### Gitleaks (Secret Detection) Gitleaks detects hardcoded secrets, API keys, passwords, and tokens in the source code. It runs as the second step of the `security` task. **Command:** ```bash gitleaks detect --source . --report-format=json \ --report-path=gitleaks-report.json \ --no-git --verbose --exit-code=0 --config=gitleaks.toml ``` Key behaviors: - `--no-git` scans file content only (not git history), which is faster and avoids needing a full git clone. - `--exit-code=0` ensures the pipeline does not fail on findings — all results are sent to DefectDojo. - Uses a custom `gitleaks.toml` config that excludes the `.docker/` directory. **DefectDojo mapping:** scan type = `Gitleaks Scan`, engagement = `code-security-{branch}`. #### CycloneDX cdxgen (SCA / SBOM) [CycloneDX cdxgen](https://github.com/CycloneDX/cdxgen) generates a Software Bill of Materials (SBOM) from the project source and uploads it directly to [Dependency-Track](./dependency-track.md). It runs as the third step of the `security` task and is also available as a standalone `cdxgen` Tekton task. **Command:** ```bash /opt/cdxgen/bin/cdxgen.js \ --api-key=$API_TOKEN \ --server-url=$DEPTRACK_URL \ --project-name=$PROJECT_NAME \ --project-version=$PROJECT_BRANCH \ --print ``` Key behaviors: - Automatically detects the project type (Maven, npm, pip, Go, etc.) and generates an appropriate SBOM. - Uploads the SBOM directly to Dependency-Track via its API (not through DefectDojo). - `--project-name` maps to the KubeRocketCI Codebase name (e.g., `my-java-app`). - `--project-version` maps to the git branch being scanned (e.g., `main`, `release/1.0`). - If the `ci-dependency-track` secret is missing or empty, the step is skipped gracefully. After upload, Dependency-Track provides: - Full dependency tree visualization - Known vulnerability (CVE) matching against NVD and other feeds - License risk analysis (GPL, Apache, MIT compliance) - Policy evaluation (banned components, outdated versions) - Audit workflow for triaging findings - Project-level risk scoring #### SonarQube (Code Quality and Security) SonarQube scanning is embedded in every build and review pipeline. It serves a dual purpose: code quality enforcement and security vulnerability detection. SonarQube is the **only scanner that acts as a hard gate** — if the Quality Gate fails, the pipeline fails. SonarQube provides language-specific task variants: | Task Name | Language | Build Tool | |-----------|----------|------------| | `sonarqube-maven` | Java (Maven) | Maven plugin (`sonar:sonar`) | | `sonarqube-gradle` | Java (Gradle) | Gradle plugin (`sonarqube`) | | `sonarqube-general` | Go, Python, JS/TS | Standalone `sonar-scanner-cli` | | `sonarqube-dotnet` | C# / .NET | `dotnet sonarscanner` | Each task performs three phases: 1. **Project setup** — verifies SonarQube availability, auto-creates the project if missing (using the Codebase's default branch). 2. **Analysis** — runs the scanner with branch-mode parameters (build pipeline) or pull-request-mode parameters (review pipeline). 3. **Quality Gate** — with `sonar.qualitygate.wait=true`, the scanner polls SonarQube until evaluation completes. A failed Quality Gate blocks artifact publishing (build) or sets a failing PR status (review). SonarQube detects bugs, security vulnerabilities, security hotspots, code smells, coverage gaps, and code duplications. :::info For installation, configuration, and integration instructions, refer to the [SonarQube Integration](../code-quality/sonarqube.md) page. ::: ### Container Image Scanners Container image scanning discovers and reports vulnerabilities in OS packages and application libraries baked into container images. KubeRocketCI provides multiple scanning approaches depending on the context. #### Helm-Based Image Discovery (image-scan-chart) This is the primary approach, used in all security-scan pipelines. It renders the application's Helm chart to discover which container images are deployed, then scans each one. **Execution flow:** | Step | Description | |------|-------------| | **discover-image-repository** | Queries Kubernetes for the `CodebaseImageStream` CR to extract the image repository and latest tag | | **helm-template-and-extract** | Runs `helm template` with image overrides from the previous step, then parses rendered manifests for all `image:` references | | **trivy** | Scans each discovered image using [Trivy](https://github.com/aquasecurity/trivy) in client/server mode, producing a JSON report per image | | **grype** | Scans each discovered image using [Anchore Grype](https://github.com/anchore/grype), producing a JSON report per image | | **upload-report** | Uploads all Trivy and Grype reports to DefectDojo with per-image engagements | Key details: - **CodebaseImageStream lookup** normalizes the branch name (lowercase, non-alphanumeric characters replaced with `-`) to locate the correct resource. - **Helm template** uses `--set image.repository` and `--set image.tag` overrides so rendered manifests contain real production image references. - If no Helm chart exists at `deploy-templates/`, the scan is skipped gracefully. - **Trivy** runs in client/server mode by default (server at `http://trivy-service.trivy-system:4954`) for performance. It falls back to standalone mode if no server is configured. - **Grype** always runs standalone (no server mode). - Both scanners use **soft failure** — a scan error on one image does not block other images from being scanned. Upload errors **do** fail the task. #### Remote Registry Scanning (image-scan-remote) For scanning images that already exist in a remote registry, without needing source code or Helm charts. This approach uses Trivy only (no Grype) and creates per-image engagements in DefectDojo. #### Build-Time TAR Scanning (image-scan) For scanning container images saved as TAR archives (e.g., output of `kaniko --tarPath`). Runs both Trivy and Grype. Uses a fixed DefectDojo engagement name `container-security`. #### Standalone Trivy Check (trivy-scan) A lightweight task that scans a single image for **HIGH and CRITICAL** vulnerabilities only: ```bash trivy image --scanners vuln --severity HIGH,CRITICAL "${TARGET_IMAGE}" ``` Results are printed to the pipeline log. No DefectDojo integration. ### Infrastructure-as-Code Linting These tools run during review pipelines to catch issues before merge: - **Hadolint** — validates Dockerfile best practices: pinned base images, proper instruction ordering, and shell best practices. - **Helm chart-testing** — validates Helm chart structure, values schema, and template rendering using `ct lint`. Both tools block the review pipeline on failure. ## DefectDojo Reporting Model Understanding how KubeRocketCI organizes findings in DefectDojo is essential for navigating scan results and configuring alerts. ### Data Hierarchy ```text Product Type: "KubeRocketCI" └─ Product: "{CODEBASE_NAME}" ├─ Engagement: "code-security-{branch}" │ ├─ Test: "Semgrep JSON Report" │ │ └─ Findings: [...] │ └─ Test: "Gitleaks Scan" │ └─ Findings: [...] │ ├─ Engagement: "image:registry.example.com/myapp:1.0.0" │ ├─ Test: "Trivy Scan" │ │ └─ Findings: [...] │ └─ Test: "Anchore Grype" │ └─ Findings: [...] │ └─ Engagement: "image:nginx:1.25" ├─ Test: "Trivy Scan" └─ Test: "Anchore Grype" ``` - **Product Type** — `KubeRocketCI` (configurable via `DD_PRODUCT_TYPE_NAME`). Groups all platform codebases. - **Product** — one per KubeRocketCI Codebase. Named after the Codebase (e.g., `my-java-app`). - **Engagement** — groups related scans. Naming depends on the scan source (see table below). - **Test** — one per scanner execution. Holds the actual findings. ### Engagement Naming Strategy | Scan Source | Engagement Name | Rationale | |:----------:|----------------|-----------| | Code security scans | `code-security` or `code-security-{branch}` | One engagement per branch for SAST and secret findings | | Container image scans | `image:{full-image-reference}` | One engagement per image to prevent `close_old_findings` from cross-contaminating findings between different images | | TAR image scans | `container-security` | Fixed engagement for build-time scanning | ### Upload Mechanism All uploads use the DefectDojo **reimport-scan** API: ```text POST {DD_HOST_URL}/api/v2/reimport-scan/ Authorization: Token {DD_TOKEN} Content-Type: multipart/form-data ``` Standard parameters sent with every upload: | Parameter | Value | Purpose | |-----------|-------|---------| | `scan_date` | Current date (`YYYY-MM-DD`) | When the scan was performed | | `minimum_severity` | `Info` | Import all severity levels | | `active` | `true` | Mark findings as active | | `verified` | `false` | Findings require human verification | | `auto_create_context` | `true` | Auto-create product, engagement, and test if missing | | `close_old_findings` | `true` | Close findings from previous scans that are no longer present | | `push_to_jira` | `false` | No automatic Jira integration | | `environment` | `Development` | Environment classification | | `product_type_name` | `KubeRocketCI` | Product type grouping | | `product_name` | `{CODEBASE_NAME}` | Maps to the KubeRocketCI Codebase | ### Result URLs After upload, pipeline tasks return clickable DefectDojo URLs as results: - **Single image scanned:** `{DD_HOST_URL}/engagement/{ENGAGEMENT_ID}` - **Multiple images scanned:** `{DD_HOST_URL}/product/{PRODUCT_ID}` (shows all engagements) These URLs appear in the KubeRocketCI portal pipeline run details. ## Configuration ### Required Secrets | Secret Name | Keys | Used By | Purpose | |:-----------:|------|---------|---------| | `ci-defectdojo` | `token`, `url` | `security`, `image-scan-chart`, `image-scan-remote`, `image-scan` tasks | DefectDojo report upload | | `ci-dependency-track` | `token`, `url` | `security` task (cdxgen step), standalone `cdxgen` task | SBOM upload to Dependency-Track | | `ci-sonarqube` | `token`, `url` | All `sonarqube-*` tasks | SonarQube analysis | | `ci-nexus` | `username`, `password` | `sonarqube-maven`, `sonarqube-gradle` tasks | Maven/Gradle artifact resolution during Sonar analysis | :::tip All security-related secrets are optional at the task level. If a secret is missing, the corresponding step is skipped gracefully. This means you can incrementally adopt tools: start with SonarQube only, then add DefectDojo, then Dependency-Track. ::: For secret provisioning instructions, refer to: - [Integrate DefectDojo](./defectdojo.md) - [Integrate Dependency-Track](./dependency-track.md) - [SonarQube Integration](../code-quality/sonarqube.md) ### Helm Chart Settings Security scanning pipelines are controlled by the following `values.yaml` settings in the `pipelines-library` Helm chart: ```yaml title="values.yaml" global: gitProviders: - github # Determines which security-scan pipeline variants are deployed - gitlab pipelines: deployableResources: security: true # Enable/disable all security-scan pipelines and tasks java: java25SkipSonar: true # Skip SonarQube for Java 25 tekton: workspaceSize: 7Gi # PVC size for pipeline workspaces ``` | Setting | Effect | |---------|--------| | `pipelines.deployableResources.security` | When `true`, deploys all security-scan pipelines, tasks, and triggers | | `global.gitProviders` | Controls which VCS-specific pipeline variants are deployed (e.g., `github-security-scan`) | | `java25SkipSonar` | When `true`, omits SonarQube from Java 25 pipelines (SonarQube does not yet support Java 25) | ### External Services | Service | Default Endpoint | Purpose | |---------|-----------------|---------| | Trivy Server | `http://trivy-service.trivy-system:4954` | Centralized vulnerability database for faster container scans | | SonarQube | From `ci-sonarqube` secret | Static analysis and Quality Gate evaluation | | DefectDojo | From `ci-defectdojo` secret | Vulnerability management and reporting | | Dependency-Track | From `ci-dependency-track` secret | SBOM storage, dependency vulnerability tracking, license analysis | ## End-to-End Example: Java Maven Application This section walks through the complete security scanning lifecycle of a Java Maven application called `payment-service` on KubeRocketCI. **Setup:** - **Codebase:** `payment-service` (registered in KubeRocketCI) - **Git provider:** GitHub - **Branch:** `main` - **Container image:** `registry.example.com/payment-service:0.1.0-SNAPSHOT.42` - **Helm chart:** `deploy-templates/` with `image.repository` and `image.tag` values ### Phase 1: Developer Opens a Pull Request A developer pushes a feature branch and opens a PR against `main`. GitHub fires a webhook, and the review pipeline starts. ```text Pipeline: github-maven-java17-app-review ``` | Step | Task | What Happens | |:----:|------|-------------| | 1 | `fetch-repository` | Clones the PR branch | | 2 | `init-values` | Reads Codebase/CodebaseBranch CRs for pipeline config | | 3 | `get-cache` | Restores Maven `.m2/repository` cache | | 4 | `compile` | Runs `mvn compile` | | 5 | `test` | Runs `mvn verify` with JaCoCo coverage | | 6 | **`sonar`** | **SonarQube PR analysis** (see below) | | 7 | `build` | Runs `mvn package -DskipTests` | | 8 | `save-cache` | Saves Maven cache for next run | | 9 | `dockerfile-lint` | **Hadolint** validates `Dockerfile` | | 10 | `dockerbuild-verify` | Dry-run container build (no push) | | 11 | `helm-lint` | **chart-testing** validates `deploy-templates/` | | finally | `set-status` | Reports pass/fail back to GitHub PR | **SonarQube during review** — the `sonarqube-maven` task runs: ```bash mvn -s /var/configmap/settings.xml -B \ -Dsonar.projectKey=payment-service \ -Dsonar.projectName=payment-service \ -Dsonar.pullrequest.key=42 \ -Dsonar.pullrequest.branch=feature/add-refunds \ -Dsonar.pullrequest.base=main \ -Dsonar.qualitygate.wait=true \ -DskipTests verify sonar:sonar ``` **Result in SonarQube:** PR #42 appears as a decoration on the `payment-service` project. The Quality Gate evaluates new code coverage, bugs, vulnerabilities, and security hotspots. If the Quality Gate fails, the pipeline fails at step 6 and the GitHub PR status shows a red check. ### Phase 2: PR Is Merged — Build Pipeline Runs After the PR is approved and merged, the build pipeline triggers. ```text Pipeline: github-maven-java17-app-build-default ``` | Step | Task | What Happens | |:----:|------|-------------| | 1 | `fetch-repository` | Clones `main` branch | | 2 | `init-values` | Reads Codebase CR metadata | | 3 | `get-version` | Generates version: `0.1.0-SNAPSHOT.42` | | 4 | `get-cache` | Restores Maven cache | | 5 | `update-build-number` | Stamps version into the project | | 6 | `compile` | Runs `mvn compile` | | 7 | `test` | Runs `mvn verify` with JaCoCo | | 8 | **`sonar`** | **SonarQube branch analysis** | | 9 | `build` | Runs `mvn package -DskipTests` | | 10 | `push` | Publishes JAR/WAR to Nexus | | 11 | `container-build` | Kaniko builds and pushes container image | | 12 | `save-cache` | Saves Maven cache | | 13 | `git-tag` | Tags commit with `0.1.0-SNAPSHOT.42` | | 14 | `update-cbis` | Updates CodebaseImageStream with new image tag and digest | | finally | `report-status` | Reports to JIRA and GitHub | **SonarQube during build** — branch analysis updates the `main` branch in SonarQube with the latest merged code, enabling long-lived branch trend tracking. ### Phase 3: Security-Scan Pipeline Runs The security-scan pipeline is triggered via webhook or manually from the portal. ```text Pipeline: github-security-scan ``` Two tasks run in parallel after `fetch-repository`: #### Task A: security (SAST + Secrets + SCA) | Step | Tool | What Happens | |:----:|------|-------------| | A.1 | **Semgrep** | Scans all source files for vulnerabilities. Writes `semgrep-report.json`. | | A.2 | **Gitleaks** | Scans file content for hardcoded secrets. Writes `gitleaks-report.json`. | | A.3 | **cdxgen** | Analyzes `pom.xml` and transitive dependencies. Generates CycloneDX SBOM. Uploads to Dependency-Track as project `payment-service`, version `main`. | | A.4 | **Upload** | Uploads Semgrep and Gitleaks reports to DefectDojo. | **What appears in DefectDojo after Task A:** ```text Product: payment-service └─ Engagement: "code-security-main" ├─ Test: "Semgrep JSON Report" │ ├─ Finding: SQL Injection in PaymentDao.java:45 [HIGH] │ └─ Finding: Insecure Deserialization in WebConfig.java:112 [CRITICAL] └─ Test: "Gitleaks Scan" └─ Finding: AWS Access Key in application-dev.properties:7 [HIGH] ``` **What appears in Dependency-Track after Task A:** ```text Project: payment-service (version: main) Components: 247 Vulnerabilities: ├─ CVE-2024-22262 — spring-web 6.1.4 [HIGH] ├─ CVE-2024-22243 — spring-web 6.1.4 [MEDIUM] └─ CVE-2023-44487 — netty-codec-http2 4.1.97 [HIGH] License Risk: ├─ 198 components: Apache-2.0 ├─ 41 components: MIT └─ 8 components: LGPL-2.1 (review required) ``` #### Task B: image-scan-chart (Container Scanning) | Step | What Happens | |------|-------------| | B.1 `discover-image-repository` | Queries CodebaseImageStream for `payment-service-main`. Finds image: `registry.example.com/payment-service`, tag: `0.1.0-SNAPSHOT.42`. | | B.2 `helm-template-and-extract` | Renders the Helm chart with image overrides. Discovers 2 images: `registry.example.com/payment-service:0.1.0-SNAPSHOT.42` and `nginx:1.25-alpine` (sidecar). | | B.3 `trivy` | Scans both images against Trivy server. | | B.4 `grype` | Scans both images with Grype. | | B.5 `upload-report` | Uploads all 4 reports (2 images x 2 scanners) to DefectDojo. | **What appears in DefectDojo after Task B:** ```text Product: payment-service ├─ Engagement: "code-security-main" │ └─ (Semgrep + Gitleaks findings from Task A) │ ├─ Engagement: "image:registry.example.com/payment-service:0.1.0-SNAPSHOT.42" │ ├─ Test: "Trivy Scan" (version: 0.1.0-SNAPSHOT.42, branch: main) │ │ ├─ Finding: CVE-2024-32002 — git 2.43.0 [HIGH] │ │ └─ Finding: CVE-2024-6345 — setuptools 69.0.3 [HIGH] │ └─ Test: "Anchore Grype" (version: 0.1.0-SNAPSHOT.42, branch: main) │ └─ Finding: CVE-2024-32002 — git 2.43.0 [High] │ └─ Engagement: "image:nginx:1.25-alpine" ├─ Test: "Trivy Scan" │ └─ Finding: CVE-2024-24790 — Go standard library [CRITICAL] └─ Test: "Anchore Grype" └─ Finding: CVE-2024-24790 — Go standard library [Critical] ``` ### Phase 4: On-Demand Remote Image Scan If the security team needs to re-scan specific images without source code (e.g., after a Trivy DB update), they trigger the `image-scan-remote` pipeline: ```yaml params: IMAGE_NAMES: - "registry.example.com/payment-service:0.1.0-SNAPSHOT.42" CODEBASE_NAME: "payment-service" ``` This runs Trivy against the remote registry and uploads results to the existing `payment-service` product in DefectDojo, updating the per-image engagement. ### Security Coverage Summary | Scanner | What It Detects | Where Results Live | Blocks Pipeline? | |---------|----------------|-------------------|:---:| | **SonarQube** | Code quality, bugs, security vulnerabilities, coverage | SonarQube server | Yes (Quality Gate) | | **Semgrep** | SAST findings (injection, deserialization, etc.) | DefectDojo | No | | **Gitleaks** | Hardcoded secrets and credentials | DefectDojo | No | | **cdxgen + Dependency-Track** | Vulnerable dependencies, license risks | Dependency-Track | No | | **Trivy** | OS-level and library CVEs in container images | DefectDojo | No | | **Grype** | OS-level and library CVEs in container images | DefectDojo | No | | **Hadolint** | Dockerfile best practices | Pipeline log | Yes (review only) | | **Helm Lint** | Chart structure and template errors | Pipeline log | Yes (review only) | :::note SonarQube is the only scanner that acts as a hard gate. All other security scanners operate in a "report and continue" mode — findings are sent to DefectDojo and Dependency-Track for triage by the security team. ::: ## Related Articles * [Security and Compliance: Overview](./overview.md) * [Integrate DefectDojo](./defectdojo.md) * [Integrate Dependency-Track](./dependency-track.md) * [SonarQube Integration](../code-quality/sonarqube.md) * [Tekton Overview](../ci/tekton-overview.md) --- ## Install Velero # Automated Kubernetes Backup and Restore Workflows With Velero [Velero](https://velero.io/) is an open source tool to safely back up, recover, and migrate Kubernetes clusters and persistent volumes. It works both on premises and in a public cloud. Velero consists of a server process running as a deployment in your Kubernetes cluster and a command-line interface (CLI) with which DevOps teams and platform operators configure scheduled backups, trigger ad-hoc backups, perform restores, and more. Configuring and installing Velero involves two key steps: - **Setting up AWS resources** – configuring IAM roles and policies. - **Installing Velero** – deploying the Velero components into the Kubernetes cluster. ## Configuring AWS IAM Roles and Policies Velero requires specific IAM permissions to interact with AWS resources. These permissions allow Velero to create and manage EBS snapshots and volumes, as well as read and write backup data to the designated S3 bucket. This policy ensures Velero can perform backup and restore operations within your AWS account. To simplify the setup, we recommend using the [terraform-module/velero/kubernetes](https://registry.terraform.io/modules/terraform-module/velero/kubernetes/latest) module. This module automates the creation of the necessary IAM roles and policies for Velero. You can find Velero configuration parameters in the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/eks/irsa.tf#L367) repository. With this configuration, the following IAM role will be created: - **AWSIRSA_\_Velero**: This role will be used by Velero to perform backup and restore operations by managing EBS snapshots and accessing the S3 backup bucket. The attached policy grants Velero the required permissions to perform backup and restore operations in AWS. It includes the following capabilities: - **EC2 permissions** — allow Velero to create, describe, tag, and delete EBS snapshots and volumes used during backup and restore processes. - **S3 bucket access** — allows listing the backup bucket and performing read/write operations on objects, including uploading backup data, retrieving it during restores, and deleting or tagging objects as needed. ## Install Velero To deploy Velero in your Kubernetes cluster, we recommend using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/velero). Here's an example of how to configure values.yaml before installation: ```yaml title="values.yaml" velero: # -- Velero service account settings for AWS access via IRSA. Replace the role ARN with your own IAM role. serviceAccount: server: create: true name: velero-server annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::01234567890:role/AWSIRSA_Core_Velero" # -- Use Kubernetes secrets for AWS credentials (false = use IRSA) credentials: useSecret: false configuration: # -- Backup storage settings for Velero backupStorageLocation: - name: default provider: aws bucket: velero-core prefix: velero config: region: eu-central-1 # -- Settings for volume snapshots volumeSnapshotLocation: - name: default provider: aws config: region: eu-central-1 # -- Init container to load AWS plugin for Velero initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.13.1 volumeMounts: - mountPath: /target name: plugins # -------------------------------------------- # Velero Schedule: krci # -------------------------------------------- # Description: # Creates daily backups of the "krci" namespace # on weekdays (Monday–Friday) at 13:00 UTC. # Excludes PersistentVolumes and PVCs, # does not snapshot volumes, # stores backups in the "default" location, # and retains them for 72 hours. # -------------------------------------------- schedules: krci: disabled: true paused: false labels: cluster: core region: eu-central-1 schedule: "0 13 * * 1-5" useOwnerReferencesInBackup: false template: excludedClusterScopedResources: - persistentvolumes excludedNamespaceScopedResources: - persistentvolumeclaims includedNamespaces: - krci snapshotVolumes: false storageLocation: default ttl: 72h0m0s ``` At the end of the file, you will find backup schedule settings that you can use as a template for creating your own schedules. ## Related Articles * [Associate IAM Roles With Service Accounts](../infrastructure-providers/aws/enable-irsa.md) --- ## Restore KubeRocketCI Tenant With Velero # Restore KubeRocketCI Tenant With Velero You can use the Velero tool to restore a platform tenant. Explore the main steps for backup and restoring below. 1. Delete all related entities in **Keycloak**: realm and clients from **master/openshift** realms. Navigate to the entities list in the Keycloak, select the necessary ones, and click the deletion icon on the entity overview page. If there are customized configs in Keycloak, save them before making backup. ![Remove keycloak realm](../../assets/operator-guide/delete-keycloak-realm.png "Remove keycloak realm") 2. To restore KubeRocketCI, install and configure the Velero tool. Please refer to the [Install Velero](install-velero-add-ons.md) documentation for details. 3. Remove all locks for operators. Delete all config maps that have `-operator-lock` names. Then restart all pods with operators, or simply run the following command: ```bash kubectl -n krci delete cm $(kubectl -n krci get cm | grep 'operator-lock' | awk '{print $1}') ``` --- ## Enable Git Resource Discovery # Enable Git Resource Discovery This page describes installation of the [GitFusion](https://github.com/KubeRocketCI/gitfusion) component, allowing the portal to access Git-related information without requiring direct access to the Version Control System. GitFusion enhances the developer experience by facilitating automated discovery of Git repositories, remote branches, and organizational structures across multiple Git providers. The service operates as an intermediary layer, securing communication between the KubeRocketCI portal and Git infrastructure while enabling rich repository browsing capabilities. ## Installation :::warning When integrating GitFusion with Bitbucket repositories, ensure your Bitbucket API token includes the `account:read` permission scope. Detailed instructions for configuring Bitbucket API tokens with appropriate permissions are available in the [Add Git Server](../../user-guide/add-git-server.md) documentation. ::: To enable the GitFusion integration in KubeRocketCI, follow the steps below: 1. Activate GitFusion service deployment by modifying the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml#L87) repository. Set the `gitfusion.enable` flag to `true` in the `values.yaml` file: ```yaml title="values.yaml" gitfusion: enable: true ``` ## Verification Once GitFusion deployment is complete, validate the integration functionality through the following testing procedures: ### Component Discovery 1. Navigate to **KubeRocketCI portal** -> **Components**. 2. On the **Components** page, click **+ Create component** -> **Application** -> **Import**. 3. Within the **Create Application** dialog, use the **Owner** dropdown to select your Git account (configured during Git Server setup): ![Git account discovery](../../assets/operator-guide/extensions/account-discovery.png "Git account discovery") 4. Browse available repositories using the **Repository** dropdown to select the desired application for import: ![Git repository discovery](../../assets/operator-guide/extensions/repo-discovery.png "Git repository discovery") ### Creating Branches 1. Return to the **KubeRocketCI portal**, access **Components**, and select a specific component by clicking its name. 2. Switch to the **Branches** tab within the component details view and click **+ Create branch**. 3. In the **Create branch** dialog, interact with the branch name field to display the list of remotely available branches: ![Git branch discovery](../../assets/operator-guide/extensions/branch-discovery.png "Git branch discovery") ## Related Articles * [Add Application](../../user-guide/add-application.md) * [Manage Branches](../../user-guide/manage-branches.md) --- ## KubeRocketCI Operator Guide: Installation, Configuration, and Administration # KubeRocketCI Operator Guide: Installation, Configuration, and Administration The Operator guide is intended for DevOps and provides information on KubeRocketCI installation, configuration and customization, as well as the platform support. Inspect the documentation to adjust the KubeRocketCI according to your business needs: - **Installation** - Provides step-by-step guidelines for the platform prerequisites and platform installation process, including various methods such as [Helm Chart](./install-kuberocketci.md), [AWS Marketplace](./aws-marketplace-install.md), [Civo Marketplace](./install-via-civo.md), and via [Cluster Add-Ons](./add-ons-overview.md). - **Authentication and Authorization** - Covers the platform's authentication models, including configurations for [Keycloak](auth/keycloak.md), [Namespace Management](auth/namespace-management.md), and [Microsoft Entra ID](microsoft-entra/oidc-authentication-overview.md). - **Artifacts Management** - Describes artifacts management using [Nexus](artifacts-management/nexus-image-registry.md) or [Harbor](artifacts-management/harbor-installation.md). - **Code Quality** - Provides guidelines on how to integrate platform with the mandatory [SonarQube](code-quality/sonarqube.md) component. - **Project Management and Reporting** - Provides integration options with project management tools like [Jira](project-management-and-reporting/jira-integration.md) and [ReportPortal](project-management-and-reporting/install-reportportal.md). - **Secrets Management** - Explains the installation and integration of the [External Secrets Operator](secrets-management/install-external-secrets-operator.md). - **Security and Compliance** - Includes guides on integrating security and compliance tools such as [Dependency-Track](devsecops/dependency-track.md) and [DefectDojo](devsecops/defectdojo.md). - **Monitoring and Observability** - Provides installation and configuration guides for monitoring and logging tools like [Grafana Loki](monitoring-and-observability/install-loki.md) and [Elasticsearch](monitoring-and-observability/kibana-ilm-rollover.md) stacks. - **Continuous Integration (CI)** - Covers the setup and configuration of [Tekton](ci/tekton-overview.md). - **Continuous Deployment (CD)** - This section contains guidelines on how to integrate a mandatory component [Argo CD](cd/argocd-integration.md). Besides, it also contains information on creating [custom pipelines](cd/customize-deploy-pipeline.md), and [deploying RPM packages](cd/deploy-rpm.md). - **Infrastructure Providers** - Provides information on integrating with various infrastructure providers, including [AWS](deploy-aws-eks.md) and [OpenShift](infrastructure-providers/okd/deploy-okd-4.10.md). - **Disaster Recovery** - Explains the installation and configuration of [Velero](disaster-recovery/install-velero-add-ons.md) as a solution for disaster recovery. - **Upgrade** - Provides guides on upgrading KubeRocketCI and related components, including historical versions. - **Troubleshooting** - Includes common troubleshooting guides for various issues that may occur during the setup and operation of KubeRocketCI. --- ## Atlantis Installation and Configuration Guide for Kubernetes: Terraform Automation # Atlantis: Enterprise-Grade Terraform Automation for Kubernetes Atlantis is an enterprise-ready tool for automating Terraform workflows in Kubernetes environments. It provides a robust web interface and seamlessly integrates with version control systems (VCS) to automatically run Terraform operations as part of the pull request process. This enables teams to systematically review, approve, and apply infrastructure changes in a consistent, secure, and auditable manner. This guide covers the full process of installing and configuring Atlantis as Helm chart. It includes setting up authentication, configuring integration with supported version control systems (such as Bitbucket, GitHub, or GitLab), and preparing Atlantis to automate Terraform workflows triggered by pull/merge requests. ## Prerequisites To install and configure Atlantis, ensure the following prerequisites are met: - A running [AWS EKS](../deploy-aws-eks.md) cluster with the necessary [IAM](https://aws.amazon.com/iam) permissions to create IAM roles. - A Git repository containing Terraform code with the required repository [structure](https://www.runatlantis.io/docs/requirements.html#repository-structure) is created. - An Atlantis Git [user](https://www.runatlantis.io/docs/access-credentials.html#create-an-atlantis-user-optional) is created with a generated [Access Token](https://www.runatlantis.io/docs/access-credentials.html#generating-an-access-token). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository for managing the Atlantis installation is cloned to the local machine. - [Helm](https://helm.sh) is installed and configured. - [kubectl](https://kubernetes.io/docs/tasks/tools/) is installed and configured. - (Optional) [Argo CD](../install-argocd.md) is installed and configured. - (Optional) [External Secrets Operator](../secrets-management/external-secrets-operator-integration) is installed and configured. - (Optional) [Keycloak](../auth/keycloak) and [keycloak-operator](https://github.com/epam/edp-keycloak-operator) is installed and configured. ## Configuring Atlantis :::important All configuration steps described in this section are based on the [Atlantis](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/atlantis) Helm chart that can be found in the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main) repository. ::: This section describes how to configure Atlantis for integration with supported version control systems (VCS), including setting up authentication for Atlantis Web Server, configuring Webhooks, and creating the AWS IAM Role which is required for Atlantis to perform Terraform operations on the selected provider. ## Authentication Configuration By default, Atlantis Web Server is deployed without any authentication mechanism, which means it is publicly accessible. This setup is not recommended, as it exposes the Atlantis UI and related operations to unauthorized access. This section describes two approaches to configure authentication for Atlantis Web Server: 1. **Basic Authentication**: A simple method that restricts access using a static username and password, stored as a Kubernetes secret. 2. **External Authentication Proxy**: A more advanced method that delegates authentication to an external identity provider (e.g., Keycloak) using an authentication proxy tools like [oauth2-proxy](https://oauth2-proxy.github.io/oauth2-proxy/). This method provides role-based access control and centralized user management. ### Basic Authentication :::note For more details on configuring basic authentication, refer to the [Atlantis Security documentation](https://www.runatlantis.io/docs/security#enable-authentication-on-atlantis-web-server) page. ::: To configure Basic Authentication for Atlantis Web Server, follow the steps below: 1. Create a Kubernetes secret containing the username and password for Basic Authentication: There are two ways to create a secret: manually using the `kubectl` command or using the External Secrets Operator (ESO).
kubectl Run the following command to create a secret: ```bash kubectl create secret generic atlantis-creds -n atlantis \ --from-literal=username= \ --from-literal=password= ``` The `username` and `password` are the credentials that will be used to access the Atlantis Web Server.
External Secrets Operator Enable ESO section in the `values.yaml` file and configure the necessary parameters. For example: ```yaml eso: # -- Install components of the ESO. enabled: true # -- Defines provider type. One of `aws`, `generic`, or `vault`. provider: "aws" # -- Defines the path to the secret in the provider. If provider is `vault`, this is the path must be prefixed with `secret/`. secretPath: "/infra/core/addons/atlantis" # -- AWS configuration (if provider is `aws`). aws: # -- AWS region. region: "eu-central-1" # -- AWS role ARN for the ExternalSecretOperator to assume. roleArn: "arn:aws:iam::012345678910:role/AWSIRSA_Shared_ExternalSecretOperatorAccess" ``` Specify the `username` and `password` fields in the AWS Parameter Store object. For example: ```json { "oauth2-proxy": { "client-id": "", "cookie-secret": "" }, "bitbucket_token": "", "bitbucket_secret": "", "username": "", "password": "", "clientSecret": "" } ``` After configuring the ESO section, the External Secrets Operator will automatically create the Kubernetes secret `atlantis-creds` during installation or upgrade of the Atlantis Helm chart.
2. Update the `values.yaml` file to enable Basic Authentication by setting the `atlantis.basicAuthSecretName` field to the name of the secret created in the previous step. For example: ```yaml title="values.yaml" atlantis: basicAuthSecretName: atlantis-creds ``` 3. Apply the changes by using the `helm upgrade` command or syncing the Argo CD application. After configuring and enabling Basic Authentication, users will be required to provide the username and password to access the Atlantis Web Server. ### External Authentication Proxy (OAuth2 Proxy) To configure an external authentication proxy (e.g., OAuth2 Proxy) with Keycloak for Atlantis Web Server, follow the steps below: 1. Ensure that the Keycloak and [keycloak-operator](https://github.com/epam/edp-keycloak-operator) are installed and configured. 2. Update the `values.yaml` by setting the `oidc.enabled` field to `true` for enabling KeycloakClient resource creation. For example: ```yaml title="values.yaml" oidc: enabled: true ``` 3. Create a Kubernetes secret containing the Keycloak client credentials. The secret should include the `clientSecret` field with the value of the Keycloak client secret. For example: :::note The `keycloak-client-atlantis-secret` is used by keycloak-operator to create the `KeycloakClient` resource. :::
kubectl Run the following command to create a secret: ```bash kubectl create secret generic keycloak-client-atlantis-secret -n atlantis \ --from-literal=clientSecret= ``` The `clientSecret` is a random generated string that is used to authenticate the Atlantis client with Keycloak. It can be generated using the following command: ```bash openssl rand -base64 32 | head -c 32 ```
External Secrets Operator Enable ESO section in the `values.yaml` file and configure the necessary parameters. For example: ```yaml eso: # -- Install components of the ESO. enabled: true # -- Defines provider type. One of `aws`, `generic`, or `vault`. provider: "aws" # -- Defines the path to the secret in the provider. If provider is `vault`, this is the path must be prefixed with `secret/`. secretPath: "/infra/core/addons/atlantis" # -- AWS configuration (if provider is `aws`). aws: # -- AWS region. region: "eu-central-1" # -- AWS role ARN for the ExternalSecretOperator to assume. roleArn: "arn:aws:iam::012345678910:role/AWSIRSA_Shared_ExternalSecretOperatorAccess" ``` Specify the `clientSecret` field in the AWS Parameter Store object. For example: ```json { "oauth2-proxy": { "client-id": "", "cookie-secret": "" }, "bitbucket_token": "", "bitbucket_secret": "", "username": "", "password": "", "clientSecret": "" } ``` The `clientSecret` is a random generated string that is used to authenticate the Atlantis client with Keycloak. It can be generated using the following command: ```bash openssl rand -base64 32 | head -c 32 ``` After configuring the ESO section, the External Secrets Operator will automatically create the Kubernetes secret `keycloak-client-atlantis-secret` during installation or upgrade of the Atlantis Helm chart.
4. Enable the `oauth2-proxy` section in the `values.yaml` file and configure the necessary parameters for the external authentication proxy. For example: ```yaml title="values.yaml" oauth2-proxy: enabled: true extraArgs: # Skip authentication for /events endpoint to allow Git webhook access skip-auth-regex: "^/events$" config: configFile: |- allowed_roles = ["administrator", "developer"] client_id = "atlantis" code_challenge_method="S256" cookie_csrf_expire="5m" cookie_csrf_per_request="true" cookie_domains = ["atlantis.example.com"] cookie_secure = "true" email_domains = [ "*" ] insecure_oidc_allow_unverified_email = "true" oidc_issuer_url = "https://keycloak.example.com/realms/" pass_access_token = "true" pass_authorization_header = "true" pass_basic_auth = "false" provider = "keycloak-oidc" redirect_url = "https://atlantis.example.com/oauth2/callback" skip_jwt_bearer_tokens = "true" upstreams = [ "http://atlantis:80" ] whitelist_domains = [".example.com"] silence_ping_logging = "true" existingSecret: oauth2-proxy ingress: enabled: true hosts: - atlantis.example.com ``` 5. Create a Kubernetes secret `oauth2-proxy` containing the `client-id`, `client-secret`, and `cookie-secret` values. The `client-id` and `client-secret` values should match the Keycloak client credentials. The `cookie-secret` value is a random string used to encrypt the session cookies:
kubectl Run the following command to create a secret: ```bash kubectl create secret generic oauth2-proxy -n atlantis \ --from-literal=client-id=atlantis \ --from-literal=client-secret= \ --from-literal=cookie-secret= ``` The `client-secret` is the random string generated in the previous step. The `cookie-secret` is a random generated string that is used to encrypt the session cookies. It can be generated using the following command: ```bash openssl rand -base64 32 | head -c 32 ```
External Secrets Operator Enable ESO section in the `values.yaml` file and configure the necessary parameters. For example: ```yaml eso: # -- Install components of the ESO. enabled: true # -- Defines provider type. One of `aws`, `generic`, or `vault`. provider: "aws" # -- Defines the path to the secret in the provider. If provider is `vault`, this is the path must be prefixed with `secret/`. secretPath: "/infra/core/addons/atlantis" aws: # -- AWS region. region: "eu-central-1" # -- AWS role ARN for the ExternalSecretOperator to assume. roleArn: "arn:aws:iam::012345678910:role/AWSIRSA_Shared_ExternalSecretOperatorAccess" ``` Specify the `oauth2-proxy` section in the AWS Parameter Store object. Also, make sure that the `clientSecret` field is included. For example: ```json { "oauth2-proxy": { "client-id": "atlantis", "cookie-secret": "" }, "bitbucket_token": "", "bitbucket_secret": "", "username": "", "password": "", "clientSecret": "" } ``` The `clientSecret` is the random string generated in the previous step. The `oauth2-proxy.cookie-secret` is a random generated string that is used to encrypt the session cookies. It can be generated using the following command: ```bash openssl rand -base64 32 | head -c 32 ``` After configuring the ESO section, the External Secrets Operator will automatically create the Kubernetes secret `oauth2-proxy` during installation or upgrade of the Atlantis Helm chart.
6. Apply the changes by using the `helm upgrade` command or syncing the Argo CD application. 7. After configuring the external authentication proxy, users will be required to authenticate using Keycloak credentials to access the Atlantis Web Server: ![OAuth2 Proxy](../../assets/operator-guide/infrastructure-providers/oauth2-proxy.png) ## Webhook Configuration :::note For more details on configuring webhooks, refer to the [Atlantis Webhook documentation](https://www.runatlantis.io/docs/configuring-webhooks.html). ::: Atlantis requires a webhook to be configured in the Git repository to trigger Terraform operations. The webhook should be set up to send events to the Atlantis Web Server URL. For each supported VCS, follow the steps below to configure the webhook: To configure the Atlantis webhook in GitHub, follow these steps: - In the GitHub repository, navigate to **Settings** -> **Webhooks**. - Click **Add webhook**. - Set the **Payload URL** to `http:///events` or `https:///events` (if using SSL). - Set the **Content type** to `application/json`. - In the **Secret** field, provide the [secret](https://www.runatlantis.io/docs/webhook-secrets.html) generated for the webhook. - Select **Let me select individual events** and choose the events you want to trigger the webhook, such as: - **Pull request reviews** - **Pushes** - **Issue comments** - **Pull requests** - Leave the **Active** option enabled. - Click **Add webhook** to save the configuration. ![GitHub Webhook](../../assets/operator-guide/infrastructure-providers/github-webhook.png "GitHub Webhook") To configure the Atlantis webhook in GitLab, follow these steps: - In the GitLab repository, navigate to **Settings** -> **Webhooks**. - Click **Add new webhook**. - Set the **Name** to `atlantis`. - Set the **URL** to `http:///events` or `https:///events` (if using SSL). - In the **Secret token** field, provide the [secret](https://www.runatlantis.io/docs/webhook-secrets.html) generated for the webhook. - Select the events you want to trigger the webhook, such as: - **Push events** - **Comments** - **Merge Request events** - Click **Add webhook** to save the configuration. ![GitLab Webhook](../../assets/operator-guide/infrastructure-providers/gitlab-webhook.png "GitLab Webhook") To configure the Atlantis webhook in Bitbucket, follow these steps: - In the Bitbucket repository, navigate to **Repository Settings**. Under the **Workflow** section, select **Webhooks**. - Click **Add webhook**. - Set the **Title** to `atlantis`. - Set the **URL** to `http:///events` or `https:///events` (if using SSL). - In the **Secret** field, provide the [secret](https://www.runatlantis.io/docs/webhook-secrets.html) generated for the webhook. - In the **Triggers** section, uncheck all options for **Repository** and **Issue** events. For **Pull Request** events, select the following: - **Created** - **Updated** - **Merged** - **Declined** - **Comment** - Click **Save** to save the configuration. ![Bitbucket Webhook](../../assets/operator-guide/infrastructure-providers/bitbucket-webhook.png "Bitbucket Webhook") After configuring the webhook, it is necessary to create the Atlantis webhook Kubernetes secret. This secret should contain the **Access Token** and **Secret** fields, which are used to authenticate the webhook requests sent from the VCS to the Atlantis Web Server. 1. Create a Kubernetes webhook secret containing the **Access Token** and **Secret** parameters.
kubectl :::note It is also possible to specify tokens and webhook secrets for multiple providers in the same Kubernetes secret. Atlantis will automatically use the appropriate fields based on the provider specified in the webhook configuration. ::: Run the following command to create a secret with the Git provider credentials. For Bitbucket, run the following command: ```bash kubectl create secret generic atlantis-webhook \ --from-literal=bitbucket_token= \ --from-literal=bitbucket_secret= ``` The `bitbucket_token` is the generated [API Token](https://support.atlassian.com/bitbucket-cloud/docs/using-api-tokens/) for the Bitbucket user. The `bitbucket_secret` is the generated [secret](https://www.runatlantis.io/docs/webhook-secrets.html) for the Bitbucket webhook. For GitHub, run the following command: ```bash kubectl create secret generic atlantis-webhook \ --from-literal=github_token= \ --from-literal=github_secret= ``` The `github_token` is the generated [Access Token](https://www.runatlantis.io/docs/access-credentials.html#generating-an-access-token) for the GitHub user. The `github_secret` is the generated [secret](https://www.runatlantis.io/docs/webhook-secrets.html) for the GitHub webhook. For GitLab, run the following command: ```bash kubectl create secret generic atlantis-webhook \ --from-literal=gitlab_token= \ --from-literal=gitlab_secret= ``` The `gitlab_token` is the generated [Access Token](https://www.runatlantis.io/docs/access-credentials.html#generating-an-access-token) for the GitLab user. The `gitlab_secret` is the generated [secret](https://www.runatlantis.io/docs/webhook-secrets.html) for the GitLab webhook.
External Secrets Operator Enable ESO section in the `values.yaml` file and configure the necessary parameters. For example: ```yaml eso: # -- Install components of the ESO. enabled: true # -- Defines provider type. One of `aws`, `generic`, or `vault`. provider: "aws" # -- Defines the path to the secret in the provider. If provider is `vault`, this is the path must be prefixed with `secret/`. secretPath: "/infra/core/addons/atlantis" aws: # -- AWS region. region: "eu-central-1" # -- AWS role ARN for the ExternalSecretOperator to assume. roleArn: "arn:aws:iam::012345678910:role/AWSIRSA_Shared_ExternalSecretOperatorAccess" ``` Specify the Git provider token and Webhook secret in the AWS Parameter Store object. :::note It is also possible to specify tokens and webhook secrets for multiple providers in the same AWS Parameter Store object. The External Secrets Operator will automatically create the `atlantis-webhook` Kubernetes secret with the multiple fields based on the provided parameters. ::: For Bitbucket: ```json { "oauth2-proxy": { "client-id": "", "cookie-secret": "" }, "bitbucket_token": "", "bitbucket_secret": "", "username": "", "password": "", "clientSecret": "" } ``` The `bitbucket_token` is the generated [API Token](https://support.atlassian.com/bitbucket-cloud/docs/using-api-tokens/) for the Bitbucket user. The `bitbucket_secret` is the generated [secret](https://www.runatlantis.io/docs/webhook-secrets.html) for the Bitbucket webhook. For GitHub: ```json { "oauth2-proxy": { "client-id": "", "cookie-secret": "" }, "github_token": "", "github_secret": "", "username": "", "password": "", "clientSecret": "" } ``` The `github_token` is the generated [Access Token](https://www.runatlantis.io/docs/access-credentials.html#generating-an-access-token) for the GitHub user. The `github_secret` is the generated [secret](https://www.runatlantis.io/docs/webhook-secrets.html) for the GitHub webhook. For GitLab: ```json { "oauth2-proxy": { "client-id": "", "cookie-secret": "" }, "gitlab_token": "", "gitlab_secret": "", "username": "", "password": "", "clientSecret": "" } ``` The `gitlab_token` is the generated [Access Token](https://www.runatlantis.io/docs/access-credentials.html#generating-an-access-token) for the GitLab user. The `gitlab_secret` is the generated [secret](https://www.runatlantis.io/docs/webhook-secrets.html) for the GitLab webhook. After configuring the ESO section, the External Secrets Operator will automatically create the Kubernetes secret `atlantis-webhook` during installation or upgrade of the Atlantis Helm chart.
2. Update the `values.yaml` file to enable the webhook secret by setting the `atlantis.vcsSecretName` field to the name of the secret created in the previous step. For example: ```yaml title="values.yaml" atlantis: vcsSecretName: atlantis-webhook ``` 3. Apply the changes by using the `helm upgrade` command or syncing the Argo CD application. After configuring the webhook secret, Atlantis will be able to authenticate incoming webhook requests from the VCS. ## Provider Credentials :::note For more details on configuring provider credentials, refer to the [Atlantis Provider Credentials documentation](https://www.runatlantis.io/docs/provider-credentials). ::: Depending on the selected provider, Atlantis needs to be provided with the necessary permissions to perform Terraform operations. In this section, it is described how to create the required AWS IAM Role or use the built-in Atlantis IAM Role from the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform) repository. ### Creating Atlantis IAM Role To create the AWS IAM Role for Atlantis, follow the steps below: 1. Navigate to the AWS Management Console and open the [IAM](https://aws.amazon.com/iam/) service. 2. Under the **Access management** section, select **Roles**. 3. Click **Create role**. 4. Select **Web identity** as the trusted entity type. 5. In the **Web identity** section, select the OIDC provider for appropriate EKS cluster and the audience (`sts.amazonaws.com`). 6. Add the condition to the trust policy to allow the Atlantis service account to assume the role. The condition should include the following values: - **Key** - `oidc.eks..amazonaws.com/id/:sub` - **Condition** - `StringLike` - **Value** - `system:serviceaccount::` ![AWS IAM Role](../../assets/operator-guide/infrastructure-providers/aws-iam-role.png "AWS IAM Role") 7. Click **Next** to proceed to the permissions page. 8. On the **Add permissions** page, attach the necessary policies for Atlantis to perform Terraform operations. Select the permissions boundary if required. ![AWS IAM Role Permissions](../../assets/operator-guide/infrastructure-providers/aws-iam-role-permissions.png "AWS IAM Role Permissions") 9. Click **Next** to proceed to the review page. 10. Provide a name for the role and review the configuration. Click **Create role** to create the role. ![AWS IAM Role Review](../../assets/operator-guide/infrastructure-providers/aws-iam-role-review.png "AWS IAM Role Review") After creating the AWS IAM Role, it is necessary to attach the role to the Atlantis service account. This can be done by updating the `values.yaml` file of the Atlantis Helm chart. 1. Navigate to the `clusters/core/addons/atlantis` directory of the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. 2. Open the `values.yaml` file and update the `atlantis.serviceAccount.annotations` field to include the ARN of the created AWS IAM Role. For example: ```yaml title="values.yaml" atlantis: serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/ ``` 3. Apply the changes by using the `helm upgrade` command or syncing the Argo CD application. After configuring the AWS IAM Role, Atlantis will be able to assume the role and perform Terraform operations using the provided permissions. ### Using built-in Atlantis IAM Role :::note For more details about Atlantis IAM Role creation, refer to the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform/tree/master/eks/atlantis.tf) repository. ::: In case of using the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform) repository to manage the AWS infrastructure, Atlantis can use the built-in **Atlantis** IAM Role created by the repository. This IAM Role is automatically created and configured with the necessary permissions to assume the **KRCIDeployerRole** IAM Role for performing Terraform operations. To use the built-in Atlantis IAM Role, follow the steps below: 1. Navigate to the AWS Management Console and open the [IAM](https://aws.amazon.com/iam/) service. 2. Under the **Access management** section, select **Roles**. Ensure that the **Atlantis** IAM Role is created. If not, create the role using the [Deploy AWS EKS Cluster](../deploy-aws-eks.md) guide. 3. Navigate to the `clusters/core/addons/atlantis` directory of the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. 4. Open the `values.yaml` file and update the `atlantis.serviceAccount.annotations` field to include the ARN of the Atlantis IAM Role. For example: ```yaml title="values.yaml" atlantis: serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/Atlantis ``` 5. Apply the changes by using the `helm upgrade` command or syncing the Argo CD application. After configuring the AWS IAM Role, Atlantis will be able to assume the **KRCIDeployerRole** IAM Role and perform Terraform operations using the provided permissions. ## Installing Atlantis There are two approaches to install Atlantis using [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository: - **Using Argo CD**: This approach implies installing Atlantis as an Argo CD Application resource. - **Using Helm**: This approach involves installing Atlantis via common Helm commands. ### Approach 1: Using Argo CD :::note For more information on configuring [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository, refer to the [Install via Add-Ons](../add-ons-overview.md) page. ::: The first approach is to deploy Atlantis using Argo CD. Follow the steps below to install Atlantis using Argo CD: 1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. 2. Navigate to the `clusters/core/addons/atlantis` directory and configure the `values.yaml` file with the necessary values for the Atlantis installation. 3. After configuring the Atlantis Helm chart values, navigate to the `clusters/core/apps` directory. In the `values.yaml` file, update the `atlantis` section by specifying the `enable` field as `true` to enable Argo CD Application creation for the Atlantis. Modify the `createNamespace` and `namespace` fields as needed. For example: ```yaml atlantis: createNamespace: true enable: true namespace: atlantis ``` 4. Commit and push the changes to the remote repository. After the changes are pushed, navigate to the Argo CD and sync the Atlantis application. Verify that the Atlantis is successfully deployed: ![Argo CD Atlantis](../../assets/operator-guide/infrastructure-providers/argo-cd-atlantis.png) ### Approach 2: Deploy Using Helm The second approach is to deploy Atlantis using Helm. Follow the steps below to install Atlantis using Helm: 1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. 2. Navigate to the `clusters/core/addons/atlantis` directory and configure the `values.yaml` file with the necessary values for the Atlantis installation. 3. After configuring the Atlantis Helm chart values, run the following command to deploy the Atlantis: ```bash helm upgrade --install atlantis . -n atlantis --create-namespace ``` 4. Verify that the Atlantis is successfully deployed. ## Related Articles * [Install via Add-Ons](../add-ons-overview.md) * [Install Keycloak](../auth/keycloak.md) * [Install Argo CD](../install-argocd.md) --- ## AWS EBS CSI Driver Setup for EKS # Amazon EBS CSI Driver This guide shows how to install the AWS EBS CSI driver on an Amazon EKS cluster, step by step: creating the IAM OIDC trust policy, attaching the AWS-managed IAM policy, and deploying the driver as an EKS add-on so Kubernetes can manage Amazon EBS-backed Persistent Volumes. ## Prerequisites An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have an OIDC provider or to create a new one, see [Creating an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). To add an Amazon EBS CSI add-on, please follow the steps below: 1. Check your cluster details (the random value in the cluster name will be required in the next step): ```bash kubectl cluster-info ``` 2. Create Kubernetes IAM Trust Policy for Amazon EBS CSI Driver. Replace `AWS_ACCOUNT_ID` with your account ID, `AWS_REGION` with your AWS Region, and `EXAMPLED539D4633E53DE1B71EXAMPLE` with the value that was returned in the previous step. Save this Trust Policy into a file `aws-ebs-csi-driver-trust-policy.json`.
aws-ebs-csi-driver-trust-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com", "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa" } } } ] } ```
To get the notion of the IAM Role creation, please refer to the [official documentation](https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html). 3. Create the IAM role, for example: ```bash aws iam create-role \ --role-name AmazonEKS_EBS_CSI_DriverRole \ --assume-role-policy-document file://"aws-ebs-csi-driver-trust-policy.json" ``` 4. Attach the required AWS Managed Policy `AmazonEBSCSIDriverPolicyV2` to the role with the following command: ```bash aws iam attach-role-policy \ --policy-arn arn:aws:iam::aws:policy/AmazonEBSCSIDriverPolicyV2 \ --role-name AmazonEKS_EBS_CSI_DriverRole ``` 5. Add the Amazon EBS CSI add-on using the AWS CLI. Replace `my-cluster` with the name of your cluster, `AWS_ACCOUNT_ID` with your account ID, and `AmazonEKS_EBS_CSI_DriverRole` with the name of the role that was created earlier: ```bash aws eks create-addon --cluster-name my-cluster --addon-name aws-ebs-csi-driver \ --service-account-role-arn arn:aws:iam::AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole ``` :::note When the plugin is deployed, it creates the `ebs-csi-controller-sa` service account. The service account is bound to a Kubernetes `ClusterRole` with the required Kubernetes permissions. The `ebs-csi-controller-sa` service account should already be annotated with `arn:aws:iam::AWS_ACCOUNT_ID:role/AmazonEKS_EBS_CSI_DriverRole`. To check the annotation, please run: ```bash kubectl get sa ebs-csi-controller-sa -n kube-system -o=jsonpath='{.metadata.annotations}' ``` In case pods have errors, restart the `ebs-csi-controller` deployment: ```bash kubectl rollout restart deployment ebs-csi-controller -n kube-system ``` ::: ### Related Articles * [Creating an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) * [Creating the Amazon EBS CSI driver IAM role for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html) * [Managing the Amazon EBS CSI driver as an Amazon EKS add-on](https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html) --- ## Associate IAM Roles With Service Accounts # Associate IAM Roles With Service Accounts This page contains information on how to associate an IAM role with the service account (IRSA). Get acquainted with the [AWS Official Documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html) on the subject before proceeding. ## Common Configuration of IAM Roles With Service Accounts To successfully associate the IAM role with the service account, follow the steps below: 1. Create an IAM role that will further be associated with the service account. This role must have the following trust policy: **IAM Role** ```bash { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { ":sub": "system:serviceaccount::" } } } ] } ``` View cluster's `` URL. ```bash aws eks describe-cluster --name --query "cluster.identity.oidc.issuer" --output text ``` Example output: ```bash https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E ``` `` in this example will be: ```bash oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E ``` 2. Annotate the created service account with the IAM role: **Service Account** ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: namespace: annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/" ``` 3. All newly launched pods with this service account will be modified and then use the associated IAM role. Find below the pod specification template: **Pod Template** ```yaml apiVersion: v1 kind: Pod metadata: name: irsa-test namespace: default spec: serviceAccountName: containers: - name: irsa image: public.ecr.aws/aws-cli/aws-cli:2.25.4 command: ['sh', '-c', 'aws sts "get-caller-identity" && sleep 3600'] ``` 4. Check the logs of the created pod from the template above. Example output: ```json { "UserId": "XXXXXXXXXXXXXXXXXXXXX:botocore-session-XXXXXXXXXX", "Account": "XXXXXXXXXXXX", "Arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/AWSIRSATestRole/botocore-session-XXXXXXXXXX" } ``` As a result, it is possible to perform actions in AWS under the **AWSIRSATestRole** role. --- ## IAM Roles for Kaniko Service Accounts # IAM Roles for Kaniko Service Accounts :::note Alternatively, the Kaniko IAM Role can also be created automatically during the cluster installation process. For more details, refer to the [Deploy AWS EKS Cluster](../../deploy-aws-eks.md) page. ::: The `build-image-kaniko` stage, used in Tekton build pipelines, manages [ECR](https://aws.amazon.com/ecr/) through IRSA that should be available on the cluster. Follow the steps below to create a required role: 1. Create AWS IAM Policy `AWSIRSA__Kaniko_policy`: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:*", "cloudtrail:LookupEvents" ], "Resource": "arn:aws:ecr:::repository/*" }, { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" }, { "Effect": "Allow", "Action": [ "ecr:DescribeRepositories", "ecr:CreateRepository" ], "Resource": "arn:aws:ecr:::repository/*" } ] } ``` 2. Create AWS IAM Role `AWSIRSA__Kaniko` with trust relationships: :::note `OIDC_PROVIDER` value can be found in the AWS Management Console under the Elastic Kubernetes Service (EKS) service. Select the appropriate cluster and navigate to the **Overview** tab to find the `OpenID Connect provider URL` value. ![OIDC_PROVIDER](../../../assets/operator-guide/oidc-provider.png) ::: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { ":sub": "system:serviceaccount:*" } } } ] } ``` 3. Attach the `AWSIRSA__Kaniko_policy` policy to the `AWSIRSA__Kaniko` role. 4. Set the resulting Amazon Resource Name (ARN) of the role in the `edp-tekton.kaniko.roleArn` field within the `values.yaml` file during the [KubeRocketCI installation](../../install-kuberocketci.md). ## Related Articles * [Associate IAM Roles With Service Accounts](enable-irsa.md) --- ## Configure AWS WAF With Terraform # Configure AWS WAF With Terraform This page contains accurate information on how to configure [AWS WAF](https://aws.amazon.com/waf/) using Terraform with the aim to have a secured traffic exposure and to prevent the Host Header vulnerabilities. ## Prerequisites To follow the instruction, check the following prerequisites: 1. Deployed infrastructure includes Nginx Ingress Controller 2. Deployed services for testing 3. Separate and exposed AWS ALB 4. terraform 0.14.10 5. hashicorp/aws = 4.8.0 ## Solution Overview The solution includes two parts: 1. Prerequisites (mostly the left part of the scheme) - AWS ALB, Compute Resources (EC2, EKS, etc.). 2. WAF configuration (the right part of the scheme). The WAF ACL resource is the main resource used for the configuration; The default web ACL option is Block. ![Overview WAF Solution](../../../assets/operator-guide/waf-configuration.drawio.png) The ACL includes three managed AWS rules that secure the exposed traffic: - AWS-AWSManagedRulesCommonRuleSet - AWS-AWSManagedRulesLinuxRuleSet - AWS-AWSManagedRulesKnownBadInputsRuleSet AWS provides a lot of rules such as baseline and use-case specific rules, for details, please refer to the [Baseline rule groups](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.html). There is the _PreventHostInjections_ rule that prevents the Host Header vulnerabilities. This rule includes one statement that declares that the Host Header should match Regex Pattern Set, thus only in this case it will be passed. The Regex Pattern Set is another resource that helps to organize regexes, in fact, is a set of regexes. All regexes added to the single set are matched by the OR statement, i.e. when exposing several URLs, it is necessary to add this statement to the set and refer to it in the rule. ## WAF ACL Configuration To create the Regex Pattern Set, inspect the following code: ```terraform resource "aws_wafv2_regex_pattern_set" "common" { name = "Common" scope = "REGIONAL" regular_expression { regex_string = "^.*(some-url).*((.example)+)\\.com$" } # Add here additional regular expressions for other endpoints, they are merging with OR operator, e.g. /* regular_expression { regex_string = "^.*(keycloak).*((.example)+)\\.com$" } */ tags = var.tags } ``` It includes 'regex_string', for example: url - some-url.example.com, In addition, it is possible to add other links to the same resource using the regular_expression element. There is the Terraform code for the aws_wafv2_web_acl resource: ```terraform resource "aws_wafv2_web_acl" "external" { name = "ExternalACL" scope = "REGIONAL" default_action { block {} } rule { name = "AWS-AWSManagedRulesCommonRuleSet" priority = 1 override_action { none {} } statement { managed_rule_group_statement { name = "AWSManagedRulesCommonRuleSet" vendor_name = "AWS" } } visibility_config { cloudwatch_metrics_enabled = true metric_name = "AWS-AWSManagedRulesCommonRuleSet" sampled_requests_enabled = true } } rule { name = "AWS-AWSManagedRulesLinuxRuleSet" priority = 2 statement { managed_rule_group_statement { name = "AWSManagedRulesLinuxRuleSet" vendor_name = "AWS" } } override_action { none {} } visibility_config { cloudwatch_metrics_enabled = true metric_name = "AWS-AWSManagedRulesLinuxRuleSet" sampled_requests_enabled = true } } rule { name = "AWS-AWSManagedRulesKnownBadInputsRuleSet" priority = 3 override_action { none {} } statement { managed_rule_group_statement { name = "AWSManagedRulesKnownBadInputsRuleSet" vendor_name = "AWS" } } visibility_config { cloudwatch_metrics_enabled = true metric_name = "AWS-AWSManagedRulesKnownBadInputsRuleSet" sampled_requests_enabled = true } } rule { name = "PreventHostInjections" priority = 0 statement { regex_pattern_set_reference_statement { arn = aws_wafv2_regex_pattern_set.common.arn field_to_match { single_header { name = "host" } } text_transformation { priority = 0 type = "NONE" } } } action { allow {} } visibility_config { cloudwatch_metrics_enabled = true metric_name = "PreventHostInjections" sampled_requests_enabled = true } } visibility_config { cloudwatch_metrics_enabled = true metric_name = "ExternalACL" sampled_requests_enabled = true } tags = var.tags } ``` As mentioned previously, ACL includes three managed AWS rules (group rules), for visibility, enabling sampling, and CloudWatch in the config. The 'PreventHostInjections' custom rule refers to the created pattern set and declares the Host Header, as well as sets the 'Action' if matched to 'Allow'. ## Associate AWS Resource To have the created ACL working, it is necessary to associate an AWS resource with it, in this case, it is AWS ALB: ```terraform resource "aws_wafv2_web_acl_association" "waf_alb" { resource_arn = aws_lb..arn web_acl_arn = aws_wafv2_web_acl.external.arn } ``` :::note AWS ALB can be created in the scope of this Terraform code or created previously. When creating ALB to expose links, the ALB should have a security group that allows some external traffic. ::: When ALB is associated with the WAF ACL, direct the traffic to the ALB by the Route53 CNAME record: ```terraform module "some_url_exposure" { source = "terraform-aws-modules/route53/aws//modules/records" version = "2.0.0" zone_name = "example.com" records = [ { name = "some-url" type = "CNAME" ttl = 300 records = [aws_lb..dns_name] } ] } ``` In the sample above, the module is used, but it is also possible to use a Terraform resource. --- ## Deploy OKD 4.10 Cluster # Deploy OKD 4.10 Cluster This instruction provides detailed information on the OKD 4.10 cluster deployment in the AWS Cloud and contains the additional setup necessary for the managed infrastructure. A full description of the cluster deployment can be found in the [official documentation](https://docs.openshift.com/container-platform/4.10/authentication/managing_cloud_provider_credentials/cco-mode-sts.html). ## Prerequisites Before the OKD cluster deployment and configuration, make sure to check the prerequisites. ### Required Tools 1. Install the following tools listed below: - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) - [OpenShift CLI](https://docs.openshift.com/container-platform/4.10/cli_reference/openshift_cli/getting-started-cli.html) 2. Create the AWS IAM user with [the required permissions](https://docs.okd.io/4.9/installing/installing_aws/installing-aws-account.html#installation-aws-permissions_installing-aws-account). Make sure the AWS account is active, and the user doesn't have a permission boundary. Remove any Service Control Policy (SCP) restrictions from the AWS account. 3. Generate a key pair for cluster node SSH access. Please perform the steps below: - Generate the SSH key. Specify the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If there is an existing key pair, ensure that the public key is in the ~/.ssh directory. ```bash ssh-keygen -t ed25519 -N '' -f / ``` - Add the SSH private key identity to the SSH agent for a local user if it has not already been added. ```bash eval "$(ssh-agent -s)" ``` - Add the SSH private key to the ssh-agent: ```bash ssh-add / ``` 4. Build the `ccoctl` tool: - Clone the `cloud-credential-operator` repository. ```bash git clone https://github.com/openshift/cloud-credential-operator.git ``` - Move to the `cloud-credential-operator` folder and build the `ccoctl` tool. ```bash cd cloud-credential-operator && git checkout release-4.10 GO_PACKAGE='github.com/openshift/cloud-credential-operator' go build -ldflags "-X $GO_PACKAGE/pkg/version.versionFromGit=$(git describe --long --tags --abbrev=7 --match 'v[0-9]*')" ./cmd/ccoctl ``` ## Prepare for the Deployment Process Before deploying the OKD cluster, please perform the steps below: ### Create AWS Resources Create the AWS resources with the Cloud Credential Operator utility (the `ccoctl` tool): 1. Generate the public and private RSA key files that are used to set up the OpenID Connect identity provider for the cluster: ```bash ./ccoctl aws create-key-pair ``` 2. Create an OpenID Connect identity provider and an S3 bucket on AWS: ```bash ./ccoctl aws create-identity-provider \ --name= \ --region= \ --public-key-file=./serviceaccount-signer.public ``` where: - NAME - is the name used to tag any cloud resources created for tracking, - AWS_REGION - is the AWS region in which cloud resources will be created. 3. Create the IAM roles for each component in the cluster: - Extract the list of the `CredentialsRequest` objects from the OpenShift Container Platform release image: ```bash oc adm release extract \ --credentials-requests \ --cloud=aws \ --to=./credrequests \ --quay.io/openshift-release-dev/ocp-release:4.10.25-x86_64 ``` :::note A version of the openshift-release-dev docker image can be found in the [Quay registry](https://quay.io/repository/openshift-release-dev/ocp-release?tab=tags). ::: - Use the `ccoctl` tool to process all `CredentialsRequest` objects in the `credrequests` directory: ```bash ccoctl aws create-iam-roles \ --name= \ --region= \ --credentials-requests-dir=./credrequests --identity-provider-arn=arn:aws:iam:::oidc-provider/-oidc.s3..amazonaws.com ``` ### Create OKD Manifests Before deploying the OKD cluster, please perform the steps below: 1. Download the [OKD installer](https://github.com/openshift/okd/releases/tag/4.10.0-0.okd-2022-07-09-073606). 2. Extract the installation program: ```bash tar -xvf openshift-install-linux.tar.gz ``` 3. Download the installation pull secret for any private registry. This pull secret allows to authenticate with the services that are provided by the authorities, including [Quay.io](https://quay.io/), serving the container images for OKD components. For example, here is a pull secret for Docker Hub:
The pull secret for the private registry ```json { "auths":{ "https://index.docker.io/v1/":{ "auth":"$TOKEN" } } } ```
4. Create a deployment directory and the **install-config.yaml** file: ```bash mkdir okd-deployment touch okd-deployment/install-config.yaml ``` To specify more details about the OKD cluster platform or to modify the values of the required parameters, customize the **install-config.yaml** file for the AWS. Please see below an example of the customized file:
install-config.yaml - OKD cluster’s platform installation configuration file ```yaml apiVersion: v1 baseDomain: credentialsMode: Manual compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: aws: rootVolume: size: 30 zones: - eu-central-1a type: r5.large replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: rootVolume: size: 50 zones: - eu-central-1a type: m5.xlarge replicas: 3 metadata: creationTimestamp: null name: 4-10-okd-sandbox networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OVNKubernetes serviceNetwork: - 172.30.0.0/16 platform: aws: region: eu-central-1 userTags: user:tag: 4-10-okd-sandbox publish: External pullSecret: sshKey: | ```
where: - YOUR_DOMAIN - is a base domain, - PULL_SECRET - is a created pull secret for a private registry, - SSH_KEY - is a created SSH key. 5. Create the required OpenShift Container Platform installation manifests: ```bash ./openshift-install create manifests --dir okd-deployment ``` 6. Copy the manifests generated by the `ccoctl` tool to the `manifests` directory created by the installation program: ```bash cp ./manifests/* ./okd-deployment/manifests/ ``` 7. Copy the private key generated in the `tls` directory by the `ccoctl` tool to the installation directory: ```bash cp -a ./tls ./okd-deployment ``` ## Deploy the Cluster To initialize the cluster deployment, run the following command: ```bash ./openshift-install create cluster --dir okd-deployment --log-level=info ``` :::note If the cloud provider account configured on the host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed. ::: When the cluster deployment is completed, directions for accessing the cluster are displayed in the terminal, including a link to the web console and credentials for the **kubeadmin** user. The `kubeconfig` for the cluster will be located in **okd-deployment/auth/kubeconfig**.
Example output ```bash ... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.mycluster.example.com INFO Login to the console with the user: "kubeadmin", and password: "4vYBz-Ee6gm-ymBZj-Wt5AL" INFO Time elapsed: 36m22s: ```
:::warning The Ignition config files contain certificates that expire after 24 hours, which are then renewed at that time. Do not turn off the cluster for this time, or you will have to update the certificates manually. See [OpenShift Container Platform documentation](https://docs.openshift.com/container-platform/4.10/installing/installing_aws/installing-aws-customizations.html#installation-launching-installer_installing-aws-customizations) for more information. ::: ## Log Into the Cluster To log into the cluster, export the `kubeconfig`: ```bash export KUBECONFIG=/auth/kubeconfig ``` ## Manage OKD Cluster Without the Inbound Rules In order to manage the OKD cluster without the `0.0.0.0/0` inbound rules, please perform the steps below: 1. Create a Security Group with a list of your external IPs: ```bash aws ec2 create-security-group --group-name --description "" --vpc-id aws ec2 authorize-security-group-ingress \ --group-id '' \ --ip-permissions 'IpProtocol=all,PrefixListIds=[{PrefixListId=}]' ``` 2. Manually attach this new Security Group to all master nodes of the cluster. 3. Create another Security Group with an Elastic IP of the Cluster VPC: ```bash aws ec2 create-security-group --group-name custom-okd-4-10 --description "Cluster Ip to 80, 443" --vpc-id aws ec2 authorize-security-group-ingress \ --group-id '' \ --protocol all \ --port 80 \ --cidr aws ec2 authorize-security-group-ingress \ --group-id '' \ --protocol all \ --port 443 \ --cidr ``` 4. Modify the cluster load balancer via the `router-default` svc in the `openshift-ingress` namespace, paste two Security Groups created on previous steps:
The pull secret for the private registry ```yaml apiVersion: v1 kind: Service metadata: name: router-default namespace: openshift-ingress annotations: service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "tag_name=some_value" service.beta.kubernetes.io/aws-load-balancer-security-groups: "" ... ```
## Optimize Spot Instances Usage In order to optimize the usage of Spot Instances on the AWS, add the following line under the `providerSpec` field in the MachineSet of Worker Nodes: ```yaml providerSpec: value: spotMarketOptions: {} ``` ## Related Articles * [Deploy AWS EKS Cluster](../../deploy-aws-eks.md) * [Associate IAM Roles With Service Accounts](../aws/enable-irsa.md) * [Deploy OKD 4.9 Cluster](deploy-okd-4.9.md) --- ## Deploy OKD 4.9 Cluster # Deploy OKD 4.9 Cluster This instruction provides detailed information on the OKD 4.9 cluster deployment in the AWS Cloud and contains the additional setup necessary for the managed infrastructure. A full description of the cluster deployment can be found in the [official documentation](https://docs.okd.io/4.9/installing/installing_aws/installing-aws-customizations.html). ## Prerequisites Before the OKD cluster deployment and configuration, make sure to check the prerequisites. ### Required Tools 1. Install the following tools listed below: - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) - [OpenShift CLI](https://docs.openshift.com/container-platform/4.9/cli_reference/openshift_cli/getting-started-cli.html) 2. Create the AWS IAM user with [the required permissions](https://docs.okd.io/4.9/installing/installing_aws/installing-aws-account.html#installation-aws-permissions_installing-aws-account). Make sure the AWS account is active, and the user doesn't have a permission boundary. Remove any Service Control Policy (SCP) restrictions from the AWS account. 3. Generate a key pair for cluster node SSH access. Please perform the steps below: - Generate the SSH key. Specify the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If there is an existing key pair, ensure that the public key is in the ~/.ssh directory. ```bash ssh-keygen -t ed25519 -N '' -f / ``` - Add the SSH private key identity to the SSH agent for a local user if it has not already been added. ```bash eval "$(ssh-agent -s)" ``` - Add the SSH private key to the ssh-agent: ```bash ssh-add / ``` ## Prepare for the Deployment Process Before deploying the OKD cluster, please perform the steps below: 1. Download the [OKD installer](https://github.com/openshift/okd/releases/tag/4.9.0-0.okd-2022-02-12-140851). 2. Extract the installation program: ```bash tar -xvf openshift-install-linux.tar.gz ``` 3. Download the installation pull secret for any private registry. This pull secret allows to authenticate with the services that are provided by the included authorities, including [Quay.io](https://quay.io/) serving container images for OKD components. For example, here is a pull secret for Docker Hub:
The pull secret for the private registry ```json { "auths":{ "https://index.docker.io/v1/":{ "auth":"$TOKEN" } } } ```
4. Create the deployment directory and the **install-config.yaml** file: ```bash mkdir okd-deployment touch okd-deployment/install-config.yaml ``` To specify more details about the OKD cluster platform or to modify the values of the required parameters, customize the **install-config.yaml** file for AWS. Please see an example of the customized file below:
install-config.yaml - OKD cluster’s platform installation configuration file ```yaml apiVersion: v1 baseDomain: compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: aws: zones: - eu-central-1a rootVolume: size: 50 type: r5.large replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: aws: rootVolume: size: 50 zones: - eu-central-1a type: m5.xlarge replicas: 3 metadata: creationTimestamp: null name: 4-9-okd-sandbox platform: aws: region: eu-central-1 userTags: user:tag: 4-9-okd-sandbox publish: External pullSecret: sshKey: | ```
where: - YOUR_DOMAIN - is a base domain, - PULL_SECRET - is a created pull secret for a private registry, - SSH_KEY - is a created SSH key. ## Deploy the Cluster To initialize the cluster deployment, run the following command: ```bash ./openshift-install create cluster --dir --log-level=info ``` :::note If the cloud provider account configured on the host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed. ::: When the cluster deployment is completed, directions for accessing the cluster are displayed in the terminal, including a link to the web console and credentials for the **kubeadmin** user. The `kubeconfig` for the cluster will be located in **okd-deployment/auth/kubeconfig**.
Example output ```bash ... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/myuser/install_dir/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.mycluster.example.com INFO Login to the console with the user: "kubeadmin", and password: "4vYBz-Ee6gm-ymBZj-Wt5AL" INFO Time elapsed: 36m22s: ```
:::warning The Ignition config files contain certificates that expire after 24 hours, which are then renewed at that time. Do not turn off the cluster for this time, or you will have to update the certificates manually. See [OpenShift Container Platform documentation](https://docs.openshift.com/container-platform/4.9/installing/installing_aws/installing-aws-customizations.html#installation-launching-installer_installing-aws-customizations) for more information. ::: ## Log Into the Cluster To log into the cluster, export the `kubeconfig`: ```bash export KUBECONFIG=/auth/kubeconfig ``` ## Related Articles * [Deploy AWS EKS Cluster](../../deploy-aws-eks.md) * [Deploy OKD 4.10 Cluster](deploy-okd-4.10.md) --- ## Use Cert-Manager in OpenShift # Use Cert-Manager in OpenShift The following material covers [Let's Encrypt](https://letsencrypt.org/) certificate automation with [cert-manager](https://github.com/cert-manager/cert-manager) using [AWS Route53](https://aws.amazon.com/route53/). The [cert-manager](https://cert-manager.io/docs/) is a Kubernetes/OpenShift operator that allows to issue and automatically renew SSL certificates. In this tutorial, the steps to secure DNS Name will be demonstrated. Below is an instruction on how to automatically issue and install wildcard certificates on OpenShift Ingress Controller and API Server covering all cluster Routes. To secure separate OpenShift Routes, please refer to the [OpenShift Route Support](https://github.com/cert-manager/openshift-routes) project for `cert-manager`. ## Prerequisites - The cert-manager - OpenShift v4.7 - v4.11 - Connection to the OpenShift Cluster - Enabled AWS IRSA - The latest [`oc` utility](https://github.com/openshift/okd/releases). The `kubectl` tool can also be used for most of the commands. ### Install Cert-Manager Operator [Install](https://docs.openshift.com/container-platform/4.11/security/cert_manager_operator/cert-manager-operator-install.html) the `cert-manager` operator via OpenShift [OperatorHub](https://docs.openshift.com/container-platform/4.11/operators/understanding/olm-understanding-operatorhub.html) that uses [Operator Lifecycle Manager (OLM)](https://docs.openshift.com/container-platform/4.11/operators/understanding/olm/olm-understanding-olm.html): 1. Go to the **OpenShift Admin Console** → **OperatorHub**, search for the `cert-manager`, and click **Install**: ![Cert-Manager Installation](../../../assets/operator-guide/ssl-okd-config-01.png "Cert-Manager Installation") 2. Modify the `ClusterServiceVersion` OLM resource, by selecting the **Update approval** → **Manual**. If selecting **Update approval** → **Automatic** after the automatic operator update, the parameters in the `ClusterServiceVersion` will be reset to default. :::note Installing an operator with **Manual approval** causes all operators installed in namespace `openshift-operators` to function as [manual approval strategy](https://docs.openshift.com/container-platform/4.11/operators/user/olm-installing-operators-in-namespace.html). In case the **Manual approval** is chosen, review the manual installation plan and approve it. ::: ![Cert-Manager Installation](../../../assets/operator-guide/ssl-okd-config-02.png "Cert-Manager Installation") 3. Navigate to **Operators** → **Installed Operators** and check the operator status to be **Succeeded**: ![Cert-Manager Installation](../../../assets/operator-guide/ssl-okd-config-03.png "Cert-Manager Installation") 4. In case of errors, [troubleshoot](https://docs.openshift.com/container-platform/4.11/operators/admin/olm-status.html) the Operator issues: ```bash oc describe operator cert-manager -n openshift-operators oc describe sub cert-manager -n openshift-operators ``` ### Create AWS Role for Route53 The `cert-manager` should be configured to validate Wildcard certificates using the [DNS-based](https://cert-manager.io/docs/tutorials/acme/dns-validation/) method. 1. Check the DNS **Hosted zone ID** in AWS Route53 for your domain. ![Hosted Zone ID](../../../assets/operator-guide/ssl-okd-config-04.png "Hosted Zone ID") 2. Create [Route53 Permissions policy](https://cert-manager.io/docs/configuration/acme/dns01/route53/#set-up-an-iam-role) in AWS for `cert-manager` to be able to create DNS TXT records for the certificate validation. In this example, `cert-manager` permissions are given for a particular DNS zone only. Replace **Hosted zone ID** *XXXXXXXX* in the *"Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXX"*. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "route53:GetChange", "Resource": "arn:aws:route53:::change/*" }, { "Effect": "Allow", "Action": [ "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets" ], "Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXX" } ] } ``` 3. Create an AWS Role with **Custom trust policy** for the `cert-manager` service account to use the [AWS IRSA](https://cert-manager.io/docs/configuration/acme/dns01/route53/#eks-iam-role-for-service-accounts-irsa) feature and then attach the created policy. Replace the following: - `${aws-account-id}` with the AWS account ID of the EKS cluster. - `${aws-region}` with the region where the EKS cluster is located. - `${eks-hash}` with the hash in the EKS API URL; this will be a random 32 character hex string, for example, 45DABD88EEE3A227AF0FA468BE4EF0B5. - `${namespace}` with the namespace where cert-manager is running. - `${service-account-name}` with the name of the ServiceAccount object created by cert-manager. - By default, it is *"system:serviceaccount:openshift-operators:cert-manager"* if `cert-manager` is installed via OperatorHub. - Attach the created **Permission policy** for Route53 to the Role. - Optionally, add **Permissions boundary** to the Role. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRoleWithWebIdentity", "Principal": { "Federated": "arn:aws:iam::* ${aws-account-id}:oidc-provider/oidc.eks.${aws-region}.amazonaws.com/id/${eks-hash}" }, "Condition": { "StringEquals": { "oidc.eks.${aws-region}.amazonaws.com/id/${eks-hash}:sub": "system:serviceaccount:${namespace}:${service-account-name}" } } } ] } ``` 4. Copy the created **Role ARN**. ### Configure Cert-Manager Integration With AWS Route53 1. [Annotate](https://cert-manager.io/docs/configuration/acme/dns01/route53/#service-annotation) the `ServiceAccount` created by `cert-manager` (required for [AWS IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)), and restart the `cert-manager` pod. 2. Replace the `eks.amazonaws.com/role-arn` annotation value with your own Role ARN. ```bash oc edit sa cert-manager -n openshift-operators ``` ```yaml hl_lines="5" apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/cert-manager ``` 3. Modify the `cert-manager` `Deployment` with the correct file system permissions `fsGroup: 1001`, so that the `ServiceAccount` token can be read. :::note In case the `ServiceAccount` token cannot be read and the operator is installed using the OperatorHub, add `fsGroup: 1001` via OpenShift [ClusterServiceVersion](https://cert-manager.io/docs/installation/operator-lifecycle-manager/#configuration-via-clusterserviceversion-csv) OLM resource. It should be a `cert-manager` controller spec. These actions are not required for OpenShift v4.10. ::: ```bash oc get csv oc edit csv cert-manager.${VERSION} ``` ```yaml spec: template: spec: securityContext: fsGroup: 1001 serviceAccountName: cert-manager ``` ![Cert-Manager System Permissions](../../../assets/operator-guide/ssl-okd-config-05.png "Cert-Manager System Permissions") :::info A mutating admission controller will automatically modify all pods running with the service account:
cert-manager controller pod ```yaml apiVersion: apps/v1 kind: Pod # ... spec: # ... serviceAccountName: cert-manager serviceAccount: cert-manager containers: - name: ... # ... env: - name: AWS_ROLE_ARN value: >- arn:aws:iam::XXXXXXXXXXX:role/cert-manager - name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token volumeMounts: - name: aws-iam-token readOnly: true mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount volumes: - name: aws-iam-token projected: sources: - serviceAccountToken: audience: sts.amazonaws.com expirationSeconds: 86400 path: token defaultMode: 420 ```
::: 4. If you have separate public and private DNS zones for the same domain (split-horizon DNS), modify the `cert-manager` `Deployment` in order to validate DNS TXT records via [public recursive nameservers](https://cert-manager.io/docs/configuration/acme/dns01/#setting-nameservers-for-dns01-self-check). :::note Otherwise, you will be getting an error during a record validation: ```bash Waiting for DNS-01 challenge propagation: NS ns-123.awsdns-00.net.:53 returned REFUSED for `_acme-challenge`. ``` To avoid the error, add `--dns01-recursive-nameservers-only --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53` as ARGs to the `cert-manager` controller `Deployment`. ::: ```bash oc get csv oc edit csv cert-manager.${VERSION} ``` ```yaml labels: app: cert-manager app.kubernetes.io/component: controller app.kubernetes.io/instance: cert-manager app.kubernetes.io/name: cert-manager app.kubernetes.io/version: v1.9.1 spec: containers: - args: - '--v=2' - '--cluster-resource-namespace=$(POD_NAMESPACE)' - '--leader-election-namespace=kube-system' - '--dns01-recursive-nameservers-only' - '--dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53' ``` :::note The `Deployment` must be modified via OpenShift [ClusterServiceVersion](https://cert-manager.io/docs/installation/operator-lifecycle-manager/#configuration-via-clusterserviceversion-csv) OLM resource if the operator was installed using the OperatorHub. The OpenShift `ClusterServiceVersion` OLM resource includes several Deployments, and the ARGs must be modified only for the `cert-manager` controller. ::: - Save the resource. After that, OLM will try to reload the resource automatically and save it to the YAML file. If OLM resets the config file, double-check the entered values. ![Cert-Manager Nameservers](../../../assets/operator-guide/ssl-okd-config-06.png "Cert-Manager Nameservers") ### Configure ClusterIssuers `ClusterIssuer` is available on the whole cluster. 1. Create the [ClusterIssuer](https://cert-manager.io/docs/configuration/acme/) resource for Let's Encrypt Staging and Prod environments that signs a Certificate using `cert-manager`. :::note Let's Encrypt has a [limit](https://letsencrypt.org/docs/rate-limits/) of duplicate certificates in the Prod environment. Therefore, a `ClusterIssuer` has been created for [Let's Encrypt Staging](https://letsencrypt.org/docs/staging-environment/) environment. By default, Let's Encrypt Staging certificates will not be trusted in your browser. The certificate validation cannot be tested in the Let's Encrypt Staging environment. ::: - Change `user@example.com` with your contact email. - Replace `hostedZoneID` XXXXXXXXXXX with the DNS **Hosted zone ID** in AWS for your domain. - Replace the region value `${region}`. - The secret under `privateKeySecretRef` will be created automatically by the `cert-manager` operator. ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: email: user@example.com server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-staging-issuer-account-key solvers: - dns01: route53: region: ${region} hostedZoneID: XXXXXXXXXXX ``` ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: email: user@example.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-prod-issuer-account-key solvers: - dns01: route53: region: ${region} hostedZoneID: XXXXXXXXXXX ``` ![Cert-Manager ClusterIssuer](../../../assets/operator-guide/ssl-okd-config-07.png "Cert-Manager ClusterIssuer") 2. Check the `ClusterIssuer` status: ![Cert-Manager ClusterIssuer](../../../assets/operator-guide/ssl-okd-config-08.png "Cert-Manager ClusterIssuer") ```bash oc describe clusterissuer letsencrypt-prod oc describe clusterissuer letsencrypt-staging ``` 3. If the `ClusterIssuer` state is not ready, investigate `cert-manager` controller pod logs: ```bash oc get pod -n openshift-operators | grep 'cert-manager' oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators ``` ### Configure Certificates 1. In two different namespaces, create a [Certificate](https://cert-manager.io/docs/usage/certificate/) resource for the OpenShift Router (Ingress controller for OpenShift) and for the OpenShift APIServer. - OpenShift Router supports a single wildcard certificate for Ingress/Route resources in different namespaces (so called, [default SSL certificate](https://cert-manager.io/docs/faq/sync-secrets/#serving-a-wildcard-to-ingress-resources-in-different-namespaces-default-ssl-certificate)). The Ingress controller expects the certificates in a `Secret` to be created in the `openshift-ingress` namespace; the API Server, in the `openshift-config` namespace. The `cert-manager` operator will automatically create these secrets from the `Certificate` resource. - Replace `${DOMAIN}` with your domain name. It can be checked with `oc whoami --show-server`. Put domain names in quotes.
The certificate for OpenShift Router in the `openshift-ingress` namespace ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: router-certs namespace: openshift-ingress labels: app: cert-manager spec: secretName: router-certs secretTemplate: labels: app: cert-manager duration: 2160h # 90d renewBefore: 360h # 15d subject: organizations: - Org Name commonName: '*.${DOMAIN}' privateKey: algorithm: RSA encoding: PKCS1 size: 2048 rotationPolicy: Always usages: - server auth - client auth dnsNames: - '*.${DOMAIN}' - '*.apps.${DOMAIN}' issuerRef: name: letsencrypt-staging kind: ClusterIssuer ```
The certificate for OpenShift APIServer in the `openshift-config` namespace ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: api-certs namespace: openshift-config labels: app: cert-manager spec: secretName: api-certs secretTemplate: labels: app: cert-manager duration: 2160h # 90d renewBefore: 360h # 15d subject: organizations: - Org Name commonName: '*.${DOMAIN}' privateKey: algorithm: RSA encoding: PKCS1 size: 2048 rotationPolicy: Always usages: - server auth - client auth dnsNames: - '*.${DOMAIN}' - '*.apps.${DOMAIN}' issuerRef: name: letsencrypt-staging kind: ClusterIssuer ```
:::info - `cert-manager` supports [ECDSA](https://cert-manager.io/docs/faq/#is-ecc-elliptic-curve-cryptography-supported) key pairs in the `Certificate` resource. To use it, change RSA `privateKey` to ECDSA: ```yaml privateKey: algorithm: ECDSA encoding: PKCS1 size: 256 rotationPolicy: Always ``` - `rotationPolicy: Always` is highly recommended since `cert-manager` does not [rotate private keys](https://cert-manager.io/docs/usage/certificate/#the-rotationpolicy-setting) by default. - Full `Certificate` spec is described in the `cert-manager` [API documentation](https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificateSpec). ::: 2. Check that the certificates in the namespaces are ready: ![Cert-Manager Certificate Status](../../../assets/operator-guide/ssl-okd-config-09.png "Cert-Manager Certificate Status") ![Cert-Manager Certificate Status](../../../assets/operator-guide/ssl-okd-config-10.png "Cert-Manager Certificate Status") 3. Check the details of the certificates via CLI: ```bash oc describe certificate api-certs -n openshift-config oc describe certificate router-certs -n openshift-ingress ``` 4. Check the cert-manager controller pod logs if the Staging Certificate condition is not ready for more than 7 minutes: ```bash oc get pod -n openshift-operators | grep 'cert-manager' oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators ``` 5. When the certificate is ready, its private key will be put into the OpenShift `Secret` in the namespace indicated in the `Certificate` resource: ```bash oc describe secret api-certs -n openshift-config oc describe secret router-certs -n openshift-ingress ``` ### Modify OpenShift Router and API Server Custom Resources 1. Update the Custom Resource of your Router (Ingress controller). Patch the `defaultCertificate` object value with `{ "name": "router-certs" }`: ```bash oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certs" }}}' --insecure-skip-tls-verify ``` :::info After updating the `IngressController` object, the OpenShift Ingress operator redeploys the router. ::: 2. Update the Custom Resource for the OpenShift API Server: - Export the name of `APIServer`: ```bash export OKD_API=$(oc whoami --show-server --insecure-skip-tls-verify | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././') ``` - Patch the `servingCertificate` object value with `{ "name": "api-certs" }`: ```bash oc patch apiserver cluster --type merge --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [ \"$OKD_API\" ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}" --insecure-skip-tls-verify ``` ### Move From Let's Encrypt Staging Environment to Prod 1. Test the Staging certificate on the OpenShift Admin Console. The `--insecure` flag is used because Let's Encrypt Staging certificates are not trusted in browsers by default: ```bash curl -v --insecure https://console-openshift-console.apps.${DOMAIN} ``` 2. Change `issuerRef` to `letsencrypt-prod` in both `Certificate` resources: ```bash oc edit certificate api-certs -n openshift-config oc edit certificate router-certs -n openshift-ingress ``` ```yaml issuerRef: name: letsencrypt-prod kind: ClusterIssuer ``` :::note In case the certificate reissue is not triggered after that, try to force the certificate renewal with [cmctl](https://cert-manager.io/docs/usage/cmctl/): ```bash cmctl renew router-certs -n openshift-ingress cmctl renew api-certs -n openshift-config ``` If this won't work, delete the `api-certs` and `router-certs` secrets. It should trigger the Prod certificates issuance: ```bash oc delete secret router-certs -n openshift-ingress oc delete secret api-certs -n openshift-config ``` Please note that these actions will lead to logging your account out of the OpenShift Admin Console, since certificates will be deleted. Accept the certificate warning in the browser and log in again after that. ::: 3. Check the status of the Prod certificates: ```bash oc describe certificate api-certs -n openshift-config oc describe certificate router-certs -n openshift-ingress ``` ```bash cmctl status certificate api-certs -n openshift-config cmctl status certificate router-certs -n openshift-ingress ``` 4. Check the web console and make sure it has secure connection: ```bash curl -v https://console-openshift-console.apps.${DOMAIN} ``` ### Troubleshoot Certificates Below is an example of the DNS TXT `challenge` record created by the `cert-manager` operator: ![DNS Validation](../../../assets/operator-guide/ssl-okd-config-11.png "DNS Validation") Use `nslookup` or `dig` tools to check if the DNS propagation for the TXT record is complete: ```bash nslookup -type=txt _acme-challenge.${DOMAIN} dig txt _acme-challenge.${DOMAIN} ``` Otherwise, use web tools like [Google Admin Toolbox](https://toolbox.googleapps.com/apps/dig/#TXT/): ![DNS Validation](../../../assets/operator-guide/ssl-okd-config-12.png "DNS Validation") If the correct TXT value is shown (the value corresponds to the current TXT value in the DNS zone), it means that the DNS propagation is complete and Let's Encrypt is able to access the record in order to validate it and issue a trusted certificate. :::note If the DNS validation challenge self check fails, `cert-manager` will retry the self check with a fixed 10-second retry interval. [Challenges](https://cert-manager.io/docs/concepts/acme-orders-challenges/#challenge-lifecycle) that do not ever complete the self check will continue retrying until the user intervenes by either retrying the `Order` (by deleting the `Order` resource) or amending the associated `Certificate` resource to resolve any configuration errors. ::: As soon as the domain ownership has been verified, any `cert-manager` affected validation TXT records in the AWS Route53 DNS zone will be cleaned up. Please find below the issues that may occur and their troubleshooting: - When certificates are not issued for a long time, or a `cert-manager` resource is not in a Ready state, [describing](https://cert-manager.io/docs/faq/troubleshooting/) a resource may show the reason for the error. - Basically, the `cert-manager` creates the following resources during a `Certificate` issuance: `CertificateRequest`, `Order`, and `Challenge`. [Investigate](https://cert-manager.io/docs/faq/acme/) each of them in case of errors. - Use the [cmctl](https://cert-manager.io/docs/usage/cmctl/) tool to show the state of a `Certificate` and its associated resources. - Check the `cert-manager` controller pod logs: ```bash oc get pod -n openshift-operators | grep 'cert-manager' oc logs -f cert-manager-${replica_set}-${random_string} -n openshift-operators ``` - Certificate error debugging: a. Decode certificate chain located in the secrets: ```bash oc get secret router-certs -n openshift-ingress -o 'go-template={{index .data "tls.crt"}}' | base64 -d | while openssl x509 -noout -text; do :; done 2>/dev/null oc get secret api-certs -n openshift-config -o 'go-template={{index .data "tls.crt"}}' | base64 -d | while openssl x509 -noout -text; do :; done 2>/dev/null ``` ```bash cmctl inspect secret router-certs -n openshift-ingress cmctl inspect secret api-certs -n openshift-config ``` b. Check the SSL RSA private key consistency: ```bash oc get secret router-certs -n openshift-ingress -o 'go-template={{index .data "tls.key"}}' | base64 -d | openssl rsa -check -noout oc get secret api-certs -n openshift-config -o 'go-template={{index .data "tls.key"}}' | base64 -d | openssl rsa -check -noout ``` c. Match the SSL certificate public key against its RSA private key. Their modulus must be identical: ```bash diff <(oc get secret api-certs -n openshift-config -o 'go-template={{index .data "tls.crt"}}' | base64 -d | openssl x509 -noout -modulus | openssl md5) <(oc get secret api-certs -n openshift-config -o 'go-template={{index .data "tls.key"}}' | base64 -d | openssl rsa -noout -modulus | openssl md5) diff <(oc get secret router-certs -n openshift-ingress -o 'go-template={{index .data "tls.crt"}}' | base64 -d | openssl x509 -noout -modulus | openssl md5) <(oc get secret router-certs -n openshift-ingress -o 'go-template={{index .data "tls.key"}}' | base64 -d | openssl rsa -noout -modulus | openssl md5) ``` ### Remove Obsolete Certificate Authority Data From Kubeconfig After updating the certificates, the access to the cluster via Lens or CLI will be denied because of the untrusted certificate errors: ```bash $ oc whoami Unable to connect to the server: x509: certificate signed by unknown authority ``` Such behavior appears because the `oc` tool references an old CA data in the kubeconfig file. :::note Examine the Certificate Authority data using the following command: ```bash oc config view --minify --raw -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d | openssl x509 -text ``` This certificate has the `CA:TRUE` parameter, which means that this is a self-signed root CA certificate. ::: To fix the error, remove the old CA data from your OpenShift kubeconfig file: ```bash sed -i "/certificate-authority-data/d" $KUBECONFIG ``` Since this field will be absent in the kubeconfig file, system root SSL certificate will be used to validate the cluster certificate trust chain. On Ubuntu, Let's Encrypt OpenShift cluster certificates will be validated against `Internet Security Research Group` root in `/etc/ssl/certs/ca-certificates.crt`. ### Certificate Renewals The `cert-manager` automatically [renews](https://cert-manager.io/docs/usage/certificate/#renewal) the certificates based on the X.509 certificate's duration and the `renewBefore` value. The minimum value for the `spec.duration` is 1 hour; for `spec.renewBefore`, 5 minutes. It is also required that `spec.duration` > `spec.renewBefore`. Use the [cmctl](https://cert-manager.io/docs/usage/cmctl/) tool to manually trigger a single instant certificate renewal: ```bash cmctl renew router-certs -n openshift-ingress cmctl renew api-certs -n openshift-config ``` Otherwise, manually renew all certificates in all namespaces with the `app=cert-manager` label: ```bash cmctl renew --all-namespaces -l app=cert-manager ``` Run the `cmctl renew --help` command to get more details. ## Related Articles * [Cert-Manager Official Documentation](https://cert-manager.io/docs/) * [Installing the Cert-Manager Operator for Red Hat OpenShift](https://docs.openshift.com/container-platform/4.11/security/cert_manager_operator/cert-manager-operator-install.html) * [Checking Issued Certificate Details](https://crt.sh/) --- ## KubeRocketCI Infrastructure Providers: Overview # KubeRocketCI Infrastructure Providers: AWS, Azure, GCP, and On-Premises Overview This section provides a technical overview and integration guidelines for Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and on-premises solutions through the perspective of KubeRocketCI. It introduces cross-cutting features such as automated provisioning, scalable deployment configurations, and security best practices specific to each provider within the KubeRocketCI framework. It aims to equip users with the necessary knowledge to deploy and manage KubeRocketCI in various environments efficiently, focusing on leveraging cloud-native technologies and optimizing infrastructure utilization. This concise guide is essential for technical users seeking to understand and implement KubeRocketCI across different infrastructure platforms. --- ## Install Argo CD # Install Argo CD Review the necessary prerequisites and follow the steps to enable Argo CD within KubeRocketCI. ## Prerequisites The following tools must be installed: - [KubeRocketCI](./install-kuberocketci.md) - [Kubectl version 1.34+](https://kubernetes.io/docs/tasks/tools/) - [Helm version 3.10+](https://github.com/helm/helm/releases) - [Keycloak](./auth/keycloak.md) (optional) ## Installation Enabling Argo CD on the platform involves two main steps: - Argo CD installation - Argo CD integration with Add-Ons Argo CD can be installed in several ways, please follow the [official documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/) for more details. It is also possible to install Argo CD using the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/argo-cd). ### Install With Helm Follow the steps below to install Argo CD using Helm: :::warning "For the OpenShift users" When using the OpenShift platform, apply the `SecurityContextConstraints` resource. Change the namespace in the `users` section if required.
View: SecurityContextConstraints ```yaml allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: null apiVersion: security.openshift.io/v1 allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - min: 99 max: 65543 groups: [] kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: argo-redis-ha priority: 1 readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsRange uidRangeMin: 1 uidRangeMax: 65543 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny seccompProfiles: - '*' users: - system:serviceaccount:argocd:argo-redis-ha - system:serviceaccount:argocd:argo-redis-ha-haproxy - system:serviceaccount:argocd:argocd-notifications-controller - system:serviceaccount:argocd:argo-argocd-repo-server - system:serviceaccount:argocd:argocd-server volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
::: 1. Fork the [Add-Ons](https://github.com/epam/edp-cluster-add-ons/tree/main/argo-cd) repository to personal account. 2. Run the installation: ```bash helm dependency update argo-cd helm install argocd argo-cd -n argocd --create-namespace ``` 3. Port-forward Argo CD service using kubectl: ```bash kubectl port-forward svc/argocd-server 8080:80 -n argocd ``` 4. Login to the Argo CD server in browser using login and password: :::info By default, to access the console with administrative privileges, use the following credentials: - URL: localhost:8080 - Login: admin ::: ```bash title="password" kubectl get secret -n argocd argocd-initial-admin-secret --template={{.data.password}} | base64 -d ``` 5. Integrate Argo CD with [Add-Ons](add-ons-overview.md) and Install Core Add-Ons To ensure the environment functions correctly, it is necessary to install the following core components: - **Nginx Ingress Controller**: Provides external endpoint management through ingress resources. - **External Secrets Operator (ESO)**: Manages secure provisioning and access to secrets. - **Keycloak (KK)**: Enables Single Sign-On (SSO) for authentication. - **Keycloak Operator**: Simplifies the configuration and management of Keycloak instances. These components can be installed using the prepared [add-ons repository](https://github.com/epam/edp-cluster-add-ons). Detailed installation instructions are available in the [add-ons installation guide](add-ons-overview.md). To configure these components, refer to the relevant sections in `values.yaml`, which include the required parameters for enabling and customizing the add-ons. ![Add-Ons Overview](../assets/operator-guide/argocd-addons.png "Add-Ons Overview") After completing the installation of these components, proceed with the configuration of Argo CD. 6. Update Argo CD Helm chart: :::note After installing the necessary add-ons, such as the External Secrets Operator, it is recommended to add ESO components for all secrets that were manually created during the integration of [add-ons with Argo CD](add-ons-overview.md). An example can be found [here](https://github.com/epam/edp-cluster-add-ons/blob/main/argo-cd/templates/external-secrets/externalsecret-argocd-github.yaml). :::info Make sure Kubernetes resources have the required labels for Argo CD to manage them. ```yaml metadata: label: app.kubernetes.io/part-of: argocd ``` ::: To enable features such as ingress, login via OIDC provider, provisioning secret using ESO need to update Argo CD with [values](https://github.com/epam/edp-cluster-add-ons/blob/main/argo-cd/values.yaml) below: ```yaml title=" kubernetes values.yaml" argo-cd: global: # -- Default domain used by all components ## Used for ingresses, certificates, SSO, notifications, etc. domain: argocd.example.com configs: cm: # required when SSO is enabled url: "https://argocd.example.com" oidc.config: | issuer: https://keycloak.example/auth/realms/shared server: ingress: enabled: true hosts: - "argocd.example.com" oidc: enabled: true eso: enabled: true ```
View: openshift values.yaml ```yaml argo-cd: global: # -- Default domain used by all components ## Used for ingresses, certificates, SSO, notifications, etc. domain: argocd.example.com configs: cm: # required when SSO is enabled url: "https://argocd.example.com" oidc.config: | issuer: https://keycloak.example/auth/realms/shared server: route: enabled: true hostname: "argocd.example.com" termination_type: edge termination_policy: Redirect oidc: enabled: true eso: enabled: true ```
## Next Step - Install third-party tools via [Add-Ons](add-ons-overview.md) - Install [KubeRocketCI](install-kuberocketci.md) After completing the steps outlined in this documentation, the following results should be achieved: 1. **Centralized Argo CD Instance**: - A fully deployed and operational Argo CD instance installed using Helm. 2. **Add-Ons Repository Configured**: - Add-ons repository integrated and configured with Argo CD for managing additional components. 3. **Core Components Deployed**: - Essential infrastructure components, such as: - **External Secrets Operator (ESO)** for secure secret management. - **Keycloak** and **Keycloak Operator** for Single Sign-On (SSO). - **Nginx Ingress Controller** for external endpoint management. ## Related Articles * [Argo CD Integration](cd/argocd-integration.md) * [Install via Add-Ons](add-ons-overview.md) --- ## Install NGINX Ingress Controller # Install NGINX Ingress Controller Inspect the prerequisites and the main steps to perform for installing [Install NGINX Ingress Controller](https://docs.nginx.com/nginx-ingress-controller/intro/overview/) on Kubernetes. ## Prerequisites - Kubectl version 1.34+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/) for details. - [Helm](https://helm.sh) version 3.10+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation To install the ingress-nginx chart, follow the steps below: 1. Create an ingress-nginx namespace: ```bash kubectl create namespace ingress-nginx ``` 2. Add a chart repository: ```bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update ``` 3. Install the ingress-nginx chart: ```bash helm install ingress ingress-nginx/ingress-nginx \ --version 4.7.0 \ --values values.yaml \ --namespace ingress-nginx ``` Check out the _values.yaml_ file sample of the ingress-nginx chart customization:
View: values.yaml ```yaml controller: addHeaders: X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN resources: limits: memory: "256Mi" requests: cpu: "50m" memory: "128M" config: ssl-redirect: 'true' client-header-buffer-size: '64k' http2-max-field-size: '64k' http2-max-header-size: '64k' large-client-header-buffers: '4 64k' upstream-keepalive-timeout: '120' keep-alive: '10' use-forwarded-headers: 'true' proxy-real-ip-cidr: '172.32.0.0/16' proxy-buffer-size: '8k' # To watch Ingress objects without the ingressClassName field set parameter value to true. # https://kubernetes.github.io/ingress-nginx/#i-have-only-one-ingress-controller-in-my-cluster-what-should-i-do watchIngressWithoutClass: true service: type: NodePort nodePorts: http: 32080 https: 32443 updateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate metrics: enabled: true defaultBackend: enabled: true serviceAccount: create: true name: nginx-ingress-service-account ```
:::info Align value **controller.config.proxy-real-ip-cidr** with your [network settings](https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#source-ip-address). ::: :::note It is also possible to install the ingress controller via cluster add-ons. For details, please refer to the [Install via Add-Ons](add-ons-overview.md) page. ::: ## Related Articles * [Install via Add-Ons](add-ons-overview.md) * [Install KubeRocketCI](install-kuberocketci.md) --- ## How to Install KubeRocketCI: Advanced Setup Guide # Install KubeRocketCI Inspect the main steps to install KubeRocketCI. Please check the [Prerequisites Overview](prerequisites.md) page before starting the installation. Also, to authenticate each of the release artifacts, please refer to the [Verification of KubeRocketCI Artifacts](../developer-guide/artifacts-verification.md) guide. There are multiple ways to deploy KubeRocketCI: - Using Helm (see below) - Using [Argo CD GitOps approach](./add-ons-overview.md) - Using [AWS Marketplace](./aws-marketplace-install.md) :::note The installation process below is given for a Kubernetes cluster. The steps that differ for an OpenShift cluster are indicated in the notes. ::: :::warning KubeRocketCI is aligned with industry standards for storing and managing sensitive data, ensuring optimal security. However, the use of custom solutions introduces uncertainties, thus the responsibility for the safety of your data is fully covered by the platform administrator. ::: 1. (Optional) KubeRocketCI manages secrets via `External Secret Operator` to integrate with a variety of tools. For more information about the secrets being used and how they are utilized, please refer to the [External Secrets Operator Integration](./secrets-management/external-secrets-operator-integration.md) guide. 2. (Optional) Deploy and configure Keycloak to enable Single Sign-On approach. To see the details on how to configure Keycloak correctly, please refer to the [Install Keycloak](./auth/keycloak.md#configuration) page. 3. Add the Helm charts repository: ```bash helm repo add epamedp https://epam.github.io/edp-helm-charts/stable ``` 4. Choose the required Helm chart version: ```bash helm search repo epamedp/edp-install NAME CHART VERSION APP VERSION DESCRIPTION epamedp/edp-install 3.14.1 3.14.1 A Helm chart for KubeRocketCI Platform ``` :::note It is highly recommended to use the latest released version. ::: 5. KubeRocketCI can be integrated with the following version control systems: - [GitHub](https://docs.github.com/en) (by default) - [GitLab](https://docs.gitlab.com/) - [Gerrit](https://gerrit-review.googlesource.com/Documentation/) This integration implies in what system the development of the application will be or is already being carried out. The `global.gitProviders` flag in the edp-install controls this integration: Global VCS configuration (can be multiple values): ```yaml title="values.yaml" ... global: gitProviders: - gerrit - github - gitlab - bitbucket ... ``` Tekton event listner configuration: ``` yaml title="values.yaml" ... edp-tekton: gitServers: github: gitProvider: github host: github.com webhook: skipWebhookSSLVerification: false eventListener: enabled: true resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "500m" ingress: enabled: true ... ``` ``` yaml title="values.yaml" ... edp-tekton: gitServers: gitlab: gitProvider: gitlab host: gitlab.com webhook: skipWebhookSSLVerification: false eventListener: enabled: true resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "500m" ingress: enabled: true ... ``` ``` yaml title="values.yaml" ... edp-tekton: gitServers: bitbucket: gitProvider: bitbucket host: bitbucket.org webhook: skipWebhookSSLVerification: false eventListener: enabled: true resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "500m" ingress: enabled: true ... ``` ``` yaml title="values.yaml" ... gerrit-operator: enabled: true ... edp-tekton: gitServers: gerrit: eventListener: enabled: true ingress: enabled: false resources: limits: cpu: 500m memory: 128Mi requests: cpu: 50m memory: 64Mi gitProvider: gerrit gitUser: edp-ci host: gerrit.edp nameSshKeySecret: gerrit-ciuser-sshkey quickLink: host: gerrit.example.com sshPort: webhook: skipWebhookSSLVerification: false ... ``` Gerrit server can be deployed as a part of KubeRocketCI platform when the Gerrit is enabled. For more details on how to integrate KubeRocketCI with GitLab or GitHub, please refer to the [Add GitServer](../user-guide/add-git-server.md) page. 6. Configure Container Registry. KubeRocketCI supports integration with multiple container registries, providing flexibility for projects using either OpenShift or Kubernetes clusters. The supported container registries include AWS ECR, DockerHub, Harbor, OpenShift (OCR), Nexus, and GitHub (GHCR). This integration simplifies the process of storing and managing container images across various environments. For more details on integrating these container registries with KubeRocketCI, refer to the following documentation: [Container Registries Integration Guide](../user-guide/change-container-registry.md) and [Manage Container Registries](../user-guide/manage-container-registries.md) For example, to enable [Harbor](artifacts-management/harbor-integration.md) as a registry storage, use the values below: ```yaml global: dockerRegistry: type: "harbor" url: "harbor.example.com" ``` 7. Install platform in the **krci** namespace with the Helm tool: Check the parameters in the installation chart [values.yaml](https://github.com/epam/edp-install/blob/v3.14.1/deploy-templates/values.yaml) file. ```bash helm install krci epamedp/edp-install --wait --timeout=900s \ --version 3.14.1 \ --values values.yaml \ --namespace krci \ --create-namespace ``` See the example on the parameters below: ```yaml title="Example values.yaml file" global: # -- platform type that can be either "kubernetes" or "openshift" platform: "kubernetes" # DNS wildcard for routing in the Kubernetes cluster; dnsWildCard: "example.com" # -- Administrators of your tenant # -- Can be gerrit, github or gitlab. By default: github gitProviders: - github dockerRegistry: # -- Docker Registry endpoint url: ".dkr.ecr..amazonaws.com" type: "ecr" space: "edp" sso: enabled: false # Keycloak address with which the platform will be integrated keycloakUrl: "https://keycloak.example.com" admins: - "stub_user_one@example.com" developers: - "stub_user_one@example.com" - "stub_user_two@example.com" edp-tekton: # Tekton Kaniko configuration section kaniko: # -- AWS IAM role to be used for kaniko pod service account (IRSA). Format: arn:aws:iam:::role/ roleArn: arn:aws:iam:::role/ gitServers: github: gitProvider: github host: github.com webhook: skipWebhookSSLVerification: false eventListener: enabled: true resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "500m" ingress: enabled: true krci-portal: ingress: enabled: true configEnv: PORTAL_URL: https://krci-portal.example.com GITFUSION_URL: http://gitfusion.:8080 DEFAULT_CLUSTER_NAMESPACE: DEFAULT_CLUSTER_NAME: core ``` :::note Set `global.platform=openshift` while deploying KubeRocketCI in OpenShift. ::: :::info The full installation with integration between tools will take at least 5 minutes. ::: 8. To check if the installation is successful, run the command below: ```bash helm status krci -n krci ``` You can also check ingress endpoints to get the Portal endpoint to enter KubeRocketCI portal: ```bash kubectl get ingress portal -n krci ``` 9. Once KubeRocketCI is successfully installed, you can explore our [Use Cases](../use-cases/index.md) to try out the functionality of the KubeRocketCI platform. ## Related Articles * [Quick Start](../quick-start/platform-installation.md) * [Manage Git Providers](../user-guide/add-git-server.md) * [Set Up Kubernetes](kubernetes-cluster-settings.md) * [Set Up OpenShift](openshift-cluster-settings.md) * [KubeRocketCI Installation Prerequisites Overview](prerequisites.md) * [UI Portal OIDC Integration](auth/ui-portal-oidc.md) * [Verification of KubeRocketCI Artifacts](../developer-guide/artifacts-verification.md) --- ## Install Tekton # Install Tekton EPAM KubeRocketCI uses Tekton resources, such as Tasks, Pipelines, Triggers, Interceptors, and Chains for running the CI/CD pipelines. Here are the main steps to follow for installing the Tekton resources using the Tekton release files. ## Prerequisites - Kubectl version 1.34+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/) for details. - For Openshift/OKD, the latest version of the `oc` utility is required. Please refer to the [OKD page](https://github.com/okd-project/okd/releases) on GitHub for details. ## Installation on Kubernetes Cluster To install Tekton resources, follow the steps below: :::info Please refer to the [Install Tekton Pipelines](https://tekton.dev/docs/installation/pipelines/) and [Install and set up Tekton Triggers](https://tekton.dev/docs/installation/triggers/) sections for details. ::: 1. Install Tekton pipelines v1.6.2 using the release file: :::note Tekton Pipeline resources are used for managing and running KubeRocketCI Tekton Pipelines and Tasks. Please refer to the [KubeRocketCI Tekton Pipelines](https://github.com/epam/edp-tekton/tree/master/charts/pipelines-library/templates/pipelines) and [KubeRocketCI Tekton Tasks](https://github.com/epam/edp-tekton/tree/master/charts/pipelines-library/templates/tasks) pages for details. ::: ```bash kubectl apply -f https://infra.tekton.dev/tekton-releases/pipeline/previous/v1.6.2/release.yaml ``` 2. Install Tekton Triggers v0.36.0 using the release file: :::note Tekton Trigger resources are used for managing and running KubeRocketCI Tekton EventListeners, Triggers, TriggerBindings and TriggerTemplates. Please refer to the [KubeRocketCI Tekton Triggers](https://github.com/epam/edp-tekton/tree/master/charts/pipelines-library/templates/triggers) page for details. ::: ```bash kubectl apply -f https://infra.tekton.dev/tekton-releases/triggers/previous/v0.36.0/release.yaml ``` 3. Install Tekton Interceptors v0.36.0 using the release file: :::note The Platform uses GitLab, GitHub and Cel ClusterInterceptors for processing requests from webhooks. ::: ```bash kubectl apply -f https://infra.tekton.dev/tekton-releases/triggers/previous/v0.36.0/interceptors.yaml ``` 4. Install Tekton Chains v0.28.1 using the release file: ```bash kubectl apply -f https://infra.tekton.dev/tekton-releases/chains/previous/v0.28.1/release.yaml ``` 5. Install Tekton Results v0.20.0 using the KubeRocketCI manifest: :::warning Unlike the components above, Tekton Results **must not** be installed from the plain upstream release file. The Platform customizes it, and the upstream `release.yaml` bundles its own PostgreSQL instance that conflicts with the Platform database. Use the [results.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results.yaml) manifest from the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository instead. ::: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-cluster-add-ons/main/clusters/core/addons/tekton/results.yaml ``` The manifest is based on the upstream `release_base.yaml` (the variant without a bundled database) and adds the following Platform customizations: - **External database.** Connection settings point to the PostgreSQL cluster defined in [results-pg.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results-pg.yaml), which must be deployed first. See [Tekton Long-Term Storage](./ci/tekton-long-term-storage.md) for details. - **Summary labels.** The watcher runs with `-summary_labels`, which records the codebase, branch, pipeline type, and CD pipeline labels on archived runs. The KubeRocketCI Portal relies on these to render the pipeline run history, so omitting the flag leaves history entries without their metadata. - **Log storage.** The Results API serves pipeline logs from a persistent volume, and a CronJob prunes entries older than 30 days. ## Installation on OKD cluster To install Tekton resources, follow the steps below: :::info Please refer to the [Install Tekton Operator](https://tekton.dev/docs/operator/) documentation for details. ::: :::note Tekton Operator also deploys [Pipelines as Code CI](https://pipelinesascode.com/) that requires OpenShift v4.11 (based on Kubernetes v1.24) or higher. This feature is optional and its deployments can be scaled to zero replicas. ::: Install Tekton Operator v0.67.0 using the release file: ```bash kubectl apply -f https://github.com/tektoncd/operator/releases/download/v0.67.0/openshift-release.yaml ``` After the installation, the Tekton Operator will install the following components: Pipeline, Trigger, and Addons. :::note If there is the following error in the `openshift-operators` namespace for `openshift-pipelines-operator` and `tekton-operator-webhook` deployments: ```bash Error: container has runAsNonRoot and image will run as root ``` Patch the deployments with the following commands: ```bash kubectl -n openshift-operators patch deployment openshift-pipelines-operator -p '{"spec": {"template": {"spec": {"securityContext": {"runAsUser": 1000}}}}}' kubectl -n openshift-operators patch deployment tekton-operator-webhook -p '{"spec": {"template": {"spec": {"securityContext": {"runAsUser": 1000}}}}}' ``` ::: Grant access for Tekton Service Accounts in the `openshift-pipelines` namespace to the Privileged SCC: ```bash oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-operators-proxy-webhook oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-pipelines-controller oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-pipelines-resolvers oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-pipelines-webhook oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-triggers-controller oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-triggers-core-interceptors oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:tekton-triggers-webhook oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:pipelines-as-code-controller oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:pipelines-as-code-watcher oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:pipelines-as-code-webhook oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-pipelines:default ``` ## Related Articles * [Install via Add-Ons](add-ons-overview.md) --- ## Install via Civo Marketplace # Install via Civo Marketplace This documentation provides detailed instructions on how to install the KubeRocketCI platform via [Civo Marketplace](https://www.civo.com/marketplace/kuberocketci). As a prerequisite, make sure to [sign up](https://dashboard.civo.com/signup) on Civo. ## Launch Cluster The first step of the installation procedure is to launch the cluster. Please refer to the [official instructions](https://www.civo.com/docs/kubernetes/create-a-cluster) that describe this process in detail. To succeed, follow the steps below: 1. Log in to your personal Civo account. 2. Create a new Kubernetes cluster with the following parameters. Please refer to the [official guidelines](https://www.civo.com/docs/kubernetes/create-a-cluster) for more details: - **Name**: `demo` - **How many nodes**: `1` - **Select size**: Type: `Standard`, Size: `Medium` - **Network**: `Default` - **Firewall**: `Create a new firewall` with the `6443` and `443` ports opened - **Advanced options**: Kubernetes version: latest (currently 1.34.0) - **Marketplace**: From the `CI/CD` selection choose `Argo CD` and `Tekton` for pre-installation. 3. Wait till the cluster is created. :::note The process of deploying the cluster typically takes about two minutes. Following this, an extra five minutes are needed for the deployment of both the Argo CD and Tekton. ::: 4. As soon as cluster is deployed, ensure all the marketplace applications are installed, too: ![Check applications](../assets/operator-guide/civo-installed-applications.png "Check applications") 5. Download and apply the kubeconfig file: ![Download kubeconfig](../assets/operator-guide/download-kubeconfig.png "Download kubeconfig") 6. Ensure all the pods are up and running in both the Tekton and Argo CD namespaces after 5 minutes of waiting. Restart deployments if the pods are failed to deploy: ```bash kubectl get ns kubectl get pods -n tekton-pipelines kubectl get pods -n argocd ``` ![Verify installation](../assets/operator-guide/check-ns-and-pods.png "Verify installation") ## Install KubeRocketCI As soon as the cluster is deployed, it is time to install the KubeRocketCI application. 1. In the Civo portal, navigate to **Marketplace** -> **CI/CD**: ![Civo Marketplace](../assets/operator-guide/civo-marketplace.png "Civo Marketplace") 2. Select `KubeRocketCI` and choose which Version Control Systems you would prefer to integrate it with and click the **Install Apps** button: ![Add KubeRocketCI](../assets/operator-guide/civo-add-kuberocketci.png "Add KubeRocketCI") 3. Wait till the KubeRocketCI app appears in the **Installed applications** list: ![KubeRocketCI installed](../assets/operator-guide/civo-kuberocketci-installed.png "KubeRocketCI installed") 4. Wait till all the pods are up and running. Use the `kubectl get pods` command to check the status of the pods: ```bash kubectl get pods -n krci ``` ![KubeRocketCI pods](../assets/operator-guide/civo-check-kuberocketci-ns.png "KubeRocketCI pods") 5. As soon as all the pods are deployed. Navigate to the **Cluster Information** tab and copy the DNS name: ![Getting DNS](../assets/operator-guide/civo-get-dns.png "Getting DNS") 6. In the new browser tab, access the KubeRocketCI portal by typing the URL according to the `https://krci-portal.` format. :::note You can find the exact portal URL in the krci-portal deployment details. ::: 7. Create the edp-admin service account and generate an access token to open the KubeRocketCI: ```bash kubectl -n krci create serviceaccount edp-admin kubectl create clusterrolebinding edp-admin --serviceaccount=krci:edp-admin --clusterrole=cluster-admin kubectl create token edp-admin -n krci ``` 8. In the login menu, paste the generated token in the **Access token** field and click the **Sign in** button. ![Portal login menu](../assets/quick-start/edp_portal_login_menu.png "Portal login menu") 9. Upon logging in, open the namespaces window by clicking the **Manage Namespaces** button in the top right corner of the UI: ![Specify namespaces](../assets/quick-start/edp_portal_ui.png "Specify namespaces") 10. Ensure the `krci` namespace is specified in both default and allowed namespaces. If not, define them manually: - Default namespace: `krci` - Allowed namespaces: `krci` ![Cluster Settings menu](../assets/quick-start/cluster_settings.png "Cluster Settings menu") :::note Remember to press **Enter** to add the allowed namespace to the list. ::: Well done! You've successfully installed the KubeRocketCI platform on the Civo cluster. You're now set to integrate KubeRocketCI with the necessary third-party tools. Proceed to the [SonarQube Integration](../quick-start/integrate-sonarcloud.md) page. ## Related Articles * [Install via AWS Marketplace](aws-marketplace-install.md) * [KubeRocketCI Installation](install-kuberocketci.md) * [SonarQube Integration](../quick-start/integrate-sonarcloud.md) * [Argo CD Integration](cd/argocd-integration.md) --- ## Installation Overview # Installation Overview This documentation provides a concise introduction to the KubeRocketCI installation options. KubeRocketCI can be installed using different approaches. Each of them comes with its own benefits: - [Quick Start Guide](../quick-start/platform-installation.md): This option is ideal for users looking to quickly deploy KubeRocketCI with minimal prerequisites. Watch our [video tutorial](https://www.youtube.com/watch?v=ILlY4niCWeU) for a guided installation experience. - [Install via Add-Ons](add-ons-overview.md): Recommended for advanced KubeRocketCI users, this installation streamlines the setup of supported integrations and follows the GitOps approach. Familiarity with Argo CD is recommended. - [Install via Helm Charts](install-kuberocketci.md): While more complex, this method offers the highest level of flexibility, allowing for customization to suit specific needs. It requires a good understanding of KubeRocketCI. - [Install via AWS Marketplace](aws-marketplace-install.md): Optimal for users leveraging AWS infrastructure, offering a minimal setup for testing KubeRocketCI functionality. - [Install via Civo Marketplace](install-via-civo.md): Preferred for users deploying KubeRocketCI in a Civo Kubernetes cluster. Follow our step-by-step [video tutorial](https://www.youtube.com/watch?v=QjZoPnIKDtA) for onboarding KubeRocketCI to a new Civo cluster. ## Related Articles * [Quick Start Guide](../quick-start/platform-installation.md) * [Quick Start Overview](../quick-start/quick-start-overview.md) * [Install KubeRocketCI](install-kuberocketci.md) * [Install via Add-Ons](add-ons-overview.md) * [Install via AWS Marketplace](aws-marketplace-install.md) * [Install via Civo Marketplace](install-via-civo.md) * [Deploy AWS EKS Cluster](deploy-aws-eks.md) --- ## Efficient Kubernetes Autoscaling With Karpenter and KEDA: A Comprehensive Guide # Efficient Kubernetes Autoscaling With Karpenter and KEDA: A Comprehensive Guide In modern cloud environments, efficient resource management is essential to maintain performance while minimizing costs. This documentation provides a step-by-step guide to configuring pods and cluster autoscaling in a Kubernetes environment using Karpenter and KEDA. Karpenter dynamically provisions and removes nodes based on real-time demand, ensuring that the cluster has just the right amount of capacity. KEDA, on the other hand, scales workloads by adjusting pod replicas based on key metrics, such as pipeline activity and user requests. By implementing this approach, the cluster remains responsive to workload fluctuations, preventing over-provisioning and reducing unnecessary resource usage. This results in a cost-effective, scalable, and efficient infrastructure that adapts to real-time demands without manual intervention. ## Prerequisites Before setting up AutoScaling, ensure the following requirements are met: - [**Prometheus is installed**](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/prometheus-operator) and exporting cluster metrics. - [**Nginx Ingress is installed**](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/ingress-nginx) and configured to export its [metrics](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/ingress-nginx/values.yaml#L38) to Prometheus. ## Karpenter Configuring and installing Karpenter involves several key steps: - **Setting up AWS resources** – configuring IAM roles, permissions, and networking. - **Installing Karpenter** – deploying Karpenter controller in the Kubernetes cluster. - **Configuring essential components** – setting up `Node Pools` and `Node Classes` to enable automatic node provisioning. ### AWS Configuration To enable Karpenter to manage node provisioning effectively, it is essential to properly configure the necessary AWS resources. These steps include: - **Creating IAM roles and policies** – Karpenter requires specific permissions to provision and manage EC2 instances. - **Tagging VPC Subnets and Security Groups** – Karpenter uses these tags to determine which network subnets and security groups should be applied when provisioning new nodes. :::info Karpenter does not use Auto Scaling Groups (ASG) for node creation. Instead, it provisions EC2 instances directly and registers them with the Kubernetes cluster. ::: #### IAM Role and Policies Karpenter needs specific IAM permissions to manage EC2 instances, including the ability to create and terminate EC2 instances in your AWS account. It also requires permissions to register and manage nodes in the Kubernetes cluster. To simplify the setup, we recommend using the [terraform-aws-karpenter](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest/submodules/karpenter) module. This module automates the creation of the necessary IAM roles and policies for Karpenter. This setup assumes your cluster uses on-demand instances with 24/7 availability. If your cluster configuration differs, adjust the settings to match your needs. You can find Karpenter configuration parameters in the [terraform-aws-platform](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/eks/irsa.tf#L345) repository. With this configuration, the following IAM roles will be created: - **KarpenterControllerRole-\**: This role also includes `system:nodes` access to the Kubernetes cluster and will be used by the [Karpenter controller](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/karpenter/values.yaml#L33) deployed within the cluster. This role contains the `KarpenterController` custom policy. - **KarpenterNodeRole-\**: This role will be used by Karpenter to manage EC2 instances. We will configure this role later in the Karpenter [Node Class setup](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/karpenter-np/templates/node-class.yaml#L8). This role contains the following policies: - `AmazonEC2ContainerRegistryReadOnly` - `AmazonEKS_CNI_Policy` - `AmazonEKSWorkerNodePolicy` - `AmazonSSMManagedInstanceCore` #### Configure Network and Security Groups Karpenter needs to know which subnets and security groups to use when provisioning new nodes. This is done by tagging your VPC subnets and security groups. Verify the tags in your required subnets and security groups. If necessary, modify your [terraform configuration](https://github.com/KubeRocketCI/terraform-aws-platform/blob/master/vpc/main.tf#L37) to add new, unique tags. These tags will be used in the Karpenter [Node Class configuration](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/karpenter-np/templates/node-class.yaml). ### Install Karpenter To deploy Karpenter in your Kubernetes cluster, we recommend using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/karpenter). For best practices, Karpenter should be deployed on nodes that are not managed by Karpenter itself. This ensures that Karpenter can manage the cluster independently of its own scaling actions. To achieve this, you should use the affinity configuration in the values.yaml file. Additionally, Karpenter requires the `KarpenterControllerRole-` IAM role to manage the cluster's nodes. This role must be specified in the Service Account configuration. The clusterName should also be set in the settings section. Here's an example of how to configure values.yaml before installation: ```yaml title="values.yaml" karpenter: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: karpenter.sh/nodepool operator: DoesNotExist # -- Karpenter IAM role to manage cluster nodes serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::0123456789:role/KarpenterControllerRole- # -- EKS cluster name settings: clusterName: ``` ### Deploy Karpenter Resources Once Karpenter is installed, you need to create a Node Class and Node Pool. The Node Class defines the configuration for nodes that Karpenter will provision, while the Node Pool specifies the criteria for selecting which nodes to use. We recommend using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/karpenter-np/values.yaml) to deploy Karpenter resources. First, adjust the values.yaml file according to your specific requirements: ```yaml title="values.yaml" karpenter: # -- AMI used by nodes in the EKS cluster amiID: ami-XXXXXXXXXXXXXXXXX # -- EKS cluster name (must match the Karpenter configuration) clusterName: "" # Instance type settings instanceType: category: ["m"] family: ["m7i"] size: ["xlarge"] type: ["on-demand"] ``` You can adjust the Node Pools and Node Class configuration based on your workload requirements. For instance, you can specify different instance types, categories, and families. More detailed information on configuring Node Pools and Node Classes can be found in the [Karpenter documentation](https://karpenter.sh/docs/). ### Verify Karpenter Functionality To verify the functionality of Karpenter, you can create a pod and adjust the number of replicas in its configuration. 1. Create a deployment with the zero replicas: ```bash cat < # Kubernetes Cluster Scaling KubeRocketCI provides a solution for dynamically scaling pods and nodes in a Kubernetes cluster. This approach ensures resources are provisioned based on demand, reducing latency and optimizing costs. By automatically adjusting capacity, the cluster remains responsive to workload fluctuations, improving performance during peak loads while minimizing resource waste during low activity. This solution enhances reliability by preventing resource shortages, supports high availability by efficiently distributing workloads, and simplifies infrastructure management by automating scaling decisions. ## Architecture Overview KubeRocketCI can use these tools for automatic resource scaling: - **Karpenter** – dynamically manages node scaling in the cluster, creating new instances as the load increases and removing resources when demand decreases. - **KEDA** – adjusts the number of pods based on workload, ensuring flexibility and optimal resource utilization. The diagram below illustrates the AWS EKS cluster architecture, including key components, dependencies, and scaling mechanisms: - **System Node** – contains control components such as Tekton Controller, KEDA, Ingress Controller, and Karpenter. - **Tekton Node** – runs CI/CD pipelines. - **Dynamic Node** – hosts operators and services that are scaled by Karpenter. - **Auto Scaling Groups (ASG)** – manage system and Tekton Nodes. ![Architecture diagram](../../assets/operator-guide/kubernetes-cluster-scaling/architecture-diagram.png "Architecture diagram") Automatic scaling ensures efficient resource allocation, reduces latency, and optimizes costs. Read the [Efficient Kubernetes Autoscaling with Karpenter and KEDA: A Comprehensive Guide](./namespace-and-cluster-autoscaling.md) to set up automatic scaling in your cluster. --- ## Set Up Kubernetes # Set Up Kubernetes Make sure the cluster meets the following conditions: 1. Kubernetes cluster is installed with minimum 2 worker nodes with total capacity 8 Cores and 32Gb RAM. 2. Machine with [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) is installed with a cluster-admin access to the Kubernetes cluster. 3. Ingress controller is installed in a cluster, for example [ingress-nginx](./install-ingress-nginx.md). 4. Ingress controller is configured with the disabled HTTP/2 protocol and header size of 64k support. Find below an example of the Config Map for the NGINX Ingress controller: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: client-header-buffer-size: 64k large-client-header-buffers: 4 64k use-http2: "false" ``` 5. Load balancer (if any exists in front of the Ingress controller) is configured with session stickiness, disabled HTTP/2 protocol and header size of 32k support. 6. Keycloak instance is installed. To get accurate information on how to install Keycloak, please refer to the [Install Keycloak](auth/keycloak.md) instruction. 7. Helm 3.10 or higher is installed on the installation machine with the help of the [Installing Helm](https://v3.helm.sh/docs/intro/install/) instruction. 8. Storage classes are used with the [Retain Reclaim Policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain) and [Delete Reclaim Policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#delete). 9. We recommended using our storage class as [default storage class](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/#changing-the-default-storageclass). :::info By default, KubeRocketCI uses the `default Storage Class` in a cluster. We recommend using the following Storage Classes. Storage class templates with the Retain and Delete Reclaim Policies: ``` yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc annotations: storageclass.kubernetes.io/is-default-class: 'true' allowedTopologies: [] mountOptions: [] provisioner: ebs.csi.aws.com reclaimPolicy: Retain volumeBindingMode: Immediate ``` ``` yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 annotations: storageclass.kubernetes.io/is-default-class: 'true' allowedTopologies: [] mountOptions: [] provisioner: ebs.csi.aws.com reclaimPolicy: Delete volumeBindingMode: Immediate allowVolumeExpansion: true ``` ``` yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3-retain allowedTopologies: [] mountOptions: [] provisioner: ebs.csi.aws.com reclaimPolicy: Retain volumeBindingMode: Immediate allowVolumeExpansion: true ``` ::: ## Related Articles * [Install Amazon EBS CSI Driver](infrastructure-providers/aws/ebs-csi-driver.md) * [Install NGINX Ingress Controller](install-ingress-nginx.md) * [Install Keycloak](auth/keycloak.md) --- ## Argo CD Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With Argo CD Configure Argo CD single sign-on using Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application, mapping Entra groups to Argo CD RBAC roles, and updating the Argo CD Helm chart — the same platform-access pattern used for the [AWS EKS cluster and KubeRocketCI Portal](./aws-eks-portal-authentication.md). ## Prerequisites for Argo CD SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [Argo CD](../install-argocd.md) is installed (can be installed during **Configuring Helm chart** step). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. - (Optional) [External Secrets Operator](../secrets-management/install-external-secrets-operator.md) is installed. ## Registering the Argo CD Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for the Argo CD, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** (you can skip setting the **Redirect URI** if you don't deploy Argo CD yet). Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///auth/callback`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**: ![Authentication settings](../../assets/operator-guide/microsoft-entra-auth/argocd-authentication-settings.png) 6. Navigate to the **Certificates & secrets** section. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/argocd-client-secrets.png) 7. Copy the generated client secret value and store it securely. You will need this value to configure the Argo CD Helm chart: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/argocd-client-secret.png) 8. Navigate to the **Token configuration** section and click on **Add group claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/argocd-token-configuration.png) Also, add the **preferred_username** and **email** optional claims: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/argocd-token-configuration-2.png) 9. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/argocd-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. ## Creating the Groups To manage access to the Argo CD, it is necessary to create groups in the Microsoft Entra Admin Center and assign users to them. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to Argo CD (e.g., `ArgoCDAdmins`, `ArgoCDReadOnly`, etc.): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will be part of the group. ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. Click on the **Create** button and repeat this process for each required group. ## Configuring the Argo CD Helm Chart for OIDC To integrate Argo CD with configured Microsoft Entra Application, it is necessary to configure the Argo CD Helm chart. In this example, we will use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository to deploy Argo CD to the Kubernetes (e.g. AWS EKS) cluster. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/argocd-application-data.png) ::: :::note The **Object ID** can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/argocd-group-object-id.png) ::: 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `argo-cd` directory: Update the `values.yaml` file with the following values: ```yaml title="argo-cd/values.yaml" argo-cd: configs: cm: url: "https://" application.instanceLabelKey: argocd.argoproj.io/instance-edp exec.enabled: true oidc.config: | name: Entra issuer: https://login.microsoftonline.com//v2.0 clientID: clientSecret: $keycloak-client-argocd-secret:clientSecret requestedIDTokenClaims: groups: essential: true value: "SecurityGroup" requestedScopes: - openid - profile - email rbac: scopes: '[groups, email]' policy.csv: | # default global admins g, "", role:admin # Default global developers g, "", role:readonly ``` Replace the following placeholders with the actual values: - `` - Endpoint of the Argo CD. - `` - Directory (tenant) ID of the Microsoft Entra Tenant. - `` - Application (client) ID from the Microsoft Entra Application settings. - `` - Object ID of the **ArgoCDAdmins** group created in the Microsoft Entra Admin Center. - `` - Object ID of the **ArgoCDReadOnly** group created in the Microsoft Entra Admin Center. 2. Update or create the `keycloak-client-argocd-secret` secret with the Application Client Secret value: - Using External Secrets Operator Be sure to update the AWS Parameter Store object path specified in the `argo-cd/values.yaml` file in the `eso.secretName` field with the Application Client Secret value. ```json title="AWS Parameter Store object" { "clientSecret": "" } ``` - Manual approach Create the `keycloak-client-argocd-secret` secret manually using the following template: ```yaml title="keycloak-client-argocd-secret.yaml" apiVersion: v1 kind: Secret metadata: name: keycloak-client-argocd-secret namespace: label: app.kubernetes.io/part-of: argocd data: clientSecret: ``` 3. After updating the `values.yaml` file, commit the changes to the repository and apply the changes with Helm or Argo CD. 4. Navigate to the Microsoft Entra Application and add the **Redirect URI** in the **Authentication** section if you haven't done it before: :::note The **Redirect URI** should be in the format `https:///auth/callback`. ::: ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/argocd-redirect-uri.png) 5. Verify that the OIDC authentication is configured correctly by logging in to Argo CD using the **Log in via Entra** option: ![Argo CD login](../../assets/operator-guide/microsoft-entra-auth/argocd-login.png) After completing these steps, the Argo CD will be configured to use Microsoft Entra as the Identity Provider for authentication. Users will be able to log in to the Argo CD using their Microsoft Entra credentials. ## Troubleshooting Argo CD SSO with Microsoft Entra If the login flow fails after completing the steps above, check the following common causes: - **Error `AADSTS50011` (redirect URI mismatch) after clicking Log in via Entra.** The Redirect URI registered in the **Authentication** section of the Microsoft Entra Application must exactly match `https:///auth/callback`, including the scheme and any trailing path. Update the application registration and retry. - **Login succeeds, but the user sees no applications or projects.** Argo CD RBAC did not match any group. Verify that the **groups claim** is added in the **Token configuration** section with the **Group ID** type, that the group **Object ID** used in the Argo CD RBAC policy matches the group in Microsoft Entra, and that `scopes` in the Helm values includes `groups`. - **`invalid client credentials` or an endless login loop.** The client secret has expired or the secret **Secret ID** was copied instead of the secret **Value**. Generate a new client secret in **Certificates & secrets** and update the secret referenced by the Argo CD Helm chart. - **The groups claim is missing for some users.** When a user belongs to a large number of groups, Microsoft Entra replaces the group list in the token with an overage claim, and Argo CD cannot evaluate it. Use dedicated, smaller groups for Argo CD access. To inspect the exact OIDC error, check the `argocd-server` pod logs while reproducing the login attempt. ## Related Articles * [AWS EKS & KubeRocketCI Portal Microsoft Entra OIDC SSO](./aws-eks-portal-authentication.md) * [Grafana Microsoft Entra OIDC SSO Setup](./grafana-authentication.md) * [OpenID Connect Authentication Overview](./oidc-authentication-overview.md) --- ## AWS EKS & Portal Microsoft Entra OIDC SSO # Guide: Microsoft Entra SSO Integration With AWS EKS & KubeRocketCI Portal Configure Microsoft Entra as the OIDC identity provider for both the AWS EKS cluster and the KubeRocketCI Portal, then use Entra extension attributes to auto-populate each user's default namespace and profile picture on login. This platform-access setup pairs naturally with [Argo CD Microsoft Entra SSO](./argo-cd-authentication.md) for a single sign-on experience across the deployment stack. ## Prerequisites for AWS EKS & Portal SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - A running [AWS EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) cluster with the necessary permissions for access and management. - The [kubelogin](https://github.com/int128/kubelogin) plugin installed for authenticating to the EKS cluster using OIDC. - The [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) CLI tool installed. - The [aws cli](https://aws.amazon.com/cli/) tool installed. - The [curl](https://curl.se/) tool installed. ## Integrate AWS EKS and KubeRocketCI Portal with Microsoft Entra For a detailed step-by-step guide on integrating OIDC authentication for AWS EKS and the KubeRocketCI Portal with Microsoft Entra, please refer to the [Integrating OIDC Authentication With Microsoft Entra in AWS EKS](/blog/integrating-oidc-authentication-microsoft-entra-aws-eks) blog post. The blog covers the process of creating a Microsoft Entra tenant, configuring an application, and managing users and groups. It also explains how to integrate the created application with the AWS EKS cluster and set up authentication for the KubeRocketCI Portal. ## Configure Default Namespace After logging in to the KubeRocketCI Portal, it is necessary to manually configure the **Default namespace** and **Allowed Namespaces**. To automate this process, it is possible to use the Microsoft Entra extension attributes. Follow the steps below to configure the default namespace extension attribute in Microsoft Entra and assign it to the user: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and navigate to the application created during the [OIDC integration process](/blog/integrating-oidc-authentication-microsoft-entra-aws-eks) with AWS EKS and the KubeRocketCI Portal (e.g., `eks`): ![Microsoft Entra Application](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-application.png) 3. In the application tab, navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Application permissions**. Add the following permissions: - **Application.ReadWrite.All** - **User.ReadWrite.All** ![Graph API Permissions](../../assets/operator-guide/microsoft-entra-auth/graph-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. 4. Navigate to the local terminal and run the following command to get the access token: :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application Data](../../assets/operator-guide/microsoft-entra-auth/application-data-overview.png) ::: :::note You can use the [jq](https://stedolan.github.io/jq/) tool to parse the JSON response. If you do not have it installed, you can drop the `| jq -r '.access_token'` part from the command. ::: ```bash curl -X POST "https://login.microsoftonline.com//oauth2/v2.0/token" \ -d "client_id=" \ -d "client_secret=" \ -d "scope=https://graph.microsoft.com/.default" \ -d "grant_type=client_credentials" | jq -r '.access_token' ``` Replace ``, ``, and `` with the corresponding values from the Microsoft Entra application. Save the access token for further use. 5. Create the `default_namespace` extension attribute for the Microsoft Entra application by running the following command: :::note The Application parameter **Object ID** can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application Data](../../assets/operator-guide/microsoft-entra-auth/application-data-overview.png) ::: ```bash curl -X POST "https://graph.microsoft.com/v1.0/applications//extensionProperties" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "default_namespace", "dataType": "String", "targetObjects": ["User"] }' ``` Replace `` with the Object ID of the Microsoft Entra application and `` with the saved access token. 6. Assign the `default_namespace` extension attribute to the user who needs to have the default namespace configured by running the following command: :::note The user **Object ID** parameter can be found in the **Overview** section of the user in the Microsoft Entra Admin Center. ![User Data](../../assets/operator-guide/microsoft-entra-auth/user-data-overview.png) ::: ```bash curl -X PATCH "https://graph.microsoft.com/v1.0/users/" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "extension__default_namespace": "" }' ``` Replace ``, ``, ``, and `` with the user Object ID, application Client ID, saved access token, and the name of the desired default namespace, respectively. 7. Navigate to the Microsoft Entra Admin Center. In the left sidebar menu, select **Applications** and navigate to the Enterprise application created for OIDC integration with AWS EKS and the KubeRocketCI Portal (e.g., `eks`): ![Enterprise Application](../../assets/operator-guide/microsoft-entra-auth/enterprise-application.png) 8. In the application tab, navigate to the **Single sign-on** section. In the **Attributes & Claims** tab, click on the **Edit** button: ![Attributes & Claims](../../assets/operator-guide/microsoft-entra-auth/attributes-claims.png) After that, click on the **Add new claim** button: ![Add new claim](../../assets/operator-guide/microsoft-entra-auth/add-new-claim.png) Fill in the **Name** field with `default_namespace` and select the **Source** as `Directory schema extension`: ![Claim settings](../../assets/operator-guide/microsoft-entra-auth/claim-settings.png) In the **Select Application** tab, choose the application created for OIDC integration with AWS EKS and the KubeRocketCI Portal (e.g., `eks`). Click on the **Select** button: ![Select Application](../../assets/operator-guide/microsoft-entra-auth/select-application.png) In the **Add Extension Attributes** tab, select the `user.default_namespace` attribute and click on the **Add** button: ![Add Extension Attributes](../../assets/operator-guide/microsoft-entra-auth/add-extension-attributes.png) Click on the **Save** button to apply the changes: ![Save Changes](../../assets/operator-guide/microsoft-entra-auth/save-changes.png) 9. In the left sidebar menu, select **Applications** and navigate to the application created for OIDC integration with AWS EKS and the KubeRocketCI Portal (e.g., `eks`). In the application tab, navigate to the **Manifest** section: Find the `acceptMappedClaims` parameter and set it to `true`. Click on the **Save** button to apply the changes: ![Accept Mapped Claims](../../assets/operator-guide/microsoft-entra-auth/accept-mapped-claims.png) 10. (Optional) Get the application access token and verify the `default_namespace` extension attribute by running the following command: :::note Before running the command, ensure that the `Redirect URI` in the Microsoft Entra application is set to `http://localhost:8000`. ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/redirect-uri.png) ::: ```bash kubelogin get-token --oidc-issuer-url=https://sts.windows.net// --oidc-client-id= --oidc-client-secret= --force-refresh ``` Replace ``, ``, and `` with the corresponding values from the Microsoft Entra application. After that, encode the token using the [jwt.io](https://jwt.io/) tool and verify the `default_namespace` attribute in the decoded token: ![Decoded Token](../../assets/operator-guide/microsoft-entra-auth/decoded-token.png) 11. After configuring the `default_namespace` extension attribute, the account settings, such as the **Default namespace** and **Allowed namespaces**, will be automatically configured during login to the KubeRocketCI Portal: ![Portal user default namespace](../../assets/operator-guide/krci-portal-ui.png "Portal user default namespace") ## Configure User Profile Picture To enhance account personalization, it is possible to configure the user profile picture in the KubeRocketCI Portal using the Microsoft Entra extension attributes. Follow the steps below to configure the user profile picture extension attribute in Microsoft Entra and assign it to the user: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and navigate to the application created during the [OIDC integration process](/blog/integrating-oidc-authentication-microsoft-entra-aws-eks) with AWS EKS and the KubeRocketCI Portal (e.g., `eks`): ![Microsoft Entra Application](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-application.png) 3. In the application tab, navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Application permissions**. Add the following permissions: - **Application.ReadWrite.All** - **User.ReadWrite.All** ![Graph API Permissions](../../assets/operator-guide/microsoft-entra-auth/graph-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. 4. Navigate to the local terminal and run the following command to get the access token: :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application Data](../../assets/operator-guide/microsoft-entra-auth/application-data-overview.png) ::: :::note You can use the [jq](https://stedolan.github.io/jq/) tool to parse the JSON response. If you do not have it installed, you can drop the `| jq -r '.access_token'` part from the command. ::: ```bash curl -X POST "https://login.microsoftonline.com//oauth2/v2.0/token" \ -d "client_id=" \ -d "client_secret=" \ -d "scope=https://graph.microsoft.com/.default" \ -d "grant_type=client_credentials" | jq -r '.access_token' ``` Replace ``, ``, and `` with the corresponding values from the Microsoft Entra application. Save the access token for further use. 5. Create the `picture` extension attribute for the Microsoft Entra application by running the following command: :::note The Application parameter **Object ID** can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application Data](../../assets/operator-guide/microsoft-entra-auth/application-data-overview.png) ::: ```bash curl -X POST "https://graph.microsoft.com/v1.0/applications//extensionProperties" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "name": "picture", "dataType": "String", "targetObjects": ["User"] }' ``` Replace `` with the Object ID of the Microsoft Entra application and `` with the saved access token. 6. Assign the `picture` extension attribute to the user who needs to have the profile picture configured by running the following command: :::note The user **Object ID** parameter can be found in the **Overview** section of the user in the Microsoft Entra Admin Center. ![User Data](../../assets/operator-guide/microsoft-entra-auth/user-data-overview.png) ::: ```bash curl -X PATCH "https://graph.microsoft.com/v1.0/users/" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "extension__picture": "" }' ``` Replace ``, ``, ``, and `` with the user Object ID, application Client ID, saved access token, and the URL of the user profile picture, respectively. :::note The profile picture URL should refer to an image accessible via a public URL. ::: 7. Navigate to the Microsoft Entra Admin Center. In the left sidebar menu, select **Applications** and navigate to the Enterprise application created for OIDC integration with AWS EKS and the KubeRocketCI Portal (e.g., `eks`): ![Enterprise Application](../../assets/operator-guide/microsoft-entra-auth/enterprise-application.png) 8. In the application tab, navigate to the **Single sign-on** section. In the **Attributes & Claims** tab, click on the **Edit** button: ![Attributes & Claims](../../assets/operator-guide/microsoft-entra-auth/attributes-claims.png) After that, click on the **Add new claim** button: ![Add new claim](../../assets/operator-guide/microsoft-entra-auth/add-new-claim.png) Fill in the **Name** field with `picture` and select the **Source** as `Directory schema extension`: ![Picture Claim Settings](../../assets/operator-guide/microsoft-entra-auth/picture-claim-settings.png) In the **Select Application** tab, choose the application created for OIDC integration with AWS EKS and the KubeRocketCI Portal (e.g., `eks`). Click on the **Select** button: ![Select Application](../../assets/operator-guide/microsoft-entra-auth/picture-select-application.png) In the **Add Extension Attributes** tab, select the `user.picture` attribute and click on the **Add** button: ![Add Extension Attributes](../../assets/operator-guide/microsoft-entra-auth/picture-add-extension-attributes.png) Click on the **Save** button to apply the changes: ![Save Changes](../../assets/operator-guide/microsoft-entra-auth/picture-save-changes.png) 9. In the left sidebar menu, select **Applications** and navigate to the application created for OIDC integration with AWS EKS and the KubeRocketCI Portal (e.g., `eks`). In the application tab, navigate to the **Manifest** section: Find the `acceptMappedClaims` parameter and set it to `true`. Click on the **Save** button to apply the changes: ![Accept Mapped Claims](../../assets/operator-guide/microsoft-entra-auth/accept-mapped-claims.png) 10. (Optional) Get the application access token and verify the `picture` extension attribute by running the following command: :::note Before running the command, ensure that the `Redirect URI` in the Microsoft Entra application is set to `http://localhost:8000`. ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/redirect-uri.png) ::: ```bash kubelogin get-token --oidc-issuer-url=https://sts.windows.net// --oidc-client-id= --oidc-client-secret= --force-refresh ``` Replace ``, ``, and `` with the corresponding values from the Microsoft Entra application. After that, encode the token using the [jwt.io](https://jwt.io/) tool and verify the `picture` attribute in the decoded token: ![Decoded Token](../../assets/operator-guide/microsoft-entra-auth/picture-decoded-token.png) 11. After configuring the `picture` extension attribute, the user profile picture will be displayed in the KubeRocketCI Portal: ![Portal user profile picture](../../assets/operator-guide/keycloak-user-attributes/portal-user-profile-picture.png "Portal user profile picture") ## Configure Token Lifetime for Application The default token lifetime in Microsoft Entra ID is set to 1 hour. This can cause issues for applications needing long-running processes or extended user sessions, leading to frequent re-authentication. To avoid this, a custom Token Lifetime Policy can be created and assigned to the application to extend the lifespan of ID Tokens or Access Tokens, improving the user experience and application efficiency. Follow the steps below to configure and assign a custom Token Lifetime Policy to the application: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and navigate to the application for which the Token Lifetime Policy needs to be configured (e.g., `eks`): ![Microsoft Entra Application](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-application.png) 3. In the application tab, navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Application permissions**. Add the following permissions: - **Policy.ReadWrite.ApplicationConfiguration** - **Application.ReadWrite.All** ![Lifetime Policy Permissions](../../assets/operator-guide/microsoft-entra-auth/lifetime-policy-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. 4. Navigate to the local terminal and run the following command to get the access token: :::note You can use the [jq](https://stedolan.github.io/jq/) tool to parse the JSON response. If you do not have it installed, you can drop the `| jq -r '.access_token'` part from the command. ::: ```bash curl -X POST "https://login.microsoftonline.com//oauth2/v2.0/token" \ -d "client_id=" \ -d "client_secret=" \ -d "scope=https://graph.microsoft.com/.default" \ -d "grant_type=client_credentials" | jq -r '.access_token' ``` Replace ``, ``, and `` with the corresponding values from the Microsoft Entra application. Save the access token for further use. 5. Create the custom Token Lifetime Policy by running the following command: :::note The `AccessTokenLifetime` value should be set in the format `HH:MM:SS` (e.g., `04:00:00` for 4 hours). ::: ```bash curl -X POST https://graph.microsoft.com/v1.0/policies/tokenLifetimePolicies \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "definition": [ "{ \"TokenLifetimePolicy\": { \"Version\": 1, \"AccessTokenLifetime\": \"04:00:00\" }}" ], "displayName": "Custom Access Token Lifetime", "isOrganizationDefault": false }' ``` Replace `` with the saved access token and set the `AccessTokenLifetime` value as needed. After running the command, save the `id` value from the response for further use. ![Token Lifetime Policy](../../assets/operator-guide/microsoft-entra-auth/token-lifetime-policy.png) 6. Assign the custom Token Lifetime Policy to the application by running the following command: :::note For UNIX-based systems, it is necessary to escape the `$` character in the `\$ref` value. ::: ```bash curl -X POST "https://graph.microsoft.com/v1.0/applications//tokenLifetimePolicies/\$ref" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "@odata.id": "https://graph.microsoft.com/v1.0/policies/tokenLifetimePolicies/" }' ``` Replace ``, ``, and `` with the Object ID of the Microsoft Entra application, the saved access token, and the `id` value of the custom Token Lifetime Policy, respectively. 7. (Optional) Get the application access or ID token and verify the token lifetime by running the following command: :::note Before running the command, ensure that the `Redirect URI` in the Microsoft Entra application is set to `http://localhost:8000`. ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/redirect-uri.png) ::: ```bash kubelogin get-token --oidc-issuer-url=https://sts.windows.net// --oidc-client-id= --oidc-client-secret= --force-refresh ``` Replace ``, ``, and `` with the corresponding values from the Microsoft Entra application. After that, decode the token using the [jwt.io](https://jwt.io/) tool and verify the `exp` claim to check the token expiration time. The lifetime of the token should match the `AccessTokenLifetime` value set in the custom Token Lifetime Policy (e.g., the token should expire after 4 hours after issuance). ![Decoded Token](../../assets/operator-guide/microsoft-entra-auth/lifetime-decoded-token.png) 8. After configuring the custom Token Lifetime Policy, the application will issue tokens with the extended lifespan as defined in the policy. ## Related Articles * [Argo CD Microsoft Entra OIDC SSO Setup](./argo-cd-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## Ansible AWX Microsoft Entra OIDC SSO # Guide: Microsoft Entra SSO Integration With Ansible AWX Configure Ansible AWX single sign-on using Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application and entering the resulting client credentials into the AWX Azure AD authentication settings — see the [Microsoft Entra OIDC overview](./oidc-authentication-overview.md) for the full list of KubeRocketCI add-ons that follow this same pattern. ## Prerequisites for Ansible AWX SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - Ansible AWX is installed using the [AWX Operator](https://github.com/ansible-community/awx-operator-helm). ## Registering the Ansible AWX Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for the Ansible AWX, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI**. Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///sso/complete/azuread-oauth2/`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Certificates & secrets** section from the left sidebar menu. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/awx-client-secrets.png) 6. Copy the generated Client secret value and store it securely: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/awx-client-secret.png) 7. Navigate to the **API permissions** section. Ensure that the **User.Read** permission is added under the **Microsoft Graph** API. If not, click on the **Add a permission** button, select **Microsoft Graph**, and add the **User.Read** permission. After adding the permission, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions: ![API permissions](../../assets/operator-guide/microsoft-entra-auth/awx-api-permissions.png) ## Configuring Ansible AWX To integrate Ansible AWX with the configured Microsoft Entra Application, it is necessary to configure the Ansible AWX to use OIDC authentication. 1. Open the Ansible AWX web interface and log in as an administrator: ![AWX Login](../../assets/operator-guide/microsoft-entra-auth/awx-login.png) 2. In the left sidebar menu, navigate to **Settings** section. In the **Authentication** tab, click on the **Azure AD settings**: ![AWX Settings](../../assets/operator-guide/microsoft-entra-auth/awx-settings.png) 3. Fill in the required fields, such as **Azure AD OAuth2 Key** and **Azure AD OAuth2 Secret**. Click on the **Save** button to apply the changes: :::note - **Azure AD OAuth2 Key** refers to the **Application (client) ID** of your Microsoft Entra Application. - **Azure AD OAuth2 Secret** refers to the **Application Client Secret** value of your Microsoft Entra Application. ::: ![AWX Azure AD Settings](../../assets/operator-guide/microsoft-entra-auth/awx-entra-settings.png) 4. Verify that the OIDC authentication is configured correctly by logging in to the AWX using **Sign in with Azure AD** button: ![AWX Login with Azure AD](../../assets/operator-guide/microsoft-entra-auth/awx-login-entra.png) After completing these steps, the Ansible AWX will be configured to use OIDC authentication with Microsoft Entra as the Identity Provider. ## Related Articles * [OpenSearch Microsoft Entra OIDC SSO Setup](./opensearch-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## DefectDojo Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With DefectDojo Set up DefectDojo single sign-on using Microsoft Entra as the OIDC identity provider. This guide walks through registering the Microsoft Entra application, configuring redirect URIs, and updating the DefectDojo Helm chart so vulnerability-management access is centrally managed alongside the rest of your DevSecOps stack, including [Dependency-Track](./dependency-track-authentication.md). ## Prerequisites for DefectDojo SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [DefectDojo](../devsecops/defectdojo.md) is installed (can be installed during **Configuring Helm chart** step). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. - (Optional) [External Secrets Operator](../secrets-management/install-external-secrets-operator.md) is installed. ## Registering the DefectDojo Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for DefectDojo, it is necessary to create and configure an Application in the Microsoft Entra Admin Center. 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home). ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**. ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button. ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** (You can skip setting the **Redirect URI** if you don't deploy DefectDojo yet). Click **Register** to create the application. :::note The **Redirect URI** should be in the format `https:///complete/azuread-tenant-oauth2/`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Certificates & secrets** section from the left sidebar menu. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**. ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/defectdojo-client-secrets.png) 6. Copy the generated Client secret value and store it securely. You will need this value to configure the DefectDojo Helm chart. ![Client secret](../../assets/operator-guide/microsoft-entra-auth/defectdojo-client-secret.png) 7. Navigate to the **API permissions** section. Ensure that the **User.Read** permission is added under the **Microsoft Graph** API. If not, click on the **Add a permission** button, select **Microsoft Graph**, and add the **User.Read** permission. After adding the permission, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. ![API permissions](../../assets/operator-guide/microsoft-entra-auth/defectdojo-api-permissions.png) ## Configuring the DefectDojo Helm Chart for OIDC To integrate DefectDojo with configured Microsoft Entra Application, it is necessary to configure the DefectDojo Helm chart. In this example, we will use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository to deploy DefectDojo to the Kubernetes (e.g. AWS EKS) cluster. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/defectdojo-application-data.png) ::: 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `clusters/core/addons/defectdojo` directory. Update the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/defectdojo/values.yaml" defectdojo: extraConfigs: DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_KEY: '' DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET: "defectdojo-extrasecrets" DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_TENANT_ID: '' DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_ENABLED: 'True' DD_SESSION_COOKIE_SECURE: 'True' DD_CSRF_COOKIE_SECURE: 'True' DD_SECURE_SSL_REDIRECT: 'False' ``` Replace the `` and `` placeholders with the appropriate values from the Microsoft Entra Application created in the previous step. 2. Update or create the `defectdojo-extrasecrets` secret with the Application Client Secret value. - Using External Secrets Operator Navigate to the `clusters/core/addons/defectdojo/templates/external-secrets/externalsecret-defectdojo-extrasecrets.yaml` file and set the `.spec.data.secretKey` field to the `DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET` value. ```yaml title="clusters/core/addons/defectdojo/templates/external-secrets/externalsecret-defectdojo-extrasecrets.yaml" apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: defectdojo-extrasecrets spec: refreshInterval: 1h secretStoreRef: kind: SecretStore name: {{ .Values.eso.secretStoreName }} data: - secretKey: DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET remoteRef: key: {{ $secretName }} property: defectdojo.oidcClientSecret ``` Be sure to update the AWS Parameter Store object path specified in the `clusters/core/addons/defectdojo/values.yaml` file in the `eso.secretName` field with the Application Client Secret value. ```json title="AWS Parameter Store object" { "defectdojo.oidcClientSecret": "" } ``` - Manual approach Create the `defectdojo-extrasecrets` secret manually using the following template: ```yaml title="defectdojo-extrasecrets.yaml" apiVersion: v1 kind: Secret metadata: name: defectdojo-extrasecrets namespace: type: Opaque data: DD_SOCIAL_AUTH_AZUREAD_TENANT_OAUTH2_SECRET: ``` Replace `` with the generated Client secret value from the Microsoft Entra Application. 3. After updating the `values.yaml` file and creating the `defectdojo-extrasecrets` secret, commit the changes to the repository and apply the changes with Helm or Argo CD. 4. Navigate to the Microsoft Entra Application and add the **Redirect URI** in the **Authentication** section if you haven't done it before. :::note The **Redirect URI** should be in the format `https:///complete/azuread-tenant-oauth2/`. ::: ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/defectdojo-redirect-uri.png) 5. Verify that the OIDC authentication is configured correctly by logging in to DefectDojo using the **Log in with Azure AD** option. ![DefectDojo login](../../assets/operator-guide/microsoft-entra-auth/defectdojo-login.png) 6. After successfully verifying the login, you can proceed with configuring specific permissions for each logged-in user directly within the DefectDojo application. ## Related Articles - [Dependency-Track Microsoft Entra OIDC SSO](./dependency-track-authentication.md) - [OpenID Connect Authentication Overview](./oidc-authentication-overview.md) --- ## Dependency-Track Microsoft Entra OIDC SSO # Guide: Microsoft Entra SSO Integration With Dependency-Track Configure Dependency-Track single sign-on using Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application, mapping access groups, and updating the Dependency-Track Helm chart — pairing naturally with [DefectDojo SSO](./defectdojo-oidc-authentication.md) if you run both scanners in your DevSecOps pipeline. ## Prerequisites for Dependency-Track SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [Dependency-Track](../devsecops/dependency-track.md) is installed (can be installed during **Configuring Helm chart** step). - [Crunchy PostgreSQL Operator](https://github.com/CrunchyData/postgres-operator) is installed. - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. ## Registering the Dependency-Track Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for Dependency-Track, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** with platform type `Single-page application` (You can skip setting the **Redirect URI** if you don't deploy Dependency-Track yet). Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///static/oidc-callback.html`. **Platform type** should be `Single-page application`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/deptrack-register-application.png) 5. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**: ![Authentication settings](../../assets/operator-guide/microsoft-entra-auth/deptrack-authentication-settings.png) 6. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **sAMAaccountName**: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/deptrack-token-configuration.png) 7. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** - **GroupMember.Read.All** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/deptrack-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. After configuring the Microsoft Entra Application, you can proceed with the Dependency-Track configuration. ## Creating the Groups To manage access to Dependency-Track, it is necessary to create the groups in Microsoft Entra and assign users to it. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to Dependency-Track (e.g., `administrator`, `developer`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will be part of the group: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. Repeat this process for each required group. ## Configuring the Dependency-Track Helm Chart for OIDC To integrate Dependency-Track with configured Microsoft Entra Application, it is necessary to configure the Dependency-Track Helm chart. In this example, we will use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository to deploy Dependency-Track to the Kubernetes (e.g. AWS EKS) cluster. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/deptrack-application-data.png) ::: 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `clusters/core/addons/dependency-track` directory: Update the `frontend` section in the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/dependency-track/values.yaml" dependency-track: frontend: apiBaseUrl: "https://deptrack.example.com" extraEnv: - name: OIDC_ISSUER value: "https://login.microsoftonline.com//v2.0" - name: OIDC_CLIENT_ID value: "" - name: OIDC_LOGIN_BUTTON_TEXT value: "Login with Entra" ``` Replace the following values with the data from the Microsoft Entra Application: - ``: The Directory ID of your Microsoft Entra Tenant. - ``: The Application (client) ID of your Microsoft Entra Application. - `https://deptrack.example.com`: The URL of your Dependency-Track instance. Update the `apiServer` section in the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/dependency-track/values.yaml" dependency-track: apiServer: extraEnv: - name: ALPINE_OIDC_ENABLED value: "true" - name: ALPINE_OIDC_CLIENT_ID value: "" - name: ALPINE_OIDC_ISSUER value: "https://login.microsoftonline.com//v2.0" - name: ALPINE_OIDC_USERNAME_CLAIM value: "name" - name: ALPINE_OIDC_USER_PROVISIONING value: "true" - name: ALPINE_OIDC_TEAMS_CLAIM value: "groups" - name: ALPINE_OIDC_TEAM_SYNCHRONIZATION value: "true" ``` Replace the following values with the data from the Microsoft Entra Application: - ``: The Directory ID of your Microsoft Entra Tenant. - ``: The Application (client) ID of your Microsoft Entra Application. 2. After updating the `values.yaml` file, commit the changes to the repository and apply the changes with Helm or Argo CD. 3. Navigate to the Microsoft Entra Application and add the **Redirect URI** in the **Authentication** section if you haven't done it before: :::note The **Redirect URI** should be in the format `https:///static/oidc-callback.html`. **Platform type** should be `Single-page application`. ::: ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/deptrack-redirect-uri.png) 4. Log in to the Dependency-Track web interface as an Admin user. Navigate to the **Administration** tab and click on the **Access Management** section. Select the **OpenID Connect Groups** tab and click on the **Create Group** button to create a new OpenID Connect group: :::note The **Object ID** can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-group-object-id.png) ::: Make sure that the group name matches the **Group ID** of the group in the Microsoft Entra Admin Center. Here is as example of how the **Administrator** group should appear in Dependency-Track: ![Dependency-Track groups](../../assets/operator-guide/microsoft-entra-auth/deptrack-groups.png) 5. Verify that the OpenID Connect Group is correctly mapped to the Dependency-Track Team. Here is an example of how the **Administrator** OpenID Connect group should map to the Dependency-Track **Administrators** Team: ![Dependency-Track team](../../assets/operator-guide/microsoft-entra-auth/deptrack-team.png) 6. Verify that the OIDC authentication is working correctly by log in to the Dependency-Track web interface using the **Login with Entra** button: ![Dependency-Track Login](../../assets/operator-guide/microsoft-entra-auth/deptrack-login.png) After completing these steps, Dependency-Track is configured to use Microsoft Entra as the Identity Provider for OIDC authentication. ## Related Articles * [DefectDojo SSO guide](./defectdojo-oidc-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## Grafana Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With Grafana Set up Grafana single sign-on with Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application, mapping Entra groups to Grafana roles, and updating the Grafana Helm chart — the same Microsoft Entra pattern used across [Nexus](./nexus-authentication.md), [DefectDojo](./defectdojo-oidc-authentication.md), and other KubeRocketCI add-ons. ## Prerequisites for Grafana SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [Grafana](https://github.com/epam/edp-cluster-add-ons/blob/3fc8a60ccfbca1c7aa757cb36226a0daf2c8a224/clusters/core/addons/prometheus-operator/values.yaml#L74) is installed (can be installed during **Configuring Helm chart** step). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. - (Optional) [External Secrets Operator](../secrets-management/install-external-secrets-operator.md) is installed. ## Registering the Grafana Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for Grafana, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** (You can skip setting the **Redirect URI** if you don't deploy Grafana yet). Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///login/generic_oauth`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**: ![Authentication settings](../../assets/operator-guide/microsoft-entra-auth/grafana-authentication-settings.png) 6. Navigate to the **Certificates & secrets** section. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/grafana-client-secrets.png) 7. Copy the generated client secret value and store it securely. You will need this value to configure the Grafana Helm chart: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/grafana-client-secret.png) 8. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**. Also, choose the **Emit groups as role claims** option: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/grafana-token-configuration.png) Also, add the **preferred_username**, **email** and **upn** optional claims: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/grafana-token-configuration-2.png) 9. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** - **GroupMember.Read.All** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/grafana-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. After configuring the Microsoft Entra Application, you can proceed with configuring the Grafana Helm chart. ## Creating the Groups To manage access to Grafana, it is necessary to create the groups in Microsoft Entra and assign users to it. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to Grafana (e.g., `administrator`, `developer`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will be part of the group: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. Repeat this process for each required group. ## Configuring the Grafana Helm Chart for OIDC To integrate Grafana with configured Microsoft Entra Application, it is necessary to configure the Grafana Helm chart. In this example, we will use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository and the **prometheus-operator** Helm chart to deploy Grafana to the Kubernetes (e.g. AWS EKS) cluster. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/grafana-application-data.png) ::: :::note The **Object ID** can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-group-object-id.png) ::: 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `clusters/core/addons/prometheus-operator` directory: Update the `grafana` section in the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/prometheus-operator/values.yaml" kube-prometheus-stack: grafana: grafana.ini: auth.generic_oauth: enabled: true client_secret: ${GF_AUTH_AZUREAD_CLIENT_SECRET} client_id: "" allow_sign_up: true scopes: openid email profile auth_url: "https://login.microsoftonline.com//oauth2/v2.0/authorize" token_url: "https://login.microsoftonline.com//oauth2/v2.0/token" role_attribute_strict: false role_attribute_path: contains(roles[*], '') && 'Admin' || contains(roles[*], 'Object ID of the Developer group') && 'Editor' || 'Viewer' skip_org_role_sync: false api_url: "https://graph.microsoft.com/oidc/userinfo" ``` Replace the following placeholders with the actual values: - `` - The Application ID from the Microsoft Entra Application settings. - `` - The Directory ID of your Microsoft Entra Tenant. - `` - The Object ID of the `administrator` group created in Microsoft Entra. - `` - The Object ID of the `developer` group created in Microsoft Entra. 2. Update or create the `keycloak-client-grafana-secret` secret with the Application Client Secret value: - Using External Secrets Operator: Navigate to the `clusters/core/addons/prometheus-operator/templates/external-secrets/externalsecret-keycloak-client-grafana-secret.yaml` file and set the `spec.data.secretKey` field to the `GF_AUTH_AZUREAD_CLIENT_SECRET` value. ```yaml title="clusters/core/addons/prometheus-operator/templates/external-secrets/externalsecret-keycloak-client-grafana-secret.yaml" {{- if .Values.eso.enabled }} {{- $secretName := .Values.eso.secretName }} apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: keycloak-client-grafana-secret spec: refreshInterval: 1h secretStoreRef: kind: SecretStore name: {{ .Values.eso.secretStoreName }} data: - secretKey: GF_AUTH_AZUREAD_CLIENT_SECRET remoteRef: conversionStrategy: Default decodingStrategy: None key: {{ $secretName }} metadataPolicy: None property: keycloak-client-grafana-secret.clientSecret {{- end }} ``` Be sure to update the AWS Parameter Store object path specified in the `clusters/core/addons/prometheus-operator/values.yaml` file in the `eso.secretName` field with the Application Client Secret value. ```json title="AWS Parameter Store object" { "keycloak-client-grafana-secret": { "clientSecret": "" } } ``` Replace the `` placeholder with the actual Client Secret value. - Manual approach: Create the `keycloak-client-grafana-secret` secret manually using the following template: ```yaml title="keycloak-client-grafana-secret.yaml" apiVersion: v1 kind: Secret metadata: name: keycloak-client-grafana-secret namespace: data: GF_AUTH_AZUREAD_CLIENT_SECRET: ``` Replace the `` placeholder with the actual Client Secret value. 3. After updating the `values.yaml` file and creating the `keycloak-client-grafana-secret` secret, commit the changes to the repository and apply the changes with Helm or Argo CD. 4. Verify that the OIDC authentication is configured correctly by logging in to Grafana using the Microsoft Entra credentials: ![Grafana login](../../assets/operator-guide/microsoft-entra-auth/grafana-login.png) After completing these steps, Grafana will be configured to use Microsoft Entra as the Identity Provider for authentication. ## Related Articles * [Nexus SSO with Microsoft Entra OIDC](./nexus-authentication.md) * [DefectDojo Microsoft Entra OIDC SSO Setup](./defectdojo-oidc-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## Harbor Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With Harbor Configure Harbor single sign-on using Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application, mapping an Entra group to the Harbor OIDC Admin Group, and entering the resulting OIDC parameters directly in the Harbor UI — the same registry-access pattern used for [Nexus SSO with Microsoft Entra OIDC](./nexus-authentication.md). ## Prerequisites for Harbor SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [Harbor](../artifacts-management/harbor-installation.md) is installed. ## Registering the Harbor Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for Harbor, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI**. Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///c/oidc/callback`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. Navigate to the **Certificates & secrets** section from the left sidebar menu. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/harbor-client-secrets.png) 6. Copy the generated Client secret value and store it securely: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/harbor-client-secret.png) 7. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/harbor-token-configuration.png) 8. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/harbor-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. After configuring the Microsoft Entra Application, you can proceed with the Harbor configuration. ## Creating the Groups To manage access to Harbor, it is necessary to create the groups in Microsoft Entra and assign users to it. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to Harbor (e.g., `administrator`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will have access to SonarQube: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. ## Configuring Harbor To integrate Harbor with the configured Microsoft Entra application, it is necessary to configure the OIDC parameters directly in Harbor. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/harbor-application-data.png) ::: :::note The **Object ID** can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-group-object-id.png) ::: 1. Log in to the Harbor web interface as an administrator. 2. In the left sidebar menu, navigate to **Administration** tab and click on **Configuration** section. In the **Authentication** tab, fill in the required fields for the OIDC configuration: ![Harbor Configuration](../../assets/operator-guide/microsoft-entra-auth/harbor-configuration.png) Replace the following values with the data from the Microsoft Entra Application: - OIDC Provider Name: `azure` - OIDC Endpoint: `https://login.microsoftonline.com//v2.0`, where `` is the Directory ID of your Microsoft Entra Tenant. - OIDC Client ID: The Application (client) ID of your Microsoft Entra Application. - OIDC Client Secret: The Client Secret value from the Microsoft Entra Application. - Group Claim Name: `groups` - OIDC Admin Group: The **Object ID** of the `administrator` group from the Microsoft Entra Admin Center. - OIDC Scope: `openid,profile,email,offline_access` 3. Click on the **Test OIDC Server** button to verify the configuration. If the test is successful, click on the **Save** button to apply the changes. 4. Verify that the OIDC authentication is configured correctly by logging in to the Harbor web interface using the **Login via OIDC Provider** button: ![Harbor Login](../../assets/operator-guide/microsoft-entra-auth/harbor-login.png) After completing these steps, Harbor is configured to use Microsoft Entra as the Identity Provider for OIDC authentication. ## Related Articles * [Nexus SSO with Microsoft Entra OIDC](./nexus-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## Nexus SSO With Microsoft Entra OIDC # Guide: Microsoft Entra SSO Integration With Nexus Configure single sign-on for Nexus Repository using Microsoft Entra as the OIDC provider and [OAuth2-Proxy](./oauth2-proxy-authentication.md) as the authentication front end. This guide covers Microsoft Entra app registration, Entra group mapping, and the combined Nexus + OAuth2-Proxy Helm chart configuration, followed by user provisioning through the Nexus Operator. ## Prerequisites for Nexus SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [Nexus](https://github.com/epam/edp-cluster-add-ons/tree/0a1bd78c52899294ef7e17bfae5d3e6477206109/clusters/core/addons/nexus) in installed (can be installed during **Configuring Helm chart** step). - [Nexus Operator](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/nexus-operator) is installed (can be installed during **Configuring Helm chart** step). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. - (Optional) [External Secrets Operator](../secrets-management/install-external-secrets-operator.md) is installed. ## Registering the Nexus Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for Nexus, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** (You can skip setting the **Redirect URI** if you don't deploy OAuth2-proxy yet). Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///oauth2/callback`. `` can be found in the `values.yaml` file of the [Nexus Helm chart](https://github.com/epam/edp-cluster-add-ons/blob/0a1bd78c52899294ef7e17bfae5d3e6477206109/clusters/core/addons/nexus/values.yaml#L89) in the `clusters/core/addons/nexus` directory. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**: ![Authentication settings](../../assets/operator-guide/microsoft-entra-auth/nexus-authentication-settings.png) 6. Navigate to the **Certificates & secrets** section. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/nexus-client-secrets.png) 7. Copy the generated client secret value and store it securely. You will need this value to configure the OAuth2-proxy Helm chart: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/nexus-client-secret.png) 8. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/nexus-token-configuration.png) 9. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/nexus-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. ## Creating the Groups To manage access to Nexus using OAuth2-proxy, it is necessary to create groups in the Microsoft Entra Admin Center and add users to these groups. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to Nexus (e.g., `administrator`, `developer`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will have access to SonarQube: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. Repeat this process for each required group. ## Configuring Nexus and OAuth2-Proxy Helm Charts To integrate Nexus with the configured Microsoft Entra Application, it is necessary to configure the Nexus and OAuth2-proxy Helm charts. In this example, we will use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository to deploy Nexus and OAuth2-proxy to the Kubernetes (e.g. AWS EKS) cluster. 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `clusters/core/addons/nexus` directory: Update the `oauth2-proxy` section in the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/nexus/values.yaml" oauth2-proxy: enabled: true config: configFile: |- allowed_groups = ["", ""] azure_tenant = "" client_id = "" cookie_csrf_expire="5m" code_challenge_method="S256" insecure_oidc_allow_unverified_email = "true" cookie_csrf_per_request="true" cookie_secure = "false" email_domains = [ "*" ] oidc_issuer_url = "https://login.microsoftonline.com//v2.0" pass_access_token = "true" pass_authorization_header = "true" pass_basic_auth = "false" oidc_email_claim = "preferred_username" provider = "oidc" redirect_url = "https://nexus.example.com/oauth2/callback" skip_jwt_bearer_tokens = "true" upstreams = [ "http://nexus:8081" ] whitelist_domains = ["*"] silence_ping_logging = "true" scope = "openid email profile" existingSecret: oauth2-proxy ingress: enabled: true hosts: - nexus.example.com ``` Replace the placeholders with the actual values from the Microsoft Entra Admin Center: - `` and `` with the Object IDs of the groups created in the Microsoft Entra Admin Center. - `` with the Directory ID of the Microsoft Entra Tenant. - `` with the Application ID of the Microsoft Entra Application. - `nexus.example.com` with the desired redirect URL for the OAuth2-proxy. 2. Update or create the `oauth2-proxy` secret with the Client Secret and other necessary values: - Using External Secrets Operator Be sure to update the AWS Parameter Store object path specified in the `clusters/core/addons/nexus/values.yaml` file in the `eso.secretName` field with the `client-id`, `client-secret` and `cookie-secret` values. :::note - `client-id` refers to the **Application (client) ID** of your Microsoft Entra Application. - `client-secret` refers to the **Application Client Secret** value of your Microsoft Entra Application. - `cookie-secret` is a randomly generated secret value. ::: ```json title="AWS Parameter Store object" { "oauth2-proxy": { "client-id": "", "client-secret": "", "cookie-secret": "" } } ``` - Manual approach Create the `oauth2-proxy` secret manually using the following template: ```yaml title="oauth2-proxy.yaml" apiVersion: v1 kind: Secret metadata: name: oauth2-proxy namespace: type: Opaque data: client-id: client-secret: cookie-secret: ``` Replace ``, ``, and `` with the actual values. 3. After updating the `values.yaml` file and creating the `oauth2-proxy` secret, commit the changes to the repository and apply the changes with Helm or Argo CD. ## Creating Nexus Users with the Nexus Operator :::note Nexus users can also be created directly from the Nexus UI instead of using the Nexus Operator Helm chart. ::: To be able to access Nexus using OAuth2-proxy and Microsoft Entra, it is necessary to create users in Nexus with the appropriate roles. In this example, we will demonstrate how to create users with Nexus Operator. 1. Make sure that the [Nexus Operator](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/nexus-operator) is installed. If not, follow the instructions in the [Sonatype Nexus Repository OSS Integration documentation](../artifacts-management/nexus-sonatype.md): 2. Configure the following `NexusUser` custom resource to create a user with the `nx-admin` and `edp-admin` roles: ```yaml title="nexus-user.yaml" apiVersion: edp.epam.com/v1alpha1 kind: NexusUser metadata: name: test-user namespace: nexus spec: email: firstName: id: lastName: nexusRef: kind: Nexus name: nexus roles: - edp-admin - nx-admin secret: $ci-user:password status: active ``` Replace the placeholders with the actual values. 3. After configuring the `NexusUser` custom resource, apply the changes to the Kubernetes cluster. 4. Repeat the process for each user you want to grant access to Nexus. 5. Verify that the OIDC Authentication is configured correctly by logging in to Nexus using the OAuth2-proxy endpoint. Click on the **Sign in with Open ID Connect** button to authenticate with Microsoft Entra credentials: ![Nexus login](../../assets/operator-guide/microsoft-entra-auth/login-nexus.png) After completing these steps, Nexus will be configured with OIDC authentication using Microsoft Entra as the Identity Provider. Users will be able to log in to Nexus using OAuth2-proxy endpoint with their Microsoft Entra credentials. ## Related Articles * [OAuth2-Proxy configuration for Microsoft Entra SSO](./oauth2-proxy-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## OAuth2-Proxy Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With OAuth2-proxy (Tekton Dashboard) Configure OAuth2-Proxy to enforce Microsoft Entra OIDC single sign-on for the Tekton Dashboard. This guide covers registering the app in Microsoft Entra, mapping Entra groups to dashboard access, and deploying the OAuth2-Proxy Helm chart. It also applies if you're securing Nexus, which uses OAuth2-Proxy as its authentication front end — see the [Nexus SSO guide](./nexus-authentication.md). ## Prerequisites for OAuth2-Proxy SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [Tekton Dashboard](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml#L267) is installed. - [OAuth2-proxy](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml#L202) is installed (can be installed during **Configuring Helm chart** step). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. - (Optional) [External Secrets Operator](../secrets-management/install-external-secrets-operator.md) is installed. ## Registering the OAuth2-Proxy Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for the OAuth2-proxy, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** (You can skip setting the **Redirect URI** if you don't deploy OAuth2-proxy yet). Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///oauth2/callback`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Certificates & secrets** section from the left sidebar menu. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-client-secrets.png) 6. Copy the generated Client secret value and store it securely. You will need this value to configure the OAuth2-proxy Helm chart: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-client-secret.png) 7. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-token-configuration.png) 8. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. After the application is configured, you can proceed with the OAuth2-proxy Helm chart configuration. ## Mapping Microsoft Entra Groups to OAuth2-Proxy Access To manage access to the Tekton Dashboard (or any other application with OAuth2-proxy), it is necessary to create groups in the Microsoft Entra Admin Center and assign users to it. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to Tekton Dashboard (e.g., `administrator`, `developer`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will have access to SonarQube: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. Repeat this process for each required group. ## Deploying the OAuth2-Proxy Helm Chart for Tekton Dashboard To integrate OAuth2-proxy with the configured Microsoft Entra Application, it is necessary to configure the OAuth2-proxy Helm chart. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-application-data.png) ::: :::note The **Object ID** can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-group-object-id.png) ::: 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `clusters/core/addons/oauth2-proxy` directory: Update the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/oauth2-proxy/values.yaml" oauth2-proxy: config: configFile: |- allowed_groups = ["", ""] code_challenge_method="S256" azure_tenant = "" client_id = "" cookie_domains = ["example.domain.com"] cookie_secure = "false" email_domains = [ "*" ] insecure_oidc_allow_unverified_email = "true" oidc_issuer_url = "https://login.microsoftonline.com//v2.0" pass_authorization_header = "true" pass_basic_auth = "false" pass_user_headers = "true" oidc_email_claim = "preferred_username" provider = "oidc" redirect_url = "https://oauth-oauth2-proxy.example.domain.com/oauth2/callback" reverse_proxy = "true" skip_jwt_bearer_tokens = "true" skip_provider_button = "true" whitelist_domains = ".example.domain.com" scope = "openid email profile" existingSecret: oauth2-proxy ingress: enabled: true hosts: - oauth-oauth2-proxy.example.domain.com ``` Replace the following placeholders with the actual values: - `` - Object ID of the `administrator` group created in the Microsoft Entra Admin Center. - `` - Object ID of the `developer` group created in the Microsoft Entra Admin Center. - `` - Directory (tenant) ID of the Microsoft Entra Tenant. - `` - Application (client) ID of the Microsoft Entra Application created in the previous step. - `example.domain.com` - Domain name associated with the OAuth2-proxy. 2. Update or create the `oauth2-proxy` secret with the Application Client Secret value: - Using External Secrets Operator: Be sure to update the AWS Parameter Store object path specified in the `clusters/core/addons/oauth2-proxy/values.yaml` file in the `eso.secretName` field with the `client-id`, `client-secret` and `cookie-secret` values. :::note - `client-id` refers to the **Application (client) ID** of your Microsoft Entra Application. - `client-secret` refers to the **Application Client Secret** value of your Microsoft Entra Application. - `cookie-secret` is a randomly generated secret value. ::: ```json title="AWS Parameter Store object" { "oauth2-proxy": { "client-id": "", "client-secret": "", "cookie-secret": "" } } ``` - Manual approach: Create the `oauth2-proxy` secret manually using the following template: ```yaml title="oauth2-proxy-secret.yaml" apiVersion: v1 kind: Secret metadata: name: oauth2-proxy namespace: type: Opaque data: client-id: client-secret: cookie-secret: ``` Replace ``, ``, and `` with the appropriate values. 3. After updating the `values.yaml` file and creating the `oauth2-proxy` secret, commit the changes to the repository and apply the changes with Helm or Argo CD. After the OAuth2-proxy is configured, it can serve as a reverse proxy, enabling secure access to various applications by handling authentication and passing validated user identity data. ## Configuring Tekton Dashboard To configure the Tekton Dashboard to use OAuth2-proxy for OIDC authentication, it is necessary to update the Tekton Dashboard Ingress resource. 1. Log in to the Kubernetes cluster where the Tekton Dashboard is installed. 2. Edit the Ingress resource associated with the Tekton Dashboard by adding `auth-signin` and `auth-url` of oauth2-proxy to the annotations: ```bash kubectl annotate ingress -n \ nginx.ingress.kubernetes.io/auth-signin='https:///oauth2/start?rd=https://$host$request_uri' \ nginx.ingress.kubernetes.io/auth-url='http://oauth2-proxy.oauth2-proxy.svc.cluster.local:80/oauth2/auth' ``` Replace the following placeholders with the actual values. 3. Verify that the OIDC authentication is configured correctly by accessing the Tekton Dashboard URL and logging in with the Microsoft Entra credentials: ![Tekton Dashboard login](../../assets/operator-guide/microsoft-entra-auth/tekton-dashboard-login.png) After completing these steps, the Tekton Dashboard will be configured to use OAuth2-proxy with Microsoft Entra as the Identity Provider for authentication. Users will be able to log in to the Tekton Dashboard using their Microsoft Entra credentials. ## Related Articles * [Nexus SSO with Microsoft Entra OIDC](./nexus-authentication.md) * [OpenID Connect (OIDC) Authentication Overview](./oidc-authentication-overview.md) --- ## Microsoft Entra OIDC SSO Guides # Microsoft Entra OIDC Authentication: Overview OpenID Connect (OIDC) is a standard protocol for enabling secure and seamless authentication across applications. OIDC ensures consistent identity management, enhances security, and streamlines user access to critical services. This section focuses on integrating Microsoft Entra as the Identity Provider for various components from the [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons). Microsoft Entra enables centralized identity management, providing single sign-on (SSO), multi-factor authentication (MFA), and strong security measures. Configuring OIDC authentication with Microsoft Entra brings significant benefits, including improved security, simplified access management, and compliance with enterprise standards. ## Supported Components Below is the list of components with OIDC integration instructions, grouped by function. ### Security & DevSecOps Tools Secure vulnerability-management and code-quality platforms with centralized Microsoft Entra sign-in. - [DefectDojo](defectdojo-oidc-authentication.md) - [Dependency-Track](./dependency-track-authentication.md) - [SonarQube](./sonar-oidc-authentication.md) ### Artifact & Registry Tools Control access to artifact repositories and image registries through Microsoft Entra groups. - [Nexus](./nexus-authentication.md) - [Harbor](harbor-authentication.md) ### Platform & Observability Tools Bring platform, deployment, and monitoring tools under the same Microsoft Entra identity provider. - [Argo CD](argo-cd-authentication.md) - [Grafana](grafana-authentication.md) - [OpenSearch](opensearch-authentication.md) - [Ansible AWX](awx-operator-authentication.md) - [OAuth2-proxy](oauth2-proxy-authentication.md) - [AWS EKS and KubeRocketCI Portal](aws-eks-portal-authentication.md) --- ## OpenSearch Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With OpenSearch Configure OpenSearch single sign-on using Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application, mapping Entra groups to OpenSearch security roles, and updating the OpenSearch and OpenSearch Dashboards Helm chart values — an observability-stack setup that pairs naturally with [Grafana Microsoft Entra OIDC SSO](./grafana-authentication.md). ## Prerequisites for OpenSearch SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [OpenSearch](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml#L208) is installed (can be installed during **Configuring Helm chart** step). - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. - (Optional) [External Secrets Operator](../secrets-management/install-external-secrets-operator.md) is installed. ## Registering the OpenSearch Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for OpenSearch, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI** (You can skip setting the **Redirect URI** if you don't deploy OpenSearch yet). Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///auth/openid/login`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**: ![Authentication settings](../../assets/operator-guide/microsoft-entra-auth/opensearch-authentication-settings.png) 6. Navigate to the **Certificates & secrets** section. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/opensearch-client-secrets.png) 7. Copy the generated client secret value and store it securely. You will need this value to configure the OpenSearch Helm chart: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/opensearch-client-secret.png) 8. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**. Also, choose the **Emit groups as role claims** option: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/opensearch-token-configuration.png) 9. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **openid** - **profile** - **User.Read** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/opensearch-api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. After configuring the Microsoft Entra Application, you can proceed with the OpenSearch configuration. ## Creating the Groups To manage access to OpenSearch, it is necessary to create the groups in Microsoft Entra and assign users to it. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to OpenSearch (e.g., `administrator`, `developer`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will be part of the group: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. Repeat this process for each required group. ## Configuring the OpenSearch Helm Chart for OIDC To integrate OpenSearch with configured Microsoft Entra Application, it is necessary to configure the OpenSearch Helm chart. In this example, we will use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository to deploy OpenSearch to the Kubernetes (e.g. AWS EKS) cluster. :::note The Application data, such as **Application (client) ID** and **Directory (tenant) ID**, can be found in the **Overview** section of the Application in the Microsoft Entra Admin Center. ![Application data](../../assets/operator-guide/microsoft-entra-auth/opensearch-application-data.png) ::: :::note The **Object ID** can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/oauth2-proxy-group-object-id.png) ::: 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `values.yaml` file in the `clusters/core/addons/opensearch` directory: Update the `opensearch` section in the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/opensearch/values.yaml" opensearch: securityConfig: config: data: config.yml: |- config: dynamic: authc: openid_auth_domain: http_enabled: true transport_enabled: true order: 1 http_authenticator: type: openid challenge: false config: subject_key: preferred_username roles_key: roles openid_connect_url: https://login.microsoftonline.com//v2.0/.well-known/openid-configuration openid_connect_idp: enable_ssl: false verify_hostnames: false authentication_backend: type: noop roles_mapping.yml: |- _meta: type: "rolesmapping" config_version: 2 all_access: reserved: true backend_roles: - "" - "admin" description: "Maps EDP administrator role and internal admin role to all_access" logstash: reserved: true backend_roles: - "logstash" description: "Maps log injection role to logstash. It might be logstash, fluent-bit, fluentd, ..." kibana_user: reserved: false backend_roles: - "kibanauser" - "" description: "Maps kibanauser and EDP developer role to kibana_user" kibana_server: reserved: true users: - "kibanaserver" edp_developer: reserved: false backend_roles: - "" description: "Maps EDP developer role to view kube index" ``` Replace the following placeholders with the actual values: - ``: The Directory ID of your Microsoft Entra Tenant. - ``: The Object ID of the `administrator` group from the Microsoft Entra Admin Center. - ``: The Object ID of the `developer` group from the Microsoft Entra Admin Center. Update the `opensearch-dashboards` section in the `values.yaml` file with the following values: ```yaml title="clusters/core/addons/opensearch/values.yaml" opensearch-dashboards: config: opensearch_dashboards.yml: server: auth.type: openid openid: connect_url: https://login.microsoftonline.com//v2.0/.well-known/openid-configuration base_redirect_url: https://opensearch-dashboards.example.com client_id: client_secret: ${OIDC_CLIENT_SECRET} scope: openid profile email header: Authorization verify_hostnames: false trust_dynamic_headers: true ``` Replace the following placeholders with the actual values: - `https://opensearch-dashboards.example.com`: The base URL of the OpenSearch Dashboards. - ``: The Directory ID of your Microsoft Entra Tenant. - ``: The Application ID from the Microsoft Entra Application settings. 2. Update or create the `opensearch-dashboards-account` secret with the Application Client Secret value: - Using External Secrets Operator: Be sure to update the AWS Parameter Store object path specified in the `clusters/core/addons/opensearch/values.yaml` file in the `eso.secretName` field with the `username`, `password`, `cookie` and `OIDC_CLIENT_SECRET` values. ```json title="AWS Parameter Store object" { "opensearch-dashboard": { "username": "", "password": "", "cookie": "", "OIDC_CLIENT_SECRET": "" } } ``` - Manual approach: Create the `opensearch-dashboards-account` secret manually using the following template: ```yaml title="opensearch-dashboards-account.yaml" apiVersion: v1 kind: Secret metadata: name: opensearch-dashboards-account namespace: type: Opaque data: username: password: cookie: OIDC_CLIENT_SECRET: ``` 3. After updating the `values.yaml` file and creating the `opensearch-dashboards-account` secret, commit the changes to the repository and apply the changes with Helm or Argo CD. 4. Navigate to the Microsoft Entra Application and add the **Redirect URI** in the **Authentication** section if you haven't done it before: :::note The **Redirect URI** should be in the format `https:///auth/openid/login`. ::: ![Redirect URI](../../assets/operator-guide/microsoft-entra-auth/opensearch-redirect-uri.png) 5. Verify that the OIDC authentication is configured correctly by logging in to OpenSearch using the Microsoft Entra credentials: ![OpenSearch login](../../assets/operator-guide/microsoft-entra-auth/opensearch-login.png) After completing these steps, OpenSearch will be configured with OIDC authentication using Microsoft Entra as the Identity Provider. ## Related Articles * [Grafana Microsoft Entra OIDC SSO Setup](./grafana-authentication.md) * [Ansible AWX Microsoft Entra OIDC SSO](./awx-operator-authentication.md) * [OpenID Connect Authentication Overview](./oidc-authentication-overview.md) --- ## SonarQube Microsoft Entra OIDC SSO Setup # Guide: Microsoft Entra SSO Integration With SonarQube Configure SonarQube single sign-on using Microsoft Entra as the OIDC identity provider. This guide covers registering the Microsoft Entra application, syncing Entra groups into SonarQube by Object ID, and configuring OIDC through the [edp-sonar-operator](https://github.com/epam/edp-sonar-operator) Helm chart — a quality-gate setup that fits alongside [DefectDojo](./defectdojo-oidc-authentication.md) and [Dependency-Track](./dependency-track-authentication.md) in a DevSecOps pipeline. ## Prerequisites for SonarQube SSO with Microsoft Entra Before you begin, make sure the following prerequisites are met: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - [Microsoft Entra](https://learn.microsoft.com/en-us/entra/fundamentals/create-new-tenant) Tenant is created. - [SonarQube](../code-quality/sonarqube.md) is installed. - A forked copy of the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository is created. ## Registering the SonarQube Application in Microsoft Entra To configure Microsoft Entra as the Identity Provider for SonarQube, it is necessary to create and configure an Application in the Microsoft Entra Admin Center: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/?feature.msaljs=true#home): ![Microsoft Entra Admin Center](../../assets/operator-guide/microsoft-entra-auth/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Applications** and click **App registrations**: ![App registrations](../../assets/operator-guide/microsoft-entra-auth/app-registrations.png) 3. Click on the **New registration** button: ![New registration](../../assets/operator-guide/microsoft-entra-auth/new-registration.png) 4. Fill in the required fields, such as **Name**, **Supported account types** and **Redirect URI**. Click **Register** to create the application: :::note The **Redirect URI** should be in the format `https:///oauth2/callback/oidc`. ::: ![Register application](../../assets/operator-guide/microsoft-entra-auth/register-application.png) 5. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**: ![Authentication settings](../../assets/operator-guide/microsoft-entra-auth/authentication-settings.png) 6. Navigate to the **Certificates & secrets** section. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. Fill in the required fields and click **Add**: ![Client secrets](../../assets/operator-guide/microsoft-entra-auth/client-secrets.png) 7. Copy the generated client secret value and store it securely. You will need this value to configure the SonarQube Helm chart: ![Client secret](../../assets/operator-guide/microsoft-entra-auth/client-secret.png) 8. Navigate to the **Token configuration** section and click on **Add groups claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/token-configuration.png) Also, add the **preferred_username** optional claim: ![Token configuration](../../assets/operator-guide/microsoft-entra-auth/token-configuration-2.png) 9. Navigate to the **API permissions** section. Click on the **Add a permission** button. Select **Microsoft Graph** and then **Delegated permissions**. Add the following permissions: - **email** - **offline_access** - **openid** - **profile** - **User.Read** ![API permissions](../../assets/operator-guide/microsoft-entra-auth/api-permissions.png) After adding the permissions, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. ## Creating the Groups To manage access to SonarQube, it is necessary to create the groups in Microsoft Entra and assign users to it. 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to SonarQube (e.g., `sonar-administrators`, `sonar-developers`): ![New group](../../assets/operator-guide/microsoft-entra-auth/new-group.png) 2. Fill in the required fields, such as **Groups type** and **Group name**. In the **Members** section, add users who will be part of the group: ![Create group](../../assets/operator-guide/microsoft-entra-auth/create-group.png) 3. After adding the necessary members, review the group settings and click **Create** to save the group. Repeat this process for each required group: :::important Ensure that the groups created in Microsoft Entra correspond to the groups configured in SonarQube. In SonarQube, the group name **must exactly match the Object ID** of the corresponding Microsoft Entra group, as this is required for proper synchronization and access control. ::: :::note The **Object ID** of the Microsoft Entra group can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../../assets/operator-guide/microsoft-entra-auth/group-object-id.png) ::: Below is an example of how the `sonar-administrators` group should appear in SonarQube: ![SonarQube groups](../../assets/operator-guide/microsoft-entra-auth/sonarqube-groups.png) ## Configuring SonarQube with Sonar-Operator :::note SonarQube OIDC configuration can also be configured directly from the SonarQube UI instead of using the Sonar-Operator. ::: In this section, we will demonstrate how to set up OIDC authentication for SonarQube using the [edp-sonar-operator](https://github.com/epam/edp-sonar-operator) Helm chart. This approach aligns with the automation and consistency principles of the KubeRocketCI platform. 1. Navigate to the forked [Cluster Add-Ons repository](https://github.com/epam/edp-cluster-add-ons) and locate the `sonar.yaml` file at the following path `clusters/core/addons/sonar-operator/templates/sonar/sonar.yaml`: Update the `sonar.yaml` file to match the configuration below: ```yaml title="sonar.yaml" apiVersion: edp.epam.com/v1alpha1 kind: Sonar metadata: name: sonar spec: url: {{ .Values.sonarUrl }} secret: {{ .Values.sonarSecret }} defaultPermissionTemplate: edp-default settings: - key: sonar.dbcleaner.hoursBeforeKeepingOnlyOneSnapshotByDay value: "20" - key: sonar.auth.oidc.enabled value: "true" - key: sonar.auth.oidc.issuerUri value: "https://login.microsoftonline.com//v2.0" - key: sonar.auth.oidc.clientId.secured value: "" - key: sonar.auth.oidc.clientSecret.secured value: "" - key: sonar.core.serverBaseURL value: {{ .Values.sonarUrl }} - key: sonar.auth.oidc.groupsSync.claimName value: "groups" - key: sonar.auth.oidc.groupsSync value: "true" - key: sonar.auth.oidc.idTokenSigAlg value: "RS256" ``` Replace the placeholders with the following values: - `` - Microsoft Entra Tenant ID. - `` - Application ID from the Microsoft Entra Application settings. - `` - Generated client secret value from the Microsoft Entra Application settings. 2. After updating the `sonar.yaml` file, commit the changes to the repository and apply the changes with Helm or Argo CD. 3. Verify that the OIDC authentication is configured correctly by logging in to SonarQube using **Log in with OpenID Connect** option: ![SonarQube login](../../assets/operator-guide/microsoft-entra-auth/sonar-login.png) After completing these steps, SonarQube will be configured with OIDC authentication using Microsoft Entra as the Identity Provider. Users will be able to log in to SonarQube using their Microsoft Entra credentials. ## Troubleshooting SonarQube SSO with Microsoft Entra If the login flow fails after completing the steps above, check the following common causes: - **Error `AADSTS50011` (redirect URI mismatch) after clicking Log in with OpenID Connect.** The Redirect URI registered in the Microsoft Entra Application must exactly match the format shown in the registration step above, including the scheme and callback path. Update the application registration and retry. - **Login succeeds, but the user has no expected permissions.** Group synchronization did not match. Verify that the **groups claim** is added in the **Token configuration** section with the **Group ID** type, that the SonarQube group corresponds to the value emitted in the token as described in the groups section above, and that the user is a member of the Microsoft Entra group. - **Authentication fails immediately after redirect back to SonarQube.** The client secret has expired or the secret **Secret ID** was copied instead of the secret **Value**. Generate a new client secret in **Certificates & secrets** and update the SonarQube configuration. - **The Log in with OpenID Connect button is missing.** The OIDC configuration was not applied. Confirm the updated values were committed and deployed, and that the SonarQube pod restarted after the change. ## Related Articles * [DefectDojo Microsoft Entra OIDC SSO Setup](./defectdojo-oidc-authentication.md) * [Dependency-Track Microsoft Entra OIDC SSO](./dependency-track-authentication.md) * [OpenID Connect Authentication Overview](./oidc-authentication-overview.md) --- ## Kubernetes Audit Trails Setup # Audit Trails Setup The **krci-audit** add-on records an append-only audit trail of admission changes to the Kubernetes resources managed by KubeRocketCI — who created, updated, or deleted an object, and when. It is self-contained and never blocks platform operations: if auditing is unavailable, the change still proceeds. ## Architecture krci-audit is composed of three parts and a read API: * **Capture** — a `ValidatingWebhookConfiguration` (the native Kubernetes admission-control mechanism, backed by [kube-audit-rest](https://github.com/RichardoC/kube-audit-rest)) receives the `AdmissionReview` for each matching change. Its `failurePolicy` is `Ignore` with a short timeout, so platform mutations are never blocked. * **Ship** — a [Vector](https://vector.dev/) sidecar tails the capture log, keeps only the events selected by the store filter, and writes them to PostgreSQL. * **Store** — a dedicated PostgreSQL database holds the events in a monthly-partitioned, append-only table. Writes use a least-privilege role that cannot update or delete existing records. * **Read API** — a separate, read-only service exposes initiator lookup (who created a resource) and an audit-events query. It connects to PostgreSQL as a SELECT-only role, so the read path can never modify the trail. Data flows one way: ```mermaid flowchart LR A[Resource change] --> B[Capture webhook] B --> C[Vector] C --> D[(PostgreSQL)] D --> E[Read API] E --> F[Consumers] ``` A Kubernetes resource change triggers the capture webhook, which Vector ships to PostgreSQL; the read API then serves that stored data to consumers. ## Installation krci-audit is delivered as a KubeRocketCI [add-on](../add-ons-overview.md). It is disabled by default. **Prerequisites:** [cert-manager](https://cert-manager.io/) (bundled with KubeRocketCI) issues the webhook's serving certificate. For `pgo` database mode you also need the Crunchydata PostgreSQL operator; to deliver credentials through the [External Secrets Operator](../secrets-management/install-external-secrets-operator.md), install that add-on first. 1. Provide the database credentials Secret. The chart never generates credentials — it only reads a Secret you supply, in every database mode. Create it in the `krci-audit` namespace (or populate it with the [External Secrets Operator](../secrets-management/install-external-secrets-operator.md)): ```bash kubectl -n krci-audit create secret generic krci-audit-db-access \ --from-literal=db-owner-username=krci-audit \ --from-literal=db-owner-password="$(openssl rand -base64 24)" \ --from-literal=writer-password="$(openssl rand -base64 24)" \ --from-literal=reader-password="$(openssl rand -base64 24)" ``` The `db-owner-*` keys are required for `simple` mode; `writer-password` is always required; `reader-password` is required when the read API is enabled. 2. Enable the add-on in `clusters/core/apps/values.yaml`: ```yaml krci-audit: enable: true ``` 3. Sync the add-on in Argo CD. The chart applies the schema migration, provisions PostgreSQL (for `simple`/`pgo` modes), and starts the capture and read API workloads. ## Configuration The add-on wrapper overrides only `db.mode`; the full configuration surface lives in the component chart's `values.yaml`. The most common settings: | Setting | Default | Purpose | |---------|---------|---------| | `capture.rules` | `v2.edp.epam.com/*`, `tekton.dev/pipelineruns` | Which admissions the API server sends to the webhook. | | `capture.filter` | KubeRocketCI groups + Tekton PipelineRuns | Which received events are actually stored (change any time with `helm upgrade`, no rebuild). | | `capture.namespaces` | `[]` (all) | Restrict auditing to named namespaces. | | `capture.level` | `metadata` | Object body captured: `metadata` (bounds size and PII) or `full`. | | `db.mode` | `simple` (add-on) | PostgreSQL provisioning: `external` (bring your own), `pgo` (Crunchydata operator), or `simple` (in-cluster Postgres for dev/small installs). | | `retention.months` | `12` | Months of history to keep (see [Retention](#retention)). | ### Retention A scheduled CronJob (nightly at `02:00` by default) drops audit history older than `retention.months`. Retention is **month-granular**: expired data is removed a whole monthly partition at a time, so the smallest effective window is about one month — sub-monthly windows (for example, three weeks) are not supported. The job only drops whole partitions and never deletes individual rows, preserving append-only integrity. ## Access control The read API is exposed only inside the cluster (a `ClusterIP` Service, no Ingress); consumers reach it by in-cluster DNS. It connects to the database as a SELECT-only role, so it cannot alter the trail. Network exposure is the interim access boundary; API authentication and authorization are planned. ## Portal integration The portal uses the read API for two features: * **Triggered By** on PipelineRun details — initiator lookup for who created the run. * **Administration → Audit Events** — the filterable audit events UI for portal administrators. Wire the portal to the in-cluster API (adjust host if your release or namespace differs): ```yaml title="krci-portal values (configEnv)" configEnv: KRCI_AUDIT_URL: http://krci-audit-api.krci-audit:8080 ``` **Administration → Audit Events** is shown only when the signed-in user has the portal `administrator` role. That role is resolved from OIDC group membership (default Keycloak group: `administrator`; override with `PORTAL_ADMIN_GROUPS`). Sessions that use a Kubernetes Service Account token do not receive portal roles, so the Administration section stays hidden for token login even when krci-audit is running. Without the add-on or without a reachable `KRCI_AUDIT_URL`, Triggered By shows **N/A** when no initiator can be resolved, and the Audit Events API returns a configuration error. ## Querying the audit trail The read API is reachable in-cluster at `-api.:8080`. To query it from your workstation, port-forward the Service: ```bash kubectl -n krci-audit port-forward svc/krci-audit-api 18080:8080 ``` Look up who created a resource (initiator lookup) by `kind`, `namespace`, and `name`: ```bash curl "http://localhost:18080/api/v1/audit/initiator?kind=PipelineRun&namespace=krci&name=build-test-go-app-main-8a2c" ``` ```json { "actor": "system:serviceaccount:krci:krci-admin", "operation": "CREATE", "found": true, "timestamp": "2026-07-07T12:41:00Z" } ``` Query the recorded events (filterable by `kind`, `operation`, `actor`, and paginated): ```bash curl "http://localhost:18080/api/v1/audit/events?kind=PipelineRun&operation=UPDATE&limit=20" ``` ## Compliance usage The audit trail provides evidence for compliance frameworks such as **SOC 2**, **ISO 27001**, **GDPR**, and **PCI DSS**: an immutable, time-stamped record of who changed which resources. Auditors retrieve evidence through the read API — initiator lookup ("who created this resource") and the filterable audit-events query — without direct database access. ## Related Articles * [Upgrade KubeRocketCI v3.13 to 3.14](../upgrade/upgrade-krci-3.14.md) * [Install via Add-Ons](../add-ons-overview.md) * [Authentication and Authorization](../auth/platform-auth-model.md) * [Pipelines Overview](../../user-guide/pipelines.md) --- ## Install Grafana Loki # Install Grafana Loki KubeRocketCI configures the logging with the help of [Grafana Loki](https://grafana.com/oss/loki/) aggregation system. ## Installation To install Loki, follow the steps below: 1. Create **logging** namespace: ```bash kubectl create namespace logging ``` 2. Add a chart repository: ```bash helm repo add grafana https://grafana.github.io/helm-charts helm repo update ``` :::note It is possible to use Amazon Simple Storage Service [Amazon S3](https://aws.amazon.com/s3/) as an object storage for Loki. To configure access, please refer to the [IRSA for Loki](./loki-irsa.md) documentation. ::: 3. Install **Loki v.2.6.0**: ```bash helm install loki grafana/loki \ --version 2.6.0 \ --values values.yaml \ --namespace logging ``` Check out the *values.yaml* file sample of the Loki customization:
View: values.yaml ```yaml image: repository: grafana/loki tag: 2.3.0 config: auth_enabled: false schema_config: configs: - from: 2021-06-01 store: boltdb-shipper object_store: s3 schema: v11 index: prefix: loki_index_ period: 24h storage_config: aws: s3: s3:///loki- boltdb_shipper: active_index_directory: /data/loki/index cache_location: /data/loki/boltdb-cache shared_store: s3 chunk_store_config: max_look_back_period: 24h resources: limits: memory: "128Mi" requests: cpu: "50m" memory: "128Mi" serviceAccount: create: true name: edp-loki annotations: eks.amazonaws.com/role-arn: "arn:aws:iam:::role/AWSIRSA‹CLUSTER_NAME›‹LOKI_NAMESPACE›Loki persistence: enabled: false ```
4. Configure [custom bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) to delete the old data. --- ## Configuring Fluent Bit for Elasticsearch ILM Rollover # Configuring Fluent Bit for Elasticsearch ILM Rollover This guide delves into the benefits of adopting the EFK stack over the conventional ELK stack, highlighting the significant enhancements it brings to KubeRocketCI. It provides a comprehensive walkthrough on configuring the EFK stack, enabling the integration of a sophisticated logging system with your applications. ## Overview of the ELK Stack The **ELK** (Elasticsearch, Logstash, and Kibana) stack is renowned for its capability to consolidate logs from various systems and applications. It facilitates the analysis of these logs and the creation of visualizations for monitoring applications and infrastructure, expediting troubleshooting, enhancing security analytics, and more. Below is an overview of the default components within the ELK stack: - **Beats family** - A suite of log shippers that transport logs from their sources to the stack, including Filebeat, Metricbeat, Packetbeat, among others. Beats can either replace Logstash or complement it. - **Logstash** - A robust log processing framework that collects, processes, stores, and searches logs. - **Elasticsearch** - A highly scalable search and analytics engine built on the [Lucene](https://lucene.apache.org/) Java library. - **Kibana** - A powerful visualization tool that queries data from Elasticsearch, enabling insightful data analysis. ![Overview of ELK Stack Components](../../assets/drawio-diagrams/elk_stack01.drawio.png "ELK Stack") ### EFK Stack Overview We use **FEK (also called EFK)** (Fluent Bit, Elasticsearch, Kibana) stack in Kubernetes instead of **ELK** because this stack provides us with the support for [Logsight](https://logsight.ai/#/) for Stage Verification and Incident Detection. In addition to it, **Fluent Bit** has a smaller memory fingerprint than Logstash. **Fluent Bit** has the Inputs, Parsers, Filters and Outputs plugins similarly to Logstash. ![Default components of FEK Stack](../../assets/drawio-diagrams/elk_stack02.drawio.png "FEK Stack") ### Automate Elasticsearch Index Rollover With ILM In this guide, index rollover with the Index Lifecycle Management [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) is automated in the FEK stack. The resources can be created via API using curl, Postman, Kibana Dev Tools console or via GUI. They are going to be created them using Kibana Dev Tools. 1. Go to **Management** -> **Dev Tools** in the Kibana dashboard: ![Index Pattern](../../assets/operator-guide/dev-tools.png "Dev Tools") 2. Create index lifecycle policy with the index [rollover](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/ilm-rollover.html): :::note This policy can also be created in GUI in **Management** -> **Stack Management** -> **Index Lifecycle Policies** ::: [Index Lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html) has several phases: Hot, Warm, Cold, Frozen, Delete. Indices also have different [priorities](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-set-priority.html) in each phase. The warmer the phase, the higher the priority is supposed to be, e.g., 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase. In this Use Case, only the Hot and Delete phases are configured. So an index will be created, rolled over to a new index when 1gb in size or 1day in time and deleted in 7 days. The rollover may not happen exactly at 1GB because it depends on how often Kibana checks the index size. Kibana usually checks the index size every 10 minutes but this can be changed by setting the [indices.lifecycle.poll_interval](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-settings.html) monitoring timer. The index lifecycle policy example:
Index Lifecycle Policy ```bash PUT _ilm/policy/fluent-bit-policy { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 }, "rollover": { "max_size": "1gb", "max_primary_shard_size": "1gb", "max_age": "1d" } } }, "delete": { "min_age": "7d", "actions": { "delete": { "delete_searchable_snapshot": true } } } } } } ```
Insert the code above into the `Dev Tools` and click the arrow to send the `PUT` request. 3. Create an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) so that a new index is created according to this template after the rollover: :::note This policy can also be created in GUI in `Management` -> `Stack Management` -> `Index Management` -> `Index Templates`. ::: Expand the menu below to see the index template example:
Index Template ```bash PUT /_index_template/fluent-bit { "index_patterns": ["fluent-bit-kube-*"], "template": { "settings": { "index": { "lifecycle": { "name": "fluent-bit-policy", "rollover_alias": "fluent-bit-kube" }, "number_of_shards": "1", "number_of_replicas": "0" } } } } ```
:::note - `index.lifecycle.rollover_alias` is required when using a policy containing the rollover action and specifies which alias to rollover on behalf of this index. The intention here is that the rollover [alias](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/aliases.html) is also defined on the index. - `number_of_shards` is the quantity of the primary shards. Elasticsearch index is really just a logical grouping of one or more physical shards, where each shard is actually a self-contained index. By distributing the documents in an index across multiple shards and distributing those shards across multiple nodes, Elasticsearch can ensure redundancy, which both protects against hardware failures and increases query capacity as nodes are added to a cluster. As the cluster grows (or shrinks), Elasticsearch automatically migrates shards to re-balance the cluster. Please refer to the official documentation [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html). - `number_of_replicas` is the number of replica shards. A replica shard is a copy of a primary shard. Elasticsearch will never assign a replica to the same node as the primary shard, so make sure you have more than one node in your Elasticsearch cluster if you need to use replica shards. The Elasticsearch cluster details and the quantity of nodes can be checked with: ```bash GET _cluster/health ``` Since we use one node, the number_of_shards is 1 and number_of_replicas is 0. If you put more replicas within one node, your index will get yellow status in Kibana, yet still be working. ::: 4. Create an empty [index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) with write permissions: :::note This index can also be created in GUI in `Management` -> `Stack Management` -> `Index Management` -> `Indices`. ::: Index example with the [date math](https://www.elastic.co/guide/en/elasticsearch/reference/current/api-conventions.html) format:
Index ```bash # URI encoded / PUT /%3Cfluent-bit-kube-%7Bnow%2Fd%7D-000001%3E { "aliases": { "fluent-bit-kube": { "is_write_index": true } } } ```
The code above will create an index in the`{index_name}-{current_date}-{rollover_index_increment}` format. For example: `fluent-bit-kube-2023.03.17-000001`. Please refer to the official documentation on the index rollover with Date Math [here](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html#_using_date_math_with_the_rollover_api). :::note It is also possible to use index pattern below if the date math format does not seem applicable:
Index ```bash PUT fluent-bit-kube-000001 { "aliases": { "fluent-bit-kube": { "is_write_index": true } } } ```
Check the status of the created index: ```bash GET fluent-bit-kube*-000001/_ilm/explain ``` ::: 5. Configure Fluent Bit. Play attention to the [Elasticsearch Output](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch) plugin configuration. The important fields in the `[OUTPUT]` section are `Index fluent-bit-kube` since we should use the index with the same name as Rollover Alias in Kibana and `Logstash_Format Off` as we use the Rollover index pattern in Kibana that increments by 1. ConfigMap example with [Configuration Variables](https://docs.fluentbit.io/manual/v/1.0/configuration/variables) for `HTTP_User` and `HTTP_Passwd`:
ConfigMap fluent-bit ```yaml data: fluent-bit.conf: | [SERVICE] Daemon Off Flush 10 Log_Level info Parsers_File parsers.conf Parsers_File custom_parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 Health_Check On [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker Mem_Buf_Limit 5MB Skip_Long_Lines Off Refresh_Interval 10 [INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=kubelet.service Read_From_Tail On Strip_Underscores On [FILTER] Name kubernetes Match kube.* Kube_Tag_Prefix kube.var.log.containers. Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Merge_Log Off Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude On [FILTER] Name nest Match kube.* Operation lift Nested_under kubernetes Add_prefix kubernetes. [FILTER] Name modify Match kube.* Copy kubernetes.container_name tags.container Copy log message Copy kubernetes.container_image tags.image Copy kubernetes.namespace_name tags.namespace [FILTER] Name nest Match kube.* Operation nest Wildcard tags.* Nested_under tags Remove_prefix tags. [OUTPUT] Name es Match kube.* Index fluent-bit-kube Host elasticsearch-master Port 9200 HTTP_User ${ES_USER} HTTP_Passwd ${ES_PASSWORD} Logstash_Format Off Time_Key @timestamp Type flb_type Replace_Dots On Retry_Limit False Trace_Error Off ```
6. Create index pattern (Data View starting from Kibana v8.0): Go to **Management** -> **Stack Management** -> **Kibana** -> **Index patterns** and create an index with the `fluent-bit-kube-*` pattern: ![Index Pattern](../../assets/operator-guide/elk-stack01.png "Index Pattern") 7. Check logs in Kibana. Navigate to **Analytics** -> **Discover**: ![Logs in Kibana](../../assets/operator-guide/elk-stack02.png "Logs in Kibana") :::note In addition, in the top-right corner of the `Discover` window, there is a button called `Inspect`. Clicking on it will reveal the query that Kibana is sending to Elasticsearch. These queries can be used in Dev Tools. ::: 8. Monitor the created indices: ```bash GET _cat/indices/fluent-bit-kube-* ``` :::note Physically, the indices are located on the `elasticsearch` Kubernetes pod in `/usr/share/elasticsearch/data/nodes/0/indices`. It is recommended to backup indices only via [Snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html). ::: We've configured the index rollover process. Now the index will be rolled over to a new one once it reaches the indicated size or time in the policy, and old indices will be removed according to the policy as well. When you create an empty index that corresponds to the pattern indicated in the index template, the index template attaches `rollover_alias` with the `fluent-bit-kube` name, policy and other configured data. Then the Fluent Bit Elasticsearch output plugin sends logs to the `Index fluent-bit-kube` rollover alias. The index rollover process is managed by ILM that increments our indices united by the `rollover_alias` and distributes the log data to the latest index. ### ILM Without Rollover Policy It is also possible to manage index lifecycle without rollover indicated in the policy. If this is the case, this section will explain how to refactor the index to make it look that way: `fluent-bit-kube-2023.03.18`. :::note The main drawback of this method is that the indices can be managed only by their creation date. ::: To manage index lifecycle without rollover policy, follow the steps below: 1. Create a Policy without `rollover` but with indices deletion:
Index Lifecycle Policy ```json PUT _ilm/policy/fluent-bit-policy { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } }, "delete": { "min_age": "7d", "actions": { "delete": { "delete_searchable_snapshot": true } } } } } } ```
2. Create an index template with the `rollover_alias` parameter:
Index Template ```json PUT /_index_template/fluent-bit { "index_patterns": ["fluent-bit-kube-*"], "template": { "settings": { "index": { "lifecycle": { "name": "fluent-bit-policy", "rollover_alias": "fluent-bit-kube" }, "number_of_shards": "1", "number_of_replicas": "0" } } } } ```
3. Change the Fluent Bit `[OUTPUT]` config to this one:
ConfigMap fluent-bit ```yaml [OUTPUT] Name es Match kube.* Host elasticsearch-master Port 9200 HTTP_User ${ES_USER} HTTP_Passwd ${ES_PASSWORD} Logstash_Format On Logstash_Prefix fluent-bit-kube Logstash_DateFormat %Y.%m.%d Time_Key @timestamp Type flb_type Replace_Dots On Retry_Limit False Trace_Error On ```
4. Restart Fluent Bit pods. Fluent Bit will be producing a new index every day with the new date in its name like in the `fluent-bit-kube-2023.03.18` name. Index deleting will be performed according to the policy. ### Tips on Fluent Bit Debugging If you experience a lot of difficulties when dealing with Fluent Bit, this section may help you. Fluent Bit has docker images labelled `-debug`, e.g., `cr.fluentbit.io/fluent/fluent-bit:2.0.9-debug`. Change that image in the Kubernetes Fluent Bit DaemonSet and add the `Trace_Error On` parameter to the `[OUTPUT]` section in the Fluent Bit configmap: ```yaml [OUTPUT] Trace_Error On ``` After adding the parameter above, you will start seeing more informative logs that will probably help you find out the reason of the problem. ### Related Articles * [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) * [Elasticsearch Output](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch) --- ## IAM Roles for Loki Service Accounts # IAM Roles for Loki Service Accounts Loki supports Amazon Simple Storage Service [Amazon S3](https://aws.amazon.com/s3/) as an object storage solution. In this case [Loki](https://grafana.com/docs/loki/latest/configure/examples/configuration-examples/) requires access to AWS resources. Follow the steps below to create a required role: 1. Create AWS IAM Policy `AWSIRSALoki_policy`: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListObjects", "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::loki-*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::loki-*" ] } ] } ``` 2. Create AWS IAM Role `AWSIRSALoki` with trust relationships: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { ":sub": "system:serviceaccount::edp-loki" } } } ] } ``` 3. Attach the `AWSIRSALoki_policy` policy to the `AWSIRSALoki` role. 4. Make sure that [Amazon S3](https://aws.amazon.com/s3/) bucket with name `loki-` exists. 5. Provide key value **eks.amazonaws.com/role-arn: "arn:aws:iam::\:role/AWSIRSA‹CLUSTER_NAME›‹LOKI_NAMESPACE›Loki"** into the **serviceAccount.annotations** parameter in *values.yaml* during the [Loki Installation](./install-loki.md#installation). ## Related Articles * [Associate IAM Roles With Service Accounts](../infrastructure-providers/aws/enable-irsa.md) * [Install Grafana Loki](install-loki.md) --- ## Multitenant Logging # Multitenant Logging Get acquainted with the multitenant logging components. ## Logging Components To configure the multitenant logging, it is necessary to deploy the following components: - [Grafana](https://grafana.com/) - [Loki](https://grafana.com/oss/loki/) - [Logging-operator](https://kube-logging.dev/docs/) In Grafana, every tenant represents an organization, i.e. it is necessary to create an organization for every namespace in the cluster. To get more details regarding the architecture of the Logging Operator, review the diagram below: ![Logging operator scheme](../../assets/operator-guide/logging-operator-architecture.png "Logging operator scheme") :::note It is necessary to deploy Loki with the `auth_enabled: true` flag with the aim to ensure that the logs are separated for each tenant. For the authentication, Loki requires the HTTP header `X-Scope-OrgID`. ::: ## Review Project Logs in Grafana To find the project logs, navigate to Grafana and follow the steps below: :::note Grafana is a common service for different customers where each customer works in its own separated Grafana Organization and doesn't have any access to another project. ::: = v9.5.0', value: '> v9.5.0'}, ]}> 1. Choose the organization by clicking the **Current Organization** drop-down list. If a user is assigned to several organizations, switch easily by using the Switch button: ![Current organization](../../assets/operator-guide/multitenant-logging/grafana-organization-user-info.png "Current organization") 2. Navigate to the left-side menu and click the **Explore** button to see the Log Browser: ![Grafana explore](../../assets/operator-guide/multitenant-logging/grafana-explore.png "Grafana explore") 3. Click the **Log Browser** button to see the labels that can be used to filter logs (e.g., hostname, namespace, application name, pod, etc.): :::note Enable the correct data source, select the relevant logging data from the top left-side corner, and pay attention that the data source name always follows the `project_name-logging` pattern. ::: ![Log browser](../../assets/operator-guide/multitenant-logging/grafana-log-browser-original.png "Log browser") 4. Filter out logs by clicking the **Show logs** button or write the query and click the **Run query** button. 5. Review the results with the quantity of logs per time, see the example below: ![Logs example](../../assets/operator-guide/multitenant-logging/grafana-logs-example.png "Logs example") - Expand the logs to get detailed information about the object entry: ![Expand logs](../../assets/operator-guide/multitenant-logging/grafana-expand-logs.png "Expand logs") - Use the following buttons to include or remove the labels from the query: ![Addition button](../../assets/operator-guide/multitenant-logging/grafana-addition-button.png "Addition button") - See the ad-hoc statistics for a particular label: ![Ad-hoc stat example](../../assets/operator-guide/multitenant-logging/grafana-ad-hoc-stat-example.png "Ad-hoc stat example") 1. Choose the organization from the drop-down list in the upper left corner: ![Current organization](../../assets/operator-guide/multitenant-logging/grafana-organizations.png "Current organization") 2. Navigate to the left-side menu and click the **Explore** button: ![Grafana explore](../../assets/operator-guide/multitenant-logging/grafana-explore-tab.png "Grafana explore") 3. Click the **Label Browser** button to see the labels that can be used to filter logs: :::note Enable the correct data source, select the relevant logging data from the top left-side corner, and pay attention that the data source name always follows the `project_name-logging` pattern. ::: ![Log browser](../../assets/operator-guide/multitenant-logging/grafana-label-browser.png "Log browser") 4. In the appeared window, select the labels to filter logs (e.g., hostname, namespace, application name, pod, etc.): ![labels example](../../assets/operator-guide/multitenant-logging/grafana-label-browser-tab.png "Logs example") 5. Close the **Label Browser** window and click the **Run query** button. Review the results with the quantity of logs per time, see the example below: ![Run query](../../assets/operator-guide/multitenant-logging/grafana-query-result.png "Run query") - Expand the logs to get detailed information about the object entry. Use the following buttons to include or remove the labels from the query: ![Expand logs](../../assets/operator-guide/multitenant-logging/grafana-expand-logs-tab.png "Expand logs") - See the ad-hoc statistics for a particular label: ![Ad-hoc stat example](../../assets/operator-guide/multitenant-logging/grafana-label-statistics.png "Ad-hoc stat example") ## Related Articles * [Grafana Documentation](https://grafana.com/docs/grafana/latest/) --- ## Set Up OpenShift # Set Up OpenShift Make sure the cluster meets the following conditions: 1. OpenShift cluster is installed with minimum 2 worker nodes with total capacity 8 Cores and 32Gb RAM. 2. Load balancer (if any exists in front of OpenShift router or ingress controller) is configured with session stickiness, disabled HTTP/2 protocol and header size of 64k support. Find below an example of the Config Map for the NGINX Ingress Controller: ``` yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: client-header-buffer-size: 64k large-client-header-buffers: 4 64k use-http2: "false" ``` 3. Keycloak instance is installed. To get accurate information on how to install Keycloak, please refer to the [Install Keycloak](auth/keycloak.md) instruction. 4. The installation machine with [oc](https://docs.openshift.com/container-platform/4.10/cli_reference/openshift_cli/getting-started-cli.html) is installed with the cluster-admin access to the OpenShift cluster. 5. Helm 3.10 is installed on the installation machine with the help of the [Installing Helm](https://v3.helm.sh/docs/intro/install/) instruction. 6. Storage classes are used with the [Retain Reclaim Policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain) and [Delete Reclaim Policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#delete). 7. We recommended using our storage class as [default storage class](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/#changing-the-default-storageclass). :::info By default, KubeRocketCI uses the `default Storage Class` in a cluster. We recommend using the following Storage Classes. Storage class templates with the Retain and Delete Reclaim Policies: ``` yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc annotations: storageclass.kubernetes.io/is-default-class: 'true' allowedTopologies: [] mountOptions: [] provisioner: ebs.csi.aws.com reclaimPolicy: Retain volumeBindingMode: Immediate ``` ``` yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 annotations: storageclass.kubernetes.io/is-default-class: 'true' allowedTopologies: [] mountOptions: [] provisioner: ebs.csi.aws.com reclaimPolicy: Delete volumeBindingMode: Immediate allowVolumeExpansion: true ``` ``` yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3-retain allowedTopologies: [] mountOptions: [] provisioner: ebs.csi.aws.com reclaimPolicy: Retain volumeBindingMode: Immediate allowVolumeExpansion: true ``` ::: ## Related Articles * [Install Amazon EBS CSI Driver](infrastructure-providers/aws/ebs-csi-driver.md) * [Install Keycloak](auth/keycloak.md) --- ## Overview # Overview Before installing KubeRocketCI via Helm Chart, make sure to complete the following steps: - Install and configure [Kubernetes](kubernetes-cluster-settings.md) or [OpenShift](openshift-cluster-settings.md) cluster - Install the [Nginx ingress controller](install-ingress-nginx.md) - Install [Tekton](install-tekton.md) - Install [Argo CD](install-argocd.md) :::note Alternatively, use the [cluster add-ons](add-ons-overview.md) approach to install the platform components. ::: After setting up the cluster and installing KubeRocketCI components according to the scenario, proceed to the [KubeRocketCI installation](install-kuberocketci.md). ## Related Articles * [Set Up Kubernetes](kubernetes-cluster-settings.md) * [Set Up OpenShift](openshift-cluster-settings.md) * [Install KubeRocketCI](install-kuberocketci.md) --- ## Install ReportPortal # Install ReportPortal Inspect the prerequisites and the main steps to perform for installing ReportPortal. ## Prerequisites - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. :::info Please refer to the [ReportPortal Helm Chart](https://github.com/reportportal/kubernetes/tree/develop/reportportal) section for details. ::: ## MinIO Installation To install MinIO, follow the steps below: 1. Check that `krci` namespace is created. If not, run the following command to create it: ```bash kubectl create namespace krci ``` :::warning For the OpenShift users When using the OpenShift platform, install the `SecurityContextConstraints` resources. In case of using a custom namespace for the `reportportal`, change the namespace in the `users` section. :::
View: report-portal-third-party-resources-scc.yaml ```yaml apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: report-portal-minio-rabbitmq-postgresql allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: null allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - min: 999 max: 65543 groups: [] priority: 1 readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsRange uidRangeMin: 1 uidRangeMax: 65543 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: - system:serviceaccount:report-portal:minio - system:serviceaccount:report-portal:rabbitmq - system:serviceaccount:report-portal:postgresql volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
View: report-portal-elasticsearch-scc.yaml ```yaml apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: report-portal-elasticsearch allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegedContainer: true allowedCapabilities: [] allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - max: 1000 min: 1000 groups: [] priority: 0 readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: type: MustRunAsRange uidRangeMax: 1000 uidRangeMin: 0 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: - system:serviceaccount:report-portal:elasticsearch-master volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
2. Add a chart repository: ```bash helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update ``` 3. Create MinIO admin secret: ```bash kubectl -n krci create secret generic reportportal-minio-creds \ --from-literal=root-password= \ --from-literal=root-user= ``` 4. Install MinIO v.11.10.3 using [bitnami/minio](https://artifacthub.io/packages/helm/bitnami/minio) Helm chart v.11.10.3: ```bash helm install minio bitnami/minio \ --version 11.10.3 \ --values values.yaml \ --namespace krci ``` Check out the _values.yaml_ file sample of the MinIO customization:
View: values.yaml ```yaml auth: existingSecret: reportportal-minio-creds persistence: size: 1Gi ```
## RabbitMQ Installation To install RabbitMQ, follow the steps below: 1. Use the `krci` namespace from the MinIO installation. 2. Use `bitnami` chart repository from the MinIO installation. 3. Create RabbitMQ admin secret: ```bash kubectl -n krci create secret generic reportportal-rabbitmq-creds \ --from-literal=rabbitmq-password= \ --from-literal=rabbitmq-erlang-cookie= ``` :::warning The `rabbitmq_password` password must be 10 characters long. The `rabbitmq_erlang_cookie` password must be 32 characters long. ::: 4. Install RabbitMQ v.10.3.8 using [bitnami/rabbitmq](https://artifacthub.io/packages/helm/bitnami/rabbitmq) Helm chart v.10.3.8: ```bash helm install rabbitmq bitnami/rabbitmq \ --version 10.3.8 \ --values values.yaml \ --namespace krci ``` Check out the _values.yaml_ file sample of the RabbitMQ customization:
View: values.yaml ```yaml auth: existingPasswordSecret: reportportal-rabbitmq-creds existingErlangSecret: reportportal-rabbitmq-creds persistence: size: 1Gi ```
5. After the rabbitmq pod gets the status Running, you need to configure the RabbitMQ memory threshold ```bash kubectl -n krci exec -it rabbitmq-0 -- rabbitmqctl set_vm_memory_high_watermark 0.8 ``` ## Elasticsearch Installation To install Elasticsearch, follow the steps below: 1. Use `krci` namespace from the MinIO installation. 2. Add a chart repository: ```bash helm repo add elastic https://helm.elastic.co helm repo update ``` 3. Install Elasticsearch v.7.17.3 using [elastic/elasticsearch](https://artifacthub.io/packages/helm/elastic/elasticsearch) Helm chart v.7.17.3: ```bash helm install elasticsearch elastic/elasticsearch \ --version 7.17.3 \ --values values.yaml \ --namespace krci ``` Check out the _values.yaml_ file sample of the Elasticsearch customization:
View: values.yaml ```yaml replicas: 1 extraEnvs: - name: discovery.type value: single-node - name: cluster.initial_master_nodes value: "" rbac: create: true resources: requests: cpu: "100m" memory: "2Gi" volumeClaimTemplate: resources: requests: storage: 3Gi ```
## PostgreSQL Installation To install PostgreSQL, follow the steps below: 1. Use `krci` namespace from the MinIO installation. 2. Add a chart repository: ```bash helm repo add bitnami-archive https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami helm repo update ``` 3. Create PostgreSQL admin secret: ```bash kubectl -n krci create secret generic reportportal-postgresql-creds \ --from-literal=postgresql-password= \ --from-literal=postgresql-postgres-password= ``` :::warning The `postgresql_password` and `postgresql_postgres_password` passwords must be 16 characters long. ::: 4. Install PostgreSQL v.10.9.4 using Helm chart v.10.9.4: ```bash helm install postgresql bitnami-archive/postgresql \ --version 10.9.4 \ --values values.yaml \ --namespace krci ``` Check out the _values.yaml_ file sample of the PostgreSQL customization:
View: values.yaml ```yaml persistence: size: 1Gi resources: requests: cpu: "100m" serviceAccount: enabled: true postgresqlUsername: "rpuser" postgresqlDatabase: "reportportal" existingSecret: "reportportal-postgresql-creds" initdbScripts: init_postgres.sh: | #!/bin/sh /opt/bitnami/postgresql/bin/psql -U postgres -d ${POSTGRES_DB} -c 'CREATE EXTENSION IF NOT EXISTS ltree; CREATE EXTENSION IF NOT EXISTS pgcrypto; CREATE EXTENSION IF NOT EXISTS pg_trgm;' ```
## ReportPortal Installation To install ReportPortal, follow the steps below: 1. Use `krci` namespace from the MinIO installation. :::warning For the OpenShift users When using the OpenShift platform, install the `SecurityContextConstraints` resource. In case of using a custom namespace for the `reportportal`, change the namespace in the `users` section. :::
View: report-portal-reportportal-scc.yaml ```yaml apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: annotations: "helm.sh/hook": "pre-install" name: report-portal allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegedContainer: true allowedCapabilities: [] allowedFlexVolumes: [] defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - max: 1000 min: 1000 groups: [] priority: 0 readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: type: MustRunAsRange uidRangeMax: 1000 uidRangeMin: 0 seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: - system:serviceaccount:report-portal:reportportal volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret ```
2. Add a chart repository: ```bash helm repo add report-portal "https://reportportal.github.io/kubernetes" helm repo update ``` 3. Install ReportPortal v.5.8.0 using Helm chart v.5.8.0: ```bash helm install report-portal report-portal/reportportal \ --values values.yaml \ --namespace krci ``` Check out the _values.yaml_ file sample of the ReportPortal customization:
View: values.yaml ```yaml serviceindex: resources: requests: cpu: 50m uat: resources: requests: cpu: 50m serviceui: resources: requests: cpu: 50m serviceAccountName: "reportportal" securityContext: runAsUser: 0 serviceapi: resources: requests: cpu: 50m serviceanalyzer: resources: requests: cpu: 50m serviceanalyzertrain: resources: requests: cpu: 50m rabbitmq: SecretName: "reportportal-rabbitmq-creds" endpoint: address: rabbitmq..svc.cluster.local user: user apiuser: user postgresql: SecretName: "reportportal-postgresql-creds" endpoint: address: postgresql..svc.cluster.local elasticsearch: endpoint: http://elasticsearch-master..svc.cluster.local:9200 minio: secretName: "reportportal-minio-creds" endpoint: http://minio..svc.cluster.local:9000 endpointshort: minio..svc.cluster.local:9000 accesskeyName: "root-user" secretkeyName: "root-password" ingress: # IF YOU HAVE SOME DOMAIN NAME SET INGRESS.USEDOMAINNAME to true usedomainname: true hosts: - report-portal-. ```
4. For the OpenShift platform, install a Gateway with Route:
View: gateway-config-cm.yaml ```yaml kind: ConfigMap metadata: name: gateway-config namespace: report-portal apiVersion: v1 data: traefik-dynamic-config.yml: | http: middlewares: strip-ui: stripPrefix: prefixes: - "/ui" forceSlash: false strip-api: stripPrefix: prefixes: - "/api" forceSlash: false strip-uat: stripPrefix: prefixes: - "/uat" forceSlash: false routers: index-router: rule: "Path(`/`)" service: "index" ui-router: rule: "PathPrefix(`/ui`)" middlewares: - strip-ui service: "ui" uat-router: rule: "PathPrefix(`/uat`)" middlewares: - strip-uat service: "uat" api-router: rule: "PathPrefix(`/api`)" middlewares: - strip-api service: "api" services: uat: loadBalancer: servers: - url: "http://report-portal-reportportal-uat:9999/" index: loadBalancer: servers: - url: "http://report-portal-reportportal-index:8080/" api: loadBalancer: servers: - url: "http://report-portal-reportportal-api:8585/" ui: loadBalancer: servers: - url: "http://report-portal-reportportal-ui:8080/" traefik.yml: | entryPoints: http: address: ":8081" metrics: address: ":8082" metrics: prometheus: entryPoint: metrics addEntryPointsLabels: true addServicesLabels: true buckets: - 0.1 - 0.3 - 1.2 - 5.0 providers: file: filename: /etc/traefik/traefik-dynamic-config.yml ```
View: gateway-deployment.yaml ```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: reportportal name: gateway namespace: report-portal spec: replicas: 1 selector: matchLabels: component: gateway template: metadata: labels: component: gateway spec: containers: - image: quay.io/waynesun09/traefik:2.3.6 name: traefik ports: - containerPort: 8080 protocol: TCP resources: {} volumeMounts: - mountPath: /etc/traefik/ name: config readOnly: true volumes: - name: config configMap: defaultMode: 420 name: gateway-config ```
View: gateway-route.yaml ```yaml kind: Route apiVersion: route.openshift.io/v1 metadata: labels: app: reportportal name: reportportal namespace: report-portal spec: host: report-portal. port: targetPort: http tls: insecureEdgeTerminationPolicy: Redirect termination: edge to: kind: Service name: gateway weight: 100 wildcardPolicy: None ```
View: gateway-service.yaml ```yaml apiVersion: v1 kind: Service metadata: labels: app: reportportal component: gateway name: gateway namespace: report-portal spec: ports: # use 8081 to allow for usage of the dashboard which is on port 8080 - name: http port: 8081 protocol: TCP targetPort: 8081 selector: component: gateway sessionAffinity: None type: ClusterIP ```
:::note For user access: default/1q2w3e For admin access: superadmin/erebus Please refer to the [ReportPortal.io](https://reportportal.io/installation/) page for details. ::: :::note It is also possible to install ReportPortal via cluster add-ons. For details, please refer to the [Install via Add-Ons](../add-ons-overview.md) page. ::: ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install via AWS Marketplace](../aws-marketplace-install.md) * [Install via Civo Marketplace](../install-via-civo.md) * [Install via Add-Ons](../add-ons-overview.md) --- ## Jira Integration # Jira Integration This documentation guide provides step-by-step instructions for enabling the Jira integration option in the KubeRocketCI portal for KubeRocketCI. Jira integration allows including useful metadata in Jira tickets. ## Overview Integrating Jira can provide a number of benefits, such as increased visibility and traceability, automatic linking code changes to relevant Jira issues, streamlining the management and tracking of development progress. By linking CI pipelines to Jira issues, teams can get a better understanding of the status of their work and how it relates to the overall development process. This can help to improve communication and collaboration, and ultimately lead to faster and more efficient delivery of software. Enabling Jira integration allows for the automatic population of three fields in Jira tickets: Fix Versions, Components, and Labels. Each of these fields provides distinct benefits: - **Fix Versions**: helps track progress against release schedules - **Components**: allows grouping related issues together - **Labels**: enables identification of specific types of work. Teams can utilize these fields to enhance their work prioritization, identify dependencies, improve collaboration, and ultimately achieve faster software delivery. ## Integration Procedure In order to adjust the Jira server integration, add the JiraServer CR by performing the following: 1. Provision the **ci-jira** secret using `KubeRocketCI portal`, `Manifest` or with the `externalSecrets` operator: Go to **KubeRocketCI** -> **Configuration** -> **Management Tool**. ![Jira update manual secret](../../assets/operator-guide/project-management-and-reporting/jira-edp-portal-secret.png "Jira update manual secret") Fill in the **URL**, **User**, **Password** fields and click the **Save** button: ![Add Jira Server](../../assets/operator-guide/project-management-and-reporting/add_jira_server.png "Add Jira Server") 1. Create Kubernetes Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: ci-jira namespace: krci labels: app.edp.epam.com/secret-type=jira stringData: username: username password: password ``` 2. Create `JiraServer` CR with the **apiUrl**, **credentialName** and **rootUrl** fields: ```yaml apiVersion: v2.edp.epam.com/v1 kind: JiraServer metadata: name: jira-server namespace: krci spec: apiUrl: 'https://jira-api.example.com' credentialName: ci-jira rootUrl: 'https://jira.example.com' ``` :::note The value of the **credentialName** property is the name of the Kubernetes Secret, created on the previous step. ::: 1. Configure secret in Parameter Store: ```json "ci-jira": { "username": "username", "password": "password" } ``` 2. Create `JiraServer` CR with the **apiUrl**, **credentialName** and **rootUrl** fields: ```yaml apiVersion: v2.edp.epam.com/v1 kind: JiraServer metadata: name: jira-server namespace: krci spec: apiUrl: 'https://jira-api.example.com' credentialName: ci-jira rootUrl: 'https://jira.example.com' ``` :::note The value of the **credentialName** property is the name of the Kubernetes Secret, created on the previous step. ::: :::note Required Permissions for Issue Management To manage issue labels, components, and add links in Jira, please make sure the user has the following permissions: 1. **Edit Issues:** This permission is necessary to modify issue fields, including adding or removing labels and components. 2. **Link Issues:** You must have this permission to create and manage links between issues. 3. **Add Comments:** Required for adding external links and comments to issues. ::: ## Enable Jira During Platform Deployment KubeRocketCI offers the capability to incorporate Jira integration as part of its deployment process. To follow this approach, please review the following parameters of the [values.yaml](https://github.com/epam/edp-install/blob/release/3.9/deploy-templates/values.yaml#L145) file. Enabling the `jira.integration` parameter creates the following custom resources: - QuickLink - JiraServer - External Secrets Operator (in case it is used) To set up Jira integration along with the platform, follow the steps below: 1. Create the **ci-jira** secret in the platform namespace (e.g., `krci`) as it's described [above](#integration-procedure). 2. Deploy the platform with the `jira.integration` parameter set to `true` in the [values.yaml](https://github.com/epam/edp-install/blob/release/3.8/deploy-templates/values.yaml#L138) file. ## Jira Usage To use Jira, you need to set up your codebases properly. During the codebase creation process, proceed to the **Advanced Settings** tab. Here, ensure to mark the **Integrate with Jira server** checkbox and complete the necessary fields accordingly: ![Advanced settings](../../assets/operator-guide/project-management-and-reporting/jira_integration_ac.png "Advanced settings") There are four predefined variables, each with specific values, that can be utilized individually or in combination. These variables display varying data based on the versioning type in use (`semver` or `default`): When using the `semver versioning` scheme: - **EDP_COMPONENT** – yields the name of the application - **EDP_VERSION** – produces either 0.0.0-SNAPSHOT or 0.0.0-RC as output - **EDP_SEM_VERSION** – generates a semantic version, specifically 0.0.0 - **EDP_GITTAG** – returns a Git tag format such as build/0.0.0-SNAPSHOT.2 or build/0.0.0-RC.2 If the `default versioning` type is used: - **EDP_COMPONENT** – returns application-name - **EDP_VERSION** – returns the date when the application was tagged. (Example: 20231023-131217) - **EDP_SEM_VERSION** – returns the date when the application was tagged. (Example: 20231023-131217) - **EDP_GITTAG** – returns the date when the application was tagged. (Example: 20231023-131217). :::note There are no character restrictions when combining the variables. You can concatenate them using the dash sign. Combination samples: **EDP_SEM_VERSION-EDP_COMPONENT**; **EDP_COMPONENT-hello-world/EDP_VERSION**; etc. ::: Upon successful configuration of the Jira integration, your tickets will be enriched with additional metadata: ![Additional Metadata](../../assets/operator-guide/project-management-and-reporting/jira_versioning_type_example.png "Additional Metadata") Should you encounter an issue where metadata is not appearing within a Jira ticket, it's advisable to inspect the status field of the `JiraIssueMetadata` custom resources located in the platform namespace (e.g, `krci`). The codebase-operator typically removes this resource after post-processing, it may persist in case of reconciliation issues, remaining within the namespace. ## Related Articles - [Add Application](../../user-guide/add-application.md) --- ## ReportPortal SSO with Keycloak (SAML) # Integrating ReportPortal with Keycloak: A Step-by-Step Guide Configure single sign-on for ReportPortal using Keycloak as the SAML identity provider. This guide covers creating the Keycloak SAML client and enabling the integration on the ReportPortal side. :::info It is also possible to install ReportPortal using the cluster add-ons. For details, please refer to the [Install via Add-Ons](../add-ons-overview.md) page. ::: ## Prerequisites - Keycloak is installed. For detailed instructions, please refer to the provided [documentation](../auth/keycloak.md). - ReportPortal is installed. To install it using the Helm Chart, please follow the instructions outlined in the corresponding [documentation](../project-management-and-reporting/install-reportportal.md). ## Keycloak Configuration :::info The Keycloak configuration can also be applied using Keycloak resources from the [Cluster Add-Ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/report-portal/templates/saml) repository. To apply the Keycloak configuration, the Keycloak Operator is required. For installation details, refer to the [Install Keycloak Operator](/docs/operator-guide/auth/eks-oidc-integration.md#install-keycloak-operator) page. ::: 1. Navigate to **Client Scopes** > **Create client scope** and create a new scope with the _SAML_ protocol type. 2. Navigate to **Client Scopes** > **your_scope_name** > **Mappers** > **Configure a new mapper** > select the _User Attribute_ mapper type. Add three mappers for the email, first name, and last name by typing `lastName`, `firstName`, and `email` in the _User Attribute_ field: - _Name_ is a display name in Keycloak. - _User Attribute_ is a user property for mapping. - _SAML Attribute Name_ is an attribute used for requesting information in the ReportPortal configuration. - _SAML Attribute NameFormat_: `Basic`. - _Aggregate attribute values_: `Off`. ![User mapper sample](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-1.png "User mapper sample") ![Scope mappers](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-2.png "Scope mappers") 3. Navigate to **Clients** > **Create client** and fill in the following fields: - _Client type_: `SAML`. - _Client ID_: `report.portal.sp.id`. :::warning The `report.portal.sp.id` Client ID is a constant value. ::: 4. Navigate to **Client** > **your_client** > **Settings** and add `https:///*` to the _Valid redirect URIs_. 5. Navigate to **Client** > **your_client** > **Keys** and disable _Client signature required_. ![Client keys](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-3.png "Client keys") 6. Navigate to **Client** > **your_client** > **Client scopes** and add the scope created on step 3 with the default _Assigned type_. ![Client scopes](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-4.png "Client scopes") ## ReportPortal Configuration 1. Log in to the ReportPortal with the admin permissions. 2. Navigate to **Client** > **Administrate** > **Plugins** and select the SAML plugin. ![Plugins menu](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-5.png "Plugins menu") 3. To add a new integration, fill in the following fields: ![Add SAML configuration](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-6.png "Add SAML configuration") - _Provider name_ is the display name in the ReportPortal login page. - _Metadata URL_ `https:///auth/realms//protocol/saml/descriptor`. - _Email_ is the value from the _SAML Attribute Name_ field in the Keycloak mapper. - _RP callback URL_ `https:///uat`. - _Name attributes mode_ is the first & last name (type based on your mapper). - _First name_ is the value from the _SAML Attribute Name_ field in the Keycloak mapper. - _Last name_ is the value from the _SAML Attribute Name_ field in the Keycloak mapper. 4. Log in to the ReportPortal. :::note By default, after the first login, ReportPortal creates the `_personal` project and adds an account with the `Project manager` role. ::: ![Report portal login page](../../assets/operator-guide/project-management-and-reporting/reportportal-keycloak-7.png "Report portal login page") ## Related Articles * [Install ReportPortal](../project-management-and-reporting/install-reportportal.md) * [Integration With Tekton](reportportal-tekton.md) * [Install Keycloak](../auth/keycloak.md) --- ## Integrating ReportPortal with Tekton: A Step-by-Step Guide # Tekton Integration Integrating ReportPortal with Tekton centralizes the management of all automation outcomes and reports, enabling the visualization of metrics and analytics. This integration facilitates team collaboration by associating statistical results, enhancing the overall efficiency and insight into the automation processes. For integration, take the following steps: 1. Log in to the ReportPortal console and navigate to the **User Profile** menu: ![ReportPortal profile](../../assets/operator-guide/project-management-and-reporting/report-portal-profile.png "ReportPortal profile") 2. Copy the **Access token** and use it as a value while creating a kubernetes secret for the ReportPortal credentials: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: rp-credentials namespace: krci stringData: rp_uuid: ``` 3. In the **Configuration examples** section of the ReportPortal **User Profile** menu, copy the following `REQUIRED` fields: `rp.endpoint`, `rp.launch` and `rp.project`. Insert these fields to the `pytest.ini` file in root directory of your project: ```bash [pytest] addopts = -rsxX -l --tb=short --junitxml test-report.xml rp_endpoint = rp_launch = rp_project = ``` 4. In root directory of the project create/update **requirements.txt** file and fill with following. It's mandatory to install report-portal python library (version may vary): ```bash pytest-reportportal == 5.1.2 ``` 5. Create a custom Tekton task:
View: Custom Tekton task ```yaml apiVersion: tekton.dev/v1beta1 kind: Task metadata: labels: app.kubernetes.io/version: '0.1' name: pytest-reportportal namespace: krci spec: description: |- This task can be used to run pytest integrated with report portal. params: - default: . description: The path where package.json of the project is defined. name: PATH_CONTEXT type: string - name: EXTRA_COMMANDS type: string - default: python:3.8-alpine3.16 description: The python image you want to use. name: BASE_IMAGE type: string - default: rp-credentials description: name of the secret holding the rp token name: rp-secret type: string steps: - env: - name: HOME value: $(workspaces.source.path) - name: RP_UUID valueFrom: secretKeyRef: key: rp_uuid name: $(params.rp-secret) image: $(params.BASE_IMAGE) name: pytest resources: {} script: > #!/usr/bin/env sh set -e export PATH=$PATH:$HOME/.local/bin $(params.EXTRA_COMMANDS) # tests are being run from ./test directory in the project pytest ./tests --reportportal workingDir: $(workspaces.source.path)/$(params.PATH_CONTEXT) workspaces: - name: source ```
6. Add this task ref to your Tekton pipeline after *tasks*:
View: Tekton pipeline ```yaml - name: pytest params: - name: BASE_IMAGE value: $(params.image) - name: EXTRA_COMMANDS value: | set -ex pip3 install -r requirements.txt [ -f run_service.py ] && python run_service.py & runAfter: - compile taskRef: kind: Task name: pytest-reportportal workspaces: - name: source workspace: shared-workspace ```
7. Launch your Tekton pipeline and check that the custom task has been successfully executed: ![Tekton task successfully executed](../../assets/operator-guide/project-management-and-reporting/tekton-task-success.png "Tekton task successfully executed") 8. Test reports will be displayed in the **Launches** section of the ReportPortal: ![Test report results](../../assets/operator-guide/project-management-and-reporting/report-portal-results.png "Test report results") ## Related Articles * [Install ReportPortal](../project-management-and-reporting/install-reportportal.md) * [Keycloak Integration](reportportal-keycloak.md) * [Pytest Integration With ReportPortal](https://github.com/reportportal/agent-python-pytest) --- ## External Secrets Operator Integration # External Secrets Operator Integration [External Secrets Operator (ESO)](https://external-secrets.io/) can be integrated with KubeRocketCI. There are [multiple Secrets Providers](https://external-secrets.io/latest/introduction/stability-support) that can be used within ESO. On this page, we detail the integration of KubeRocketCI with the two foremost providers: - [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) - [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) KubeRocketCI leverages a variety of secrets to facilitate the integration of different solutions. This document provides a comprehensive overview of the secrets utilized within the KubeRocketCI platform, along with their respective descriptions. Please note that all secrets are encoded in Base64 format. |Secret Name|Fields|Description|Used by| |:-|:-|:-|:-| |keycloak|usernamepassword|Username and password with [specific permissions](../auth/keycloak.md#configuration) for the Platform tenant management in Keycloak|keycloak-operator| |ci-defectdojo|tokenurl|DefectDojo tokenDefectDojo URL|edp-tekton| |kaniko-docker-config|.dockerconfigjson|Serialized JSON that follows docker config patterns|edp-tekton| |regcred|.dockerconfigjson|Serialized JSON that follows docker config patterns|cd-pipeline-operator| |ci-github|id_rsatokensecretString|Private key from github repo API tokenRandom string|edp-tekton| |ci-gitlab|id_rsatokensecretString|Private key from gitlab repo API tokenRandom string|edp-tekton| |ci-bitbucket|id_rsatokensecretString|Private key from bitbucket repo API tokenRandom string|edp-tekton| |ci-jira|usernamepassword|Jira username Jira password|edp-codebase-operator| |ci-sonarqube|tokenurl|SonarQube tokenSonarQube URL|edp-tekton| |ci-nexus|usernamepasswordurl|Nexus usernameNexus passwordNexus URL|edp-tekton| |ci-dependency-track|tokenurl|Dependency-Track tokenDependency-Track URL|edp-tekton| |oauth2-proxy-cookie-secret|cookie-secret|Secret key for oauth2-proxy|edp-install| |keycloak-client-headlamp-secret|clientSecret|Secret key for keycloak client |keycloak-operator| |ci-argocd|tokenurl|Argo CD tokenArgo CD URL|edp-tekton| ## Platform Core Secrets The list below represents the baseline required for full operation within platform core components: - kaniko-docker-config: Used for pushing container images to a specific registry. - ci-sonarqube: Used in the CI process for SonarQube integration. - ci-nexus: Used for pushing artifacts to the Nexus storage. :::warning These secrets are mandatory for Tekton pipelines to work properly. ::: ## Kubernetes Provider All secrets are stored in Kubernetes in pre-defined namespaces. Platform suggests using the following approach for secrets management: - `KRCI_NAMESPACE-vault`, where `KRCI_NAMESPACE` is a name of the namespace where KubeRocketCI is deployed, such as `krci-vault`. This namespace is used by the platform. Access to secrets in the `krci-vault` is granted only for `Administrators`. - `KRCI_NAMESPACE-cicd-vault`, where `KRCI_NAMESPACE` is a name of the namespace where KubeRocketCI is deployed, such as `krci-cicd-vault`. Development team uses secrets in the `krci-cicd-vault` for microservices development. See a diagram below for more details: ![eso-with-kubernetes](../../assets/operator-guide/eso-k8s.png) In order to [Install KubeRocketCI](../install-kuberocketci.md), a list of passwords must be created. Secrets are provided automatically when using ESO. 1. Create a common namespace for secrets and platform: ```bash kubectl create namespace krci-vault kubectl create namespace krci ``` 2. Create secrets in the `krci-vault` namespace: ```yaml apiVersion: v1 kind: Secret metadata: name: keycloak namespace: krci-vault data: password: cGFzcw== # pass in base64 username: dXNlcg== # user in base64 type: Opaque ``` 3. In the `krci-vault` namespace, create a Role with a permission to read secrets: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: krci-vault name: external-secret-store rules: - apiGroups: [""] resources: - secrets verbs: - get - list - watch - apiGroups: - authorization.k8s.io resources: - selfsubjectrulesreviews verbs: - create ``` 4. In the `krci-vault` namespace, create a ServiceAccount used by `SecretStore`: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: secret-manager namespace: krci ``` 5. Connect the Role from the `krci-vault` namespace with the ServiceAccount in the `krci` namespace: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: eso-from-krci namespace: krci-vault subjects: - kind: ServiceAccount name: secret-manager namespace: krci roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: external-secret-store ``` 6. Create a SecretStore in the `krci` namespace, and use ServiceAccount for authentication: ```yaml apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: krci-vault namespace: krci spec: provider: kubernetes: remoteNamespace: krci-vault # namespace with secrets auth: serviceAccount: name: secret-manager server: caProvider: type: ConfigMap name: kube-root-ca.crt key: ca.crt ``` 7. Each secret must be defined by the `ExternalSecret` object. A code example below creates the `keycloak` secret in the `krci` namespace based on a secret with the same name in the `krci-vault` namespace: ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: keycloak namespace: krci spec: refreshInterval: 1h secretStoreRef: kind: SecretStore name: krci-vault # target: # name: secret-to-be-created # name of the k8s Secret to be created. metadata.name used if not defined data: - secretKey: username # key to be created remoteRef: key: keycloak # remote secret name property: username # value will be fetched from this field - secretKey: password # key to be created remoteRef: key: keycloak # remote secret name property: password # value will be fetched from this field ``` Apply the same approach for enabling secrets management in the namespaces used for microservices development, such as `sit` and `qa` on the diagram above. ## AWS Systems Manager Parameter Store AWS SSM Parameter Store can be used as a [Secret Provider for ESO](https://external-secrets.io/latest/provider/aws-parameter-store). For the platform, it is recommended to use the [IAM Roles For Service Accounts approach](https://external-secrets.io/latest/provider/aws-parameter-store/#eks-service-account-credentials) (see a diagram below). ![eso-with-ssm](../../assets/operator-guide/eso-ssm.png) ### AWS Parameter Store Scenario in KubeRocketCI In order to [Install KubeRocketCI](../install-kuberocketci.md), a list of passwords must be created. Follow the steps below, to get secrets from the SSM: 1. In the AWS, create an AWS IAM policy and an IAM role used by `ServiceAccount` in `SecretStore`. The IAM role must have permissions to get values from the SSM Parameter Store. a. Create an IAM policy that allows to get values from the Parameter Store with the `krci/` path. Use your `AWS Region` and `AWS Account Id`: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "ssm:GetParameter*", "Resource": "arn:aws:ssm:eu-central-1:012345678910:parameter/krci/*" } ] } ``` b. Create an AWS IAM role with trust relationships (defined below) and attach the IAM policy. Put your string for `Federated` value ([see more](../infrastructure-providers/aws/enable-irsa.md) on IRSA enablement for EKS Cluster) and AWS region. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::012345678910:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXX" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringLike": { "oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:krci:*" } } } ] } ``` 2. Create a secret in the AWS Parameter Store with the name `/krci/my-json-secret`. This secret is represented as a parameter of type string within the AWS Parameter Store:
View: Parameter Store JSON ```json { "keycloak": { "username": "keycloak-username", "password": "keycloak-password" }, "ci-defectdojo": { "token": "XXXXXXXXXXXX", "url": "https://defectdojo.example.com" }, "kaniko-docker-config": { "auths" : { "registry.com": { "username":"registry-username", "password":"registry-password", "auth": "" } }}, "regcred": { "auths": { "registry.com": { "username":"registry-username", "password":"registry-password", "auth":"" } }}, "ci-github": { "id_rsa": "id-rsa-key", "token": "github-token", "secretString": "XXXXXXXXXXXX" }, "ci-gitlab": { "id_rsa": "id-rsa-key", "token": "gitlab-token", "secretString": "XXXXXXXXXXXX" }, "ci-jira": { "username": "jira-username", "password": "jira-password" }, "ci-sonarqube": { "username": "", "secret": "" }, "ci-nexus": { "username": "", "password": "" }, "oauth2-proxy-cookie-secret": { "cookie-secret": "XXXXXXXXXXXX" }, "keycloak-client-headlamp-secret": "XXXXXXXXXXXX", "ci-argocd": { "token": "argocd-token", "url": "https://argocd.example.com" }, "ci-dependency-track": { "token": "dependency-track-token", "url": "https://dependency-track.example.com" } } ```
3. Set External Secret operator enabled by updating the values.yaml file: ```yaml install values.yaml external-secrets: createNamespace: false enable: false namespace: external-secrets ``` 4. Install/upgrade edp-install: ```bash helm upgrade --install krci epamedp/edp-install --wait --timeout=900s \ --version \ --values values.yaml \ --namespace krci \ --atomic ``` ## Related Articles * [Install External Secrets Operator](install-external-secrets-operator.md) * [Secured Secrets for Application Deployment](../../use-cases/external-secrets.md) --- ## Install External Secrets Operator # Install External Secrets Operator Inspect the prerequisites and the main steps to perform for enabling [External Secrets Operator](https://external-secrets.io/) in KubeRocketCI. ## Prerequisites - Kubectl version 1.34.0+ is installed. Please refer to the [Kubernetes official website](https://kubernetes.io/releases/download/) for details. - [Helm](https://helm.sh) version 3.19.0+ is installed. Please refer to the [Helm page](https://github.com/helm/helm/releases) on GitHub for details. ## Installation To install `External Secrets Operator` with Helm, run the following commands: ```bash helm repo add external-secrets https://charts.external-secrets.io helm install external-secrets \ external-secrets/external-secrets \ --version 0.18.2 \ -n external-secrets \ --create-namespace ``` :::info It is also possible to install External Secrets Operator using the [Cluster Add-Ons](../add-ons-overview.md) or [Operator Lifecycle Manager (OLM)](https://operatorhub.io/operator/external-secrets-operator). ::: ## Related Articles * [External Secrets Operator Integration](external-secrets-operator-integration.md) * [Install via Add-Ons](../add-ons-overview.md) --- ## Codebase Creation Issues # Codebase Creation Issues This troubleshooting page covers issues related to codebases. ## Problem Application can't be added in the KubeRocketCI portal: ![Insufficient permissions error](../../assets/operator-guide/troubleshooting/insufficient_permissions.png "Insufficient permissions error") ## Cause Multiple factors can contribute to this issue occurring: | Short error | Description | |-------------|-------------| | Need at least read:org scope | This error occurs when the token used for Git Server integration lacks specific rights. To resolve this issue, adjust the token rights or create a new token with the required set of permissions. Refer to [Manage Git Providers](../../user-guide/add-git-server.md) for details. Ensure to set correct Git Server properties as described in [Manage Git Servers](../../user-guide/add-git-server.md). | | Name already exist | This error indicates that the application name you are trying to add in the KubeRocketCI portal already exists. Check if the application or codebase with the same name has been created previously. If necessary, choose a different name for your application or codebase to avoid naming conflicts. | | Failed to push changes | This error might occur due to insufficient permissions or issues with the repository access. Ensure the token or SSH key used has the proper rights to push changes to the repository. Additionally, verify that the repository URL and branch specified in the codebase configuration are correct and accessible. | | Invalid project ID | The "Invalid project ID" error message appears when creating a codebase with an incorrectly set repository relative path. To resolve this issue, ensure that the Git repository relative path is formatted correctly, prefixing the repository name with the respective account or organization name. | ## Solution For instance, to address the `Need at least read:org` scope issue: 1. Adjust token rights or create a new token with the required set of permissions. Please refer to the [Manage Git Providers](../../user-guide/add-git-server.md) for details. 2. Ensure to set correct Git Server properties. The process of adding a Git Server is described in the [Manage Git Servers](../../user-guide/add-git-server.md) page. 3. Retry to create application. ![Codebase created](../../assets/operator-guide/troubleshooting/codebase_synced.png "Codebase created") ## Related Articles * [Manage GitServer](../../user-guide/add-git-server.md) --- ## Container Registry Error # Container Registry Error ## Problem Failed to establish a connection with the container registry. ![Container registry issue](../../assets/operator-guide/troubleshooting/registry_issue.png "Container registry issue") ## Cause The user credentials are incorrect. ## Solution 1 (External Secret Operator) 1. Navigate to Cloud Provider's Management Console. 2. Navigate to `AWS System Manager` -> `Parameter Store`. 3. Update the credentials for the container registry to ensure they are current and have the necessary permissions. 4. Wait for External Secret Operator Credential Update. Allow approximately 2 minutes for the External Secret Operator to detect and implement the credential changes. ## Solution 2 (Manually Provisioned Credentials) 1. In the KubeRocketCI portal navigate **Configuration** -> **Artifacts Storage** -> **Registry**: 2. Modify user credentials, including endpoint, user, or token, to ensure appropriate access rights. :::note For detailed instructions, visit the [managing container registries.](../../user-guide/manage-container-registries.md) ::: ## Related Articles * [Change Container Registry](../../user-guide/change-container-registry.md) --- ## Environment Creation Issues # Environment Creation Issues ## Problem Failed to create an environment due to the following error: ![Failed environment](../../assets/operator-guide/troubleshooting/failed_cd_pipeline_creation.png "Failed environment") ## Cause KubeRocketCI offers several approaches for namespace management: namespaces can be created by users, by KubeRocketCI itself, or by Argo CD. If you expect KubeRocketCI to handle namespace creation, ensure that the [cd-pipeline-operator](https://github.com/epam/edp-cd-pipeline-operator), a component of KubeRocketCI, has the necessary permissions. This permission is managed by the `manageNamespace` parameter of the [values.yaml](https://github.com/epam/edp-install/blob/v3.14.0/deploy-templates/values.yaml#L288) file. If the `manageNamespace` parameter is set to false, it means the KubeRocketCI does not have permission to create namespaces and the procedure of creating namespaces in your project is delegated to a concrete person. ## Solution Make sure the `manageNamespace` parameter is intentionally set to `false`. If not, define proper flow for namespace management. Read the [Namespace Management](../auth/namespace-management.md) page for more details. If the parameter was intentionally set to `false`, the solution would be to clarify with your management who is allowed to manage namespaces in your project and request a namespace creation: If you are allowed to create namespaces, create it manually: 1. Open the terminal. Ensure you have access to the cluster from your kubeconfig. 2. Create a namespace using the `kubectl create namespace` command: ```bash kubectl create namespace ``` Usually, the required namespace is shown on the Overview tab of the Environment: ![Failed environment](../../assets/operator-guide/troubleshooting/failed_cd_pipeline_creation_namespace.png "Failed environment") ## Related Articles * [Namespace Management](../auth/namespace-management.md) --- ## Forbidden Messages on the Overview Page # Forbidden Messages on the Overview Page ## Problem Users can't see resources within the KubeRocketCI dedicated namespace via KubeRocketCI portal: ![Resource observability issue](../../assets/operator-guide/resource_observability_issue.png "Resource observability issue") ## Cause The problem might be caused by several factors. First of all, default and allowed namespaces may be either unset or set incorrectly. Secondly, service account used for browsing the KubeRocketCI portal may have insufficient permission set. Thirdly, if Keycloak is used for as an authentication mechanism, then the problem might be related to improper group membership. ## Solution Solution can vary depending on the way users log into the platform, whether it is a service account token or an OpenID Connect mechanism. ### Solution 1 (Configure Default namespace) This solution suits those who to log into the KubeRocketCI portal Keycloak. To fix the problem, check both default and allowed namespace by navigating to the **KubeRocketCI** portal -> **Account Settings** -> **Cluster**. Refer to the [KubeRocketCI portal](../../user-guide/portal-settings.md#manage-namespaces) page for more details. ### Solution 2 (Service Account Token) This solution suits those who use a service account token to log into the KubeRocketCI portal. To fix the problem, follow the steps below: 1. Make sure the service account is set up in accordance with the [KubeRocketCI Access Model](../../operator-guide/auth/platform-auth-model.md#cluster-rbac-resources) and possesses the appropriate access rights. 2. Check the roles associated with the ServiceAccount: ```bash serviceaccount= kubectl get rolebindings -n krci -o json | jq -r --arg sa "$serviceaccount" '.items[] | select(.subjects[]? | select(.kind == "ServiceAccount" and .name == $sa)) | .metadata.name' ``` 3. Check permissions of the role used in the role binding: ```bash kubectl describe role -n krci ``` 4. Refresh the page and verify that the new RBAC settings are functioning properly. ## Related Articles * [KubeRocketCI Access Model](../../operator-guide/auth/platform-auth-model.md) --- ## Modify GitOps Repository Path # Modify GitOps Repository Path This page explains how to change the path of a [GitOps](../../user-guide/gitops.md) repository after it has been created. ## Problem The GitOps repository is located at an incorrect path: ![GitOps Repository Path](../../assets/operator-guide/troubleshooting/gitops-configuration.png "GitOps repository path") ## Cause The GitOps repository path was either misconfigured at creation or needs to be modified as part of a migration or project restructuring. ## Solution 1 (Modify the Codebase Custom Resource) This solution involves modifying the existing repository path in Codebase custom resource. To fix the problem, follow the steps below: 1. Navigate to the **Kubernetes** tab in the bottom left corner in KubeRocketCI portal: ![Kubernetes Tab](../../assets/operator-guide/troubleshooting/kubernetes-tab.png "Kubernetes tab") 2. Navigate to **Cluster** -> **Custom Resources**: ![Custom Resources Tab](../../assets/operator-guide/troubleshooting/custom-resources-tab.png "Custom resources tab") 3. Find and navigate to the `Codebase` custom resource using the search field in the top right corner: ![Search Tab](../../assets/operator-guide/troubleshooting/codebase-crd.png "Search tab") 4. Find the `Codebase` custom resource that corresponds to the GitOps repository: ![GitOps Custom Resource](../../assets/operator-guide/troubleshooting/edp-gitops-cr.png "GitOps custom resource") 5. Navigate to the resource and then press the **Edit** button in the top right corner: ![Edit Button](../../assets/operator-guide/troubleshooting/edit-button.png "Edit button") 6. In the **Edit** window modify the `gitUrlPath` field within the `spec` section to change the GitOps repository path and then press the **Save and Apply** button: ![Edit GitOps Repository Path](../../assets/operator-guide/troubleshooting/edit-codebase-resource.png "Edit GitOps repository path") 7. With the path updated, the GitOps repository will be recreated at the new path on the designated Git Server. ## Solution 2 (Delete and Recreate) This solution involves deleting the existing GitOps repository and recreating it. To fix the problem, follow the steps below: 1. Navigate to the **Kubernetes** tab in the bottom left corner in KubeRocketCI portal: ![Kubernetes Tab](../../assets/operator-guide/troubleshooting/kubernetes-tab.png "Kubernetes tab") 2. Navigate to **Cluster** -> **Custom Resources**: ![Custom Resources Tab](../../assets/operator-guide/troubleshooting/custom-resources-tab.png "Custom resources tab") 3. Find and navigate to the `Codebase` custom resource using the search field in the top right corner: ![Search Tab](../../assets/operator-guide/troubleshooting/codebase-crd.png "Search tab") 4. Find the `Codebase` custom resource that corresponds to the GitOps repository: ![GitOps Custom Resource](../../assets/operator-guide/troubleshooting/edp-gitops-cr.png "GitOps custom resource") 5. Navigate to the resource and then press the **Delete** button in the top right corner: ![Delete Button](../../assets/operator-guide/troubleshooting/delete-button.png "Delete button") 6. Confirm the deletion by pressing the **Yes** button in the confirmation window: ![Delete Confirmation](../../assets/operator-guide/troubleshooting/confirm-deletion.png "Delete confirmation") 7. Click the **Portal** tab in the bottom left corner. Navigate to **Configuration** -> **Deployment** -> **GitOps** and recreate the GitOps repository with the new desired path: ![Create GitOps Repository](../../assets/operator-guide/troubleshooting/recreate-gitops-repo.png "Create GitOps repository") ## Related Articles * [Manage GitOps](../../user-guide/gitops.md) --- ## Troubleshooting Guide Overview # Troubleshooting Guide Overview Welcome to the Troubleshooting Guide for the KubeRocketCI. Here, we offer essential information to assist you with the challenges you may encounter while using the platform. This guide is designed to address common issues and answer frequently asked questions, aiming to streamline your troubleshooting process and enhance your experience. Whether you're dealing with deployment setbacks, authentication hurdles, or configuration complexities, you'll find helpful insights and solutions here. Currently, this section covers the following issues: - [Container Registry Error](container-registry-error.md) - [Sync and Health Status Unknown](sync-health-status-unknown.md) - [Codebase Creation Issues](codebase-creation-issues.md) - [Forbidden Messages on the Overview Page](forbidden-messages.md) - [Review Pipeline is Failed](review-pipeline-is-failed.md) - [Modify GitOps Repository Path](modify-gitops-repo-path.md) --- ## Review Pipeline is Failed # Review Pipeline is Failed ## Problem Application can't succeed with review. ![Tree diagram window](../../assets/operator-guide/troubleshooting/sonar_review_failed.png "Tree diagram window") ## Cause The review pipeline can fail for various reasons related to the Merge Request, including linter errors, scanner issues, tests, and more. The standard approach to troubleshooting involves navigating to the pipeline status to examine the error, and then making adjustments based on the type of error encountered. ## Solution (Sonar Scanner Error) This error occurs because SonarQube does not have complete source code data and is unable to compare changes from the patch with the main code. To resolve this issue, execute the build pipeline to submit a comprehensive report of the source code to SonarQube. 1. Navigate to your application in KubeRocketCI portal. 2. In the main branch section, click the **Trigger build pipeline run** button to run scanning of full repository. 3. After successful finished build pipeline - re-trigger review pipeline, for example by using `/recheck` (or `/ok-to-test`) feature. ![Tree diagram window](../../assets/operator-guide/troubleshooting/successful_pass_review.png "Tree diagram window") ## Related Articles * [Create Application](../../quick-start/create-application.md) * [SonarQube Integration](../code-quality/sonarqube.md) --- ## Sync and Health Status Unknown # Sync and Health Status Unknown ## Problem In the **Stage Detail** page, the application **Health status** and **Sync Status** is unknown. The `deploy` button doesn't work when attempting to create a stage. ![Stage is not synced](../../assets/operator-guide/troubleshooting/stage_is_not_synced.png "Stage is not synced") ## Cause Incorrect Configuration of ArgoCD Integration ## Solution 1. Please verify that the [Argo CD Integration](../../operator-guide/cd/argocd-integration.md) process has been completed and all steps have been executed correctly. 2. Check the status of the application: ![Stage is now synced](../../assets/operator-guide/troubleshooting/stage_is_now_synced.png "Stage is now synced") ## Related Articles * [Argo CD Integration](../../operator-guide/cd/argocd-integration.md) --- ## Uninstall KubeRocketCI # Uninstall KubeRocketCI This guide offers comprehensive steps for effectively removing the KubeRocketCI from your system. ## Deletion Procedure To uninstall platform, perform the following steps: 1. It is highly recommended to delete all the resources created via KubeRocketCI portal first, such as: - Applications - Libraries - Autotests - Infrastructures - CD Pipelines We advise removing them through the KubeRocketCI Portal interface. However, you can also eliminate all KubeRocketCI Portal resources by running the `kubectl delete` command. 2. Uninstall KubeRocketCI: - If you deployed KubeRocketCI via Helm, run the following command: ```bash helm uninstall edp-install -n ``` Replace `` with the namespace where KubeRocketCI was deployed. - If you deployed platform via Argo Application, delete the Argo Application: ```bash kubectl delete application -n argocd ``` ## Related Articles * [Install KubeRocketCI](install-kuberocketci.md) --- ## Migrate CI Pipelines From Jenkins to Tekton # Migrate CI Pipelines From Jenkins to Tekton To migrate the CI pipelines for a codebase from Jenkins to Tekton, follow the steps below: - Migrate CI Pipelines From Jenkins to Tekton - [Deploy a Custom EDP Scenario With Tekton and Jenkins CI Tools](#deploy-a-custom-edp-scenario-with-tekton-and-jenkins-ci-tools) - [Disable Jenkins Triggers](#disable-jenkins-triggers) - [Manage Tekton Triggers the Codebase(s)](#manage-tekton-triggers-the-codebases) - [Switch CI Tool for Codebase(s)](#switch-ci-tool-for-codebases) ## Deploy a Custom EDP Scenario With Tekton and Jenkins CI Tools Make sure that Tekton stack is deployed according to the [documentation](../../operator-guide/prerequisites.md). Enable Tekton as sub component during platform installation: ```yaml title="values.yaml" edp-tekton: enabled: true ``` ## Disable Jenkins Triggers To disable Jenkins Triggers for the codebase, add the following code to the provisioner: ```groovy title="job-provisioner" def tektonCodebaseList = [""] if (!tektonCodebaseList.contains(codebaseName.toString())){ triggers { gerrit { events { if (pipelineName.contains("Build")) changeMerged() else patchsetCreated() } project("plain:${codebaseName}", ["plain:${watchBranch}"]) } } } ``` :::note The sample above shows the usage of Gerrit VCS where the `` value is your codebase name. ::: - If using GitHub or GitLab, additionally remove the webhook from the relevant repository. - If webhooks generation for new codebase(s) is not required, correct the code above so that it creates a webhook in the job-provisioner. - To recreate the pipeline in Jenkins, trigger the job-provisioner. - Check that the new pipeline is created without triggering Gerrit events. ## Manage Tekton Triggers the Codebase(s) By default, each Gerrit project inherits configuration from the **All-Projects** repository. To exclude triggering in Jenkins and Tekton CI tools simultaneously, edit the configuration in the **All-Projects** repository or in the project which inherits rights from your project. Edit the **webhooks.config** file in the **refs/meta/config** and remove all context from this configuration. :::warning The clearance of the **webhooks.config** file will disable the pipeline trigger in Tekton. ::: To use Tekton pipelines, add the configuration to the corresponding Gerrit project (**webhooks.config** file in the **refs/meta/config**): ```yaml title="webhooks.config" [remote "changemerged"] url = http://el-gerrit-listener:8080 event = change-merged [remote "patchsetcreated"] url = http://el-gerrit-listener:8080 event = patchset-created [remote "commentadded"] url = http://el-gerrit-listener:8080 event = comment-added ``` Configure webhooks plugin in Gerrit configuration (typically located at **/var/gerrit/review_site/etc/gerrit.config**): ```yaml title="gerrit.config" [plugin "webhooks"] connectionTimeout = 3000 socketTimeout = 2500 maxTries = 1 retryInterval = 5000 threadPoolSize = 3 ``` ## Switch CI Tool for Codebase(s) Go to the codebase Custom Resource and change the `spec.ciTool` field from `jenkins` to `tekton`. ## Related Articles * [Install KubeRocketCI](../install-kuberocketci.md) * [Install Tekton](../install-tekton.md) --- ## Upgrade EDP v2.12 to 3.0 # Upgrade EDP v2.12 to 3.0 :::warning Please read carefully the following notes before upgrading EDP to 3.0: - Before starting the upgrade procedure, please make the necessary backups. - Kiosk integration is disabled by default. With EDP below v.3.0.x, define the `global.kioskEnabled` parameter in the [values.yaml](https://github.cm/epam/edp-install/blob/release/3.0/deploy-templates/values.yaml) file. - The `gerrit-ssh-port` parameter is moved from the `gerrit-operator.gerrit.sshport` to `global.gerritSSHPort` [values.yaml](https://github.com/epam/edp-install/blob/master/deploy-templates/values.yaml#L30) file. - In edp-gerrit-operator, the `gitServer.user` value is changed from the `jenkins` to `edp-ci`[values.yaml](https://github.com/epam/edp-gerrit-operator/blob/release/2.13/deploy-templates/values.yaml#L96) file. ::: This section provides the details on upgrading EDP to 3.0. Explore the actions and requirements below. 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/d9a4d15244c527ef6d1d029af27574282a281b98/deploy-templates/crds/v2.edp.epam.com_gerrits.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_cdstagedeployments.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_codebasebranches.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_codebaseimagestreams.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_codebases.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_gitservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_gittags.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_imagestreamtags.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_jiraissuemetadatas.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/release/2.14/deploy-templates/crds/v2.edp.epam.com_jiraservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakauthflows.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakclients.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakclientscopes.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealmcomponents.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealmgroups.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealmidentityproviders.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealmrolebatches.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealmroles.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealms.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloakrealmusers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/release/1.14/deploy-templates/crds/v1.edp.epam.com_keycloaks.yaml ``` 2. Set the required parameters. For more details, please refer to the [values.yaml](https://github.com/epam/edp-install/blob/release/3.0/deploy-templates/values.yaml) file.
View: values.yaml ```yaml edp-tekton: enabled: false admin-console-operator: enabled: true jenkins-operator: enabled: true ```
3. Add proper Helm annotations and labels as indicated below. This step is necessary starting from the release v.3.0.x as custom resources are managed by Helm and removed from the Keycloak Controller logic. ```bash kubectl label EDPComponent main-keycloak app.kubernetes.io/managed-by=Helm -n kubectl annotate EDPComponent main-keycloak meta.helm.sh/release-name= -n kubectl annotate EDPComponent main-keycloak meta.helm.sh/release-namespace= -n kubectl label KeycloakRealm main app.kubernetes.io/managed-by=Helm -n kubectl annotate KeycloakRealm main meta.helm.sh/release-name= -n kubectl annotate KeycloakRealm main meta.helm.sh/release-namespace= -n ``` 4. To upgrade EDP to 3.0, run the following command: ```bash helm upgrade edp epamedp/edp-install -n \ --values values.yaml --version=3.0.x ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the following command: `helm upgrade edp epamedp/edp-install -n --values values.yaml --version=3.0.x --dry-run` ::: 5. Update image versions for the Jenkins agents in the *ConfigMap*: ```bash kubectl edit configmap jenkins-slaves -n ``` - The versions of the images must be the following: ```bash epamedp/edp-jenkins-codenarc-agent:3.0.10 epamedp/edp-jenkins-dotnet-31-agent:3.0.9 epamedp/edp-jenkins-go-agent:3.0.17 epamedp/edp-jenkins-gradle-java11-agent:3.0.7 epamedp/edp-jenkins-gradle-java8-agent:3.0.10 epamedp/edp-jenkins-helm-agent:3.0.11 epamedp/edp-jenkins-kaniko-docker-agent:1.0.9 epamedp/edp-jenkins-maven-java11-agent:3.0.7 epamedp/edp-jenkins-maven-java8-agent:3.0.10 epamedp/edp-jenkins-npm-agent:3.0.9 epamedp/edp-jenkins-opa-agent:3.0.7 epamedp/edp-jenkins-python-38-agent:3.0.8 epamedp/edp-jenkins-sast-agent:0.1.5 epamedp/edp-jenkins-terraform-agent:3.0.9 ``` - Remove the `edp-jenkins-dotnet-21-agent` agent manifest. - Restart the Jenkins pod. 6. Attach the `id_rsa.pub` SSH public key from the `gerrit-ciuser-sshkey` secret to the `edp-ci` Gerrit user in the `gerrit` pod: ```bash ssh -p gerrit set-account --add-ssh-key ~/id_rsa.pub ``` :::note - For this operation, use the `gerrit-admin` SSH key from secrets. - `` is admin@localhost or any other user with permissions. ::: 7. Change the username from `jenkins` to `edp-ci` in the `gerrit-ciuser-sshkey` secret: ```bash kubectl -n patch secret gerrit-ciuser-sshkey\ --patch="{\"data\": { \"username\": \"$(echo -n edp-ci |base64 -w0)\" }}" -o yaml ``` :::warning In EDP v.3.0.x, [Admin Console](../../user-guide/index.md) is deprecated, and EDP interface is available only via [EDP Portal](../../user-guide/index.md). ::: ## Related Articles * [Migrate CI Pipelines From Jenkins to Tekton](migrate-ci-pipelines-from-jenkins-to-tekton.md) --- ## Upgrade EDP v3.0 to 3.1 # Upgrade EDP v3.0 to 3.1 :::warning We suggest making a backup of the EDP environment before starting the upgrade procedure. ::: This section provides the details on the EDP upgrade to v3.1. Explore the actions and requirements below. 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.13.2/deploy-templates/crds/v2.edp.epam.com_jenkins.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.13.4/deploy-templates/crds/v2.edp.epam.com_gerrits.yaml ``` 2. To upgrade EDP to the v3.1, run the following command: ```bash helm upgrade edp epamedp/edp-install -n --values values.yaml --version=3.1.0 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the following command: ::: ```bash helm upgrade edp epamedp/edp-install -n --values values.yaml --version=3.1.0 --dry-run ``` --- ## Upgrade KubeRocketCI v3.9 to 3.10 # Upgrade KubeRocketCI v3.9 to 3.10 This section provides detailed instructions for upgrading KubeRocketCI to version 3.10. Follow the steps and requirements outlined below: :::important We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: :::important Starting from version 3.10, the Tekton Dashboard is no longer installed with KubeRocketCI. Instead, it is now installed via a Helm chart from the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. For more information, refer to the [Tekton Dashboard](#tekton-dashboard) section. ::: 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_cdstagedeployments.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_codebasebranches.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_codebaseimagestreams.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_codebases.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_gitservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_jiraissuemetadatas.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.24.0/deploy-templates/crds/v2.edp.epam.com_jiraservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-cd-pipeline-operator/v2.21.0/deploy-templates/crds/v2.edp.epam.com_cdpipelines.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-cd-pipeline-operator/v2.21.0/deploy-templates/crds/v2.edp.epam.com_stages.yaml ``` 2. (Optional) If you are using the **Gerrit Operator**, update the necessary CRDs for Gerrit. Run the following command to apply the necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerritgroupmembers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerritgroups.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerritmergerequests.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerritprojectaccesses.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerritprojects.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerritreplicationconfigs.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.22.0/deploy-templates/crds/v2.edp.epam.com_gerrits.yaml ``` 3. Update the Quick Links configuration section: In version 3.10, the section for configuring Quick Links resources has been updated. Ensure that the `quickLinks` section is correctly defined in the `values.yaml` file. Below is an example of the updated configuration: ```yaml title="values.yaml" quickLinks: # Base URL value for the argocd Quick Link argocd: "" # Base URL value for the defectdojo Quick Link defectdojo: "" # Base URL value for the dependency track Quick Link dependency_track: "" # Base URL value for the container registry Quick Link (e.g. Nexus, DockerHub, ECR, etc.) docker_registry: "" # Base URL value for the Grafana Quick Link grafana: "" # Base URL value for the Kibana Quick Link kibana: "" # Base URL value for the nexus Quick Link nexus: "" # Base URL value for the sonar Quick Link sonar: "" # Base URL value for the codemie Quick Link codemie: "" ``` ```yaml title="values.yaml" quickLinks: # Base URL value for the argocd Quick Link argocd: "" # Base URL value for the codemie Quick Link codemie: "" # Base URL value for the defectdojo Quick Link defectdojo: "" # Base URL value for the dependency track Quick Link dependency_track: "" # Base URL value for the container registry Quick Link (e.g. Nexus, DockerHub, ECR, etc.) docker_registry: "" # Configuration for the logging Quick Link logging: # -- Define the provider name for correct URL generation. # Available providers: "opensearch", "datadog". # If the provider name is not specified, the base URL will be used. provider: "" # Base URL value for the logging Quick Link url: "" # Configuration for the monitoring Quick Link monitoring: # -- Define the provider name for correct URL generation. # Available providers: "grafana", "datadog". # If the provider name is not specified, the base URL will be used. provider: "" # Base URL value for the monitoring Quick Link url: "" # Base URL value for the nexus Quick Link nexus: "" # Base URL value for the sonar Quick Link sonar: "" ``` Also, ensure that the `quickLink.enabled` field is set for each Git Server provider specified in the `edp-tekton` section. Below is an example of the updated configuration: ```yaml title="values.yaml" edp-tekton: enabled: true ... gitServers: github: # -- Enable creation of QuickLink for GitHub quickLink: enabled: true ``` 4. (Optional) Update Tekton Pipeline labels: In version 3.10, Tekton pipelines require specific labels to indicate their type: **build**, **review**, **clean**, or **deploy**. These labels allow KubeRocketCI portal to correctly display pipeline data. Below are examples of the required labels for each pipeline type: - **Build Pipeline** ```yaml metadata: name: build labels: app.edp.epam.com/pipelinetype: build ``` - **Review Pipeline** ```yaml metadata: name: review labels: app.edp.epam.com/pipelinetype: review ``` - **Clean Pipeline** ```yaml metadata: name: clean labels: app.edp.epam.com/pipelinetype: clean ``` - **Deploy Pipeline** ```yaml metadata: name: deploy labels: app.edp.epam.com/pipelinetype: deploy ``` :::note For any custom Tekton pipelines created in earlier versions, ensure that each pipeline has the appropriate label to maintain compatibility. ::: To label an existing custom pipeline, you can use the following command as a reference: ```bash kubectl label pipeline app.edp.epam.com/pipelinetype= -n ``` 5. (Optional) Migrate Git providers credentials to a new AWS Parameter Store object: :::note In version 3.9.0, Git providers credentials were included in the `manageEDPInstallSecretsName` object in AWS Parameter Store. Due to character limit restrictions, these credentials should now be migrated to a dedicated AWS Parameter Store object named `manageGitProviderSecretsName`. ::: If you are using the **External Secret Operator** to manage secrets, specify a new field in the `values.yaml` file: `manageGitProviderSecretsName`. This field should be defined as follows: ```yaml title="values.yaml" externalSecrets: enabled: true ... manageEDPInstallSecretsName: /infra/core/addons/kuberocketci ``` ```yaml title="values.yaml" externalSecrets: enabled: true ... manageEDPInstallSecretsName: /infra/core/addons/kuberocketci manageGitProviderSecretsName: /infra/core/addons/kuberocketci-git-provider-secrets ``` Below is an example of the object structure for `manageGitProviderSecretsName`: ```json { "github-config": { "id_rsa": "LS0tLS1...", "token": "ghp_fdO1joO...", "secretString": "kfr1ndk4..." }, "gitlab-config": { "id_rsa": "LS0tLS1...", "token": "RW1UG2EAdEjPEC1NSoVm3A", "secretString": "kfr1ndk4..." }, "bitbucket-config": { "id_rsa": "LS0tLS1...", "token": "YXV0b19lcG1kLWVkc...", "secretString": "kfr1ndk4..." } } ``` 6. (Optional) Enable Code Quality Widgets: :::note Starting from version 3.10, KrakenD is used to manage Code Quality widgets in KubeRocketCI portal. This migration simplifies the integration with third-party APIs, improves performance, and provides a scalable, flexible API gateway. ::: By default, in version 3.10, Code Quality widgets are disabled. To enable them, follow these steps: 1. **Install KrakenD**: For detailed installation instructions, refer to the [KrakenD documentation](https://www.krakend.io/docs/). 2. **Configure the `values.yaml` file**: In the **edp-install** chart, set the `apiGatewayUrl` key to specify the API URL of KrakenD as configured during its installation. ```yaml title="values.yaml" global: apiGatewayUrl: "https://api.example.com" ``` 7. (Optional) Migrate SSO (OAuth2-proxy) configuration: :::danger Starting from version 3.10, the platform no longer supports setting Single Sign-on (SSO) configuration through the `edp-install` chart. This includes the installation of the `oauth2-proxy` component and the creation of required Keycloak resources. ::: Deprecated fields in `values.yaml` file for SSO configuration are listed below:
View: Deprecated fields in values.yaml ```yaml sso: # -- Install OAuth2-proxy and Keycloak CRs as a part of EDP deployment. enabled: false keycloakOperatorResources: # Set to false if using the add-ons approach (refer to https://github.com/epam/edp-cluster-add-ons) # for EDP installation and if the extension-oidc is already installed. # This prevents the creation of an additional Keycloak resource and secret. # The 'kind' and 'name' must be specified in case of using an existing Keycloak/ClusterKeycloak resource. # Create kind: Keycloak as a part of chart installation createKeycloakCR: true # Can be Keycloak or ClusterKeycloak. kind: Keycloak # Name of kind: Keycloak/ClusterKeycloak CR. name: main # -- Defines the Keycloak realm name, which by default is named after the namespace where EDP is deployed. # realmName: edp # -- Defines Keycloak sso realm name that is used as the Identity Provider (IdP) realm ssoRealmName: "broker" # -- Defines Keycloak client name that is used for the Identity Provider (IdP) client ssoClientName: "edp" # -- Keycloak URL. keycloakUrl: https://keycloak.example.com/auth # -- Administrators of your tenant. admins: - "stub_user_one@example.com" # -- Developers of your tenant developers: - "stub_user_one@example.com" - "stub_user_two@example.com" # Oauth2-proxy image image: # -- OAuth2-proxy image repository repository: quay.io/oauth2-proxy/oauth2-proxy # -- OAuth2-proxy image tag tag: v7.4.0 # Create a cookie-secret with the following command: # 'openssl rand -base64 32 | head -c 32 | base64' # Use an existing secret for OAuth2 cookie-secret existingSecret: # -- Secret name which stores cookie-secret secretName: oauth2-proxy-cookie-secret # -- Secret key which stores cookie-secret secretKey: cookie-secret # -- Additional container environment variables extraEnv: [] # -- Extra arguments to provide to the OAuth2-proxy extraArgs: {} # -- Additional volumes to be added to the OAuth2-proxy pod extraVolumes: [] # - name: custom-ca # secret: # defaultMode: 420 # secretName: custom-ca # -- Additional volumeMounts to be added to the OAuth2-proxy container extraVolumeMounts: [] # - name: custom-ca # mountPath: /etc/ssl/certs/CA.crt # readOnly: true # subPath: CA.crt ingress: # -- Enable ingress controller resource enabled: true # -- Additional ingress annotations annotations: {} # -- Ingress path type. One of `Exact`, `Prefix` or `ImplementationSpecific` pathType: Prefix # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName # Ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress # -- Defines which ingress controller will implement the resource, e.g. nginx ingressClassName: "" # -- Ingress TLS configuration tls: [] # - secretName: chart-example-tls # hosts: # - chart-example.local # -- Node labels for pod assignment nodeSelector: {} # -- Toleration labels for pod assignment tolerations: [] # -- Affinity settings for pod assignment affinity: {} ```
If you previously used SSO configuration section, follow these migration steps: 1. Manually remove all Keycloak resources previously created by the **edp-install** chart. The list of resources to remove can be found [here](https://github.com/epam/edp-install/tree/v3.9.0/deploy-templates/templates/keycloak). Remove the resources sequentially, following the specified deletion order. Sample commands for deleting Keycloak resources are provided below: ```bash kubectl delete keycloakclient gerrit -n kubectl delete keycloakclient oauth2-proxy -n kubectl delete keycloakclientscope edp -n kubectl delete keycloakrealmrole default-roles-administrator -n kubectl delete keycloakrealmrole default-roles-developer -n kubectl delete keycloakrealmrolebatch default-roles -n kubectl delete keycloakauthflow keycloakauthflow-browser -n kubectl delete keycloakrealmidentityprovider broker-idp -n kubectl delete keycloakrealm main -n kubectl delete keycloak keycloak -n ``` 2. Review and configure the [kuberocketci-rbac](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) chart in the add-ons repository. Set the required values in the `values.yaml` file and install the chart. Verify that all necessary [Keycloak resources](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac/templates) have been successfully created in Keycloak. 3. Review and configure the [oauth2-proxy](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/oauth2-proxy) chart in the add-ons repository. Set the required values in the `values.yaml` file and install the chart. Ensure that **oauth2-proxy** is running correctly. 4. Once the **kuberocketci-rbac** and **oauth2-proxy** charts are installed and verified, proceed with upgrading the platform to version 3.10. 8. (Optional) Update the OIDC Integration section for **edp-headlamp** chart: In version 3.10, the OpenID Connect (OIDC) section for the **edp-headlamp** chart has been modified. Certain fields have been renamed or removed. For example, the `keycloakUrl` field has been renamed to `issuerUrl`, and the `issuerRealm` field is no longer required. Refer to the example values configurations below for versions 3.9 and 3.10 to identify the fields that need to be modified for compatibility: ```yaml title="values.yaml" edp-headlamp: enabled: true ... config: oidc: enabled: true keycloakUrl: "https://keycloak.example.com" clientID: "" clientSecretName: "keycloak-client-headlamp-secret" clientSecretKey: "clientSecret" issuerRealm: "" scopes: "" ``` ```yaml title="values.yaml" edp-headlamp: enabled: true ... config: oidc: enabled: true issuerUrl: "https://keycloak.example.com/auth/realms/" clientID: "" clientSecretName: "keycloak-client-headlamp-secret" clientSecretKey: "clientSecret" scopes: "" ``` 9. (Optional) Update Gerrit configuration: Update the following configuration for the **gerrit-operator** section in `values.yaml`: 1. Update the `extraEnv` variable. Modify the `OAUTH_KEYCLOAK_CLIENT_ID` value to follow the `gerrit-` format: ```yaml title="values.yaml" gerrit-operator: enabled: true ... gerrit: extraEnv: - name: OAUTH_KEYCLOAK_CLIENT_ID value: gerrit- ``` 2. Specify the Single Sign-on (SSO) configuration. Ensure the SSO configuration is defined as follows: ```yaml title="values.yaml" sso: enabled: true keycloakUrl: https://keycloak.example.com kind: ClusterKeycloakRealm name: main ``` 10. To upgrade KubeRocketCI to the v3.10, run the following command: ```bash helm upgrade krci epamedp/edp-install -n krci --values values.yaml --version=3.10.5 ``` :::note To verify the installation, you can test the deployment before applying it to the cluster with the `--dry-run` key: `helm upgrade krci epamedp/edp-install -n krci --values values.yaml --version=3.10.5 --dry-run` ::: ## Tekton Dashboard In version 3.10, the Tekton Dashboard is migrated from the [edp-tekton](https://github.com/epam/edp-tekton) repository to a separate Helm chart in the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. There are two approaches to install the Tekton Dashboard using the add-ons repository: :::note For more information about deploying applications using [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository, refer to the [Install via Add-Ons](../add-ons-overview.md) page. ::: ### Approach 1: Deploy Using Argo CD The first approach implies installing the Tekton Dashboard via resource synchronization in Argo CD: 1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/tekton-dashboard) repository. 2. Navigate to the `clusters/core/addons/tekton-dashboard` directory and configure the `values.yaml` file with the necessary values for the Tekton Dashboard installation. 3. After configuring the Tekton Dashboard Helm chart values, navigate to the `clusters/core/apps` directory. In the `values.yaml` file, update the `tekton-dashboard` section by specifying the `enable` field as `true` to enable Argo CD Application creation for the Tekton Dashboard. Also, specify the `namespace` field to define the target namespace where the Tekton Dashboard will be deployed. ```yaml title="clusters/core/apps/values.yaml" tekton-dashboard: enable: true namespace: krci ``` 4. Commit and push the changes to the remote repository. After the changes are pushed, navigate to the Argo CD and sync the Tekton Dashboard application. Verify that the Tekton Dashboard is successfully deployed. ### Approach 2: Deploy Using Helm The second approach deploys Tekton Dashboard as a common Helm chart: 1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/tekton-dashboard) repository. 2. Navigate to the `clusters/core/addons/tekton-dashboard` directory and configure the `values.yaml` file with the necessary values for the Tekton Dashboard installation. 3. After configuring the Tekton Dashboard Helm chart values, run the following command to deploy the Tekton Dashboard: ```bash helm upgrade --install tekton-dashboard . -n ``` Replace `` with the target namespace where the Tekton Dashboard will be deployed. 4. Verify that the Tekton Dashboard is successfully deployed. --- ## Upgrade KubeRocketCI v3.10 to 3.11 # Upgrade KubeRocketCI v3.10 to 3.11 This section provides detailed instructions for upgrading KubeRocketCI to version 3.11. Follow the steps and requirements outlined below: :::important We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: 1. Configure API Cluster Endpoint: Starting from version 3.11, the KubeRocketCI portal UI supports generating a kubeconfig file that can be used to access the cluster and manage Kubernetes resources. To enable this feature, it is necessary to set the `apiClusterEndpoint` field in the `values.yaml` file. This field should contain the API endpoint of the cluster where KubeRocketCI is installed. For example: ```yaml title="values.yaml" global: apiClusterEndpoint: "https:///EXAMPLED539D4633E53DE1B71EXAMPLE.gr7.eu-central-1.eks.amazonaws.com" ``` After upgrading to version 3.11, kubeconfig file generation will be available in the KubeRocketCI portal UI under the **Account Settings** section. ![Generate kubeconfig file](../../assets/operator-guide/upgrade/generate-kubeconfig.png) 2. Add Trigger Template Label to Tekton Custom Pipelines: :::note Pipelines without the `app.edp.epam.com/triggertemplate` label will not be available for manual triggering in the KubeRocketCI portal UI. ::: Starting from version 3.11, the KubeRocketCI portal supports manually triggering pipelines. To enable this feature for existing custom Tekton pipelines, it is necessary to specify the label `app.edp.epam.com/triggertemplate` with the name of the Tekton trigger template that will be used to trigger the pipeline. For example: ```yaml title="custom-pipeline.yaml" apiVersion: tekton.dev/v1 kind: Pipeline metadata: name: custom-pipeline labels: app.edp.epam.com/triggertemplate: github-build-template ``` After upgrading to version 3.11, the custom pipeline can be triggered manually with provided parameters from the **Pipelines** section in the KubeRocketCI portal UI. ![Trigger custom pipeline](../../assets/operator-guide/upgrade/trigger-custom-pipeline.png) 3. Update Codebase Custom Resources: :::note It is highly recommended to update the Codebase resources manually to avoid unexpected behavior when using a bash script. ::: In version 3.11, the `edp` versioning type was renamed to `semver`. Due to this change, it is necessary to update the existing Codebase resources with the new versioning type name. For example: ```yaml title="CodebaseBranch" apiVersion: v2.edp.epam.com/v1 kind: Codebase metadata: name: java-app namespace: krci spec: ... versioning: startFrom: 0.1.0-SNAPSHOT type: edp ``` ```yaml title="CodebaseBranch" apiVersion: v2.edp.epam.com/v1 kind: Codebase metadata: name: java-app namespace: krci spec: ... versioning: startFrom: 0.1.0-SNAPSHOT type: semver ``` To automate the update of the Codebase resources, it is possible to use the following bash script: ```bash title="patch-codebase.sh" #!/bin/bash codebases=$(kubectl get codebase -n krci -o jsonpath='{.items[*].metadata.name}') for codebase in $codebases; do versioning_type=$(kubectl get codebase $codebase -n krci -o jsonpath='{.spec.versioning.type}') if [[ $versioning_type == "edp" ]]; then kubectl patch codebase $codebase -n krci --type=json -p="[{\"op\": \"replace\", \"path\": \"/spec/versioning/type\", \"value\": \"semver\"}]" echo "Updated $codebase: $versioning_type -> semver" fi done ``` 4. Update CodebaseBranch Custom Resources: :::note It is highly recommended to update the CodebaseBranch resources manually to avoid unexpected behavior when using a bash script. ::: In version 3.11, Tekton build pipelines were renamed to align with the `semver` versioning type (previously `edp`). Due to this change, it is necessary to update the existing CodebaseBranch resources with the new Tekton build pipelines names. For example: ```yaml title="CodebaseBranch" apiVersion: v2.edp.epam.com/v1 kind: CodebaseBranch metadata: name: java-app-main namespace: krci spec: branchName: main codebaseName: java-app pipelines: build: gerrit-maven-java21-app-build-edp review: gerrit-maven-java21-app-review release: false version: 0.1.0-SNAPSHOT ``` ```yaml title="CodebaseBranch" apiVersion: v2.edp.epam.com/v1 kind: CodebaseBranch metadata: name: java-app-main namespace: krci spec: branchName: main codebaseName: java-app pipelines: build: gerrit-maven-java21-app-build-semver review: gerrit-maven-java21-app-review release: false version: 0.1.0-SNAPSHOT ``` To automate the update of the CodebaseBranch resources, it is possible to use the following bash script: ```bash title="patch-codebasebranch.sh" #!/bin/bash codebasebranches=$(kubectl get codebasebranch -n krci -o jsonpath='{.items[*].metadata.name}') for branch in $codebasebranches; do build_pipeline=$(kubectl get codebasebranch $branch -n krci -o jsonpath='{.spec.pipelines.build}') if [[ $build_pipeline == *-edp ]]; then new_build_pipeline="${build_pipeline%-edp}-semver" kubectl patch codebasebranch $branch -n krci --type=json -p="[{\"op\": \"replace\", \"path\": \"/spec/pipelines/build\", \"value\": \"$new_build_pipeline\"}]" echo "Updated $branch: $build_pipeline -> $new_build_pipeline" fi done ``` 5. Update ApplicationSet Custom Resources: :::note It is highly recommended to update the ApplicationSet resources manually to avoid unexpected behavior when using a bash script. ::: In version 3.11, the `edp` versioning type was renamed to `semver`. Due to this change, it is necessary to update the existing ApplicationSet resources with the new versioning type name. 1. Replace `edp` with `semver` in the `versionType` field within the `spec.generators.list.elements` section. For example: ```yaml title="ApplicationSet" apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: demo namespace: krci spec: generators: - list: elements: - cluster: in-cluster codebase: java-app ... versionType: edp ``` ```yaml title="ApplicationSet" apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: demo namespace: krci spec: generators: - list: elements: - cluster: in-cluster codebase: java-app ... versionType: semver ``` 2. Update the conditional expression in the `targetRevision` fields of the `spec.template.spec.source` and `spec.templatePatch.spec.sources` sections. For example: ```yaml title="ApplicationSet" apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: demo namespace: krci spec: template: spec: source: targetRevision: '{{ if eq .versionType "edp" }}build/{{ .imageTag }}{{ else }}{{ .imageTag }}{{ end }}' ... templatePatch: |2- {{- if .customValues }} spec: sources: - helm: parameters: - name: image.tag value: '{{ .imageTag }}' - name: image.repository value: {{ .imageRepository }} releaseName: '{{ .codebase }}' valueFiles: - $values/platform/{{ .stage }}/{{ .codebase }}-values.yaml targetRevision: '{{ if eq .versionType "edp" }}build/{{ .imageTag }}{{ else }}{{ .imageTag }}{{ end }}' ``` ```yaml title="ApplicationSet" apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: demo namespace: krci spec: template: spec: source: targetRevision: '{{ if eq .versionType "semver" }}build/{{ .imageTag }}{{ else }}{{ .imageTag }}{{ end }}' ... templatePatch: |2- {{- if .customValues }} spec: sources: - helm: parameters: - name: image.tag value: '{{ .imageTag }}' - name: image.repository value: {{ .imageRepository }} releaseName: '{{ .codebase }}' valueFiles: - $values/platform/{{ .stage }}/{{ .codebase }}-values.yaml targetRevision: '{{ if eq .versionType "semver" }}build/{{ .imageTag }}{{ else }}{{ .imageTag }}{{ end }}' ``` To automate the update of the ApplicationSet resources, it is possible to use the following bash script: ```bash title="patch-applicationset.sh" #!/bin/bash applicationsets=$(kubectl get applicationset -n krci -o jsonpath='{.items[*].metadata.name}') for applicationset_name in $applicationsets; do echo "Patching ApplicationSet: $applicationset_name" # Export the current ApplicationSet resource to a temporary file kubectl get applicationset "$applicationset_name" -n krci -o yaml > tmp-applicationset.yaml # Replace 'versionType: edp' with 'versionType: semver' sed -i 's/versionType: edp/versionType: semver/g' tmp-applicationset.yaml # Replace conditionals in targetRevision sed -i 's/{{ if eq .versionType "edp" }}/{{ if eq .versionType "semver" }}/g' tmp-applicationset.yaml # Apply the modified ApplicationSet kubectl apply -f tmp-applicationset.yaml echo "Patched $applicationset_name" done # Cleanup rm -f tmp-applicationset.yaml ``` ```bash title="patch-applicationset.sh" #!/bin/bash applicationsets=$(kubectl get applicationset -n krci -o jsonpath='{.items[*].metadata.name}') for applicationset_name in $applicationsets; do echo "Patching ApplicationSet: $applicationset_name" # Export the current ApplicationSet resource to a temporary file kubectl get applicationset "$applicationset_name" -n krci -o yaml > tmp-applicationset.yaml # Replace 'versionType: edp' with 'versionType: semver' sed -i '' 's/versionType: edp/versionType: semver/g' tmp-applicationset.yaml # Replace conditionals in targetRevision sed -i '' 's/{{ if eq .versionType "edp" }}/{{ if eq .versionType "semver" }}/g' tmp-applicationset.yaml # Apply the modified ApplicationSet kubectl apply -f tmp-applicationset.yaml echo "Patched $applicationset_name" done # Cleanup rm -f tmp-applicationset.yaml ``` 6. Replace `edp-config` ConfigMap for Tekton Custom Tasks: Starting from version 3.11, the `edp-config` ConfigMap was renamed to `krci-config`. For Tekton custom tasks, that are using the `edp-config` ConfigMap, it is necessary to replace it with the `krci-config` in the task resource. For example: ```yaml title="custom-task.yaml" apiVersion: tekton.dev/v1 kind: Task metadata: name: custom-task spec: steps: - name: custom-step env: - name: PLATFORM valueFrom: configMapKeyRef: name: edp-config key: platform ``` ```yaml title="custom-task.yaml" apiVersion: tekton.dev/v1 kind: Task metadata: name: custom-task spec: steps: - name: custom-step env: - name: PLATFORM valueFrom: configMapKeyRef: name: krci-config key: platform ``` 7. Update `pipelineUrl` parameter for Tekton Custom Pipelines: :::note The `pipelineUrl` parameter is used to update the Pull Request status with the corresponding pipeline URL, which is generated when the review or build pipeline is triggered automatically. ::: In version 3.11, KubeRocketCI portal has changed the URL format for Tekton pipelines. Due to this change, it is necessary to update the `pipelineUrl` parameter for the existing custom Tekton pipelines. For example: ```yaml title="custom-pipeline.yaml" apiVersion: tekton.dev/v1 kind: Pipeline metadata: name: custom-pipeline spec: params: - default: https://portal-{{ $.Release.Namespace }}.{{ $.Values.dnsWildCard }}/c/main/pipelines/$(context.pipelineRun.namespace)/$(context.pipelineRun.name) name: pipelineUrl type: string ``` ```yaml title="custom-pipeline.yaml" apiVersion: tekton.dev/v1 kind: Pipeline metadata: name: custom-pipeline spec: params: - default: https://portal-{{ $.Release.Namespace }}.{{ $.Values.dnsWildCard }}/c/main/pipelines/pipelineruns/$(context.pipelineRun.namespace)/$(context.pipelineRun.name) name: pipelineUrl type: string ``` 8. (Optional) Add yamllint config to the GitOps repository: :::note By default, yamllint uses the default configuration to lint YAML files in the GitOps repository. For more details on available yamllint rules, refer to the yamllint [documentation](https://yamllint.readthedocs.io/en/stable/rules.html). ::: In version 3.11, yamllint task, used in the GitOps review and build pipelines, supports custom configuration which can be added to the GitOps repository. To configure yamllint custom rules, it is necessary to create a `.yamllint` config file in the root directory of the GitOps repository. For example: ```yaml title=".yamllint" # yamllint configuration file # Extends the default configuration: # https://yamllint.readthedocs.io/en/stable/configuration.html#default-configuration extends: default ignore: - '.yamllint' rules: line-length: disable # Disable line-length rule document-start: disable # Disable rule for requiring '---' at the document start ``` After upgrading to version 3.11, the custom yamllint configuration will be used in the GitOps review and build pipelines. ![Yamllint rules](../../assets/user-guide/yamllint-rules.png "Yamllint rules") 9. Security Tekton Task migration: Starting from version 3.11, the `security` Tekton task, previously used in build pipelines, has been migrated to a separate `security-scan` pipeline. After the upgrade process, a new `security-scan` pipeline will be created for each available Git server. To trigger the `security-scan` pipeline for the appropriate component, follow the steps below: 1. Open the KubeRocketCI portal and navigate to the **Pipelines** section. 2. In the **Pipelines** tab, select the appropriate `security-scan` pipeline depending on the Git server where the component is hosted. Click the three dots icon in the **Actions** column and select the **Run with params** option. ![Security scan pipeline](../../assets/operator-guide/upgrade/security-scan-pipeline.png "Security scan pipeline") 3. In the **Editor** tab, specify the following parameters and click **Save & Apply** button to trigger the pipeline: - `git-source-url` - URL of the Git repository where the component is hosted. - `git-source-revision` - Git revision (branch) to be scanned. - `CODEBASE_NAME` - Name of the component to be scanned. ![Security scan parameters](../../assets/operator-guide/upgrade/security-scan-params.png "Security scan parameters") 4. After the pipeline is triggered, navigate to the **Pipelines** section and locate the `security-scan` pipeline run. The pipeline will contain a single main task named `security`, which will perform the security scan for the specified component. ![Security scan pipeline run](../../assets/operator-guide/upgrade/security-scan-pipeline-run.png "Security scan pipeline run") 10. To upgrade KubeRocketCI to the v3.11, run the following command: :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` key: `helm upgrade krci epamedp/edp-install -n krci --values values.yaml --version=3.11.3 --dry-run` ::: ```bash helm upgrade krci epamedp/edp-install -n krci --values values.yaml --version=3.11.3 ``` --- ## Upgrade EDP v3.1 to 3.2 # Upgrade EDP v3.1 to 3.2 :::warning We suggest making a backup of the EDP environment before starting the upgrade procedure. ::: This section provides the details on the EDP upgrade to v3.2.2. Explore the actions and requirements below. 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_cdstagedeployments.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_codebasebranches.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_codebaseimagestreams.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_codebases.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_gitservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_gittags.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_imagestreamtags.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_jiraissuemetadatas.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_jiraservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_cdstagejenkinsdeployments.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkins.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsagents.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsauthorizationrolemappings.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsauthorizationroles.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsfolders.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsjobbuildruns.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsjobs.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsscripts.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinsserviceaccounts.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_jenkinssharedlibraries.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-component-operator/v0.13.0/deploy-templates/crds/v1.edp.epam.com_edpcomponents.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-cd-pipeline-operator/v2.14.1/deploy-templates/crds/v2.edp.epam.com_cdpipelines.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-cd-pipeline-operator/v2.14.1/deploy-templates/crds/v2.edp.epam.com_stages.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-nexus-operator/v2.14.1/deploy-templates/crds/v2.edp.epam.com_nexuses.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-nexus-operator/v2.14.1/deploy-templates/crds/v2.edp.epam.com_nexususers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-sonar-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_sonargroups.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-sonar-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_sonarpermissiontemplates.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-sonar-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_sonars.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerritgroupmembers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerritgroups.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerritmergerequests.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerritprojectaccesses.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerritprojects.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerritreplicationconfigs.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.14.0/deploy-templates/crds/v2.edp.epam.com_gerrits.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-perf-operator/v2.13.0/deploy-templates/crds/v2.edp.epam.com_perfdatasourcegitlabs.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-perf-operator/v2.13.0/deploy-templates/crds/v2.edp.epam.com_perfdatasourcejenkinses.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-perf-operator/v2.13.0/deploy-templates/crds/v2.edp.epam.com_perfdatasourcesonars.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-perf-operator/v2.13.0/deploy-templates/crds/v2.edp.epam.com_perfservers.yaml ``` 2. Generate a cookie-secret for proxy with the following command: ```bash nexus_proxy_cookie_secret=$(openssl rand -base64 32 | head -c 32) ``` Create `nexus-proxy-cookie-secret` in the `` namespace: ```bash kubectl -n create secret generic nexus-proxy-cookie-secret \ --from-literal=cookie-secret=${nexus_proxy_cookie_secret} ``` 3. EDP 3.2.2 features OIDC configuration for EDP Portal. If this parameter is required, create `keycloak-client-headlamp-secret` as described in this [article](../auth/ui-portal-oidc.md): ```bash kubectl -n create secret generic keycloak-client-edp-portal-secret \ --from-literal=clientSecret= ``` 4. Delete the following resources: ```bash kubectl -n delete KeycloakClient nexus kubectl -n delete EDPComponent nexus kubectl -n delete Ingress nexus kubectl -n delete deployment edp-tekton-dashboard ``` 5. EDP release 3.2.2 uses the default cluster storageClass and we must check previous storageClass parameters. Align , if required, the `storageClassName` in EDP `values.yaml` to the same that were used by EDP PVC. For example: ```yaml edp-tekton: buildTool: go: cache: persistentVolume: # -- Specifies storageClass type. If not specified, a default storageClass for go-cache volume is used storageClass: ebs-sc jenkins-operator: enabled: true jenkins: storage: # -- Storageclass for Jenkins data volume class: gp2 sonar-operator: sonar: storage: data: # -- Storageclass for Sonar data volume class: gp2 database: # -- Storageclass for database data volume class: gp2 gerrit-operator: gerrit: storage: # -- Storageclass for Gerrit data volume class: gp2 nexus-operator: nexus: storage: # -- Storageclass for Nexus data volume class: gp2 ``` 6. To upgrade EDP to the v3.2.2, run the following command: ```bash helm upgrade edp epamedp/edp-install -n --values values.yaml --version=3.2.2 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the following command: `helm upgrade edp epamedp/edp-install -n --values values.yaml --version=3.2.2 --dry-run` ::: --- ## Upgrade EDP v3.2 to 3.3 # Upgrade EDP v3.2 to 3.3 :::warning We suggest making a backup of the EDP environment before starting the upgrade procedure. ::: :::note We currently disabled cache volumes for go and npm in the EDP 3.3 release. ::: This section provides the details on the EDP upgrade to v3.3.0. Explore the actions and requirements below. 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.16.0/deploy-templates/crds/v2.edp.epam.com_codebases.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-jenkins-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_jenkins.yaml ``` 2. If you use Gerrit VCS, delete the corresponding resource due to changes in annotations: ```bash kubectl -n edp delete EDPComponent gerrit ``` The deployment will create a new EDPComponent called `gerrit` instead. 3. To upgrade EDP to the v3.3.0, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.3.0 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.3.0 --dry-run` ::: 4. In EDP v3.3.0, a new feature was introduced allowing manual pipeline re-triggering by sending a comment with `/recheck`. To enable the re-trigger feature for applications that were added before the upgrade, please proceed with the following: 4.1 For Gerrit VCS, add the following event to the `webhooks.config` configuration file in the `All-Projects` repository: ```bash [remote "commentadded"] url = http://el-gerrit-listener:8080 event = comment-added ``` 4.2 For GitHub VCS, check the `Issue comments` permission for each webhook in every application added before the EDP upgrade to 3.3.0. 4.3 For GitLab VCS, check the `Comments` permission for each webhook in every application added before the EDP upgrade to 3.3.0. --- ## Upgrade EDP v3.3 to 3.4 # Upgrade EDP v3.3 to 3.4 :::warning We suggest making a backup of the EDP environment before starting the upgrade procedure. ::: :::note Pay attention that the following components: `perf-operator`, `edp-admin-console`, `edp-admin-console-operator`, and `edp-jenkins-operator` are deprecated and should be additionally migrated in order to avoid their deletion. For migration details, please refer to the [Migrate CI Pipelines From Jenkins to Tekton](migrate-ci-pipelines-from-jenkins-to-tekton.md) instruction. ::: This section provides the details on the EDP upgrade to v3.4.1. Explore the actions and requirements below. 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-cd-pipeline-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_cdpipelines.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-cd-pipeline-operator/v2.15.0/deploy-templates/crds/v2.edp.epam.com_stages.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_clusterkeycloakrealms.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_clusterkeycloaks.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakauthflows.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakclients.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakclientscopes.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealmcomponents.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealmgroups.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealmidentityproviders.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealmrolebatches.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealmroles.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealms.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloakrealmusers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-keycloak-operator/v1.17.0/deploy-templates/crds/v1.edp.epam.com_keycloaks.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.17.0/deploy-templates/crds/v2.edp.epam.com_templates.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.17.0/deploy-templates/crds/v2.edp.epam.com_codebasebranches.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.17.0/deploy-templates/crds/v2.edp.epam.com_codebaseimagestreams.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.17.0/deploy-templates/crds/v2.edp.epam.com_codebases.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.17.0/deploy-templates/crds/v2.edp.epam.com_gitservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.17.0/deploy-templates/crds/v2.edp.epam.com_jiraservers.yaml kubectl apply -f https://raw.githubusercontent.com/epam/edp-gerrit-operator/v2.16.0/deploy-templates/crds/v2.edp.epam.com_gerrits.yaml ``` 2. Remove deprecated components:
View: values.yaml ```yaml perf-operator: enabled: false admin-console-operator: enabled: false jenkins-operator: enabled: false ```
3. Since the [values.yaml](https://github.com/epam/edp-install/blob/master/deploy-templates/values.yaml#L37) file structure has been modified, move the **dockerRegistry** subsection to the **global** section: The dockerRegistry value has been moved to the global section: ```yaml global: dockerRegistry: # -- Define Image Registry that will to be used in Pipelines. Can be ecr (default), harbor type: "ecr" # -- Docker Registry endpoint url: ".dkr.ecr..amazonaws.com" ``` 4. (Optional) To integrate EDP with Jira, rename the default values from `epam-jira-user` to `jira-user` for a secret name. In case Jira is already integrated, it will continue working. ```yaml codebase-operator: jira: credentialName: "jira-user" ``` 5. (Optional) To switch to the Harbor registry, change the secret format for the external secret from [kaniko-docker-config v3.3.0](https://raw.githubusercontent.com/epam/edp-install/v3.3.0/deploy-templates/templates/external-secrets/externalsecret-kaniko.yaml) to [kaniko-docker-config v3.4.1](https://raw.githubusercontent.com/epam/edp-install/v3.4.0/deploy-templates/templates/external-secrets/externalsecret-kaniko.yaml):
View: old format ```json "kaniko-docker-config": {"secret-string"} //base64 format ```
View: new format ```json "kaniko-docker-config": { "auths" : { "registry.com" : {"username":"","password":"","auth":"secret-string"} } } ```
6. To upgrade EDP to the v3.4.1, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.4.1 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.4.1 --dry-run` ::: --- ## Upgrade EDP v3.4 to 3.5 # Upgrade EDP v3.4 to 3.5 :::warning We suggest making a backup of the EDP environment before starting the upgrade procedure. ::: This section provides detailed instructions for upgrading EPAM Delivery Platform to version 3.5.3. Follow the steps and requirements outlined below: 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.19.0/deploy-templates/crds/v2.edp.epam.com_gitservers.yaml ``` :::danger Codebase-operator v2.19.0 is not compatible with the previous versions. Please become familiar with the [breaking change](https://github.com/epam/edp-codebase-operator/commit/67ed1e336b7b185aba03992fd1c4fbebcd33941d) in Git Server Custom Resource Definition. ::: 2. Familiarize yourself with the updated file structure of the [values.yaml](https://raw.githubusercontent.com/epam/edp-install/v3.5.3/deploy-templates/values.yaml) file and adjust your values.yaml file accordingly: 1. By default, the deployment of sub components such as `edp-sonar-operator`, `edp-nexus-operator`, `edp-gerrit-operator`, and `keycloak-operator`, have been disabled. Set them back to `true` in case they are needed or manually deploy external tools, such as SonarQube, Nexus, Gerrit and integrate them with the EPAM Delivery Platform. 2. The default Git provider has been changed from Gerrit to GitHub: Old format: ```yaml global: gitProvider: gerrit gerritSSHPort: "22" ``` New format: ```yaml global: gitProvider: github #gerritSSHPort: "22" ``` 3. The **sonarUrl** and **nexusUrl** parameters have been deprecated. All the URLs from external components are stored in integration secrets: ```yaml global: # -- Optional parameter. Link to use custom sonarqube. Format: http://.:9000 or http://:9000 sonarUrl: "" # -- Optional parameter. Link to use custom nexus. Format: http://.:8081 or http://: nexusUrl: "" ``` 4. **Keycloak** integration has been moved from the **global** section to the **sso** section. Update the parameters accordingly: Old format: ```yaml global: # -- Keycloak URL keycloakUrl: https://keycloak.example.com # -- Administrators of your tenant admins: - "stub_user_one@example.com" # -- Developers of your tenant developers: - "stub_user_one@example.com" - "stub_user_two@example.com" ``` New format: ```yaml sso: enabled: true # -- Keycloak URL keycloakUrl: https://keycloak.example.com # -- Administrators of your tenant admins: - "stub_user_one@example.com" # -- Developers of your tenant developers: - "stub_user_one@example.com" - "stub_user_two@example.com" ``` 5. (Optional) The default secret name for Jira integration has been changed from `jira-user` to `ci-jira`. Please adjust the secret name in the parameters accordingly: ```yaml codebase-operator: jira: credentialName: "ci-jira" ``` 3. The secret naming and format have been refactored. Below are patterns of the changes for various components: Old format: ```json "sonar-ciuser-token": { "username": "xxxxx", "secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" } ``` New format: ```json "ci-sonarqube": { "token": "xxxxxxxxxxxxxxxxxxxxxxx", "url":"https://sonar.example.com" } ``` Old format: ```json "ci-dependency-track": { "token": "xxxxxxxxxxxxxxxxxx" } ``` New format: ```json "ci-dependency-track": { "token": "xxxxxxxxxxxxxxxxxx", "url": "http://dependency-track.example.com"} ``` Old format: ```json "defectdojo-ciuser-token": { "token": "xxxxxxxxxxxxxxxxxx" "url": "http://defectdojo.example.com" } ``` New format: ```json "ci-defectdojo": { "token": "xxxxxxxxxxxxxxxxxx", "url": "http://defectdojo.example.com" } ``` Old format: ```json "jira-user": { "username": "xxxxx", "password": "xxxxx" } ``` New format: ```json "ci-jira": { "username": "xxxxx", "password": "xxxxx" } ``` Old format: ```json "gitlab": { "id_rsa": "xxxxxxxxxxxxxx", "token": "xxxxxxxxxxxxxx", "secretString": "xxxxxxxxxxxxxx" } ``` New format: ```json "ci-gitlab": { "id_rsa": "xxxxxxxxxxxxxx", "token": "xxxxxxxxxxxxxx", "secretString": "xxxxxxxxxxxxxx" } ``` Old format: ```json "github": { "id_rsa": "xxxxxxxxxxxxxx", "token": "xxxxxxxxxxxxxx", "secretString": "xxxxxxxxxxxxxx" } ``` New format: ```json "ci-github": { "id_rsa": "xxxxxxxxxxxxxx", "token": "xxxxxxxxxxxxxx", "secretString": "xxxxxxxxxxxxxx" } ``` Old format: ```json "nexus-ci-user": { "username": "xxxxx", "password": "xxxxxxxxxxxxxxxxxx" } ``` New format: ```json "ci-nexus": { "username": "xxxxx", "password": "xxxxx", "url": "http://nexus.example.com" } ``` The tables below illustrate the difference between the old and new format: **Old format** |Secret Name|Username|Password|Token|Secret|URL| |:-:|:-:|:-:|:-:|:-:|:-:| |jira-user|:white_check_mark:|:white_check_mark:|||| |nexus-ci.user|:white_check_mark:|:white_check_mark:|||| |sonar-ciuser-token|:white_check_mark:|||:white_check_mark:|| |defectdojo-ciuser-token||||:white_check_mark:|:white_check_mark:| |ci-dependency-track||||:white_check_mark:|| **New format** |Secret Name|Username|Password|Token|URL| |:-:|:-:|:-:|:-:|:-:| |ci-jira|:white_check_mark:|:white_check_mark:||| |ci-nexus|:white_check_mark:|:white_check_mark:||:white_check_mark:| |ci-sonarqube|||:white_check_mark:|:white_check_mark:| |ci-defectdojo|||:white_check_mark:|:white_check_mark:| |ci-dependency-track|||:white_check_mark:|:white_check_mark:| 4. To upgrade EDP to the v3.5.3, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.5.3 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.5.3 --dry-run` ::: --- ## Upgrade EDP v3.5 to 3.6 # Upgrade EDP v3.5 to 3.6 :::warning We suggest backing up the EDP environment before starting the upgrade procedure. ::: This section provides detailed instructions for upgrading the EPAM Delivery Platform to version 3.6.0. Follow the steps and requirements outlined below: 1. Update Custom Resource Definitions (CRDs). Run the following command to apply all the necessary CRDs to the cluster: ```bash kubectl apply -f https://raw.githubusercontent.com/epam/edp-codebase-operator/v2.20.0/deploy-templates/crds/v2.edp.epam.com_codebases.yaml ``` 2. Familiarize yourself with the updated structure of the [values.yaml](https://github.com/epam/edp-install/blob/v3.6.0/deploy-templates/values.yaml#L38) file and adjust it accordingly: 2.1 A new parameter called [space](https://github.com/epam/edp-install/blob/v3.6.0/deploy-templates/values.yaml#L38) has been added to the `DockerRegistry` section. It is designed to form URLs in CodebaseImageStreams. This parameter is set the same as the `EPAM Delivery Platform` namespace name. Ensure you define the `space` parameter prior to the update. :::warning This [parameter](https://github.com/epam/edp-install/blob/v3.6.0/deploy-templates/values.yaml#L38) is a significant change and must be set before the update. ::: ```yaml global: dockerRegistry: type: "harbor" url: "registry.example.com" space: "edp" ``` 2.2 Sub components, such as [sonar-operator](https://github.com/epam/edp-sonar-operator), [nexus-operator](https://github.com/epam/edp-nexus-operator), and [keycloak-operator](https://github.com/epam/edp-keycloak-operator), have been removed since dependencies are no longer provisioned by the [edp-install](https://github.com/epam/edp-install/blob/v3.6.0/deploy-templates/values.yaml) Helm Chart. To install and integrate shared components with EDP, please use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) approach or refer to the [SonarQube Integration](../code-quality/sonarqube.md) and [Nexus Sonatype Integration](../artifacts-management/nexus-sonatype.md) documentation pages. 2.3 The Argo CD integration dependency has been deleted as now we implement it using [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) approach. To install and integrate Argo CD as a shared component, use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) approach. 2.4 The handling of secrets for stages namespaces in the [cd-pipeline-operator](https://github.com/epam/edp-cd-pipeline-operator/blob/v2.17.0/deploy-templates/values.yaml#L102) has been updated. The parameter `manageSecrets` has been replaced with `secretManager`. If your environment previously utilized this parameter, manually modify it from `manageSecrets: true` to `secretManager: own`. Otherwise, set it to `secretManager: none`: ```yaml cd-pipeline-operator: # -- flag that indicates whether the operator should manage secrets for stages; # own - just copy secrets; # eso - secrete will be managed by External Secrets Operator(operator should be installed in the cluster); # none - not enable secrets management logic; secretManager: none ``` 3. To upgrade EDP to the v3.6.0, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.6.0 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.6.0 --dry-run` ::: --- ## Upgrade EDP v3.6 to 3.7 # Upgrade EDP v3.6 to 3.7 :::warning We suggest backing up the EDP environment before starting the upgrade procedure. ::: This section provides detailed instructions for upgrading the EPAM Delivery Platform to version 3.7.5. Follow the steps and requirements outlined below: To upgrade EDP to the v3.7.5, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.7.5 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.7.5 --dry-run` ::: --- ## Upgrade KubeRocketCI v3.7 to 3.8 # Upgrade KubeRocketCI v3.7 to 3.8 :::warning We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: This section provides detailed instructions for upgrading the KubeRocketCI to the 3.8.1 version. Follow the steps and requirements outlined below: 1. Delete the following resources: ```bash kubectl -n edp delete ingress edp-headlamp kubectl -n edp delete ingress edp-tekton-dashboard kubectl -n edp delete ingress el-github-listener ``` 2. Delete the following Custom Resource Definitions (CRDs): ```bash kubectl delete CustomResourceDefinition edpcomponents.v1.edp.epam.com ``` :::info Prior to deleting CRDs, please remove all the complementing resources: ```bash kubectl -n edp get edpcomponent -o custom-columns=":metadata.name" | xargs -I {} kubectl -n edp delete edpcomponent {} ``` ::: 3. In Keycloak, update the KubeRocketCI portal URL: Navigate to the Keycloak realm with the `Portal` client. Update the `Valid redirect URIs` field: Old value: ```bash https://edp-headlamp-edp./* ``` New value: ```bash https://portal-edp./* ``` ![Keycloak client](../../assets/operator-guide/portal-keycloak.png "Keycloak client") 4. Remove all the Argo CD applications: :::warning This command will delete all the deployments. ```bash kubectl -n edp get application -o custom-columns=":metadata.name" | xargs -I {} kubectl -n edp get application {} -o json | jq 'del(.metadata.finalizers)' | kubectl replace -f - ``` ::: :::info Beginning from version 3.8.x, KubeRocketCI uses Argo CD application set instead applications to manage deploy environments, please ensure to upgrade your Argo CD instance to v2.10.3 and higher to work with this kind of resource. An example of how to install it is provided in the [edp-cluster-addons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/argo-cd/values.yaml#L30) repository. ::: 5. Familiarize yourself with the updated file structure of the [values.yaml](https://raw.githubusercontent.com/epam/edp-install/v3.8.1/deploy-templates/values.yaml) file and adjust your values.yaml file accordingly: 1. Starting from the version 3.8.x, KubeRocketCI supports multiple Git servers, so the `global.gitProvider` value was renamed: Old format: ```yaml global: gitProvider: github ``` New format: ```yaml global: gitProviders: - github # - gitlab # - gerrit ... edp-tekton: gitServers: github: gitProvider: github host: github.com webhook: skipWebhookSSLVerification: false eventListener: enabled: true resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "500m" ingress: enabled: true ``` 2. The `EDP-components` custom resources migrated to the `quick-links` section: Old format: ```yaml EDPComponents: - prometheus: url: https://ingress-prometheus.example.com visible: true icon: icon_in_base64 - kibana: url: https://ingress-kibana.example.com visible: false icon: icon_in_base64 - another_tool: url: https://ingress-anothertool.example.com visible: true icon: icon_in_base64 ``` New format for default components: ```yaml quickLinks: argocd: "https://argocd.example.com" defectdojo: "https://defectdojo.example.com" dependency_track: "https://dependency_track.example.com" docker_registry: "https://docker_registry.example.com" grafana: "https://grafana.example.com" kibana: "https://kibana.example.com" nexus: "https://nexus.example.com" sonar: "https://sonar.example.com" ``` New format for extra components: ```yaml extraQuickLinks: - prometheus: url: https://ingress-prometheus.example.com visible: true icon: icon_in_base64 - another_tool: url: https://ingress-anothertool.example.com visible: true icon: icon_in_base64 ``` 6. To upgrade KubeRocketCI to the v3.8.1, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.8.1 ``` :::note To verify the installation, test the deployment before applying it to the cluster using the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.8.1 --dry-run` ::: 7. Update webhooks for Github and GitLab: ```bash kubectl -n edp get codebases -o custom-columns=":metadata.name" | xargs -I {} kubectl patch codebases -n edp {} --subresource=status --type=json -p '[{"op": "remove", "path": "/status/webHookID"}]' kubectl delete pod -l name=codebase-operator ``` Open repository in Github, navigate to **Settings** -> **Webhooks** -> Select exist webhook and click `edit`. Change Payload URL: ```bash Old value: https://el-github-listener-edp. New value: https://el-github-edp. ``` Open repository in Github, navigate to **Settings** -> **Webhooks** -> Select exist webhook and click `edit`. Change Payload URL: ```bash Old value: https://el-gitlab-listener-edp. New value: https://el-gitlab-edp. ``` --- ## Upgrade KubeRocketCI v3.8 to 3.9 # Upgrade KubeRocketCI v3.8 to 3.9 :::important We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: This section provides detailed instructions for upgrading the KubeRocketCI to version 3.9.0. Follow the steps and requirements outlined below: :::warning [edp-install](https://github.com/epam/edp-install/tree/v3.9.0) version v3.9 is not support Kiosk as tenancy engine. Please migrate to Capsule engine or disable this options. ::: 1. (Optional) Migrate from Kiosk tenancy engine. 1. Take look how to install Capsule using [addons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/capsule). 2. [Integrate Capsule](https://epam.github.io/edp-install/operator-guide/capsule) with KubeRocketCI. 3. Update edp-install values file: ```yaml title="values.yaml" ... cd-pipeline-operator: tenancyEngine: "capsule" ... ``` 1. To disable tenancy engine update edp-install values file: ```yaml title="values.yaml" ... cd-pipeline-operator: tenancyEngine: "none" ... ``` 2. (Optional) Align Keycloak integration. In KubeRocketCI version 3.9.0, the Keycloak configuration procedure has been altered in the [values](https://github.com/epam/edp-install/blob/v3.9.0/deploy-templates/values.yaml#L461) file. Please be attentive and align the configuration for your own use case. ```yaml title="values.yaml" sso: # -- Install OAuth2-proxy and Keycloak CRs as a part of KubeRocketCI deployment. enabled: false # -- Defines Keycloak realm name that is used as the Identity Provider (IdP) realm realmName: "broker" # -- Keycloak URL. keycloakUrl: https://keycloak.example.com ``` ```yaml title="values.yaml" sso: # -- Install OAuth2-proxy and Keycloak CRs as a part of KubeRocketCI deployment. enabled: false keycloakOperatorResources: # Set to false if using the add-ons approach (refer to https://github.com/epam/edp-cluster-add-ons) # for KubeRocketCI installation and if the extension-oidc is already installed. # This prevents the creation of an additional Keycloak resource and secret. # The 'kind' and 'name' must be specified in case of using an existing Keycloak/ClusterKeycloak resource. # Create kind: Keycloak as a part of chart installation createKeycloakCR: true # Can be Keycloak or ClusterKeycloak. kind: Keycloak # Name of kind: Keycloak/ClusterKeycloak CR. name: main # -- Defines the Keycloak realm name, which by default is named after the namespace where KubeRocketCI is deployed. # realmName: edp # -- Defines Keycloak sso realm name that is used as the Identity Provider (IdP) realm ssoRealmName: "broker" # -- Defines Keycloak client name that is used for the Identity Provider (IdP) client ssoClientName: "edp" # -- Keycloak URL. keycloakUrl: https://keycloak.example.com/auth ``` 3. To upgrade KubeRocketCI to the v3.9.0, run the following command: ```bash helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.9.0 ``` :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` tag: `helm upgrade edp epamedp/edp-install -n edp --values values.yaml --version=3.9.0 --dry-run` ::: --- ## Upgrade Keycloak v17.0 to 19.0 # Upgrade Keycloak v17.0 to 19.0 Starting from Keycloak v.18.x.x, the Keycloak server has been moved from the Wildfly (JBoss) Application Server to [Quarkus](https://quarkus.io/) framework and is called Keycloak.X. There are two ways to upgrade Keycloak v.17.0.x-legacy to v.19.0.x on Kubernetes, please perform the steps described in the [Prerequisites](#prerequisites) section of this tutorial, and then select a suitable upgrade strategy for your environment: - [Upgrade Postgres database to a minor release v.11.17](#upgrade-postgres-database-to-a-minor-release-v1117) - [Migrate Postgres database from Postgres v.11.x to v.14.5](#migrate-postgres-database-from-postgres-v11x-to-v145) ## Prerequisites Before upgrading Keycloak, please perform the steps below: 1. Create a backup/snapshot of the Keycloak database volume. Locate the AWS `volumeID` and then create its [snapshot](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html) on AWS: - Find the `PVC` name attached to the Postgres pod. It can be similar to `data-keycloak-postgresql-0` if the Postgres `StatefulSet` name is `keycloak-postgresql`: ```bash kubectl get pods keycloak-postgresql-0 -n security -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}' ``` - Locate the `PV` `volumeName` in the `data-keycloak-postgresql-0` Persistent Volume Claim: ```bash kubectl get pvc data-keycloak-postgresql-0 -n security -o jsonpath='{.spec.volumeName}{"\n"}' ``` - Get `volumeID` in the Persistent Volume: ```bash kubectl get pv ${pv_name} -n security -o jsonpath='{.spec.awsElasticBlockStore.volumeID}{"\n"}' ``` 2. Add two additional keys: `password` and `postgres-password`, to the `keycloak-postgresql` secret in the Keycloak namespace. :::note - The `password` key must have the same value as the `postgresql-password` key. - The `postgres-password` key must have the same value as the `postgresql-postgres-password` key. ::: The latest chart for Keycloak.X does not have an option to override Postgres password and admin password keys in the secret, and it uses the Postgres [defaults](https://github.com/codecentric/helm-charts/blob/master/charts/keycloakx/values.yaml#L371), therefore, a new secret scheme must be implemented: ```bash kubectl -n security edit secret keycloak-postgresql ``` ```yaml data: postgresql-password: XXXXXX postgresql-postgres-password: YYYYYY password: XXXXXX postgres-password: YYYYYY ``` 3. Save Keycloak `StatefulSet` names, for example, `keycloak` and `keycloak-postgresql`. These names will be used in the new Helm deployments: ```bash $ kubectl get statefulset -n security NAME READY AGE keycloak 1/1 18h keycloak-postgresql 1/1 18h ``` ## Upgrade Postgres Database to a Minor Release v.11.17 To upgrade Keycloak by upgrading [Postgres Database](https://www.postgresql.org/) to a minor release v.11.17, perform the steps described in the [Prerequisites](#prerequisites) section of this tutorial, and then perform the following steps: ### Delete Keycloak Resources 1. Delete `Keycloak` and `Postgres` `StatefulSets`: ```bash kubectl delete statefulset keycloak keycloak-postgresql -n security ``` 2. Delete the Keycloak `Ingress`object, to prevent hostname duplication issues: ```bash kubectl delete ingress keycloak -n security ``` ### Upgrade Keycloak 1. Make sure the Keycloak chart repository is added: ```bash helm repo add codecentric https://codecentric.github.io/helm-charts helm repo update ``` 2. Create values for Keycloak: :::note Since the Keycloak.X release, Keycloak and Postgres database charts are separated. Upgrade Keycloak, and then install the Postgres database. ::: :::note - `nameOverride: "keycloak"` sets the name of the Keycloak pod. It must be the same Keycloak name as in the previous `StatefulSet`. - Change Ingress host name to the Keycloak host name. - `hostname: keycloak-postgresql` is the hostname of the pod with the Postgres database that is the same as Postgres StatefulSet name, for example, `keycloak-postgresql`. - `"/opt/keycloak/bin/kc.sh start --auto-build"` was used in the legacy Keycloak version. However, it is no longer required in the new Keycloak version since it is [deprecated](https://www.keycloak.org/docs/latest/upgrading/index.html#changes-to-the-server-configuration-and-startup) and used by default. - Optionally, use the following command for applying the old Keycloak theme: ```bash bin/kc.sh start --features-disabled=admin2 ``` :::
View: keycloak-values.yaml ```yaml nameOverride: "keycloak" replicas: 1 # Deploy the latest verion image: tag: "19.0.1" # start: create OpenShift realm which is required by KubeRocketCI extraInitContainers: | - name: realm-provider image: busybox imagePullPolicy: IfNotPresent command: - sh args: - -c - | echo '{"realm": "openshift","enabled": true}' > /opt/keycloak/data/import/openshift.json volumeMounts: - name: realm mountPath: /opt/keycloak/data/import extraVolumeMounts: | - name: realm mountPath: /opt/keycloak/data/import extraVolumes: | - name: realm emptyDir: {} command: - "/opt/keycloak/bin/kc.sh" - "--verbose" - "start" - "--http-enabled=true" - "--http-port=8080" - "--hostname-strict=false" - "--hostname-strict-https=false" - "--spi-events-listener-jboss-logging-success-level=info" - "--spi-events-listener-jboss-logging-error-level=warn" - "--import-realm" extraEnv: | - name: KC_PROXY value: "passthrough" - name: KEYCLOAK_ADMIN valueFrom: secretKeyRef: name: keycloak-admin-creds key: username - name: KEYCLOAK_ADMIN_PASSWORD valueFrom: secretKeyRef: name: keycloak-admin-creds key: password - name: JAVA_OPTS_APPEND value: >- -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0 -Djava.awt.headless=true -Djgroups.dns.query={{ include "keycloak.fullname" . }}-headless # This block should be uncommented if you install Keycloak on Kubernetes ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx ingress.kubernetes.io/affinity: cookie rules: - host: keycloak. paths: - path: '{{ tpl .Values.http.relativePath $ | trimSuffix "/" }}/' pathType: Prefix # This block should be uncommented if you set Keycloak to OpenShift and change the host field # route: # enabled: false # # Path for the Route # path: '/' # # Host name for the Route # host: "keycloak." # # TLS configuration # tls: # enabled: true resources: limits: memory: "2048Mi" requests: cpu: "50m" memory: "512Mi" # Check database readiness at startup dbchecker: enabled: true database: vendor: postgres existingSecret: keycloak-postgresql hostname: keycloak-postgresql port: 5432 username: admin database: keycloak ```
3. Upgrade the Keycloak Helm chart: :::note - The Helm chart is substituted with the new KeyacloakX instance. - Change the namespace and the values file name if required. ::: ```bash helm upgrade keycloak codecentric/keycloakx --version 1.6.0 --values keycloak-values.yaml -n security ``` :::note If there are error messages when upgrading via Helm, make sure that `StatefulSets` are removed. If they are removed and the error still persists, try to add the `--force` flag to the Helm command: ```bash helm upgrade keycloak codecentric/keycloakx --version 1.6.0 --values keycloak-values.yaml -n security --force ``` ::: ### Install Postgres 1. Add Bitnami chart repository and update Helm repos: ```bash helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update ``` 2. Create values for Postgres: :::note - Postgres v.11 and Postgres v.14.5 are not compatible. - Postgres image will be upgraded to a minor release v.11.17. - `fullnameOverride: "keycloak-postgresql"` sets the name of the Postgres StatefulSet. It must be the same as in the previous `StatefulSet`. :::
View: postgres-values.yaml ```yaml fullnameOverride: "keycloak-postgresql" # PostgreSQL read only replica parameters readReplicas: # Number of PostgreSQL read only replicas replicaCount: 1 global: postgresql: auth: username: admin existingSecret: keycloak-postgresql secretKeys: adminPasswordKey: postgres-password userPasswordKey: password database: keycloak image: registry: docker.io repository: bitnami/postgresql tag: 11.17.0-debian-11-r3 auth: existingSecret: keycloak-postgresql secretKeys: adminPasswordKey: postgres-password userPasswordKey: password primary: persistence: enabled: true size: 3Gi # If the StorageClass with reclaimPolicy: Retain is used, install an additional StorageClass before installing PostgreSQL # (the code is given below). # If the default StorageClass will be used - change "gp2-retain" to "gp2" storageClass: "gp2-retain" ```
3. Install the Postgres database chart: :::note Change the namespace and the values file name if required. ::: ```bash helm install postgresql bitnami/postgresql \ --version 11.7.6 \ --values postgres-values.yaml \ --namespace security ``` 4. Log in to Keycloak and check that everything works as expected. ### Clean and Analyze Database Optionally, run the [vacuumdb](https://www.postgresql.org/docs/current/app-vacuumdb.html) application on the database, to recover space occupied by "dead tuples" in the tables, analyze the contents of database tables, and collect statistics for PostgreSQL query engine to improve performance: ```bash PGPASSWORD="${postgresql_postgres-password}" vacuumdb --analyze --verbose -d keycloak -U postgres ``` For all databases, run the following command: ```bash PGPASSWORD="${postgresql_postgres-password}" vacuumdb --analyze --verbose --all -U postgres ``` ## Migrate Postgres Database From Postgres v.11.x to v.14.5 :::info There is a [Postgres database migration script](#postgres-database-migration-script) at the end of this tutorial. Please read the section below before using the script. ::: To upgrade Keycloak by migrating Postgres database from Postgres v.11.x to v.14.5, perform the steps described in the [Prerequisites](#prerequisites) section of this tutorial, and then perform the following steps: ### Export Postgres Databases 1. Log in to the current Keycloak Postgres pod and create a logical backup of all roles and databases using the [pg_dumpall](https://www.postgresql.org/docs/current/app-pg-dumpall.html) application. If there is no access to the Postgres Superuser, backup the Keycloak database with the [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) application: :::note - The secret key `postgresql-postgres-password` is for the `postgres` Superuser and `postgresql-password` is for `admin` user. The `admin` user is indicated by default in the Postgres Helm chart. The `admin` user may not have enough permissions to dump all Postgres databases and roles, so the preferred option for exporting all objects is using the `pg_dumpall` tool with the `postgres` Superuser. - If the `PGPASSWORD` variable is not specified before using the `pg_dumpall` tool, you will be prompted to enter a password for each database during the export. - If the `-l keycloak` parameter is specified, `pg_dumpall` will connect to the `keycloak` database for dumping global objects and discovering what other databases should be dumped. By default, `pg_dumpall` will try to connect to `postgres` or `template1` databases. This parameter is optional. - The `pg_dumpall --clean` option adds SQL commands to the dumped file for dropping databases before recreating them during import, as well as `DROP` commands for roles and tablespaces (`pg_dump` also has this option). If the `--clean` parameter is specified, connect to the `postgres` database initially during import via `psql`. The `psql` script will attempt to drop other databases immediately, and that will fail for the database you are connected to. This flag is optional, and it is not included into this tutorial. ::: ```bash PGPASSWORD="${postgresql_postgres-password}" pg_dumpall -h localhost -p 5432 -U postgres -l keycloak > /tmp/keycloak_wildfly_db_dump.sql ``` :::note If there is no working password for the `postgres` Superuser, try the `admin` user using the [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) tool to export the `keycloak` database without global roles: ```bash PGPASSWORD="${postgresql_password}" pg_dump -h localhost -p 5432 -U admin -d keycloak > /tmp/keycloak_wildfly_db_dump.sql ``` ::: :::info Double-check that the contents of the dumped file is not empty. It usually contains more than 4000 lines. ::: 2. Copy the file with the database dump to a local machine. Since `tar` may not be present in the pod and `kubectl cp` will not work without `tar`, use the following command: ```bash kubectl exec -n security ${postgresql_pod} -- cat /tmp/keycloak_wildfly_db_dump.sql > keycloak_wildfly_db_dump.sql ``` :::note Please find below the alternative commands for exporting the database to the local machine without copying the file to a pod for Postgres and admin users: ```bash kubectl exec -n security ${postgresql_pod} "--" sh -c "PGPASSWORD='"${postgresql_postgres-password}"' pg_dumpall -h localhost -p 5432 -U postgres" > keycloak_wildfly_db_dump.sql kubectl exec -n security ${postgresql_pod} "--" sh -c "PGPASSWORD='"${postgresql_password}"' pg_dump -h localhost -p 5432 -U admin -d keycloak" > keycloak_wildfly_db_dump.sql ``` ::: 3. Delete the dumped file from the pod for security reasons: ```bash kubectl exec -n security ${postgresql_pod} "--" sh -c "rm /tmp/keycloak_wildfly_db_dump.sql" ``` ### Delete Keycloak Resources 1. Delete all previous Keycloak resources along with the Postgres database and keycloak `StatefulSets`, `Ingress`, and custom resources via Helm, or via the tool used for their deployment. ```bash helm list -n security helm delete keycloak -n security ``` :::warning Don't delete the whole namespace. Keep the `keycloak-postgresql` and `keycloak-admin-creds` secrets. ::: 2. Delete the volume in AWS, from which a snapshot has been created. Then delete the PVC: ```bash kubectl delete pvc data-keycloak-postgresql-0 -n security ``` ### Install Postgres 1. Add Bitnami chart repository and update Helm repos: ```bash helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update ``` 2. Create Postgres values: :::note `fullnameOverride: "keycloak-postgresql"` sets the name of the Postgres StatefulSet. It must be same as in the previous `StatefulSet`. :::
View: postgres-values.yaml ```yaml nameOverride: "keycloak-postgresql" # PostgreSQL read only replica parameters readReplicas: # Number of PostgreSQL read only replicas replicaCount: 1 global: postgresql: auth: username: admin existingSecret: keycloak-postgresql secretKeys: adminPasswordKey: postgres-password userPasswordKey: password database: keycloak auth: existingSecret: keycloak-postgresql secretKeys: adminPasswordKey: postgres-password userPasswordKey: password primary: persistence: enabled: true size: 3Gi # If the StorageClass with reclaimPolicy: Retain is used, install an additional StorageClass before installing PostgreSQL # (the code is given below). # If the default StorageClass will be used - change "gp2-retain" to "gp2" storageClass: "gp2-retain" ```
3. Install the Postgres database: :::note Change the namespace and the values file name if required. ::: ```bash helm install postgresql bitnami/postgresql \ --version 11.7.6 \ --values postgres-values.yaml \ --namespace security ``` 4. Wait for the database to be ready. ### Import Postgres Databases 1. Upload the database dump to the new Keycloak Postgres pod: ```bash cat keycloak_wildfly_db_dump.sql | kubectl exec -i -n security ${postgresql_pod} "--" sh -c "cat > /tmp/keycloak_wildfly_db_dump.sql" ``` :::warning Database import must be done before deploying Keycloak, because Keycloak will write its own data to the database during the start, and the import will partially fail. If that happened, scale down the keycloak `StatefulSet`, and try to drop the Keycloak database in the Postgres pod: ```bash dropdb -i -e keycloak -p 5432 -h localhost -U postgres ``` If there still are some conflicting objects like roles, drop them via the [DROP ROLE](https://www.postgresql.org/docs/current/sql-droprole.html) command. If the previous steps do not help, downscale the Keycloak and Postgres `StatefulSets` and delete the attached `PVC` (save the `volumeID` before removing), and delete the volume on AWS if using `gp2-retain`. In case of using `gp2`, the volume will be deleted automatically after removing PVC. After that, redeploy the Postgres database, so that the new `PVC` is automatically created. ::: 2. Import the [SQL dump](https://www.postgresql.org/docs/current/backup-dump.html) file to the Postgres database cluster: :::info Since the databases were exported in the `sql` format, the [psql](https://www.postgresql.org/docs/current/app-psql.html) tool will be used to restore (reload) them. [pg_restore](https://www.postgresql.org/docs/current/app-pgrestore.html) does not support this plain-text format. ::: - If the entire Postgres database cluster was migrated with the `postgres` Superuser using `pg_dumpall`, use the import command without indicating the database: ```bash psql -U postgres -f /tmp/keycloak_wildfly_db_dump.sql ``` - If the database was migrated with the `admin` user using `pg_dump`, the `postgres` Superuser still can be used to restore it, but, in this case, a database must be indicated: :::warning If the database name was not indicated during the import for the file dumped with `pg_dump`, the `psql` tool will import this database to a default Postgres database called `postgres`. ::: ```bash psql -U postgres -d keycloak -f /tmp/keycloak_wildfly_db_dump.sql ``` - If the `postgres` Superuser is not accessible in the Postgres pod, run the command under the `admin` or any other user that has the database permissions. In this case, indicate the database as well: ```bash psql -U admin -d keycloak -f /tmp/keycloak_wildfly_db_dump.sql ``` 3. After a successful import, delete the dump file from the pod for security reasons: ```bash kubectl exec -n security ${postgresql_pod} "--" sh -c "rm /tmp/keycloak_wildfly_db_dump.sql" ``` :::note Please find below the alternative commands for importing the database from the local machine to the pod without storing the backup on a pod for `postgres` or `admin` users: ::: ```bash cat "keycloak_wildfly_db_dump.sql" | kubectl exec -i -n "${keycloak_namespace}" "${postgres_pod_name}" "--" sh -c "cat | PGPASSWORD='"${postgresql_superuser_password}"' psql -h "${db_host}" -p "${db_port}" -U "${postgres_username}"" cat "keycloak_wildfly_db_dump.sql" | kubectl exec -i -n "${keycloak_namespace}" "${postgres_pod_name}" "--" sh -c "cat | PGPASSWORD='"${postgresql_superuser_password}"' psql -h "${db_host}" -p "${db_port}" -U "${postgres_username}" -d "${database_name}"" cat "keycloak_wildfly_db_dump.sql" | kubectl exec -i -n "${keycloak_namespace}" "${postgres_pod_name}" "--" sh -c "cat | PGPASSWORD='"${postgresql_admin_password}"' psql -h "${db_host}" -p "${db_port}" -U "${postgres_username}" -d "${database_name}"" ``` ### Install Keycloak 1. Make sure the Keycloak chart repository is added: ```bash helm repo add codecentric https://codecentric.github.io/helm-charts helm repo update ``` 2. Create Keycloak values: :::note - `nameOverride: "keycloak"` sets the name of the Keycloak pod. It must be the same Keycloak name as in the previous `StatefulSet`. - Change Ingress host name to the Keycloak host name. - `hostname: keycloak-postgresql` is the hostname of the pod with the Postgres database that is the same as Postgres StatefulSet name, for example, `keycloak-postgresql`. - `"/opt/keycloak/bin/kc.sh start --auto-build"` was used in the legacy Keycloak version. However, it is no longer required in the new Keycloak version since it is [deprecated](https://www.keycloak.org/docs/latest/upgrading/index.html#changes-to-the-server-configuration-and-startup) and used by default. - Optionally, use the following command for applying the old Keycloak theme: ```bash bin/kc.sh start --features-disabled=admin2 ``` ::: :::info [Automatic database migration](https://www.keycloak.org/docs/latest/upgrading/index.html#automatic-relational-database-migration) will start after the Keycloak installation. :::
View: keycloak-values.yaml ```yaml nameOverride: "keycloak" replicas: 1 # Deploy the latest version image: tag: "19.0.1" # start: create OpenShift realm which is required by KubeRocketCI extraInitContainers: | - name: realm-provider image: busybox imagePullPolicy: IfNotPresent command: - sh args: - -c - | echo '{"realm": "openshift","enabled": true}' > /opt/keycloak/data/import/openshift.json volumeMounts: - name: realm mountPath: /opt/keycloak/data/import extraVolumeMounts: | - name: realm mountPath: /opt/keycloak/data/import extraVolumes: | - name: realm emptyDir: {} command: - "/opt/keycloak/bin/kc.sh" - "--verbose" - "start" - "--http-enabled=true" - "--http-port=8080" - "--hostname-strict=false" - "--hostname-strict-https=false" - "--spi-events-listener-jboss-logging-success-level=info" - "--spi-events-listener-jboss-logging-error-level=warn" - "--import-realm" extraEnv: | - name: KC_PROXY value: "passthrough" - name: KEYCLOAK_ADMIN valueFrom: secretKeyRef: name: keycloak-admin-creds key: username - name: KEYCLOAK_ADMIN_PASSWORD valueFrom: secretKeyRef: name: keycloak-admin-creds key: password - name: JAVA_OPTS_APPEND value: >- -XX:+UseContainerSupport -XX:MaxRAMPercentage=50.0 -Djava.awt.headless=true -Djgroups.dns.query={{ include "keycloak.fullname" . }}-headless # This block should be uncommented if you install Keycloak on Kubernetes ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx ingress.kubernetes.io/affinity: cookie rules: - host: keycloak. paths: - path: '{{ tpl .Values.http.relativePath $ | trimSuffix "/" }}/' pathType: Prefix # This block should be uncommented if you set Keycloak to OpenShift and change the host field # route: # enabled: false # # Path for the Route # path: '/' # # Host name for the Route # host: "keycloak." # # TLS configuration # tls: # enabled: true resources: limits: memory: "2048Mi" requests: cpu: "50m" memory: "512Mi" # Check database readiness at startup dbchecker: enabled: true database: vendor: postgres existingSecret: keycloak-postgresql hostname: keycloak-postgresql port: 5432 username: admin database: keycloak ```
3. Deploy Keycloak: :::note Change the namespace and the values file name if required. ::: ```bash helm install keycloak codecentric/keycloakx --version 1.6.0 --values keycloak-values.yaml -n security ``` 4. Log in to Keycloak and check if everything has been imported correctly. ### Clean and Analyze Database Optionally, run the [vacuumdb](https://www.postgresql.org/docs/current/app-vacuumdb.html) application on the database, to analyze the contents of database tables and collect statistics for the Postgres query optimizer: ```bash PGPASSWORD="${postgresql_postgres-password}" vacuumdb --analyze --verbose -d keycloak -U postgres ``` For all databases, run the following command: ```bash PGPASSWORD="${postgresql_postgres-password}" vacuumdb --analyze --verbose --all -U postgres ``` ### Postgres Database Migration Script :::info Please read the [Migrate Postgres Database From Postgres v.11.x to v.14.5](#migrate-postgres-database-from-postgres-v11x-to-v145) section of this tutorial before using the script. ::: :::note - The [`kubectl`](https://github.com/kubernetes/kubectl) tool is required for using this script. - This script will likely work for any other Postgres database besides Keycloak after some adjustments. It queries the `pg_dump`, `pg_dumpall`, `psql`, and `vacuumdb` commands under the hood. ::: The following script can be used for exporting and importing Postgres databases as well as optimizing them with the [vacuumdb](https://www.postgresql.org/docs/current/app-vacuumdb.html) application. Please examine the code and make the adjustments if required. - By default, the following command exports Keycloak Postgres databases from a Kubernetes pod to a local machine: ```bash ./script.sh ``` After running the command, please follow the prompt. - To import a database backup to a newly created Postgres Kubernetes pod, pass a database dump sql file to the script: ```bash ./script.sh path-to/db_dump.sql ``` - The `-h` flag prints help, and `-c|-v` runs the `vacuumdb` garbage collector and analyzer.
View: keycloak_db_migration.sh ```bash #!/bin/bash # set -x db_migration_help(){ echo "Keycloak Postgres database migration" echo echo "Usage:" echo "------------------------------------------" echo "Export Keycloak Postgres database from pod" echo "Run without parameters:" echo " $0" echo "------------------------------------------" echo "Import Keycloak Postgres database to pod" echo "Pass filename to script:" echo " $0 path/to/db_dump.sql" echo "------------------------------------------" echo "Additional options: " echo " $0 [OPTIONS...]" echo "Options:" echo "h Print Help." echo "c|v Run garbage collector and analyzer." } keycloak_ns(){ printf '%s\n' 'Enter keycloak namespace: ' read -r keycloak_namespace if [ -z "${keycloak_namespace}" ]; then echo "Don't skip namespace" exit 1 fi } postgres_pod(){ printf '%s\n' 'Enter postgres pod name: ' read -r postgres_pod_name if [ -z "${postgres_pod_name}" ]; then echo "Don't skip pod name" exit 1 fi } postgres_user(){ printf '%s\n' 'Enter postgres username: ' printf '%s' "Skip to use [postgres] superuser: " read -r postgres_username if [ -z "${postgres_username}" ]; then postgres_username='postgres' fi } pgdb_host_info(){ database_name='keycloak' db_host='localhost' db_port='5432' } postgresql_admin_pass(){ postgresql_password='POSTGRES_PASSWORD' postgresql_admin_password="$(kubectl exec -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "printenv ${postgresql_password}")" } postgresql_su_pass(){ postgresql_postgres_password='POSTGRES_POSTGRES_PASSWORD' postgresql_superuser_password="$(kubectl exec -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "printenv ${postgresql_postgres_password}")" if [ -z "${postgresql_superuser_password}" ]; then echo "SuperUser password variable does not exist. Using user password instead..." postgresql_admin_pass postgresql_superuser_password="${postgresql_admin_password}" fi } keycloak_pgdb_export(){ current_cluster="$(kubectl config current-context | tr -dc '[:alnum:]-')" exported_db_name="keycloak_db_dump_${current_cluster}_${keycloak_namespace}_${postgres_username}_$(date +"%Y%m%d%H%M").sql" if [ "${postgres_username}" == 'postgres' ]; then # call a function to get a pass for postgres user postgresql_su_pass kubectl exec -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "PGPASSWORD='"${postgresql_superuser_password}"' pg_dumpall -h "${db_host}" -p "${db_port}" -U "${postgres_username}"" > "${exported_db_name}" else # call a function to get a pass for admin user postgresql_admin_pass kubectl exec -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "PGPASSWORD='"${postgresql_admin_password}"' pg_dump -h "${db_host}" -p "${db_port}" -U "${postgres_username}" -d "${database_name}"" > "${exported_db_name}" fi separate_lines="---------------" if [ ! -s "${exported_db_name}" ]; then rm -f "${exported_db_name}" echo "${separate_lines}" echo "Something went wrong. The database dump file is empty and was not saved." else echo "${separate_lines}" grep 'Dumped' "${exported_db_name}" | sort -u echo "Database has been exported to $(pwd)/${exported_db_name}" fi } keycloak_pgdb_import(){ echo "Preparing Import" echo "----------------" if [ ! -f "$1" ]; then echo "The file $1 does not exist." exit 1 fi keycloak_ns postgres_pod postgres_user pgdb_host_info if [ "${postgres_username}" == 'postgres' ]; then # restore full backup with all databases and roles as superuser or a single database postgresql_su_pass if [ -n "$(cat "$1" | grep 'CREATE ROLE')" ]; then cat "$1" | kubectl exec -i -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "cat | PGPASSWORD='"${postgresql_superuser_password}"' psql -h "${db_host}" -p "${db_port}" -U "${postgres_username}"" else cat "$1" | kubectl exec -i -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "cat | PGPASSWORD='"${postgresql_superuser_password}"' psql -h "${db_host}" -p "${db_port}" -U "${postgres_username}" -d "${database_name}"" fi else # restore a single database postgresql_admin_pass cat "$1" | kubectl exec -i -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "cat | PGPASSWORD='"${postgresql_admin_password}"' psql -h "${db_host}" -p "${db_port}" -U "${postgres_username}" -d "${database_name}"" fi } vacuum_pgdb(){ echo "Preparing garbage collector and analyzer" echo "----------------------------------------" keycloak_ns postgres_pod postgres_user pgdb_host_info if [ "${postgres_username}" == 'postgres' ]; then postgresql_su_pass kubectl exec -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "PGPASSWORD='"${postgresql_superuser_password}"' vacuumdb --analyze --all -h "${db_host}" -p "${db_port}" -U "${postgres_username}"" else postgresql_admin_pass kubectl exec -n "${keycloak_namespace}" "${postgres_pod_name}" "--" \ sh -c "PGPASSWORD='"${postgresql_admin_password}"' vacuumdb --analyze -h "${db_host}" -p "${db_port}" -U "${postgres_username}" -d "${database_name}"" fi } while [ "$#" -eq 1 ]; do case "$1" in -h | --help) db_migration_help exit 0 ;; -c | --clean | -v | --vacuum) vacuum_pgdb exit 0 ;; --) break ;; -*) echo "Invalid option '$1'. Use -h|--help to see the valid options" >&2 exit 1 ;; *) keycloak_pgdb_import "$1" exit 0 ;; esac shift done if [ "$#" -gt 1 ]; then echo "Please pass a single file to the script" exit 1 fi echo "Preparing Export" echo "----------------" keycloak_ns postgres_pod postgres_user pgdb_host_info keycloak_pgdb_export ```
## Related Articles * [Install Keycloak](../auth/keycloak.md) --- ## Upgrade KubeRocketCI v3.11 to 3.12 # Upgrade KubeRocketCI v3.11 to 3.12 This section provides detailed instructions for upgrading KubeRocketCI to version 3.12. Follow the steps and requirements outlined below: :::important We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: :::important In version 3.12, the `docker.io/epamedp/tekton-cache` image has been [deprecated](https://github.com/epam/edp-tekton/issues/511) and replaced with `ghcr.io/kuberocketci/krci-cache`. In case of using the [tekton-cache](https://github.com/epam/edp-tekton/tree/master/charts/tekton-cache) Helm chart, the image will be automatically updated during the upgrade process. ::: :::warning Since Kaniko Docker image build tool support has been discontinued, the [BuildKit](https://docs.docker.com/build/buildkit/) tool support has been introduced. Users now have a choice between two image build tools: Kaniko and BuildKit. For this reason, the `dockerbuild-verify` and `kaniko-build` Tekton Tasks no longer exist. Rename these Tasks by the following pattern if you leverage them in your custom pipelines: - `dockerbuild-verify` → `dockerbuild-verify-kaniko` or `dockerbuild-verify-buildkit`, depending on which build tool you are using. - `kaniko-build` → `container-build` (this task works with both Kaniko and BuildKit tools). Kaniko is still used by default. To switch between Kaniko and BuildKit, set the `tekton.containerBuildTool` parameter in the [pipelines-library values](https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/values.yaml#L230). ::: 1. (Optional) Update custom Tekton Pipelines: :::note For more information about using Tekton custom pipelines in KubeRocketCI, refer to the [Create and Use Custom Tekton Pipelines](../../use-cases/custom-pipelines-flow.md) use case. ::: In case of using Tekton custom pipelines, it is necessary to update them to ensure compatibility with the new version of KubeRocketCI. 1. Update branch name validation: Starting from version 3.12, the KubeRocketCI portal [supports](https://github.com/epam/edp-headlamp/issues/753) adding branches with long names (more than 30 characters), including support for special characters and uppercase letters. Due to this change, it is necessary to update the passing parameters for the `update-cbis` task in the Tekton custom build pipelines to ensure compatibility with the new version of KubeRocketCI. The `update-cbis` Tekton task now accepts the `CODEBASEBRANCH_NAME` parameter instead of `CBIS_NAME`. Update the `update-cbis` passing parameters in the Tekton custom build pipelines as follows: ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: build app.edp.epam.com/triggertemplate: github-build-template name: custom-build-pipeline spec: ... tasks: - name: update-cbis params: - name: CBIS_NAME value: $(tasks.init-values.results.RESULT_IMAGE_NAME) - name: IMAGE_TAG value: $(tasks.get-version.results.IS_TAG) runAfter: - git-tag taskRef: kind: Task name: update-cbis ``` ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: build app.edp.epam.com/triggertemplate: github-build-template name: custom-build-pipeline spec: ... tasks: - name: update-cbis params: - name: CODEBASEBRANCH_NAME value: $(params.CODEBASEBRANCH_NAME) - name: IMAGE_TAG value: $(tasks.get-version.results.IS_TAG) runAfter: - git-tag taskRef: kind: Task name: update-cbis ``` Updated `update-cbis` task in the KubeRocketCI version 3.12 is provided below for reference:
update-cbis Updated `update-cbis` task in KubeRocketCI version 3.12: ```yaml apiVersion: tekton.dev/v1 kind: Task metadata: name: update-cbis spec: description: >- This task updates a Codebase ImageStream (CBIS) with a new image tag. It checks for the presence of tags in the specified CBIS and adds the new tag if it doesn't already exist. The task utilizes kubectl commands and is customizable with parameters for CBIS params: - name: CODEBASEBRANCH_NAME type: string description: "CodebaseBranch name with only letters and dashes" - name: IMAGE_TAG type: string - name: BASE_IMAGE description: The base image for the task. type: string default: {{ include "edp-tekton.registry" . }}/bitnami/kubectl:1.25.4 steps: - name: update-cbis image: $(params.BASE_IMAGE) env: - name: CODEBASEBRANCH_NAME value: "$(params.CODEBASEBRANCH_NAME)" - name: IMAGE_TAG value: "$(params.IMAGE_TAG)" script: | #!/usr/bin/env bash set -e cbisName=$(kubectl get cbis.v2.edp.epam.com -l app.edp.epam.com/codebasebranch="${CODEBASEBRANCH_NAME}" -o jsonpath='{.items[0].metadata.name}') if [ -z "${cbisName}" ]; then echo "[TEKTON][ERROR] No CBIS found with label app.edp.epam.com/codebasebranch=${CODEBASEBRANCH_NAME}" exit 1 fi cbisCrTags=$(kubectl get cbis.v2.edp.epam.com ${cbisName} --output=jsonpath={.spec.tags}) dateFormat=$(date -u '+%Y-%m-%dT%H:%M:%SZ') newcbisTag="{\"name\":\"${IMAGE_TAG}\",\"created\":\"${dateFormat}\"}" if [ "${cbisCrTags}" = "" ] ; then echo "[TEKTON][DEBUG] There're no tags in imageStream ${cbisName} ... the first one will be added." kubectl patch cbis.v2.edp.epam.com ${cbisName} --type=merge -p "{\"spec\":{\"tags\":[${newcbisTag}]}}" fi cbisTagsList=$(kubectl get cbis.v2.edp.epam.com ${cbisName} --output=jsonpath={.spec.tags[*].name}) if [[ ! ${cbisTagsList} == *"${IMAGE_TAG}"* ]]; then echo "[TEKTON][DEBUG] ImageStream ${cbisName} doesn't contain ${IMAGE_TAG} tag ... it will be added." kubectl patch cbis.v2.edp.epam.com ${cbisName} --type json -p="[{\"op\": \"add\", \"path\": \"/spec/tags/-\", \"value\": ${newcbisTag} }]" fi ```
2. Build Pipeline Task Condition Update: :::note This change is relevant only for build pipelines with `semver` versioning type. ::: In version 3.12, the execution condition for the `update-cbb` Tekton task has been [updated](https://github.com/epam/edp-tekton/issues/509) to ensure that the `Successful build` field for the Component branch is properly updated in the KubeRocketCI portal. Now, the `update-cbb` task will also be executed if build pipelines are triggered manually and have the `Completed` status. Update the `update-cbb` task condition in the Tekton custom build pipelines as follows: ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: build app.edp.epam.com/triggertemplate: github-build-template name: custom-build-semver spec: ... finally: - name: update-cbb params: - name: CODEBASEBRANCH_NAME value: $(params.CODEBASEBRANCH_NAME) - name: CURRENT_BUILD_NUMBER value: $(tasks.get-version.results.BUILD_ID) taskRef: kind: Task name: update-cbb when: - input: $(tasks.status) operator: in values: - Succeeded ``` ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: build app.edp.epam.com/triggertemplate: github-build-template name: custom-build-semver spec: ... finally: - name: update-cbb params: - name: CODEBASEBRANCH_NAME value: $(params.CODEBASEBRANCH_NAME) - name: CURRENT_BUILD_NUMBER value: $(tasks.get-version.results.BUILD_ID) taskRef: kind: Task name: update-cbb when: - input: $(tasks.status) operator: in values: - Succeeded - Completed ``` 3. Deploy Pipeline Parameters Renaming: :::note For detailed information about parameter changes in Tekton tasks, refer to the [edp-tekton](https://github.com/epam/edp-tekton/tree/master/charts/pipelines-library/templates/tasks) repository. ::: In version 3.12, the deprecated parameters `PIPELINE` and `STAGE` were [renamed](https://github.com/epam/edp-tekton/issues/503) in the Tekton tasks used in deployment pipelines to align with the KubeRocketCI portal naming conventions. Due to this change, it is necessary to update the passing parameters for the custom deploy pipelines that use the following Tekton tasks such as `init-autotests`, `clean`, `deploy-ansible-awx`, `deploy-ansible`, `deploy-applicationset-cli`, `run-quality-gate`, `run-clean-gate`, `sync-app`, and `promote-images`. For tasks such as `run-clean-gate`, `clean`, `run-quality-gate`, `deploy-ansible-awx`, `deploy-ansible`, `sync-app`, and `deploy-applicationset-cli`, it is necessary to rename the passing parameters from `PIPELINE` to `DEPLOYMENT_FLOW` and from `STAGE` to `ENVIRONMENT` in the Tekton custom deploy pipelines as follows: ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy app.edp.epam.com/triggertemplate: deploy name: custom-deploy-pipeline spec: ... tasks: - name: deploy-app params: - name: PIPELINE value: $(params.CDPIPELINE) - name: STAGE value: $(params.CDSTAGE) ... taskRef: kind: Task name: deploy-applicationset-cli ``` ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy app.edp.epam.com/triggertemplate: deploy name: custom-deploy-pipeline spec: ... tasks: - name: deploy-app params: - name: DEPLOYMENT_FLOW value: $(params.CDPIPELINE) - name: ENVIRONMENT value: $(params.CDSTAGE) ... taskRef: kind: Task name: deploy-applicationset-cli ``` For the `init-autotests` task, it is necessary to rename the passing parameters from `cd-pipeline-name` to `DEPLOYMENT_FLOW` and from `stage-name` to `ENVIRONMENT` in the Tekton custom deploy pipelines as follows: ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy app.edp.epam.com/triggertemplate: deploy name: custom-deploy-pipeline spec: ... tasks: - name: init-autotests params: - name: cd-pipeline-name value: $(params.CDPIPELINE) - name: stage-name value: $(params.CDSTAGE) ... taskRef: kind: Task name: init-autotests ``` ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy app.edp.epam.com/triggertemplate: deploy name: custom-deploy-pipeline spec: ... tasks: - name: init-autotests params: - name: DEPLOYMENT_FLOW value: $(params.CDPIPELINE) - name: ENVIRONMENT value: $(params.CDSTAGE) ... taskRef: kind: Task name: init-autotests ``` For the `promote-images` task, it is necessary to rename the passing parameters from `CDPIPELINE_CR` to `DEPLOYMENT_FLOW` and from `CDPIPELINE_STAGE` to `ENVIRONMENT` in the Tekton custom deploy pipelines as follows: ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy app.edp.epam.com/triggertemplate: deploy name: custom-deploy-pipeline spec: ... tasks: - name: promote-images params: - name: CDPIPELINE_CR value: $(params.CDPIPELINE) - name: CDPIPELINE_STAGE value: $(params.CDSTAGE) ... taskRef: kind: Task name: promote-images ``` ```yaml title="Pipeline" apiVersion: tekton.dev/v1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy app.edp.epam.com/triggertemplate: deploy name: custom-deploy-pipeline spec: ... tasks: - name: promote-images params: - name: DEPLOYMENT_FLOW value: $(params.CDPIPELINE) - name: ENVIRONMENT value: $(params.CDSTAGE) ... taskRef: kind: Task name: promote-images ``` 3. (Optional) Enable Repository Discovery: :::warning In case of using GitFusion with the Bitbucket Git provider, it is necessary to update the Bitbucket API token permissions to include the `read:account` scope. For more details on how to create a Bitbucket API token with the required permissions, refer to the [Add Git Server](../../user-guide/add-git-server.md) guide. ::: :::note For more information about the GitFusion microservice, refer to the [GitFusion](https://github.com/KubeRocketCI/gitfusion) repository. ::: :::note For more details about KrakenD integration with KubeRocketCI, refer to the [KrakenD documentation](https://www.krakend.io/docs/) guide. ::: Starting from version 3.12, KubeRocketCI supports integration with the [GitFusion](https://github.com/KubeRocketCI/gitfusion) microservice. This integration enables automatic discovery of repositories, branches, and organizations from various Git providers during the component or branch creation process in the KubeRocketCI portal. GitFusion act as a bridge between the KubeRocketCI portal and the Git provider, allowing the portal to access repository-related information without requiring direct access to the Git provider. To enable the GitFusion integration in KubeRocketCI, follow the steps below: 1. Enable the GitFusion dependency in the `values.yaml` file for `edp-install` Helm chart by setting the `gitfusion.enabled` parameter to `true`: ```yaml title="values.yaml" gitfusion: enabled: true ``` 2. Update the KrakenD configuration to include the GitFusion API endpoints: :::note The latest KrakenD configuration can be found in the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend) repository. ::: 1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. 2. Navigate to the `clusters/core/addons/krakend` directory and update the `values.yaml` file to include the GitFusion API endpoints:
values.yaml Add new GitFusion API endpoints to the `krakend.config` section: ```yaml krakend: config: | { "$schema": "https://www.krakend.io/schema/krakend.json", "version": 3, "name": "KrakenD - API Gateway", "timeout": "6000ms", "cache_ttl": "300s", "output_encoding": "json", "client_tls": { "@comment": "Skip SSL verification when connecting to backends", "allow_insecure_connections": true }, "extra_config": { "router": { "logger_skip_paths": [ "/__health" ] }, "auth/jwk-client": { "@comment": "Enable a JWK shared cache amongst all endpoints of 60 minutes", "shared_cache_duration": 3600 } }, "endpoints": [ { "endpoint": "/widgets/sonarqube/measures/component", "method": "GET", "output_encoding": "json", "input_query_strings": [ "component", "metricKeys" ], "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/measures/component", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "SONARQUBE_URL" }}" ], "extra_config": { "qos/http-cache": {}, "modifier/martian": { "header.Append": { "scope": [ "request" ], "name": "Authorization", "value": "Basic {{ env "SONARQUBE_TOKEN" }}" } } } } ] }, { "endpoint": "/widgets/deptrack/project", "method": "GET", "output_encoding": "json", "input_query_strings": [ "name" ], "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/project", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "DEPTRACK_URL" }}" ], "is_collection": true, "extra_config": { "qos/http-cache": {}, "modifier/martian": { "header.Append": { "scope": [ "request" ], "name": "X-Api-Key", "value": "{{ env "DEPTRACK_TOKEN" }}" } } } } ] }, { "endpoint": "/widgets/deptrack/metrics/project/{uuid}/current", "method": "GET", "output_encoding": "json", "input_query_strings": [ "name" ], "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/metrics/project/{uuid}/current", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "DEPTRACK_URL" }}" ], "is_collection": false, "extra_config": { "qos/http-cache": {}, "modifier/martian": { "header.Append": { "scope": [ "request" ], "name": "X-Api-Key", "value": "{{ env "DEPTRACK_TOKEN" }}" } } } } ] }, { "endpoint": "/search/logs", "method": "POST", "output_encoding": "json", "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/logstash-edp-*/_search", "method": "POST", "host": [ "{{ env "OPENSEARCH_URL" }}" ], "encoding": "json", "extra_config": { "qos/http-cache": {}, "modifier/martian": { "header.Append": { "scope": [ "request" ], "name": "Authorization", "value": "Basic {{ env "OPENSEARCH_CREDS" }}" } } } } ] }, { "endpoint": "/gitfusion/repositories", "method": "GET", "input_query_strings": ["*"], "output_encoding": "json", "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/repositories", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "GITFUSION_URL" }}" ], "extra_config": { "qos/http-cache": {} } } ] }, { "endpoint": "/gitfusion/repository", "method": "GET", "input_query_strings": ["*"], "output_encoding": "json", "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/repository", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "GITFUSION_URL" }}" ], "extra_config": { "qos/http-cache": {} } } ] }, { "endpoint": "/gitfusion/organizations", "method": "GET", "input_query_strings": ["*"], "output_encoding": "json", "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/user/organizations", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "GITFUSION_URL" }}" ], "extra_config": { "qos/http-cache": {} } } ] }, { "endpoint": "/gitfusion/branches", "method": "GET", "input_query_strings": ["*"], "output_encoding": "json", "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/branches", "encoding": "json", "sd": "static", "method": "GET", "host": [ "{{ env "GITFUSION_URL" }}" ], "extra_config": { "qos/http-cache": {} } } ] }, { "endpoint": "/gitfusion/invalidate", "method": "POST", "input_query_strings": ["*"], "output_encoding": "json", "extra_config": { "auth/validator": { "alg": "RS256", "cache_duration": 3600, "cache": true, "disable_jwk_security": false, "jwk_url": "{{ env "JWK_URL" }}" } }, "backend": [ { "url_pattern": "/api/v1/cache/invalidate", "encoding": "json", "sd": "static", "method": "DELETE", "host": [ "{{ env "GITFUSION_URL" }}" ], "extra_config": { "qos/http-cache": {} } } ] } ] } ```
3. Update the KrakenD secret with the GitFusion URL variable: :::note The `GITFUSION_URL` variable should point to the GitFusion service URL, e.g., `http://gitfusion.krci:8080`. ::: Update the `krakend` secret to include the `GITFUSION_URL` variable: ```yaml title="KrakenD secret" kind: Secret apiVersion: v1 metadata: name: krakend namespace: krakend data: ... GITFUSION_URL: http://gitfusion.krci:8080 type: Opaque ``` In case of using the External Secrets Operator with AWS Parameter Store, update the Parameter Store object to include the `GITFUSION_URL` variable: ```yaml title="AWS Parameter Store" { "SONARQUBE_URL": "http://sonar.sonar:9000", "SONARQUBE_TOKEN": "", "JWK_URL": "https://keycloak.example.com/realms//protocol/openid-connect/certs", "OPENSEARCH_URL": "https://opensearch-cluster-master.logging:9200", "OPENSEARCH_CREDS": "opensearch-base64-encoded-credentials", "GITFUSION_URL": "http://gitfusion.krci:8080" } ``` 4. Commit and push the changes to the `edp-cluster-add-ons` repository. After the changes are pushed, navigate to the Argo CD and sync the KrakenD application to apply the updated configuration. After upgrading KubeRocketCI to version 3.12, the GitFusion microservice will be automatically used for repository discovery in the KubeRocketCI portal. Example of repository discovery in the KubeRocketCI portal during the component onboarding process: ![GitFusion](../../assets/operator-guide/upgrade/gitfusion.png) 4. (Optional) Align Remote Cluster Names: In case of using remote clusters in KubeRocketCI, it is necessary to update the remote cluster names in KubeRocketCI portal after upgrading to version 3.12. After the upgrade, all remote cluster names will display as `default-cluster` in the KubeRocketCI portal. This happens because starting from version 3.12, the portal retrieves cluster names from the `kubeconfig` specification stored in the `` Kubernetes secret, where `default-cluster` is the default name of the cluster in the `kubeconfig` specification. ![Remote Clusters](../../assets/operator-guide/upgrade/remote-clusters.png) There are two ways to align the remote cluster names in the KubeRocketCI portal: 1. Recreate the remote cluster integration in the KubeRocketCI portal: 1. Navigate to the **Configuration** -> **Deployment** -> **Clusters** section in the KubeRocketCI portal. 2. Click on the cluster integration that needs to be updated and delete it by clicking the **Delete** (trash can) icon: ![Delete Cluster](../../assets/operator-guide/upgrade/delete-cluster.png) Confirm the deletion in the pop-up window. 3. After the cluster integration is deleted, click the **Add Cluster** button to create a new cluster integration. Fill in the required fields and click the **Save** button to add the cluster: ![Add Cluster](../../assets/operator-guide/upgrade/add-cluster.png) After the new cluster integration is created, the correct cluster name will be displayed in the KubeRocketCI portal. 2. Update the `kubeconfig` specification in the `` Kubernetes secret: :::note The `` secret is created automatically when a new cluster integration is added in the KubeRocketCI portal. The secret contains the `kubeconfig` specification used to connect to the remote cluster. ::: It is also possible to update the `kubeconfig` specification in the existing `` secret to change the cluster name, instead of recreating the cluster integration in the KubeRocketCI portal. 1. Locate the `` secret in the namespace where KubeRocketCI is installed (e.g., `krci` namespace): ```bash kubectl get secret -n krci -o yaml ``` 2. Update the `kubeconfig` specification by changing the `clusters.name` and `contexts.context.cluster` fields to match the desired cluster name: :::note The `data.config` field in the secret is base64 encoded. To update the `kubeconfig` specification, decode the `data.config` field, make the necessary changes, and then encode it back to base64 before updating the secret. ::: Example of the `kubeconfig` specification in the `` secret: ```yaml title="kubeconfig" { "apiVersion": "v1", "kind": "Config", "current-context": "...", "preferences": {}, "clusters": [ { "cluster": { "server": "...", "certificate-authority-data": "..." }, "name": "" # Change this value to the desired cluster name } ], "contexts": [ { "context": { "cluster": "", # Change this value to the desired cluster name "user": "..." }, "name": "..." } ], "users": [ { "user": { "token": "..." }, "name": "..." } ] } ``` After updating the `kubeconfig` specification in the secret, the correct cluster name will be displayed in the KubeRocketCI portal. 5. To upgrade KubeRocketCI to the v3.12, run the following commands: :::note To verify the installation, it is possible to test the deployment before applying it to the cluster with the `--dry-run` key: `helm upgrade krci epamedp/edp-install -n krci --values values.yaml --version=3.12.4 --dry-run` ::: ```bash helm repo update epamedp helm upgrade krci epamedp/edp-install -n krci --values values.yaml --version=3.12.4 ``` --- ## Upgrade KubeRocketCI v3.12 to 3.13 # Upgrade KubeRocketCI v3.12 to 3.13 This section provides detailed instructions for upgrading KubeRocketCI to version 3.13. Follow the steps and requirements outlined below: :::important We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: :::important **GitFusion** is now a mandatory platform dependency (`gitfusion.enabled: true` by default). It is required by `krci-portal` for repository and branch selection, Pull Request / Merge Request browsing, GitLab pipeline triggering, and pipeline listing. Ensure GitFusion is reachable from the portal pod and that `GITFUSION_URL` is set in `krci-portal` configuration. ::: :::important **Tekton Results** must be deployed and `TEKTON_RESULTS_URL` must be configured for the pipeline history page, live logs, and activity charts in `krci-portal`. The portal starts without it, but all pipeline history pages return errors until this is configured. ::: :::warning All Tekton resources (Tasks, Pipelines, TriggerTemplates) have been migrated from `tekton.dev/v1beta1` to `tekton.dev/v1`. Clusters running Tekton Pipelines older than **v0.44** cannot apply these manifests — the upgrade will fail. Verify your Tekton Pipelines version before proceeding (see [Step 1](#step-1-required-verify-tekton-pipelines-version)). ::: :::warning Three parameter names were renamed in the security scan tasks (`security`, `image-scan`, `image-scan-remote`). If you maintain **custom security TriggerTemplates** that pass the old names, security findings will be uploaded to DefectDojo under a blank product name or the upload will fail. Build and review pipelines are **not affected**. See [Step 2](#step-2-optional-update-custom-security-triggertemplates) for detection and migration. ::: ## Step 1. (Required) Verify Tekton Pipelines Version KubeRocketCI 3.13 migrates all Tekton resources to the `tekton.dev/v1` API, which became GA in Tekton Pipelines **v0.44**. Clusters running an older version cannot apply the updated manifests. Check your current Tekton Pipelines version: ```bash title="Check Tekton Pipelines version" kubectl get deployment tekton-pipelines-controller -n tekton-pipelines \ -o jsonpath='{.spec.template.spec.containers[0].image}' ``` If the image tag is older than `v0.44.0`, upgrade Tekton Pipelines before running the Helm upgrade. Refer to the [Tekton Pipelines release page](https://github.com/tektoncd/pipeline/releases) for upstream upgrade instructions, or deploy the managed Tekton stack from the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/tekton) repository. ## Step 2. (Optional) Update Custom Security TriggerTemplates :::note This step only affects users who maintain **custom security TriggerTemplates** that pass parameters to the `security`, `image-scan`, or `image-scan-remote` tasks. Standard build and review pipelines are not affected — they already used `CODEBASE_NAME`. ::: Three parameter names were renamed in the security scan tasks to align naming conventions across the platform: | Old name | New name | Affected tasks | |------------------|-------------------|-------------------------------------------------| | `PROJECT_NAME` | `CODEBASE_NAME` | `security`, `image-scan` | | `COMPONENT_NAME` | `CODEBASE_NAME` | `image-scan-remote` | | `PROJECT_BRANCH` | `CODEBASE_BRANCH` | `security` | Run the following command to detect affected custom TriggerTemplates in your cluster: ```bash title="Detect custom TriggerTemplates with old parameter names" kubectl get triggertemplates -A -o yaml | grep -E "COMPONENT_NAME|PROJECT_NAME|PROJECT_BRANCH" ``` If this returns results, update the affected TriggerTemplates as shown below: ```yaml title="Custom security TriggerTemplate (3.12)" apiVersion: triggers.tekton.dev/v1beta1 kind: TriggerTemplate metadata: name: custom-github-security-template spec: params: - name: gitrevision - name: reponame resourcetemplates: - apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata: generateName: github-security-$(tt.params.reponame)- spec: pipelineRef: name: github-security-scan params: - name: git-source-url value: $(tt.params.gitcloneurl) - name: git-source-revision value: $(tt.params.gitrevision) - name: PROJECT_NAME value: $(tt.params.reponame) - name: PROJECT_BRANCH value: $(tt.params.gitrevision) ``` ```yaml title="Custom security TriggerTemplate (3.13)" apiVersion: triggers.tekton.dev/v1beta1 kind: TriggerTemplate metadata: name: custom-github-security-template spec: params: - name: gitrevision - name: reponame resourcetemplates: - apiVersion: tekton.dev/v1 kind: PipelineRun metadata: generateName: github-security-$(tt.params.reponame)- spec: pipelineRef: name: github-security-scan params: - name: git-source-url value: $(tt.params.gitcloneurl) - name: git-source-revision value: $(tt.params.gitrevision) - name: CODEBASE_NAME value: $(tt.params.reponame) - name: CODEBASE_BRANCH value: $(tt.params.gitrevision) ``` ## Step 3. (Required if applicable) Remove CodeMie Configuration CodeMie integration has been removed from both `edp-install` and `codebase-operator`. There is no migration path. Check whether your current values include CodeMie configuration: ```bash title="Check for CodeMie in current values" helm get values -n | grep -A5 -i codemie ``` If the command returns any output, remove the `codemie` block from your `values.yaml` before running the upgrade. Leaving it in place will cause the Helm upgrade to fail with an unknown values error. ## Step 4. (Required if applicable) Fix Non-Standard ciTool Values on Codebase CRs The `ciTool` field on the `Codebase` CRD now enforces an enum: only `tekton` and `gitlab` are accepted. Existing CRs with other values (such as `jenkins` or an empty string) will continue to exist after the upgrade, but any subsequent `kubectl apply` or edit on those CRs will be rejected. Detect affected Codebase CRs: ```bash title="Detect Codebase CRs with non-standard ciTool values" kubectl get codebases -A \ -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.ciTool}{"\n"}{end}' \ | grep -v ": tekton" | grep -v ": gitlab" ``` For each affected Codebase, patch `ciTool` to a valid value: ```bash title="Patch ciTool to a valid value" kubectl patch codebase -n --type=merge \ -p '{"spec":{"ciTool":"tekton"}}' ``` ## Step 5. (Required) Deploy Tekton Results Tekton Results provides the backend for the pipeline history page, live logs, and activity charts in `krci-portal`. It must be deployed before the portal upgrade. Tekton Results requires a **PostgreSQL 14+** database. You are free to use any PostgreSQL provider: - **In-cluster via PGO** (Crunchy Data Postgres Operator) — recommended for self-managed clusters; add-on manifests are provided in `edp-cluster-add-ons` - **Cloud-managed database** — AWS RDS, Azure Database for PostgreSQL, Google Cloud SQL, or any other managed PostgreSQL service ### Option A: In-cluster PostgreSQL via PGO If the Crunchy Data Postgres Operator is not yet installed, deploy it from [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/tekton) first, then apply the Tekton Results manifests: ```bash title="Deploy Tekton Results with PGO" kubectl apply -f clusters/core/addons/tekton/results-pg.yaml kubectl apply -f clusters/core/addons/tekton/results.yaml kubectl apply -f clusters/core/addons/tekton/results-clean-old-logs-cronjob.yaml ``` Wait for PostgreSQL to be ready: ```bash title="Wait for PostgreSQL pod" kubectl wait --for=condition=Ready pod \ -l postgres-operator.crunchydata.com/cluster=results \ -n tekton-pipelines --timeout=300s ``` ### Option B: Cloud-managed PostgreSQL Create a PostgreSQL 14+ database on your cloud provider of choice, then configure Tekton Results to connect to it by supplying the connection string in the Tekton Results deployment. Refer to the [Tekton Results documentation](https://github.com/tektoncd/results) for the database configuration parameters. ### Verify Tekton Results is running ```bash title="Verify Tekton Results pods" kubectl get pods -n tekton-pipelines | grep tekton-results ``` The internal service URL to use in `TEKTON_RESULTS_URL` (for in-cluster portal deployments): ``` http://tekton-results-api-service.tekton-pipelines:8080 ``` :::important **Create performance indexes on the Tekton Results database.** The default schema has no indexes beyond primary keys, causing sequential scans that degrade performance under load. After the database is ready, apply the required indexes. See the [Tekton Results PostgreSQL performance indexes guide](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/README.md) in `edp-cluster-add-ons` for the exact SQL statements and instructions for both PGO and cloud-managed databases. ::: ## Step 6. (Required) Migrate from edp-headlamp to krci-portal `edp-headlamp` is disabled by default in 3.13 and replaced by `krci-portal`. All portal configuration must be migrated to the new `krci-portal` values. ### 6.1. Update OIDC Client Registration You can **reuse the existing OIDC client** registered for `edp-headlamp`. The client ID and client secret remain the same. Update the redirect URIs to match the new portal paths: | Setting | Old value (edp-headlamp) | New value (krci-portal) | |--------------------------|-------------------------------------------------|-------------------------------------------------| | Redirect URI | `https:///oidc-callback` | `https:///auth/callback` | | Post-logout Redirect URI | `https:///` | `https:///login` | | PKCE | Not required | Enable **S256** (see note below) | | Scopes | `openid`, `profile`, `email` | `openid`, `profile`, `email` *(unchanged)* | If the identity provider reports **invalid redirect URI** on login or logout, register additional valid redirect URIs as needed — for example `https://`, `https:///auth/callback`, and `https:///login`. As a single catch-all in Keycloak (and similar providers that allow wildcards), you can use `https:///*`. :::warning If the OIDC client is reconciled from Git by the Keycloak operator (for example via `kuberocketci-rbac` in [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/kuberocketci-rbac/templates/kubernetes/keycloak-client.yaml)), manual edits in the Keycloak UI will be overwritten. Update `redirectUris` (and related fields) in that manifest — or your fork/overlay — so the desired URIs persist across reconciles. ::: :::note **PKCE** (Proof Key for Code Exchange, RFC 7636) binds the authorization code to the client that started the login, which reduces risk if an authorization code is intercepted. The portal is configured with `OIDC_CODE_CHALLENGE_METHOD: S256` (SHA-256 code challenge). In your IdP, enable PKCE for this client and use the **S256** method where the UI offers it. ::: :::note **Azure AD** users: the portal automatically normalizes Azure AD's array-of-JSON-strings groups format — no additional provider configuration is needed. ::: ### 6.2. Create the portal-secret Kubernetes Secret The portal requires a Kubernetes Secret named `portal-secret` to exist before the pod starts: ```bash title="Create portal-secret" kubectl create secret generic portal-secret \ --namespace \ --from-literal=SERVER_SECRET="$(openssl rand -base64 32)" \ --from-literal=OIDC_CLIENT_SECRET="" ``` To also enable the SonarQube SAST and Dependency Track SCA pages, add their credentials: ```bash title="Create portal-secret with optional integrations" kubectl create secret generic portal-secret \ --namespace \ --from-literal=SERVER_SECRET="$(openssl rand -base64 32)" \ --from-literal=OIDC_CLIENT_SECRET="" \ --from-literal=SONAR_TOKEN="squ_xxxxxxxxxxxxxxxxxxxx" \ --from-literal=DEPENDENCY_TRACK_API_KEY="odt_xxxxxxxxxxxxxxxxxxxx" ``` | Key | Purpose | Required | |---------------------------|--------------------------------------------------------------------------------------|----------| | `SERVER_SECRET` | Session cookie encryption (min 32 chars). Changing this invalidates active sessions. | Yes | | `OIDC_CLIENT_SECRET` | OIDC client secret from your identity provider | Yes | | `SONAR_TOKEN` | SonarQube personal/analysis token (`Browse` + `View Issues` permissions sufficient) | Optional | | `DEPENDENCY_TRACK_API_KEY`| From Dependency Track → Administration → API Keys | Optional | :::note If using the External Secrets Operator (`eso.enabled: true`), the chart creates an `ExternalSecret` instead of reading this Secret directly. Ensure the secret path `eso.secretPath` (default: `/infra/core/addons/krci-portal`) contains all required keys. ::: ### 6.3. Update values.yaml Add the `krci-portal` section, configure GitFusion and Tekton Results URLs, and disable `edp-headlamp`: ```yaml title="values.yaml" # Disable the deprecated portal edp-headlamp: enabled: false # GitFusion — mandatory dependency, required by krci-portal gitfusion: enabled: true # New portal (replaces edp-headlamp) krci-portal: enabled: true ingress: enabled: true className: nginx annotations: cert-manager.io/cluster-issuer: letsencrypt-prod hosts: - host: portal.example.com paths: - path: / pathType: Prefix tls: - secretName: portal-tls hosts: - portal.example.com configEnv: API_PREFIX: /api SERVER_PORT: "3000" PORTAL_URL: https://portal.example.com # must match ingress host exactly OIDC_ISSUER_URL: https://keycloak.example.com/realms/shared OIDC_CLIENT_ID: portal # reuse existing edp-headlamp client ID OIDC_SCOPE: openid profile email OIDC_CODE_CHALLENGE_METHOD: S256 TEKTON_RESULTS_URL: http://tekton-results-api-service.tekton-pipelines:8080 GITFUSION_URL: http://gitfusion.:8080 DEFAULT_CLUSTER_NAME: core DEFAULT_CLUSTER_NAMESPACE: # Optional — enables SAST/SCA pages: SONAR_HOST_URL: https://sonar.example.com/ # trailing slash required SONAR_WEB_URL: https://sonar.example.com/ DEPENDENCY_TRACK_URL: https://deptrack.example.com DEPENDENCY_TRACK_WEB_URL: https://deptrack.example.com ``` :::warning `PORTAL_URL` in `configEnv` must exactly match the ingress hostname. A mismatch causes OIDC redirect URI validation to fail and users cannot log in. ::: :::note `SONAR_HOST_URL` requires a trailing slash. `DEPENDENCY_TRACK_URL` does not. ::: ## Step 7. (Informational) DefectDojo Engagement Naming Change Security scan engagements are now isolated per branch. The engagement name format has changed: | Scan type | Old format (3.12) | New format (3.13) | |----------------------|----------------------|------------------------------------------------| | Code security | `code-security` | `code-security-` (e.g. `code-security-main`) | | Container image scan | `container-security` | `image:` (e.g. `image:myapp`) | On the first pipeline run after upgrade, DefectDojo will create new per-branch engagements. Historical findings under the old unified engagement names are automatically closed (`close_old_findings=true`) — they are **not deleted** and remain as an audit trail. If you have DefectDojo dashboards or reports referencing the old engagement names (`code-security`, `container-security`), update them to the new per-branch format. ## Step 8. Upgrade The recommended approach is to manage the upgrade through [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) via ArgoCD. Update the chart version and values in your add-ons repository, then let ArgoCD apply the changes. **1.** In your `edp-cluster-add-ons` repository, update the chart version in `clusters/core/addons/kuberocketci/Chart.yaml`: ```yaml title="clusters/core/addons/kuberocketci/Chart.yaml" apiVersion: v2 name: edp-install version: 3.13.5 appVersion: 3.13.5 dependencies: - name: edp-install version: 3.13.5 repository: https://epam.github.io/edp-helm-charts/stable ``` **2.** Apply the values changes from [Step 6.3](#63-update-valuesyaml) to `clusters/core/addons/kuberocketci/values.yaml`. **3.** Commit and push the changes. ArgoCD will detect the diff and sync automatically, or trigger a manual sync from the ArgoCD UI. For environments managed directly with Helm: ```bash title="Run the Helm upgrade" # Update the Helm repository helm repo update # Confirm the target chart version is available helm search repo epamedp/edp-install --versions | grep 3.13.5 # Preview changes against your values helm diff upgrade epamedp/edp-install \ --version 3.13.5 \ -f values.yaml \ -n # Run the upgrade helm upgrade --install epamedp/edp-install \ --version 3.13.5 \ -f values.yaml \ -n \ --timeout 10m \ --wait ``` The upgrade applies updated CRDs, deploys `krci-portal`, updates GitFusion and all operators, and disables `edp-headlamp`. Allow 3–5 minutes for all pods to reach Running state. ## Step 9. Post-Upgrade Verification Run the following commands to verify the upgrade: ```bash title="Verify pods and deployments" # Check all pods are running kubectl get pods -n # Verify Tekton v1 resources were applied kubectl get pipelines -A | head -10 # Verify krci-portal is running kubectl get deployment -n | grep krci-portal # Verify gitfusion is running kubectl get deployment -n | grep gitfusion ``` Then verify in the browser: - [ ] Portal loads at `https://` - [ ] OIDC authentication completes (redirects back to portal after login) - [ ] **Projects** page is visible (formerly "Components") - [ ] Pipeline history page loads (requires Tekton Results) - [ ] A deployed application shows image digest in environment details - [ ] Pull Request / Merge Request tab visible on a project (requires GitFusion) - [ ] DefectDojo: after the first security pipeline run, new per-branch engagements appear (e.g. `code-security-main`) - [ ] SonarQube SAST page loads (if `SONAR_HOST_URL` is configured) - [ ] Dependency Track SCA page loads (if `DEPENDENCY_TRACK_URL` is configured) --- ## Upgrade KubeRocketCI v3.13 to 3.14 # Upgrade KubeRocketCI v3.13 to 3.14 This section provides detailed instructions for upgrading KubeRocketCI to version 3.14. Follow the steps and requirements outlined below: :::important We suggest backing up the KubeRocketCI environment before starting the upgrade procedure. ::: :::note **3.14 is a feature release.** Upgrading from 3.13 requires **no forced resource migrations** and introduces **no breaking Custom Resource schema or label changes**. The platform dependencies added in 3.13 — cert-manager for the operator admission webhooks, GitFusion, Tekton Results, and Tekton Pipelines with the `tekton.dev/v1` API — are unchanged and are already in place if you are on 3.13.x. Most of what is new in 3.14 is **opt-in**. ::: :::note **Envoy Gateway support is optional and disabled by default.** A cluster without Envoy Gateway or the Gateway API CRDs upgrades to 3.14 without any changes — by default the platform renders standard nginx `Ingress` resources and **zero** `HTTPRoute` objects. Envoy Gateway is enabled explicitly, per component, as described in [Step 4](#step-4-optional-enable-envoy-gateway-and-httproute-exposure). ::: ## Step 1. (Required) Verify Prerequisites Confirm the platform is currently on a **3.13.x** release before upgrading. The `CHART` column shows the installed `edp-install` version: ```bash title="Check the installed chart version" helm list -n ``` The following platform dependencies are carried over from 3.13 and must remain in place. If you are upgrading from 3.13.x they are already satisfied — the commands below are a quick confirmation: ```bash title="Verify carried-over dependencies" # cert-manager — required for the codebase-operator and cd-pipeline-operator admission webhooks kubectl get crd certificates.cert-manager.io # Tekton Pipelines — must expose the tekton.dev/v1 API (GA since v0.44); 3.14 is validated with v1.6.2 kubectl get deployment tekton-pipelines-controller -n tekton-pipelines \ -o jsonpath='{.spec.template.spec.containers[0].image}'; echo # GitFusion and Tekton Results — required by krci-portal (repository browsing and pipeline history) kubectl get deployment -n | grep -E 'gitfusion|krci-portal' ``` :::note Tekton Pipelines is bumped to the tested version **1.6.2** and Tekton cache to **0.4.5** in 3.14. No API migration is required — the `tekton.dev/v1` API introduced in 3.13 is unchanged. When Tekton is managed through [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/tekton), these versions are applied for you. ::: ## Step 2. (Recommended) Configure `clusterName` for Pipeline URLs 3.14 adds a `clusterName` value used to build correct `krci-portal` pipeline URLs (`/c//...`). Set it to the same value as the portal's `DEFAULT_CLUSTER_NAME`. If left empty, it falls back to the first segment of `global.dnsWildCard`, which may not match. ```yaml title="values.yaml" global: # Must match krci-portal.configEnv.DEFAULT_CLUSTER_NAME clusterName: core edp-tekton: # Same value; used to construct pipeline links surfaced by Tekton clusterName: core ``` :::note You can skip this step only if the first segment of `global.dnsWildCard` already equals your `DEFAULT_CLUSTER_NAME`. Setting `clusterName` explicitly is safe and recommended. ::: ## Step 3. (Optional) Enable Cancel-in-Progress for Review Pipelines 3.14 can automatically cancel a review PipelineRun when a newer commit arrives on the same Pull Request / Merge Request / Gerrit change, saving executor capacity. Cancellation is graceful (`spec.status: CancelledRunFinally`) — the `finally` tasks of the superseded run still execute and report status back to the git provider. It is **disabled by default**. ```yaml title="values.yaml" edp-tekton: pipelines: cancelInProgress: true ``` ## Step 4. (Optional) Enable Envoy Gateway and HTTPRoute Exposure 3.14 introduces optional **Envoy Gateway** exposure for Tekton EventListeners, GitServer webhooks, application Helm chart scaffolding, and the portal Networking tab. It is **off by default**, and the existing nginx `Ingress` path is unchanged. Skip this step entirely to keep nginx. To opt in, a Gateway API-capable Envoy Gateway must already be deployed (for example, via [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons)), then enable it per component: ```yaml title="values.yaml" global: # Parent Gateway that HTTPRoutes attach to; the Gateway itself must already exist gatewayApi: gatewayName: main-gateway gatewayNamespace: envoy-gateway-system codebase-operator: # Register new GitServer webhooks against the HTTPRoute host instead of the Ingress host ingressController: envoy edp-tekton: gitServers: my-github: # ...existing gitServer configuration... eventListener: ingress: enabled: true # kept as a fallback during cutover httproute: enabled: true # opt-in HTTPRoute for this EventListener ``` :::note The cutover is reversible and per-gitServer. The `Ingress` continues to render as a fallback while `httproute.enabled: true`, so the webhook keeps working during cutover. To roll back, set `ingressController: nginx` and `httproute.enabled: false` — the `HTTPRoute` disappears on the next reconcile and the `Ingress` remains the active path. External reachability additionally requires the edge (ALB or load balancer) to route the host to Envoy. ::: ## Step 5. (Optional) Enable Kubernetes Audit Trail The portal **Admin → Audit Events** page and the **Triggered By** field on PipelineRun details depend on the **krci-audit** add-on. The add-on is **disabled by default**. Upgrading to 3.14 does not enable it automatically. To turn it on: 1. Follow [Audit Trails Setup](../monitoring-and-observability/audit-trails-setup.md): create the database credentials Secret, set `krci-audit.enable: true` in your add-ons values, and sync the add-on. 2. Ensure the portal can reach the read API. The default in-cluster URL is: ```yaml title="krci-portal values (configEnv)" configEnv: KRCI_AUDIT_URL: http://krci-audit-api.krci-audit:8080 ``` 3. To open **Administration → Audit Events**, sign in with **OIDC** as a user in a Keycloak group mapped to the portal `administrator` role (default group name: `administrator`, overridable with `PORTAL_ADMIN_GROUPS`). Service Account token sessions do not receive portal admin roles, so that menu is hidden for token login. Skip this step if you do not need an in-platform audit trail yet. Without krci-audit, **Triggered By** shows **N/A** for runs that have no initiator in the trail. ## Step 6. (Optional) Enable In-App Notifications 3.14.1 adds an in-app notifications hub (header bell, unread badge, toasts) for platform events such as failed PipelineRuns. The UI can appear after upgrade, but ingestion stays off until you configure it. 1. Add a shared secret to the portal Secret wired through `extraEnvFrom` (see the portal chart `secrets-example`): ```yaml title="portal-secret (excerpt)" stringData: INTERNAL_EVENTS_TOKEN: "" ``` 2. Deploy or update an in-cluster **Argo Events** Sensor that `POST`s to the portal internal events endpoint and sends the same value in the `x-internal-events-token` header. Until `INTERNAL_EVENTS_TOKEN` is set, the portal responds `503` on that endpoint and no events are stored. 3. After rotating the token, update both the portal Secret and the Sensor copy, then restart the portal Deployment so the new environment variable is loaded. :::note Notifications are an **alpha** capability: storage is local SQLite on a single portal replica, and every authenticated portal user currently sees the same event stream. Skip this step if you are not ready to operate the Sensor and token. ::: ## Step 7. (Informational) Review Pipeline Reporter From 3.14.1, finished review PipelineRuns can publish a self-updating pull or merge request comment (per-task status table plus trailing logs of failed steps) via **tekton-reporter** inside `edp-tekton` / the pipelines-library chart. - The reporter ships **enabled by default** (`reporter.enabled: true`). No upgrade action is required to keep the feature. - To disable it, set: ```yaml title="values.yaml" edp-tekton: # Path may be under the pipelines-library / edp-tekton values your umbrella chart exposes reporter: enabled: false ``` - Comment links into the portal use `portalHost` and `clusterName`. Set `clusterName` as in [Step 2](#step-2-recommended-configure-clustername-for-pipeline-urls) so those URLs resolve correctly. Published logs come from steps that run pull request code. Secret masking is best-effort; enable the reporter only where that risk is acceptable for your pipelines. ## Step 8. (Informational) New Portal Capabilities `krci-portal` `0.6.0` (and the 3.14.1 portal updates) add the following capabilities. Most require **no upgrade action** beyond installing the new chart version: - **Service Account token login** — operators can sign in with a Kubernetes Service Account token; OIDC configuration is now optional. - **Expanded Kubernetes mode** — redesigned cluster overview, Custom Resource and CRD browsing for users without cluster-wide CRD access, and scale / restart / rollback actions for Deployments, StatefulSets, and DaemonSets. - **GitLab CI** — a dedicated pipeline list and log viewer for codebases with `ciTool: gitlab`. - **Networking tab** on stage details showing Gateways, HTTPRoutes, and Ingresses, plus HTTPRoute-derived external URLs in the Applications table. HTTPRoute data is most useful when [Step 4](#step-4-optional-enable-envoy-gateway-and-httproute-exposure) is enabled. - **Monitoring tab** on PipelineRun details (per-step CPU and memory from Prometheus). Set `krci-portal.configEnv.PROMETHEUS_URL` to your in-cluster Prometheus base URL (for example `http://prometheus.monitoring.svc:9090`). If unset, the tab returns a precondition error. - **Current deployed version** in the CD stage deploy dropdown, a **branch** column on the applications table, and a **stale** badge for branches missing in git. **Admin → Audit Events** and **Triggered By** are covered in [Step 5](#step-5-optional-enable-kubernetes-audit-trail). The notifications hub is covered in [Step 6](#step-6-optional-enable-in-app-notifications). ## Step 9. (Informational) Resource Migrations **No Custom Resource migrations or configuration-schema / label changes are required for this upgrade.** `Codebase`, `CodebaseBranch`, `CDPipeline`, `Stage`, Tekton resources, and Argo CD `ApplicationSet` definitions created under 3.13 remain valid under 3.14. The 3.14 chart applies the updated CRDs in place, and existing resources are reconciled without manual conversion. If you maintain **custom** Tekton pipelines or resource templates, no parameter or API changes are required in 3.14. The `tekton.dev/v1` migration and the security-task parameter renames were completed in 3.13 — see [Upgrade v3.12 to 3.13](./upgrade-krci-3.13.md). ## Step 10. (Informational) Breaking Changes and Deprecations There are **no new breaking changes** in 3.14. Deprecations carried over from 3.13 remain in effect: - `edp-headlamp` stays disabled and is replaced by `krci-portal`. If you completed the 3.13 portal migration, no further action is needed. ## Step 11. Upgrade The recommended approach is to manage the upgrade through [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) via ArgoCD. Update the chart version and values in your add-ons repository, then let ArgoCD apply the changes. **1.** In your `edp-cluster-add-ons` repository, update the chart version in `clusters/core/addons/kuberocketci/Chart.yaml`: ```yaml title="clusters/core/addons/kuberocketci/Chart.yaml" apiVersion: v2 name: edp-install version: 3.14.1 appVersion: 3.14.1 dependencies: - name: edp-install version: 3.14.1 repository: https://epam.github.io/edp-helm-charts/stable ``` **2.** Apply any optional values changes from Steps 2-7 to `clusters/core/addons/kuberocketci/values.yaml` (and enable `krci-audit` in your add-ons apps values if you follow [Step 5](#step-5-optional-enable-kubernetes-audit-trail)). **3.** Commit and push the changes. ArgoCD will detect the diff and sync automatically, or trigger a manual sync from the ArgoCD UI. For environments managed directly with Helm: ```bash title="Run the Helm upgrade" # Update the Helm repository helm repo update # Confirm the target chart version is available helm search repo epamedp/edp-install --versions | grep 3.14.1 # Preview changes against your values helm diff upgrade epamedp/edp-install \ --version 3.14.1 \ -f values.yaml \ -n # Run the upgrade helm upgrade --install epamedp/edp-install \ --version 3.14.1 \ -f values.yaml \ -n \ --timeout 10m \ --wait ``` The upgrade applies the updated CRDs and rolls the operators, `edp-tekton`, `gitfusion`, and `krci-portal` to their 3.14 versions. Allow 3-5 minutes for all pods to reach the Running state. ## Step 12. Post-Upgrade Verification ```bash title="Verify pods and component versions" # All pods should return to Running kubectl get pods -n # Confirm the operators, portal, and tekton are on their 3.14 image tags kubectl get deployment -n \ -o custom-columns=NAME:.metadata.name,IMAGE:'.spec.template.spec.containers[0].image' \ | grep -E 'codebase-operator|cd-pipeline-operator|krci-portal|gitfusion|tekton' ``` Then verify in the browser: - [ ] Portal loads at `https://` and authentication completes - [ ] Pipeline history and live logs load (requires Tekton Results) - [ ] Pull Request / Merge Request browsing works (requires GitFusion) - [ ] Pipeline URLs open the correct cluster path `/c//...` (see [Step 2](#step-2-recommended-configure-clustername-for-pipeline-urls)) - [ ] A superseded review pipeline run cancels correctly — only if `cancelInProgress: true` was enabled (see [Step 3](#step-3-optional-enable-cancel-in-progress-for-review-pipelines)) - [ ] (If Envoy Gateway was enabled) the stage details **Networking** tab lists Gateways and HTTPRoutes, and the webhook still reaches the EventListener - [ ] (If krci-audit was enabled) **Triggered By** resolves on a new PipelineRun, and an **OIDC administrator** can open **Administration → Audit Events** (see [Step 5](#step-5-optional-enable-kubernetes-audit-trail)) - [ ] (If notifications were enabled) the header bell receives a test or real platform event (see [Step 6](#step-6-optional-enable-in-app-notifications)) - [ ] (If the review reporter is enabled) a finished review PipelineRun updates the pull or merge request comment (see [Step 7](#step-7-informational-review-pipeline-reporter)) --- ## Create Application # Create Application In KubeRocketCI, all software components, such as applications, libraries, Terraform infrastructures, and automated tests, are referred to as codebases. KubeRocketCI provides flexible methods for scaffolding these components. This guide will walk you through the process of creating a Go application using the Gin framework. The [Marketplace](../user-guide/marketplace.md) will be used to streamline the application creation process. ## Application Onboarding To create the first application, complete the instructions below: 1. In the KubeRocketCI, navigate to **Projects**. 2. In the top right corner of the screen, click **+ Create project**. 3. In the **Select Ready Template** section, select **Web Applications with Gin Framework** and click **Continue ->**: ![Marketplace applications](../assets/quick-start/marketplace_application.png "Marketplace applications") 4. In the **Git and Project info** stage, define the following values and click **Continue**: - Git server: `github` - Owner: `` - Repository name: `my-go-gin-app` - Default branch: `main` - Project name: `my-go-gin-app` - Description: `My first application` - Private: `disabled` ![Application blank](../assets/quick-start/git-project-info-stage.png "Application blank") 5. In the **Build Config** stage, define the following values and click **Continue**: - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Application blank](../assets/quick-start/build-config-stage.png "Application blank") 6. In the **Review** stage, verify the configuration and click **Create Project**: ![Application blank](../assets/quick-start/review-stage.png "Application blank") 7. As soon as the codebase is created, click the **Open Project** button: ![Marketplace notification](../assets/quick-start/success_notification.png "Marketplace notification") ## Build Application Having created the Go application, proceed to build it by performing the following actions: 1. Open SonarCloud page. 2. On the [account settings](https://sonarcloud.io/account/organizations), copy your SonarCloud **organization key** (lowercase identifier used in API and properties): ![Organizations page](../assets/quick-start/sonar_org_page.png "Organizations page") 3. In SonarCloud, open the project and go to **Project Information**. Copy the **Project key** exactly as shown (spelling and casing matter). For projects created from GitHub, it is often `_` (for this guide the codebase name is `my-go-gin-app`): ![Project information](../assets/quick-start/sonarcloud_project_information.png "Project information") 4. In the component details page, expand the application and click the **GIT** button: ![Open Git repo](../assets/quick-start/go_to_source_code.png "Open Git repo") 5. In the repository on the `main` branch, edit `sonar-project.properties`. Set `sonar.projectKey` to the **Project key** from the previous step, `sonar.organization` to your **organization key**, and keep the rest aligned with the codebase name `my-go-gin-app`: ```ini sonar.projectKey=_my-go-gin-app sonar.projectName=my-go-gin-app sonar.go.coverage.reportPaths=coverage.out sonar.branch.name=main sonar.test.inclusions=**/*_test.go sonar.exclusions=**/cmd/**,**/deploy/**,**/deploy-templates/**,**/*.groovy,**/config/** sonar.language=go sonar.organization= ``` ![Sonar project properties](../assets/quick-start/sonar-project-properties.png "Sonar project properties") :::note Replace `` with your real organization key from SonarCloud. If SonarCloud shows a **Project key** that does not follow `_my-go-gin-app`, use the value from **Project Information** verbatim for `sonar.projectKey`. ::: 6. Commit and push the `sonar-project.properties` changes to the main branch. 7. (Optional) Select the SonarCloud-specific pipeline: On the Project details page, select the **Branches tab**, click the actions button and select **Configure**: ![Edit branch](../assets/quick-start/edit-main.png "Edit branch") :::note If this Pipeline is managed by Helm or Argo CD, apply the same change in the GitOps manifest or chart values so the cluster object is not reverted on sync. ::: :::note If you use a self-hosted SonarQube instance instead of SonarCloud, follow [SonarQube integration](../operator-guide/code-quality/sonarqube.md); project key rules may differ. ::: In the **Edit** window, select the **github-build-edp-sonarcloud** build Pipeline and click **Apply**: ![Change build pipeline](../assets/quick-start/change-build-pipeline.png "Change build pipeline") 8. In the Project details page, click **Build**: ![Triggering pipeline run](../assets/quick-start/trigger_pipeline_run.png "Triggering pipeline run") 9. Open the PipelineRun and wait until it finishes. Many small starter projects pass on the first run: ![Pipelinerun logs](../assets/quick-start/pipelinerun_status.png "Pipelinerun logs") ![Successful pipelinerun](../assets/quick-start/successful_pipeline.png "Successful pipelinerun") :::note After the first analysis, SonarCloud may adjust Quality Gate behavior for the project. If the `sonar` task fails with **Quality Gate failed** even though the scan completed, address the reported issues or review the Quality Gate, then **re-run** the build pipeline. A second run is not always required. ::: If the pipeline failed, ensure **Automatic Analysis** is disabled for this project in SonarCloud when using CI scans (see [Integrate SonarQube](./integrate-sonarcloud.md)): ![Analysis error](../assets/quick-start/analysis-error.png "Analysis error") Build pipelines in KubeRocketCI are specifically designed to generate an executable image of an application. Once the build process is complete, the resulting image can be deployed and run in a target environment. Now that you have successfully built an application, the next step is to create an environment for deployment. In order to deploy the application, you will need to install and integrate Argo CD. To learn how to install and integrate Argo CD, please refer to the [Integrate Argo CD](./integrate-argocd.md) page. --- ## Deploy Application # Deploy Application Now, let's proceed to deploy our first application. This page offers detailed instructions on how to create an environment and deploy the application within it. ## Create GitOps Repository As a prerequisite, create a GitOps repository in your GitHub account. KubeRocketCI Portal adheres to the GitOps approach when working with environments. In a GitOps repository, values are saved to redefine the default behavior (parameters) of deployment for each environment. The creation of a GitOps repository involves the following two steps: 1. In the UI Portal, navigate to **Configuration** -> **Deployment** -> **GitOps** and click **+ Add GitOps repository**: ![GitOps tab](../assets/quick-start/gitops_section.png "GitOps tab") 2. Define the following values and click **Save**: - Git server: `github` - Git repo relative path: `github_account_name` - Repository Name: `edp-gitops` ![GitOps tab](../assets/quick-start/add_gitops_repository.png "GitOps tab") ## Create Deployment Flow To create a deployment flow, follow the steps below: 1. In the KubeRocketCI portal, navigate to **Deployments** and click the **+ Create deployment** button: ![Deployment flows section](../assets/quick-start/create_environment.png "Deployment Flows section") 2. In the **Applications** stage, select `my-go-gin-app` application, `main` branch, and click **Continue ->**: ![Applications stage](../assets/quick-start/applications_stage.png "Applications stage") 3. In the **Pipeline configuration** tab, fill in the required fields and click **Continue ->**: ![Pipeline configuration](../assets/quick-start/pipeline_configuration.png "Pipeline configuration") 4. In the **Review and create** tab, verify the Deployment configuration and click **Create Deployment**: ![Review and create](../assets/quick-start/review_and_create.png "Review and create") 5. Once the Deployment is created, click the **Go to Deployment** button on the congratulations window. 6. In the **Deployment** details page, click the **Create environment** button: ![Create environment](../assets/quick-start/create-environment.png "Create environment") 7. On the **Basic configuration** stage, define the following values and click **Continue ->**: - Cluster: `in-cluster` - Environment name: `dev` - Namespace: `edp-my-go-gin-app-dev` - Description: `Development stage` ![Basic configuration](../assets/quick-start/basic-configuration-stage.png "Basic configuration") 8. On the **Pipeline configuration** stage, define the following values and click **Next**: - Trigger type: `Manual` - Deploy Pipeline template: `deploy` - Clean Pipeline template: `clean` ![Pipeline configuration](../assets/quick-start/pipeline_configuration_stage.png "Pipeline configuration") 9. On the **Add quality gates** tab, define the quality gates and click **Create**: - Quality gate type: `Manual` - Step name: `dev` ![Add quality gates window](../assets/quick-start/define-quality-gates.png "Add quality gates window") 10. On the congratulations window, click **Open environment**: ![Open environment](../assets/quick-start/go-to-environment.png "Open environment") The created deployment flow will be added to the deployment flows list. ## Application Deployment Once you enter the environment details page, proceed to deploy the application: 1. Click the **Configure deploy** button. 2. Select the Image stream version. 3. Click the **Start deploy** button: ![Deploying application](../assets/quick-start/deploy_application.png "Deploying application") 4. Watch the PipelineRun completion status by navigating to **Pipelines** tab and clicking the PipelineRun name: ![Deploying application](../assets/quick-start/deploy_pipeline_status.png "Deploying application") Congratulations on completing the Quick Start guide! We hope you found this journey enjoyable. Now that you have a good understanding of how KubeRocketCI works, you can further enhance its capabilities by integrating it with [Nexus](../operator-guide/artifacts-management/nexus-sonatype.md). Additionally, explore other functionalities detailed in our [Use Cases](../use-cases/index.md) section. If you're eager to unlock the full potential of KubeRocketCI, navigate to the [Operator Guide](../operator-guide/index.md) to fine-tune your KubeRocketCI for your specific requirements! --- ## Integrate Argo CD # Integrate Argo CD KubeRocketCI employs Argo CD as a Continuous Deployment tool for its purposes. This page provides guidance on the installation procedure for Argo CD. ## Installation To install Argo CD, follow the steps below: 1. Add a Helm Chart repository: ```bash helm repo add argocd https://argoproj.github.io/argo-helm helm repo update ``` 2. Create the `argocd-values.yaml` file and paste the following data:
View: argocd-values.yaml ```yaml redis-ha: enabled: false redis: enabled: true server: replicas: 1 env: - name: ARGOCD_API_SERVER_REPLICAS value: '1' ingress: enabled: true hosts: # after domain creation update - "argocd-dev.example.com" # Disabled for multitenancy env with single instance deployment applicationSet: allowAnyNamespace: true serviceAccount: # -- Annotations applied to created service account annotations: {} extraEnv: - name: ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_SCM_PROVIDERS value: 'false' # we use Keycloak so no DEX is required dex: enabled: false configs: secret: # -- Create the argocd-secret createSecret: true cm: application.instanceLabelKey: argocd.argoproj.io/instance-kuberocketci params: server.insecure: true applicationsetcontroller.namespaces: krci application.namespaces: krci ```
3. Apply the argocd-values.yaml file using the `helm install` command: ```bash helm install argo-cd argocd/argo-cd --values argocd-values.yaml --version 9.3.4 --create-namespace --atomic -n argocd ``` :::info It may take a while for the Argo CD to be installed. The installation time depends on cluster capabilities. ::: ## Integration The next step is to integrate Argo CD with the platform. Proceed with the instructions below: 1. Navigate to the directory that stores the `ed25519` key created during the [GitHub integration](./integrate-github.md). 2. Copy the SSH private key to the Argo CD namespace. Make sure to provide the appropriate value for the `ACCOUNT_NAME` variable: ```bash KRCI_NAMESPACE=krci VCS_HOST="github.com" ACCOUNT_NAME="" URL="ssh://git@${VCS_HOST}:22/${ACCOUNT_NAME}" kubectl create secret generic ${KRCI_NAMESPACE} -n argocd \ --from-file=sshPrivateKey=ed25519 \ --from-literal=url="${URL}" kubectl label --overwrite secret ${KRCI_NAMESPACE} -n argocd "argocd.argoproj.io/secret-type=repo-creds" ``` 3. Add GitHub host to the Argo CD config map with known hosts: ```bash VCS_HOST="github.com" KNOWN_HOSTS_FILE="/tmp/ssh_known_hosts" ARGOCD_KNOWN_HOSTS_NAME="argocd-ssh-known-hosts-cm" rm -f ${KNOWN_HOSTS_FILE} kubectl get cm ${ARGOCD_KNOWN_HOSTS_NAME} -n argocd -o jsonpath='{.data.ssh_known_hosts}' > ${KNOWN_HOSTS_FILE} ssh-keyscan ${VCS_HOST} >> ${KNOWN_HOSTS_FILE} kubectl create configmap ${ARGOCD_KNOWN_HOSTS_NAME} -n argocd --from-file ${KNOWN_HOSTS_FILE} -o yaml --dry-run=client | kubectl apply -f - ``` 4. Create the argocd-project.yaml file, using, for example, the `krci` name:
View: argocd-project.yaml ```yaml apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: krci namespace: argocd # Finalizer that ensures that project is not deleted until it is not referenced by any application finalizers: - resources-finalizer.argocd.argoproj.io spec: description: CD pipelines for my application roles: - name: developer description: Users for kuberocketci tenant policies: - p, proj:krci:developer, applications, create, krci/*, allow - p, proj:krci:developer, applications, delete, krci/*, allow - p, proj:krci:developer, applications, get, krci/*, allow - p, proj:krci:developer, applications, override, krci/*, allow - p, proj:krci:developer, applications, sync, krci/*, allow - p, proj:krci:developer, applications, update, krci/*, allow - p, proj:krci:developer, repositories, create, krci/*, allow - p, proj:krci:developer, repositories, delete, krci/*, allow - p, proj:krci:developer, repositories, update, krci/*, allow - p, proj:krci:developer, repositories, get, krci/*, allow destinations: # ensure we can deploy to ns with tenant prefix - namespace: 'krci-*' # allow to deploy to specific server (local in our case) server: https://kubernetes.default.svc # Deny all cluster-scoped resources from being created, except for Namespace clusterResourceWhitelist: - group: '' kind: Namespace # Allow all namespaced-scoped resources to be created, except for ResourceQuota, LimitRange, NetworkPolicy namespaceResourceBlacklist: - group: '' kind: ResourceQuota - group: '' kind: LimitRange - group: '' kind: NetworkPolicy # we are ok to create any resources inside namespace namespaceResourceWhitelist: - group: '*' kind: '*' # enable access only for specific git server. The example below 'krci' - it is namespace where KubeRocketCI is deployed sourceRepos: - ssh://git@github.com:22//* # enable capability to deploy objects from namespaces sourceNamespaces: - krci ``` :::info Please enter your GitHub account name at ``. :::
5. Retrieve the Argo CD admin secret and securely save it, as it will be required for further steps: ```bash kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ``` Now that all the necessary preparations are completed, we can proceed with the [Application Deployment](./deploy-application.md). --- ## Integrate DockerHub # Integrate DockerHub KubeRocketCI utilizes container registries for storing and distributing application images. This page provides instructions on integrating your platform with DockerHub. ## Integration Procedure To integrate KubeRocketCI with DockerHub, follow the steps below: 1. Log in to your DockerHub account. 2. In the DockerHub main page, click your user icon at the top right corner and select **Account Settings**. 3. In the **Account Settings** page, select **Settings** -> **Personal access tokens** and click **Generate new token**. 4. Enter the token description, select the **Read, Write, Delete** permission and click **Generate**. 5. Copy the generated token: ![DockerHub token](../assets/quick-start/dockerhub_token.png "DockerHub token") 6. In the UI Portal, navigate to **Configuration** -> **Registry** and click the **Add Registry** button. 7. In the **Registry Provider** field, select **DockerHub**. Define the following values and click **Save**: - Registry Endpoint: `https://docker.io` - Registry Space: `DockerHub account name` - User: `DockerHub account name` - Password/Token: `Your generated access token` - Use the Push Account's credentials: `check` ![DockerHub integration](../assets/quick-start/integrate_docker.png "DockerHub integration") :::note In the DockerHub context, the **Registry Space** field is equivalent to the account/organization name. ::: With all integrations in place, let's proceed to the [Create Application](./create-application.md) page for further instructions on how to create applications using KubeRocketCI. --- ## Integrate GitHub # Integrate GitHub To initiate work in the KubeRocketCI, integration with a Version Control System (VCS) is essential. This integration facilitates the use of create, clone, and import strategies for handling source code. This tutorial focuses on the create strategy, wherein an application is directly created in the integrated VCS solution for subsequent work. Here, you will find instructions on how to integrate KubeRocketCI with GitHub. :::note Prior to moving forward, ensure you possess an active GitHub account and have successfully deployed the platform. ::: :::note If integration with a different Version Control System (VCS) is required, refer to the [Add Git Server](../user-guide/add-git-server.md) section for details on configuring alternative VCS providers. ::: ## Integration Procedure To integrate KubeRocketCI with GitHub, follow the steps below: 1. Generate an SSH key pair: ```bash ssh-keygen -t ed25519 -C "email@example.com" ``` 2. Add the created SSH key (the public part) to the GitHub account: 1. In the GitHub main page, click your user icon. Navigate to **Settings** -> **SSH and GPG keys** and click **New SSH key**. 2. Create the `quick_start` key. Insert your `ed25519.pub` key data and click **Add SSH key**: ![Repo permission](../assets/quick-start/add_ssh_key.png "Repo permission") 3. Generate an `access token` for the GitHub account with read/write access to the API: 1. Log in to GitHub. 2. Click the profile account and navigate to **Settings** -> **Developer Settings**. 3. Select **Personal access tokens (classic)** and generate a new token with the following permissions: ![Repo permission](../assets/operator-guide/github-scopes-1.png "Repo permission") :::note The following (Admin:repo, Admin:org, and User) access is necessary for the platform to retrieve Pull Request commits, their status, and author information. ::: ![Admin:repo permission](../assets/operator-guide/github-scopes-2.png "Admin:repo permission") ![Admin:org permission](../assets/operator-guide/github-scopes-4.png "Admin:org permission") ![User permission](../assets/operator-guide/github-scopes-3.png "User permission") 4. Save a new personal access token. 4. In UI Portal, navigate to **Configuration** -> **Version Control System** -> **Git Servers**. Define the following values and click **Save**: - Git provider: select `GitHub` - Host: `github.com` - User: `git` - SSH port: `22` - HTTPS port: `443` - Private SSH key: `your generated SSH private key` - Access token: `your account token generated in GitHub` ![Git Server configuration](../assets/quick-start/github_integration.png "Git Server configuration") Ensure the Git server has a green status. To store container images, integrate platform with a container registry by navigating to the [Integrate DockerHub](./integrate-container-registry.md) page. --- ## Integrate SonarQube # Integrate SonarQube It is essential for KubeRocketCI to have SonarQube integrated with the platform as all the pipelines include the `sonar` step. SonarQube is a powerful tool used in build and code review pipelines to enhance code quality by identifying and reporting issues, as well as providing recommendations for improvement. SonarCloud, the SaaS solution offered by SonarQube, serves this purpose. This guide will walk you through the configuration process of SonarCloud for your project. :::note An alternative option is to use an independent [SonarQube instance](../operator-guide/code-quality/sonarqube.md). ::: ## Integrate SonarCloud To integrate SonarCloud with the platform, follow the steps below: 1. Sign in to [SonarCloud](https://sonarcloud.io) with your GitHub account. 2. In the top right corner of the screen, click **+** and select [Create new organization](https://sonarcloud.io/create-organization), select **GitHub**, and continue with your GitHub account/organization: ![Create organization (current UI)](../assets/quick-start/sonarcloud_step2_create_organization_current.png "Create organization (current UI)") :::note SonarCloud UI has changed: the flow is now **Create organization** -> **GitHub** instead of the old "import from GitHub" screen. ::: 3. In the Create an organization menu, choose the free plan and click **Create organization**: :::note Organization key must be unique. ::: ![Choose plan](../assets/quick-start/free_plan.png "Choose plan") 4. For each SonarCloud project that KubeRocketCI analyzes with Tekton (`sonar-scanner` in the build or review pipeline), turn off **Automatic Analysis** in SonarCloud so only CI runs the scan. If you have not created those projects yet, do this after you add each repository in SonarCloud (for example while following [Create Application](./create-application.md)): 1. In SonarCloud, open the project. 2. Go to **Administration** -> **Analysis Method**. 3. Disable **Automatic Analysis**: ![Disable automatic scans](../assets/quick-start/disable_autoscan.png "Disable automatic scans") :::warning If **Automatic Analysis** stays enabled while the pipeline runs `sonar-scanner`, SonarCloud returns an error such as *You are running manual analysis while Automatic Analysis is enabled* and the `sonar` task fails. KubeRocketCI expects a **single** analysis path: CI-driven scans with the token configured in the portal. ::: 5. In the [create project](https://sonarcloud.io/projects/create) page on the SonarCloud site, add new project to analyze it: :::note Project name in the Sonar cloud must be the same as codebase name. ::: ![Add new project](../assets/quick-start/add_sonar_project.png "Add new project") Set the **Previous** version option and click **Create project**: ![Add new project](../assets/quick-start/previous-version.png "Add new project") 6. On the project menu, navigate to **Branches** menu, select `master` branch and rename it to `main`. You can skip this step if the default branch is already `main`: ![Change default branch](../assets/quick-start/change_default_branch.png "Change default branch") 7. Click your profile icon, open [**My account -> Security**](https://sonarcloud.io/account/security), enter a token name, and click **Generate Token**: :::warning Copy the generated token right away. SonarCloud shows token value only once. ::: ![Generate token (current UI)](../assets/quick-start/sonarcloud_step4_generate_token_current.png "Generate token (current UI)") 8. In the KubeRocketCI portal, navigate to **Configuration** -> **Code Quality** -> **SonarQube**. Define the following values and click **Save**: * Quick Link URL: `https://sonarcloud.io` * URL: `https://sonarcloud.io` * Token: `account token generated in SonarCloud` ![SonarQube integration](../assets/quick-start/sonarqube_integrated.png "SonarQube integration") After completing the SonarQube integration, proceed to integrate the platform with GitHub. Navigate to the [Integrate GitHub](./integrate-github.md) page for further instructions. --- ## Install KubeRocketCI # Install KubeRocketCI This page serves as the starting point for the quick start guide, where we will install Tekton as a prerequisite and then proceed to install the KubeRocketCI itself. ## Install Tekton KubeRocketCI relies on Tekton resources, including Tasks, Pipelines, Triggers, and Interceptors to execute CI/CD pipelines. To install Tekton, run the commands below: ```bash kubectl apply -f https://infra.tekton.dev/tekton-releases/pipeline/previous/v1.6.2/release.yaml kubectl apply -f https://infra.tekton.dev/tekton-releases/triggers/previous/v0.36.0/release.yaml kubectl apply -f https://infra.tekton.dev/tekton-releases/triggers/previous/v0.36.0/interceptors.yaml ``` ## Install platform To deploy the platform, follow the steps below: 1. Add a Helm Chart repository: ```bash helm repo add epamedp https://epam.github.io/edp-helm-charts/stable helm repo update ``` 2. Deploy the platform using the `helm install` command: ```bash helm install krci epamedp/edp-install --version 3.14.1 --create-namespace --atomic -n krci --set global.dnsWildCard=example.com ``` 3. Upon successful deployment of the KubeRocketCI Helm Chart, run the `kubectl port-forward` command: ```bash kubectl port-forward service/krci-portal 59480:80 -n krci ``` 4. Enter the localhost in your browser to access the login menu and click **Use Token**: ```bash localhost:59480 ``` ![Portal login menu](../assets/quick-start/edp_portal_login_menu.png "Portal login menu") 5. Create the edp-admin service account and generate an access token to open the KubeRocketCI: ```bash kubectl -n krci create serviceaccount edp-admin kubectl create clusterrolebinding edp-admin --serviceaccount=krci:edp-admin --clusterrole=cluster-admin kubectl create token edp-admin -n krci ``` 6. In the login menu, paste the generated token in the **Access token** field and click the **Sign in** button. 7. Upon logging in, open the namespaces window by clicking the **Manage Namespaces** button in the top right corner of the page: ![Specify namespaces](../assets/quick-start/edp_portal_ui.png "Specify namespaces") 8. Ensure the `krci` namespace is specified in both default and allowed namespaces. If not, define them manually: * Default namespace: `krci` * Allowed namespaces: `krci` ![Cluster Settings menu](../assets/quick-start/cluster_settings.png "Cluster Settings menu") :::note Remember to press **Enter** to add the allowed namespace to the list. ::: After completing these steps, you will gain access to KubeRocketCI components through the KubeRocketCI portal. You can now proceed with the integration steps, starting with the [SonarQube](./integrate-sonarcloud.md) integration. --- ## Quick Start: Overview # Quick Start: Overview This page serves as an introductory part of the Quick Start guide. It outlines the core steps and reasons to perform them. ## Introduction The purpose of this guide is to provide a comprehensive overview of the KubeRocketCI platform and its key functionalities. By following this guide, users will gain a clear understanding of the platform's capabilities and be able to explore its features quickly. This guide is specifically designed for individuals who want to swiftly familiarize themselves with the KubeRocketCI platform. ## Setup Flow The installation procedure of the platform can be divided into four logical parts: ```mermaid graph LR; A(Prerequisites) --> B(Platform Installation) --> C(Integration) --> D(Application Deployment) ``` 1. **Prerequisites**: In this step, Tekton and Argo CD are installed to provide KubeRocketCI with CI/CD capabilities. 2. **Platform Installation**: Here, we deploy the platform itself and configure it for work. 3. **Integration**: This step involves integrating third-party tools with KubeRocketCI to provide the platform with a minimum functionality set. 4. **Application Deployment**: In this final step, we create and deploy an application, which serves as the target endpoint of the entire guide. The guide is formally divided into two parts: **CI** and **CD**, corresponding to the stages of the pipeline that the platform supports at each moment. ## Prerequisites Before deploying KubeRocketCI, it is important to ensure that a local Kubernetes cluster is set up. We recommend allocating 4 CPUs and 8 GB of RAM to the cluster. Alternatively, you can use a cloud provider that offers a Kubernetes service with sufficient CPU and RAM capacity. For example, you can check out our [video tutorial](https://www.youtube.com/watch?v=QjZoPnIKDtA) where we demonstrate the installation of the platform via the [Civo Marketplace](../operator-guide/install-via-civo.md). This allows you to skip the installation part of the guide and proceed with adjusting integrations. ## Starting Point The first step of the guide is to prepare the cluster environment for the KubeRocketCI setup process by installing the Tekton tool. Once this is done, you can proceed with the installation of the platform itself. Detailed guidelines for the platform installation can be found in the [Platform Installation](./platform-installation.md) page. Alternatively, you can watch our [video tutorial](https://www.youtube.com/watch?v=QjZoPnIKDtA) which provides a clear demonstration of the installation process. Good luck with your installation! --- ## Supported Versions and Compatibility # Supported Versions and Compatibility At the KubeRocketCI, our commitment to providing a robust and reliable service involves supporting the latest innovations while ensuring stability and security. To maintain this balance, we focus our active **support on our platform's three most** recent versions. This approach allows us to dedicate our resources to delivering cutting-edge features, enhancing performance, and fortifying security measures while ensuring a consistent and optimized user experience across the platform. ```mermaid gantt dateFormat YYYY-MM-DD title Release Schedule and Lifecycle section (Under Development) KubeRocketCI 3.15 SNAPSHOT: crit, r45, 2026-07-14, 2026-12-14 section (Supported) KubeRocketCI 3.14 Release: active, r44, 2026-07-14, 2026-12-14 section (Supported) KubeRocketCI 3.13 Release: active, r43, 2026-03-27, 2026-12-14 section (Supported) KubeRocketCI 3.12 Release: active, r42, 2025-07-22, 2026-12-14 section (Unsupported) KubeRocketCI 3.11 Release: active, r41, 2025-03-21, 2026-07-14 section (Unsupported) KubeRocketCI 3.10 Release: active, r40, 2024-10-18, 2026-03-27 section (Unsupported) KubeRocketCI 3.9 Release: r39, 2024-06-13, 2025-07-21 section (Unsupported) KubeRocketCI 3.8 Release: r38, 2024-03-12, 2025-03-21 ``` By concentrating our support on the last three versions, we prioritize staying current with evolving technology landscapes. This enables us to swiftly adapt to advancements in Kubernetes and OpenShift environments, incorporating the latest enhancements, fixes, and optimizations into our platform. This support model not only fosters innovation but also enables us to deliver a secure, high-performing, and dependable KubeRocketCI that meets the dynamic needs of our users and exceeds expectations for reliability and functionality. Get acquainted with the list of the latest releases and component versions on which the platform is tested and verified: :::tip Professional Services You can leverage our [professional services](https://kuberocketci.io/pricing) to upgrade your platform to the latest version or perform migration from Unsupported to Supported versions. ::: |[Release Version](https://github.com/epam/edp-install/blob/master/RELEASES.md)|Release Date|[EKS Version](https://aws.amazon.com/eks/)|[OpenShift Version](https://github.com/okd-project/okd/releases)| |:-:|:-:|:-:|:-:| |3.14 |Jul 14, 2026|1.34+|N/A| |3.13 |Mar 27, 2026|1.34+|N/A| |3.12 |Jul 22, 2025|1.32+|N/A| |3.11 |Mar 21, 2025|1.29+|N/A| |3.9 |Jun 13, 2024|1.26+|4.12| |3.8 |Mar 12, 2024|1.26+|4.12| |3.7 |Dec 15, 2023|1.26+|4.12| --- ## Scaffold and Deploy FastAPI Application # Scaffold and Deploy FastAPI Application This use case outlines the process of creating and deploying a FastAPI application using KubeRocketCI. The purpose is to provide developers with a streamlined approach to generate a functional code structure for a FastAPI web application, including basic read functionality. The process also allows for customization to meet specific requirements and deployment to a development environment. By utilizing a standardized process for code review, testing, and deployment, developers can save time and effort while ensuring high code quality and reliability. The goal is to empower the development team to release new features and applications more efficiently, while maintaining a high level of code quality and reliability. The KubeRocketCI is used to facilitate this process. ### Roles This documentation is tailored for the Developers and Team Leads. ### Goals - Create a new FastAPI application quickly. - Deploy the initial code to the DEV environment. - Check CI pipelines. - Delivery update using the GitOps approach. ### Preconditions - KubeRocketCI instance is [configured](../operator-guide/prerequisites.md) with [GitOps](../user-guide/gitops.md) repo. - Developer has access to the KubeRocketCI instances using the Single-Sign-On approach. - Developer has merge permissions in the [GitOps](../user-guide/gitops.md) repository on GitHub. ## Scenario To scaffold and deploy FastAPI Application, follow the steps below. ### Scaffold the New FastAPI Application 1. Log in to the KubeRocketCI portal: ![Logging Page](../assets/use-cases/general/login-1.png "Logging screen") 2. On the main menu, click the **Manage Namespaces** button: ![Manage Namespaces](../assets/use-cases/general/settings-1.png "Manage Namespaces button") 3. Ensure the `Namespace` value points to the namespace with the KubeRocketCI installation: ![Default and allowed namespaces](../assets/use-cases/fastapi-scaffolding/default-allowed-namespace.png "Default and allowed namespaces") :::note Don't forget to press Enter to add the namespace to the allowed namespaces list. ::: 4. Create a new Project with the `Application` type using the `Create` strategy. Select the **Projects** section and click **+ Create project**: ![Components Overview](../assets/use-cases/general/components-1.png "Components tab") 5. Select the **Custom configuration** option, choose the `Application` Codebase type as we intend to deliver our application as a container and deploy it within the Kubernetes cluster. Select the **Create** strategy to scaffold our application from the template provided by the KubeRocketCI and click **Continue**: ![Initial setup](../assets/use-cases/general/component-type-1.png "Initial setup") 6. On the **Git & project info** tab, define the following values and click **Continue**: - Git server: `github` - Owner: `github_account_name` - Repository name: `fastapi-demo` - Default branch: `main` - Project name: `fastapi-demo` - Description: `FastAPI application for demo purposes` ![Application info](../assets/use-cases/fastapi-scaffolding/create-application.png "Application info") 7. On the **Build config** tab, define the values and click **Continue**: - Code language: `Python` - Language version/framework: `FastAPI` - Build tool: `Python` - Deployment options: `helm-chart` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Build config](../assets/use-cases/fastapi-scaffolding/create-application-advance.png "Build config") 8. On the **Review** tab, verify the project configuration and click **Create project**: ![Review and create](../assets/use-cases/fastapi-scaffolding/review-and-create.png "Review and create") 9. On the congratulations menu, click **Open project**: ![Components overview page](../assets/use-cases/fastapi-scaffolding/ready-component.png "Application status") ### Deploy Application to the Development Environment This section outlines the approach for deploying the application from the latest commit on the `main` branch. The following steps are involved: 1. Build the initial version of the application from the last commit on the `main` branch. This version is generated from the provided template. 2. Set up a deployment flow to enable continuous delivery to the development environment. 3. Deploy the initial version of the application to the development environment. To succeed with the steps above, follow the instructions below: 1. Build Container from the latest branch commit. To build the initial version of the application's `main` branch, go to the **Branches** -> **main** and click the **Build** button: ![Build Main Branch](../assets/use-cases/fastapi-scaffolding/build-application.png "Application building") 2. Build pipeline for the `fastapi-demo` application starts. Track Pipeline's status by clicking the **build-fastapi-demo-main-xxxx** pipeline link or **Go to page**: ![Branch Build Pipeline](../assets/use-cases/fastapi-scaffolding/build-in-process.png "Pipeline building") 3. Observe the build pipeline details: ![Pipeline details](../assets/use-cases/fastapi-scaffolding/build-details.png "Pipeline details") 4. Ensure that build pipeline was successfully completed. 5. To enable application deployment, create a Deployment with a single environment - Development (with the name `dev`). :::note To utilize and manage various environments through the KubeRocketCI platform, the initial step is to onboard a new GitOps repository. ::: 6. Navigate to the **Deployments** tab and click the corresponding button to create new GitOps repository: ![Deployment flow tab](../assets/use-cases/fastapi-scaffolding/add-gitops-repo.png "Deployment flow tab") 7. Click to the **+ Add GitOps repository**: ![Onboard gitops repository](../assets/use-cases/fastapi-scaffolding/add-gitops-button.png "Onboard GitOps repository") 8. Select the `Create` strategy and `github` server, enter GitHub account name and click **Save**: ![Configure gitops repository](../assets/use-cases/fastapi-scaffolding/create-gitops.png "Configure gitops repository") 9. In the KubeRocketCI portal, navigate to **Deployments** tab and click the **+ Create deployment** button: ![Create deployment](../assets/use-cases/general/create-cd-pipeline-1.png "Create deployment") 10. In the **Create new deployment** dialog, fill in the required fields: - **Applications**: - Applications: Add `fastapi-demo` application - Branch: Select `main` branch ![Applications step](../assets/use-cases/fastapi-scaffolding/create-deployment-applications.png "Applications step") - **Pipeline configuration**: - Pipeline name: `mypipe` - Description: `Deployment for FastAPI application` - Deployment type: Select `Container` - Promote applications: Leave unchecked ![Pipeline configuration step](../assets/use-cases/fastapi-scaffolding/create-deployment-pipeline-configuration.png "Pipeline configuration step") - **Review**: Verify the Deployment configuration and click **Create deployment**: ![Review step](../assets/use-cases/fastapi-scaffolding/create-deployment-review.png "Review step") 11. Now click the `Open deployment` button to continue creating the environment: ![Deployment created](../assets/use-cases/fastapi-scaffolding/deployment-created.png "Deployment created") 12. In the Deployment details page, click the **+ Create environment** button to create new environment and fill in the required fields: ![Create environment](../assets/use-cases/fastapi-scaffolding/add-environment.png "Create environment") - **Basic configuration**: - Cluster: `in-cluster` - Environment name: `dev` - Deploy namespace: `krci-mypipe-dev` - Description: `Development Environment` ![Basic configuration step](../assets/use-cases/fastapi-scaffolding/create-environment-basic-configuration.png "Basic configuration step") - **Pipeline configuration**: - Trigger type: `Manual` - Deploy Pipeline template: `deploy` - Clean Pipeline template: `clean` ![Pipeline configuration step](../assets/use-cases/fastapi-scaffolding/create-environment-pipeline-configuration.png "Pipeline configuration step") - **Quality Gates**: Leave everything as is: ![Quality gates step](../assets/use-cases/fastapi-scaffolding/create-environment-quality-gates.png "Quality gates step") - **Review**: Review the specified values and click the **Create environment** button: ![Review step](../assets/use-cases/fastapi-scaffolding/create-environment-review.png "Review step") 13. As soon as the Environment is created, click **Open environment**: ![Open environment](../assets/use-cases/fastapi-scaffolding/open-environment.png "Open environment") 14. Deploy the initial version of the application to the development environment: - Select **Applications** tab. - Click the **Configure deploy** button. - In the **Select image tag** field select version `0.1.0-SNAPSHOT.1` and click the **Start deploy** button. ![CD Pipeline Deploy initial version](../assets/use-cases/fastapi-scaffolding/start-deploy.png "CD Pipeline deploy") 15. Navigate to the **Pipelines** tab and click the pipeline name to view its details: ![Deploy pipeline (clickable)](../assets/use-cases/fastapi-scaffolding/deploy-pipeline-clickable.png "Deploy pipeline (clickable)") 16. View the Pipeline details until it finishes: ![Deploy pipeline details](../assets/use-cases/fastapi-scaffolding/deploy-pipeline-details.png "Deploy pipeline details") ### Check the Application Status To ensure the application is deployed successfully, follow the steps below: 1. Ensure application status is `Healthy` and `Synced`, and the `Deployed version` points to `0.1.0-SNAPSHOT.1`: ![Deployment health status](../assets/use-cases/fastapi-scaffolding/deploy-is-healthy.png "Deployment health status") 2. Check that the selected version of the container is deployed on the `dev` environment. `${KRCI_ENV}` - is the KRCI namespace name: ```bash # Check the deployment status of fastapi-demo application $ kubectl get deployments -n ${KRCI_ENV}-mypipe-dev NAME READY UP-TO-DATE AVAILABLE AGE fastapi-demo 1/1 1 1 14m ``` ### Deliver New Code This section describes the code delivery process. We need to deploy our `fastapi-demo` application that deploys `Ingress` object to expose API outside the Kubernetes cluster. Within the [GitOps](../user-guide/gitops.md) repository, follow the naming convention `//-values.yaml` to create the chart values file. For our case, use `mypipe/dev/fastapi-demo-values.yaml` as the file name. To deploy application with custom application Helm chart values, follow he steps below: 1. Commit `mypipe/dev/fastapi-demo-values.yaml` file to the default branch of the GitOps repository: ```yaml title="mypipe/dev/fastapi-demo-values.yaml" ingress: enabled: true ``` 2. Deploy the `fastapi-demo` application to the environment. Before the new version deployment, check the ingress object in `dev` environment: ```bash $ kubectl get ingress -n ${KRCI_ENV}-mypipe-dev No resources found in ${KRCI_ENV}-mypipe-dev namespace. ``` No ingress object exists as expected. 3. Deploy the `fastapi-demo` application which has the ingress object in place. Since we use `Manual` deployment approach, we perform version upgrade manually: To ensure our application recognizes the changes in the GitOps repository related to enabling the creation of the `Ingress` resource, we need to set the **Override values** checkbox to `true` before deploying. - Go to the **Deployments** section of the KubeRocketCI portal, select `mypipe` and choose the **dev** Environment. - In the **Select image tag** section select the version `0.1.0-SNAPSHOT.1`. - Set the **Values override** checkbox to `true` and click the **Start deploy** button. - Check that the application is deployed and application status is `Healthy` and `Synced`. ![Values override](../assets/use-cases/fastapi-scaffolding/deploy-with-override.png "Values override") 4. Check that the new version with Ingress is deployed: ```bash # Check Ingress object kubectl get ingress -n ${KRCI_ENV}-mypipe-dev NAME CLASS HOSTS ADDRESS PORTS AGE fastapi-demo fastapi-demo-ko1zs-example.com 10.100.182.134 80 14m # Check application external URL curl https://your-hostname-appeared-in-hosts-column-above.example.com {"Hello":"World"} ``` ## Related Articles - [Use Cases](./index.md) - [Autotest as a Quality Gate](./autotest-as-quality-gate.md) - [Create and Use Custom Tekton Pipelines](./custom-pipelines-flow.md) --- ## Autotest as a Quality Gate # Autotest as a Quality Gate This use case outlines the process of integrating an autotest as a quality gate into a newly created Deployment. The Deployment includes a selected build version of an application that needs to be promoted. The purpose of incorporating autotests is to ensure that the application meets predefined criteria for stability and functionality, guaranteeing that only reliable versions are promoted. By implementing this feature, users can perform comprehensive testing, thereby enhancing the overall stability of the application. In KubeRocketCI, users can seamlessly add autotests as quality gates to their Deployment, enabling them to validate the application's stability and functionality before promoting it to the next stage. This ensures that only reliable versions of the application are deployed, improving the overall quality and reliability of the software. ## Roles This documentation is tailored for the Developers and Quality Assurance specialists. ## Goals - Create several applications and autotests. - Create a Deployment. - Perform testing. - Update delivery by deploying the new version. ## Preconditions - KubeRocketCI instance is [configured](../operator-guide/prerequisites.md) with GitHub, Tekton and [Argo CD](../operator-guide/cd/argocd-integration.md). - Developer has access to the platform using the Single-Sign-On approach. - Developer has the **Administrator** role (to perform merge in GitHub). ### Create Applications To implement autotests as Quality Gates, follow the steps below: 1. Log in to the KubeRocketCI portal: ![Logging Page](../assets/use-cases/general/login-1.png "Logging screen") 2. On the main menu, click the **Manage Namespaces** button: ![Manage Namespaces](../assets/use-cases/general/settings-1.png "Manage Namespaces button") 3. Ensure the `Namespace` value points to the namespace with the KubeRocketCI installation: ![Default and allowed namespaces](../assets/use-cases/fastapi-scaffolding/default-allowed-namespace.png "Default and allowed namespaces") :::note Don't forget to press Enter to add the namespace to the allowed namespaces list. ::: 4. Create a new Project with the `Application` type using the `Create` strategy. Select the **Projects** section and click **+ Create project**: ![Components Overview](../assets/use-cases/general/components-1.png "Components tab") 5. Select the **Custom configuration** option, choose the `Application` Codebase type as we intend to deliver our application as a container and deploy it within the Kubernetes cluster. Select the **Create** strategy to scaffold our application from the template provided by the KubeRocketCI and click **Continue**: ![Initial setup](../assets/use-cases/general/component-type-1.png "Initial setup") 6. On the **Git & project info** tab, define the following values and click **Continue**: - Git server: `github` - Owner: `github_account_name` - Repository name: `js-application` - Default branch: `main` - Project name: `js-application` - Description: `JavaScript application` - Private: `enabled` ![Codebase info tab](../assets/use-cases/autotest-as-quality-gate/git-and-project-info.png "Codebase info tab") 7. On the **Build config** tab, define the values and click **Continue**: - Application code language: `JavaScript` - Language version/Provider: `Vue` - Build tool: `NPM` - Deployment options: `helm-chart` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Advanced settings tab](../assets/use-cases/autotest-as-quality-gate/build-config.png "Advanced settings tab") 8. On the **Review** tab, verify the project configuration and click **Create project**: ![Review and create](../assets/use-cases/autotest-as-quality-gate/review-and-create.png "Review and create") 9. On the congratulations menu, click **View all projects**: ![View all projects](../assets/use-cases/autotest-as-quality-gate/ready-component.png "View all projects") 10. Repeat the procedure twice to create the **go-application** and **python-application** applications. These applications will have the following parameters: go-application: - Git server: `github` - Owner: `github_account_name` - Repository name: `go-application` - Project name: `go-application` - Description: `Go application` - Application code language: `Go` - Language version/Provider: `Gin` - Build tool: `Go` - Default branch: `main` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` python-application: - Git server: `github` - Owner: `github_account_name` - Repository name: `python-application` - Project name: `python-application` - Description: `Python application` - Application code language: `Python` - Language version/Provider: `FastAPI` - Build tool: `Python` - Default branch: `main` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` 11. In the **Projects** tab, click one of the applications name to enter the application menu: ![Projects list](../assets/use-cases/autotest-as-quality-gate/3-apps.png "Projects list") 12. Select the **Branches** tab and click the **Build** button: ![Build Application](../assets/use-cases/autotest-as-quality-gate/build_application.png "Build Application") 13. Click the PipelineRun name to watch the building logs: ![Application building](../assets/use-cases/autotest-as-quality-gate/app_built.png "Application building") 14. On the pipeline details page, you can find information about each step, pipeline status, and view logs: ![KubeRocketCI pipeline run](../assets/use-cases/autotest-as-quality-gate/kuberocketci_pipeline.png "KubeRocketCI pipeline run") 15. Wait till the build is successful. 16. Repeat steps 11-15 for the rest of the applications. ### Create Autotests The steps below instruct how to create autotests in KubeRocketCI: 1. Create a couple of autotests using the **Clone** strategy. Navigate to the **Projects** tab, click on the **+ Create project** button. Select **Autotest**, **Clone** and click **Next**: ![Add autotest](../assets/use-cases/autotest-as-quality-gate/add_autotest.png "Add autotest") :::note Please refer to the [Add Autotest](../user-guide/add-autotest.md) section for details. ::: 2. On the **Git & project info** tab, define the following values and click **Continue**: - Repository URL: `https://github.com/Oleksandr123234/autotests.git` - Git server: `github` - Owner: `github_account_name` - Repository name: `demo-autotest-gradle` - Default branch: `main` - Project name: `demo-autotest-gradle` - Description: `Gradle demo autotests` ![Git & project info info tab for autotests](../assets/use-cases/autotest-as-quality-gate/demo_autotest_gradle.png "Git & project info info tab for autotests") 3. On the **Build config** tab, define the values and click **Continue**: - Autotest code language: `Java` - Language version/framework: `Java25` - Build tool: `Gradle` - Autotest report framework: `Allure` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Build config tab](../assets/use-cases/autotest-as-quality-gate/autotest_advanced_settings.png "Build config tab") 4. On the **Review** tab, verify the autotest configuration and click **Create project**: ![Review and create](../assets/use-cases/autotest-as-quality-gate/autotest_review_tab.png "Review and create") 5. Repeat the steps 1-4 to create one more autotest with the parameters below: - Repository URL: `https://github.com/Oleksandr123234/autotests.git` - Git server: `github` - Repository name: `demo-autotest-maven` - Project name: `demo-autotest-maven` - Description: `Maven demo autotest` - Autotest code language: `Java` - Language version/framework: `Java25` - Build tool: `Maven` - Autotest report framework: `Allure` - Default branch: `main` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ### Create CD Pipeline Now that applications and autotests are created, create pipeline for them by following the steps below: :::note To utilize and manage various environments through the KubeRocketCI platform, the initial step is to onboard a new GitOps repository. ::: 1. Navigate to the **Deployments** section and click the corresponding button to create new GitOps repository: ![Deployments tab](../assets/use-cases/autotest-as-quality-gate/create_cd_pipeline.png "Deployments tab") 2. Click to the **+ Add GitOps repository**: ![Onboard gitops repository](../assets/use-cases/autotest-as-quality-gate/onboard_gitops.png "Onboard GitOps repository") 3. Select the `github` server, enter GitHub account name and click **Save**: ![Configure gitops repository](../assets/use-cases/autotest-as-quality-gate/add_gitops_repository.png "Configure gitops repository") 4. Return to the **Deployments** tab and click **+ Create deployment** button: ![Create new environment](../assets/use-cases/autotest-as-quality-gate/create_new_deployment.png "Create new environment") 5. On the **Applications** tab, add all the three applications, specify the **main** branch for all for them and click the **Continue** button: ![Pipeline tab](../assets/use-cases/autotest-as-quality-gate/applications-tab.png "Pipeline tab") 6. Enter `demo-deploy` name, specify description, enable the "Promote applications" option and click **Continue**: ![Applications tab](../assets/use-cases/autotest-as-quality-gate/pipeline-configuration-tab.png "Applications tab") 7. Review the Deployment configuration and click **Continue**: ![Deployment review](../assets/use-cases/autotest-as-quality-gate/deployment-review-tab.png "Deployment review") 7. Once all the steps have been completed, you may begin creating Environments: ![Deployment created](../assets/use-cases/autotest-as-quality-gate/deployment_created.png "Deployment created") 8. On the Deployment details page, click the **+ Create environment** button: ![Add new Environment](../assets/use-cases/autotest-as-quality-gate/add-new-env.png "Add new Environment") 9. On the **Basic configuration** step, define the following values and click **Continue**: - Cluster: `in-cluster` - Stage name: `dev` - Description: `Development Environment` ![Configure stage](../assets/use-cases/autotest-as-quality-gate/configure-environment.png "Configure stage") 10. On the **Pipeline configuration** step, define the following values and click **Continue**: - Trigger type: `Manual` - Deploy pipeline template: `deploy-with-autotests` - Clean pipeline template: `clean` ![Pipeline configuration tab](../assets/use-cases/autotest-as-quality-gate/pipeline-configuration.png "Pipeline configuration tab") 11. In the **Quality gates** menu click **+** button. Specify the following parameters and click **Continue**: First Quality Gate: - Quality gate type: `Autotest` - Step name: `autotest-step-1` - Autotest: `demo-autotest-gradle` - Autotest branch: `main` Second Quality Gate: - Quality gate type: `Autotest` - Step name: `autotest-step-2` - Autotest: `demo-autotest-gradle` - Autotest branch: `main` Third Quality Gate: - Quality gate type: `Autotest` - Step name: `autotest-step-3` - Autotest: `demo-autotest-maven` - Autotest branch: `main` Fourth Quality Gate: - Quality gate type: `Autotest` - Step name: `autotest-step-4` - Autotest: `demo-autotest-maven` - Autotest branch: `main` ![Quality gates tab](../assets/use-cases/autotest-as-quality-gate/quality_gates_tab.png "Quality gates tab") :::note The same automated test can be executed multiple times within a single deployment pipeline using different step names. To achieve this, specify a unique step name as a parameter when triggering the Tekton automated test pipeline. Verify that the key names in demo-autotest-gradle/run.json and demo-autotest-maven/run.json comply with the required template and update them if necessary. ```yaml title="demo-autotest-gradle/run.json" { "comment": "step-name: --", "demo-deploy-dev-autotest-step-1": "gradle -q hello", "demo-deploy-dev-autotest-step-2": "gradle -q hello", "demo-deploy-dev-autotest-step-3": "mvn antrun:run@hello -q", "demo-deploy-dev-autotest-step-4": "mvn antrun:run@hello -q" } ``` ::: 12. Review the changes and click **Create environment**: ![Review environment](../assets/use-cases/autotest-as-quality-gate/review-environment.png "Review environment") 13. Repeat the steps 8-10 to create one more stage with the parameters below: - Cluster: `in-cluster` - Stage name: `sit` - Description: `System integration testing` - Trigger type: `manual` - Deploy pipeline template: `deploy` - Clean pipeline template: `clean` - Quality gate type: `Manual` ### Run Autotests After the Deployment is created, deploy applications and run autotests by following the steps below: 1. Open the **sit** Environment name by clicking on its name: ![Sit environment](../assets/use-cases/autotest-as-quality-gate/deploy_applications_sit.png "Sit environment") 2. Select the **Applications** tab and click **Configure deploy**. Hover your cursor over the application versions to see the image that blocks you from running a deployment: ![Deployment blocked](../assets/use-cases/autotest-as-quality-gate/deployment_blocked.png "Deployment blocked") 3. Get back to the Deployment details page. Click the **dev** stage name to expand its details: ![Deploy applications](../assets/use-cases/autotest-as-quality-gate/deploy_applications.png "Deploy applications") 4. Select the **Applications** tab and click **Configure deploy**. Then select latest versions of all the applications and click **Start deploy**: ![Run deploy pipeline](../assets/use-cases/autotest-as-quality-gate/run_deploy_pipeline.png "Run deploy pipeline") 5. To find information about the running deploy pipeline, navigate to the **Pipelines** tab and click on the pipeline name: ![Deploy pipeline information](../assets/use-cases/autotest-as-quality-gate/deploy_pipeline.png "Deploy pipeline information") 6. View the pipeline details. Pay attention that the **deploy-with-autotest** pipeline features the **wait-for-autotests** task: ![Wait for autotests](../assets/use-cases/autotest-as-quality-gate/wait-for-autotests.png "Wait for autotests") 7. Wait until the deploy pipeline completes. The application statuses should be **Healthy** and **Synced**: ![Successful deployment](../assets/use-cases/autotest-as-quality-gate/successful-deployment.png "Successful deployment") 8. Once promotion procedure is finished, the promoted applications will become available in the **Sit** stage. You will be able to select image stream versions for the promoted applications: ![Sit stage](../assets/use-cases/autotest-as-quality-gate/deploy_promoted_stage.png "Sit stage") ## Related Articles - [Add Application](../user-guide/add-application.md) - [Add Autotest](../user-guide/add-autotest.md) - [Add CD Pipeline](../user-guide/add-cd-pipeline.md) - [Add Quality Gate](../user-guide/add-quality-gate.md) --- ## Test Environment Using Environment Variables # Set Test Suite Parameters Using Environment Variables in CD Pipelines This use case demonstrates how KubeRocketCI uses environment variables to flexibly configure commands in deployment pipelines. Teams can dynamically adjust parameters by adding, modifying, or removing them without changing the test suite code. This provides precise control over quality gates, streamlines workflows, and allows pipelines to adapt to different environments or requirements while maintaining efficiency and quality standards. ## Goals - Flexible and convenient preparation and validation of the testing environment using environment variables. - The ability to manage environment variables by overriding, adding, or removing their values and keys to configure test parameters. - A streamlined setup process that allows teams to focus on development without platform constraints. ## Preconditions - KubeRocketCI instance with GitHub is [configured](../operator-guide/prerequisites.md). - Developer has access to the KubeRocketCI instance using the Single-Sign-On approach. - Developer has `Write` permissions for GitHub repository to merge the code. ## Scenario To implement specific logic in deployment quality gate within the KubeRocketCI platform, follow these steps: - **Fork the Tekton Chart**: Fork, clone and deploy the prepared Tekton chart that implements the required Tekton resources using environment variables. - **Add the Application**: Add a simple application from the platform marketplace and build the application to prepare it for deployment. - **Add the Autotests**: Set up the autotests with the clone strategy to include them in the Deployment. - **Create Deployment, Environment, and Set Variables**: Create a Deployment, Set up an environment for the deployment, add Autotests as a quality gate to the deployment, and define the variable. - **Validate**: Review the pipeline logs to ensure autotests execute correctly and produce the expected output. ### Fork the Tekton Chart The first step is to clone and install the Tekton chart: 1. In GitHub, create a private repository. 2. Clone your private repository to your local machine: ![Clone repo](../assets/use-cases/cd-autotests-run-with-env-variables/clone-repo.png "Clone repo") ```bash git clone git@github.com:/variables-case-tekton.git ``` 3. Copy the [chart repository](https://github.com/KubeRocketCI/variables-case-tekton) content to your newly created repository. The repository structure should be as follows:
Repository structure ```bash variables-case-tekton . ├── Chart.lock ├── Chart.yaml ├── README.md ├── chart_schema.yaml ├── ct.yaml ├── lintconf.yaml ├── templates │ ├── pipelines │ │ └── cd │ │ ├── autotest-variables.yaml │ │ └── deploy-with-autotests-variables.yaml │ ├── tasks │ │ └── cd │ │ ├── init-autotests-variables.yaml │ │ └── run-autotests-variables-java.yaml │ └── triggers │ └── cd │ └── deploy-with-autotests-variables.yaml └── values.yaml ```
4. Update the component configuration in the `values.yaml` file: ```yaml # Replace `example.domain.com` with the appropriate wildcard domain for your setup. dnsWildCard: "example.domain.com" ``` 5. Install the chart: ```bash cd variables-case-tekton helm upgrade --install custom-tekton-chart . -n krci ``` ### Add Application As soon as the Helm chart is installed, the new deploy pipeline will appear in the Environment configuration window. The next step is to create an application from a template: 1. Create a new Project with the `Application` type using the `Create` strategy. Select the **Projects** section and click **+ Create project**: ![Projects tab](../assets/use-cases/general/components-1.png "Projects tab") 2. Select the **Select ready template** option, **Simple Spring Boot Application** and click **Continue**: ![Initial setup](../assets/use-cases/cd-autotests-run-with-env-variables/predefined-template.png "Initial setup") 3. On the **Git & project info** tab, define the following values and click **Continue**: * Git server: `github` * Owner: `GitHub_username` * Repository name: `spring-boot-app` * Default branch: `main` * Project name: `spring-boot-app` * Description: `Simple spring boot application` ![Application creation](../assets/use-cases/cd-autotests-run-with-env-variables/codebase-app.png "Application creation") 4. On the **Build config** tab, define the values and click **Continue**: * Codebase versioning type: `semver` * Start version from: `0.1.0` * Suffix: `SNAPSHOT` ![Application creation](../assets/use-cases/cd-autotests-run-with-env-variables/git-project-info.png "Application creation") 5. On the **Review** tab, verify the project configuration and click **Create project**: ![Review and create](../assets/use-cases/cd-autotests-run-with-env-variables/review-and-create.png "Review and create") 6. On the congratulations menu, click **Open project**: ![Ready project](../assets/use-cases/cd-autotests-run-with-env-variables/ready-component.png "Ready project") 7. Select the **Branches** tab and click the **Build** button: ![Trigger build](../assets/use-cases/cd-autotests-run-with-env-variables/trigger-build.png "Trigger build") 8. Click the PipelineRun name to view its details: ![Click PipelineRun name](../assets/use-cases/cd-autotests-run-with-env-variables/click-pipeline-name.png "Click PipelineRun name") 9. Wait until the build is successful and its status is **Completed**: ![PipelineRun completed](../assets/use-cases/cd-autotests-run-with-env-variables/pipeline-completed.png "PipelineRun completed") ### Add Autotests Now you need to create template autotests that will be executed when deploying the application: 1. To add an autotest, navigate to the **Projects** section on the navigation bar and click **+ Create project**. 2. On the **Create new project** dialog, select **Custom configuration** -> **Autotest** -> **Clone** and click **Continue**: ![Autotests project](../assets/use-cases/cd-autotests-run-with-env-variables/codebase-autotest.png "Autotests project") 3. On the **Git & project info** tab, define the following values and click **Continue**: * RepoUrl: `https://github.com/KubeRocketCI/demo-autotests-gradle` * Git server: `github` * Owner: `GitHub_username` * Repository name: `autotests` * Default branch: `master` * Project name: `autotests` * Description: `Simple autotests gradle` ![Autotests codebase](../assets/use-cases/cd-autotests-run-with-env-variables/autotests-project-info.png "Autotests codebase") 4. On the **Build config** tab, define the values and click **Continue**: * Autotest code language: `Java` * Language version/framework: `Java17` * Build tool: `Gradle` * Autotest report framework: `allure` * Codebase versioning type: `semver` * Start version from: `0.1.0` * Suffix: `SNAPSHOT` ![Autotests codebase](../assets/use-cases/cd-autotests-run-with-env-variables/autotests-build-config.png "Autotests codebase") 5. On the **Review** tab, verify the project configuration and click **Create project**: ![Autotests codebase advanced settings](../assets/use-cases/cd-autotests-run-with-env-variables/autotest-review.png "Autotests codebase advanced settings") 6. On the congratulations menu, click **View all projects**. 7. Click **Close**. Now the application and autotest are created: ![Projects overview](../assets/use-cases/cd-autotests-run-with-env-variables/projects-overview.png "Projects overview") ### Create Deployment, Environment, and Set Variables The next step is to configure a Deployment with appropriate deploy pipeline, application, autotests, and variables: :::note To utilize and manage various environments through the KubeRocketCI platform, the initial step is to onboard a new [GitOps repository](../user-guide/gitops.md). ::: 1. Navigate to the `Deployments` section and click the **Click here to add a repository** button: ![Deployments tab](../assets/use-cases/fastapi-scaffolding/add-gitops-repo.png "Deployments tab") 2. Click to the **+ Add GitOps repository**: ![Onboard gitops repository](../assets/use-cases/fastapi-scaffolding/add-gitops-button.png "Onboard GitOps repository") 3. Select the `Create` strategy and `github` server, enter GitHub account name and click **Save**: ![Configure gitops repository](../assets/use-cases/fastapi-scaffolding/create-gitops.png "Configure gitops repository") 4. In the KubeRocketCI portal, navigate to **Deployments** tab and click the **+ Create deployment** button: ![Create deployment](../assets/use-cases/general/create-cd-pipeline-1.png "Create deployment") 5. On the **Applications** tab, add the `spring-boot-app` application, specify the **main** branch for it, and click **Continue**: ![Applications tab](../assets/use-cases/cd-autotests-run-with-env-variables/deployment-add-application.png "Applications tab") 6. On the **Pipeline configuration** tab, fill in the required fields and click **Continue**: * Pipeline name: `demo-pipeline` * Description: `Deployment with specific variables` * Deployment type: `Container` ![Pipeline configuration tab](../assets/use-cases/cd-autotests-run-with-env-variables/deployment-pipeline-configuration.png "Pipeline configuration tab") 7. On the **Review** tab, verify the configuration and click **Create deployment**: ![Review deployment](../assets/use-cases/cd-autotests-run-with-env-variables/review-deployment.png "Review deployment") 8. On the congratulations page, select **Open deployment**: ![Deployment created](../assets/use-cases/cd-autotests-run-with-env-variables/deployment-created.png "Deployment created") 9. On the Deployment details page, click the **+ Create Environment** button: ![Create Environment](../assets/use-cases/cd-autotests-run-with-env-variables/create-environment.png "Create Environment") 10. On the **Basic configuration** tab, define the following values and click **Continue**: * Cluster: `in-cluster` * Environment name: `qa` * Description: `Testing environment` ![Basic configuration tab](../assets/use-cases/cd-autotests-run-with-env-variables/environment-basic-configuration.png "Basic configuration tab") 11. On the **Pipeline configuration** tab, define the following values and click **Continue**: * Trigger type: `Manual` * Pipeline template: `deploy-with-autotests-variables` * Clean pipeline template: `clean` ![Pipeline configuration](../assets/use-cases/cd-autotests-run-with-env-variables/environment-pipeline-configuration.png "Pipeline configuration") 12. On the **Quality gates** tab, click the pencil icon to edit the first quality gate. Specify the following parameters and click **Update quality gate** and then **Continue**: * Quality gate type: `Autotest` * Autotest: `autotests` * Step name: `Autotest` * Autotest branch: `master` ![Add quality gates](../assets/use-cases/cd-autotests-run-with-env-variables/add-quality-gates.png "Add quality gates") 13. On the **Review** tab, verify the Environment configuration and click **Create environment**: ![Environment details](../assets/use-cases/cd-autotests-run-with-env-variables/environment-review.png "Environment details") 14. Click the **Open environment** button: ![Open Environment](../assets/use-cases/cd-autotests-run-with-env-variables/open-environment.png "Open Environment") 15. Select the **Variables** tab and click the **Add variable** button: ![Add variables](../assets/use-cases/cd-autotests-run-with-env-variables/add-variables.png "Add variables") 16. Enter `MY_COMMAND`, and set the **Value** to `gradle -q hello`. Then, click the **Save** button: ![Set variable](../assets/use-cases/cd-autotests-run-with-env-variables/set-variable.png "Set variable") In the Tekton task run-autotests-gradle-variables, we have implemented logic to check for the MY_COMMAND variable. If it exists, the task uses the value of this variable as the command. If it is missing or empty, the task falls back to using the logic defined in the run.json file from the autotests repository.
run-autotests-gradle-variables.yaml ```yaml apiVersion: tekton.dev/v1 kind: Task metadata: name: run-autotests-gradle-variables labels: app.kubernetes.io/based-on: "0.2" {{- include "edp-tekton.labels" $ | nindent 4 }} annotations: tekton.dev/pipelines.minVersion: "0.12.1" tekton.dev/categories: Autotest Tools tekton.dev/tags: autotest-tool tekton.dev/platforms: "linux/amd64,linux/s390x,linux/ppc64le" spec: description: >- This task runs autotests for a specified Environment using either Maven or Gradle based on the provided configuration, extracting and executing the relevant commands from a `run.json` file. workspaces: - name: source description: A workspace that contains the repository. params: - name: DEPLOYMENT_FLOW type: string - name: ENVIRONMENT type: string - name: base-image type: string steps: - name: run-autotest image: "$(params.base-image)" workingDir: $(workspaces.source.path) env: - name: ENVIRONMENT value: $(params.ENVIRONMENT) - name: DEPLOYMENT_FLOW value: $(params.DEPLOYMENT_FLOW) envFrom: - configMapRef: name: $(params.DEPLOYMENT_FLOW)-$(params.ENVIRONMENT) script: | #!/bin/bash set -exo pipefail # Check if the MY_COMMAND variable is set and not empty if [[ -n "${MY_COMMAND}" ]]; then # If MY_COMMAND is defined, execute its content eval "${MY_COMMAND}" else # If MY_COMMAND is not set, execute the existing logic $(sed -n 's/.*"'$ENVIRONMENT'": "\(.*\)",/\1/p' run.json | awk -F '"' '{print $1}') fi ```
17. Navigate to the **Applications** tab. Then click **Configure deploy** -> **Latest** -> **Start deploy**: ![Deploy application](../assets/use-cases/cd-autotests-run-with-env-variables/start-deploy-application.png "Deploy application") ### Validate Now it is time to review the pipeline logs to ensure autotests pass successfully and application produces the expected output: 1. To find autotests logs, navigate to **CI/CD Pipelines** -> **PipelineRuns** and wait until all pipelines are done: ![Pipelines list](../assets/use-cases/cd-autotests-run-with-env-variables/pipelines-list.png "Pipelines list") 2. Click on the **autotests-variables-gradle-run-xxxxx** PipelineRun, select the **run autotests** stage and the **run autotests** task in the logs page. You will see your autotests output: ![Autotests logs](../assets/use-cases/cd-autotests-run-with-env-variables/autotest-logs.png "Autotests logs") ## Related Articles - [Add Application](../user-guide/add-application.md) - [Add Autotest](../user-guide/add-autotest.md) - [Add Deployment](../user-guide/add-cd-pipeline.md) - [Add Quality Gate](../user-guide/add-quality-gate.md) - [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) - [Manage Deployments](../user-guide/manage-environments.md) --- ## Create and Use Custom Tekton Pipelines # Create and Use Custom Tekton Pipelines This use case explains how to create and use personalized Tekton pipelines in KubeRocketCI. While KubeRocketCI offers pre-configured Tekton pipelines for common use cases, custom pipelines allow you to adapt workflows to meet unique project requirements. It also provides guidance on integrating and using these custom Tekton pipelines within your development processes. ## Goals - Provide a clear method for creating and configuring custom Tekton tasks and pipelines for build, review, deployment, and clean processes within KubeRocketCI. - Offer a detailed guide on integrating custom pipelines with KubeRocketCI, ensuring seamless connection and functionality. - Clarify the process for replacing existing build, review, deployment, and clean pipelines with custom pipelines, highlighting necessary steps and considerations. - Demonstrate the benefits of using custom Tekton pipelines in KubeRocketCI, such as improved flexibility, scalability, and efficiency in development workflows. ## Prerequisites Before proceeding with this use case, ensure the following prerequisites are met: - Access to a KubeRocketCI instance with permissions to create and edit **Projects** and **Environments**. - A configured KubeRocketCI instance with at least one active [Git Server](../user-guide/add-git-server.md) (e.g., GitHub, GitLab or Bitbucket). - Configured Argo CD instance with the [Add-ons repository](../operator-guide/add-ons-overview.md) added. ## Scenario - Create a Tekton library in KubeRocketCI for storing and managing custom Tekton resources. - Define custom Tekton pipelines for build, review, deployment, or clean processes based on specific project requirements. - Integrate custom Tekton pipelines with KubeRocketCI to automate build, review, and deployment workflows using Argo CD and the add-ons repository. - Replace existing build, review, deploy, or clean pipelines with custom pipelines. ### Creating a Tekton Library for Custom Pipelines The first step in creating and using custom Tekton pipelines is to set up a Tekton library in KubeRocketCI. This library acts as a central repository for storing and managing custom Tekton tasks, pipelines, and trigger templates. :::note The Tekton library template can be found in the [helm-helm-pipeline](https://github.com/epmd-edp/helm-helm-pipeline) repository. ::: To create a Tekton library, follow these steps: 1. Log in to the KubeRocketCI portal: ![Logging Page](../assets/use-cases/general/login-1.png "Logging screen") 2. On the main menu, click the **Manage Namespaces** button: ![Manage Namespaces](../assets/use-cases/general/settings-1.png "Manage Namespaces button") 3. Ensure the `Namespace` value points to the namespace with the KubeRocketCI installation: ![Default and allowed namespaces](../assets/use-cases/fastapi-scaffolding/default-allowed-namespace.png "Default and allowed namespaces") :::note Don't forget to press Enter to add the namespace to the allowed namespaces list. ::: 4. Create a new Project with the `Application` type using the `Create` strategy. Select the **Projects** section and click **+ Create project**: ![Components Overview](../assets/use-cases/general/components-1.png "Components tab") 5. Select **Custom configuration** -> **Library** -> **Create** and click **Continue**: ![Strategy type](../assets/use-cases/custom-tekton-pipelines/strategy-type.png "Strategy type") 6. On the **Git & project info** tab, define the following values and click **Continue**: - Git server: `github` - Owner: `github_account_name` - Repository name: `tekton-custom-pipelines` - Default branch: `main` - Project name: `tekton-custom-pipelines` - Description: `Repository for storing and managing custom Tekton resources` ![Component Info](../assets/use-cases/custom-tekton-pipelines/component-info.png "Component Info") 7. On the **Build config** tab, define the values and click **Continue**: - Library code language `Helm` - Language version/framework: `Pipeline` - Build tool: `Helm` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Advanced Settings](../assets/use-cases/custom-tekton-pipelines/build-config.png "Advanced Settings") 8. On the **Review** tab, verify the project configuration and click **Create project**: ![Advanced Settings](../assets/use-cases/custom-tekton-pipelines/library-review-and-create.png "Advanced Settings") 9. On the congratulations page, click **Open project**: ![Open project](../assets/use-cases/custom-tekton-pipelines/library-congratulations.png "Open project") Once the library is created, you can start adding custom Tekton tasks, pipelines, and trigger templates to the library. ### Defining Custom Tekton Pipelines After creating the Tekton library, the next step is to define custom Tekton pipelines for build, review or deployment processes based on your project requirements. To define custom Tekton pipelines, follow these steps: 1. In the top-right corner of the Project details page, click the **Git** button to open the Git repository for the `tekton-custom-pipelines` library: ![Git Repository](../assets/use-cases/custom-tekton-pipelines/git-repository.png "Git Repository") 2. Clone the repository to your local machine: ![Clone Repository](../assets/use-cases/custom-tekton-pipelines/clone-repository.png "Clone Repository") You can use the following command to clone the repository: ```bash git clone ``` 3. Examine the repository structure. By default, the repository structure looks as follows: ```plaintext tekton-custom-pipelines ├── Chart.yaml ├── templates │   ├── pipelines │   │   ├── deploy │   │   │   ├── custom-clean.yaml │   │   │   └── custom-deploy.yaml │   │   ├── bitbucket-build-default.yaml │   │   ├── bitbucket-build-semver.yaml │   │   ├── bitbucket-build-lib-default.yaml │   │   ├── bitbucket-build-lib-semver.yaml │   │   ├── bitbucket-review.yaml │   │   ├── gerrit-build-default.yaml │   │   ├── gerrit-build-semver.yaml │   │   ├── gerrit-build-lib-default.yaml │   │   ├── gerrit-build-lib-semver.yaml │   │   ├── gerrit-review.yaml │   │   ├── github-build-default.yaml │   │   ├── github-build-semver.yaml │   │   ├── github-build-lib-default.yaml │   │   ├── github-build-lib-semver.yaml │   │   ├── github-review.yaml │   │   ├── gitlab-build-default.yaml │   │   ├── gitlab-build-semver.yaml │   │   ├── gitlab-build-lib-default.yaml │   │   ├── gitlab-build-lib-semver.yaml │   │   └── gitlab-review.yaml │   ├── resources │   │   └── npm-settings.yaml │   ├── tasks │   │   ├── deploy │   │   │   └── hello-world-deploy.yaml │   │   └── hello-world.yaml │   └── triggers │   ├── custom-clean.yaml │   └── custom-deploy.yaml └── values.yaml ``` - The `templates` directory is a main directory that contains templates for custom Tekton resources. This directory is divided into subdirectories for pipelines, tasks, triggers, and resources. - The `templates/pipelines` directory contains Tekton pipeline templates for build and review processes. For each Git server (e.g., GitHub, GitLab, Bitbucket, or Gerrit), there are specific templates, organized by component and versioning type. - The appropriate template should be selected based on the configuration of your component. For example: - If the component is created in GitHub, has the `Application` component type, and uses `semver` versioning, you should use the `github-build-semver.yaml` template. - If the component is created in Bitbucket, has the `Library` component type, and uses the `default` versioning, you should use the `bitbucket-build-lib-default.yaml` template. Here is an example illustrating the naming conventions used in template filenames: ![Pipeline Templates](../assets/use-cases/custom-tekton-pipelines/file-structure.png "File name structure") - The `templates/pipelines/deploy` directory contains Tekton pipeline templates for deploy and clean processes. Select the appropriate template based on the deployment process requirements. - The `templates/tasks` directory contains Tekton task templates for specific tasks that are part of the build, review, or deployment processes. - The `templates/triggers` directory contains Tekton trigger templates for triggering the deployment or clean processes. - The `templates/resources` directory is intended for various templates, such as config maps and secrets, which can be mounted into custom Tekton tasks as needed. 4. Modify the existing pipeline and task templates or create new ones based on your project requirements. **Result**: This step will demonstrate an example of creating custom Tekton resources. After completing this step, you will have a custom Tekton task, pipeline, and trigger template. **Usage**: The created custom Tekton resources can be used later for components on the KubeRocketCI platform. This flow serves as an example that can be used when creating your own, specific custom Tekton resources for your project's needs Consider the scenario where a custom task `hello-world` needs to be created and used in a custom Tekton pipeline. Follow the steps below to achieve this: 1. Create a custom Tekton task. Create a custom Tekton task called `hello-world` to be used in a custom build pipeline. This task should be created in the `templates/tasks` directory with the name `hello-world.yaml`. The task will accept two parameters: `BASE_IMAGE` and `username`. The `BASE_IMAGE` parameter specifies the base Docker image to be used for the task, while the `username` parameter is used to personalize the task's output. Here is an example of the `hello-world.yaml` task template: ```yaml apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: hello-world labels: {{- include "edp-tekton.labels" . | nindent 4 }} spec: params: - name: BASE_IMAGE type: string default: "alpine" description: "The base Docker image to use for the task" - name: username type: string description: "The username to personalize the task's output" workspaces: - description: A workspace that contains fetched git repo. name: source steps: - name: hello-world image: $(params.BASE_IMAGE) workingDir: $(workspaces.source.path) script: | #!/bin/sh echo "Hello $(params.username), the repository contains the following files:" ls -al ``` 2. Create a Custom Tekton Pipeline. Define a custom Tekton pipeline that uses the `hello-world` custom task. Since the application for which this pipeline will be used is located in GitHub, has the component type `Application`, and has the versioning type `semver`, use the template `github-build-semver.yaml` in the `templates/pipelines` directory. First, set the parameters in the `spec.params` field that will be used in the custom task: ```yaml spec: params: - default: "World" description: "Example parameter" name: username type: string - default: 'alpine:3.18.9' name: image-version type: string ``` Then, in the `spec.tasks` field, after the `get-version` task, describe the custom `hello-world` task. Also, in the subsequent `git-tag` task, specify the `runAfter: hello-world` field to maintain the sequence of execution. Here is an example of part of the `github-build-semver.yaml` pipeline template: ```yaml apiVersion: tekton.dev/v1 kind: Pipeline metadata: name: github-build-semver labels: app.edp.epam.com/pipelinetype: build spec: tasks: ... - name: hello-world taskRef: name: hello-world runAfter: - get-version params: - name: BASE_IMAGE value: "$(params.image-version)" - name: username value: "$(params.username)" workspaces: - name: source workspace: shared-workspace subPath: source - name: git-tag params: - name: GIT_USER_EMAIL value: edp-ci@edp.ci-user - name: GIT_USER_NAME value: edp-ci - name: GIT_SCRIPT value: | git tag -a "$(tasks.get-version.results.VCS_TAG)" -m "Tag is added automatically by CI user" git push --tags runAfter: - hello-world # Ensure to update the runAfter field for task that should run after the custom task with the correct task name taskRef: kind: Task name: git-cli ... ``` 3. Create a Custom Tekton Trigger Template. Create a Tekton Trigger Template to invoke the custom `github-build-semver` pipeline. Use the `custom-deploy.yaml` template in the `templates/triggers` directory. Modify the `.spec.resourcetemplates.spec.pipelineRef.name` field to reference the custom pipeline that uses the `hello-world` custom Tekton task, in this case, `custom-deploy`. Here is the relevant part of the `custom-deploy.yaml` trigger template: ```yaml apiVersion: triggers.tekton.dev/v1beta1 kind: TriggerTemplate metadata: name: custom-deploy labels: app.edp.epam.com/pipelinetype: deploy ... spec: resourcetemplates: - apiVersion: tekton.dev/v1beta1 kind: PipelineRun ... spec: pipelineRef: name: custom-deploy ... ``` 4. Optional: Specify the `dnsWildCard` field in the `values.yaml` file. If custom Tekton review pipelines have been created, it is necessary to specify the `dnsWildCard` field in the `values.yaml` file. This field will be used to form the `pipelineUrl` variable, which is utilized in merge and pull requests to navigate to the KubeRocketCI Portal. Here is an example of the `values.yaml` file: ```yaml dnsWildCard: "example.com" ``` 5. Commit and push the changes to the Git repository. After modifying the pipeline and task templates, commit and push the changes to the Git repository. Use the following commands to commit and push the changes: ```bash git add . git commit -m "Add custom task and pipeline templates" git push origin main ``` ### Deliver Custom Tekton Pipelines to the Cluster To deploy custom pipelines to the cluster, you can use Argo CD, which includes a repository with [add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main). :::note Before proceeding, ensure you have added Add-Ons repository and application according to the [Install via Add-Ons](../operator-guide/add-ons-overview.md) page. Ensure to add the credentials template in the Argo CD settings to grant Argo CD access to both your Add-Ons and tekton-custom-pipelines repositories. ::: To deliver custom Tekton pipelines to the cluster, follow these steps: 1. Clone the private repository with add-ons and make the following changes. In the `clusters/core/apps/values.yaml` file, set the `repoUrl` and `namespace` fields to specify the Git URL of the `tekton-custom-pipelines` repository and the namespace where the KubeRocketCI platform is deployed. Also, set the `kuberocketci-pipelines.enabled` field to `true` to enable the deployment of the `tekton-custom-pipelines` Argo CD Application: ```yaml kuberocketci-pipelines: enable: true namespace: repoUrl: ssh://git@github.com:22//tekton-custom-pipelines.git ``` 2. Commit and push the changes to the repository. After making the necessary changes, commit and push the changes to the repository. Use the following commands to commit and push the changes: ```bash git add . git commit -m "Enable custom Tekton pipelines deployment" git push origin main ``` 3. After pushing the changes, access **Argo CD**, navigate to the Application that corresponds to the repository with the **add-ons**, and initiate the `Sync` process. This will apply the `tekton-custom-pipelines` Helm Chart to the cluster within the specified namespace: ![Argo CD Sync](../assets/use-cases/custom-tekton-pipelines/argocd-sync.png "Argo CD Sync") ### Replace Pipelines With Personalized Versions This section demonstrates how to replace existing build, review, deploy, and clean pipelines with custom pipelines in KubeRocketCI. Two cases are covered: replacing build and review pipelines for a created component, and replacing deploy and clean pipelines for an existing deployment. 1. To replace existing review or build pipelines for a component with custom pipelines, follow these steps: 1. Open the KubeRocketCI portal. Navigate to the **Projects** tab and select the component for which you want to replace the build or review pipeline: ![Component](../assets/use-cases/custom-tekton-pipelines/java-app-project.png "Component") 2. In the **Branches** tab, select the branch for which you want to replace the pipeline. Click the three dots (**⋮**) menu and click **Edit**: ![Edit Branch](../assets/use-cases/custom-tekton-pipelines/edit-branch.png "Edit Branch") 3. In the **Edit branch** dialog, select the required build or review pipeline from the dropdown list: ![Select Pipeline](../assets/use-cases/custom-tekton-pipelines/select-pipeline.png "Select Pipeline") 2. To replace existing deploy or clean pipelines for a deployment with custom pipelines, follow these steps: 1. Open the KubeRocketCI portal. Navigate to the **Deployments** tab and select the deployment for which you want to replace the deploy or clean pipeline: ![Deployment](../assets/use-cases/custom-tekton-pipelines/deployment.png "Deployment") 2. Choose the required Environment: ![Environment](../assets/use-cases/custom-tekton-pipelines/environment.png "Environment") 3. Click the **Edit** button to edit the Environment: ![Edit button](../assets/use-cases/custom-tekton-pipelines/edit-environment.png "Edit button") 4. In the **Edit Environment** dialog, select the required deploy or clean pipeline from the dropdown list: ![Select Pipeline](../assets/use-cases/custom-tekton-pipelines/custom-deploy-clean-pipelines.png "Select Pipeline") ## Related Articles - [Tekton Overview](../operator-guide/ci/tekton-overview.md) - [Add Application](../user-guide/add-application.md) - [Add Deployment](../user-guide/add-cd-pipeline.md) - [Argo CD Overview](../quick-start/integrate-argocd.md) - [Cluster Add-Ons Overview](../operator-guide/add-ons-overview.md) --- ## Deploy Application From a Feature Branch # Deploying Feature Branches With KubeRocketCI: A Comprehensive Guide for Efficient Application Testing and Deployment Deploying feature branches is crucial for testing activities, including manual testing of the product's functionality, running quality gates, and verifying dependencies or integrations with other components. KubeRocketCI enables the deployment of feature environments straight from feature branches. This guide offers comprehensive instructions for managing and deploying these branches. For a hands-on, end-to-end walkthrough on a local cluster — covering auto-triggered deployments, proof of namespace isolation, per-environment GitOps values override, and one-click teardown — see the [Ephemeral Environments on Kubernetes: Feature Branch Preview Walkthrough](/blog/ephemeral-preview-environments-kubernetes-feature-branch) blog post. ### Preconditions Before you start the use case, ensure to meet the following requirements: - KubeRocketCI instance is [configured](../operator-guide/prerequisites.md) with [GitOps](../user-guide/gitops.md) repository. - Developer has access to the KubeRocketCI instance using the [Single-Sign-On](../operator-guide/auth/platform-auth-model.md) approach or via token. - Developer has access to the KubeRocketCI platform under the [Developer](../operator-guide/auth/platform-auth-model.md#keycloak-groups) role. - Application you want to add a branch to is [onboarded](../user-guide/add-application.md) in KubeRocketCI. ### Goals Below are the goals to complete in the use case: - Onboard a feature branch for the application. - Configure the GitOps approach for the Environment by adding a values.yaml file with application parameters within it. - Deploy application with overridden parameters. ## Scenario The use case scenario contains the following stages: - **Create a feature branch**: Create a feature branch and build artifact for it. - **Create a Deployment**: Create a Deployment that contains an Environment and deploy application within the Environment. - **Configure application parameters**: Adjust a GitOps repository by adding custom application configuration into it. - **Apply custom settings for application**: Deploy application with custom parameters. ### Create Feature Branch There are two approaches to onboard a feature branch: 1. **Create a Feature Branch via KubeRocketCI**: KubeRocketCI creates a new branch in GitHub. 2. **Onboard an Existing Branch in KubeRocketCI**: Manually [create the branch in GitHub](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-and-deleting-branches-within-your-repository) using the correct naming convention and then onboard it into KubeRocketCI. In this use case, we will follow the first approach to add a branch. When creating a feature branch, ensure the branch name is lowercase to meet Kubernetes restrictions. As an example, we will use the `feature/` pattern as a naming convention for feature branches. To create a feature branch via KubeRocketCI, follow these steps: 1. Open the KubeRocketCI portal and log into the platform: ![Login page](../assets/use-cases/general/login-1.png "Login page") 2. Navigate to the **Projects** section and select the desired Project. 3. Select the **Branches** tab and click the **Create branch** button: ![Create branch](../assets/use-cases/deploy-application-from-feature-branch/create-branch-button.png "Create branch") 4. Click the **Create branch** button and fill in the form: * **Branch Name**: Specify the name (e.g., `feature/tt-000`). * **From Commit Hash**: Specify the commit hash or leave it empty for the latest commit. * **Branch Version**: Provide a version tag for the branch. * **Review Pipeline**: Leave the default pipeline. * **Build Pipeline**: Leave the default pipeline. ![Create branch button](../assets/use-cases/deploy-application-from-feature-branch/create-branch-window.png "Create branch button") 5. Click **Create** to finalize the branch creation. 6. The branch will also be created in GitHub. Verify its creation in the repository: ![Codebase branches in GitHub](../assets/use-cases/deploy-application-from-feature-branch/github-codebase-branches.png "Codebase branches in GitHub") Since we don't create a pull request to merge our feature with the main branch, we need to manually trigger the build pipeline in the KubeRocketCI portal. 7. Build the application using the **Build** button: ![Build button](../assets/use-cases/deploy-application-from-feature-branch/build-application.png "Build button") 8. View the build pipeline run details by selecting the PipelineRun name: ![Build pipeline details](../assets/use-cases/deploy-application-from-feature-branch/build-pipeline-details.png "Build pipeline details") ### Deploy Feature Branch Once you have completed the onboarding process for the feature branch, you can deploy its artifacts to a dedicated feature environment. #### Create Deployment We recommend using initials to name the Deployment. For example, a user with the email `firstname_lastname@example.com` should use `fl` as the identifier. To create a feature environment, follow the steps below: 1. Open the **Deployments** section and click **Create Deployment**: ![Create deployment button](../assets/use-cases/general/create-cd-pipeline-1.png "Create deployment button") 2. In the **Create new deployment** dialog, fill in the required fields: - **Applications**: - Applications: Add `inventory-service` application - Branch: Select `feature/tt-000` branch ![Applications step](../assets/use-cases/deploy-application-from-feature-branch/create-deployment-applications.png "Applications step") - **Pipeline configuration**: - Pipeline name: `fl` - Description: `Deploy the "inventory-service" application for the "Firstname Lastname" user` - Deployment type: Select `Container` - Promote applications: Leave unchecked ![Pipeline configuration step](../assets/use-cases/deploy-application-from-feature-branch/create-deployment-pipeline-configuration.png "Pipeline configuration step") - **Review**: Verify the Deployment configuration and click **Create deployment**: ![Review step](../assets/use-cases/deploy-application-from-feature-branch/create-deployment-review.png "Review step") 3. In the congratulations window, click **Open deployment**: ![Deployment created](../assets/use-cases/deploy-application-from-feature-branch/deployment-created.png "Deployment created") #### Create Environment The next step is to create an Environment: 1. On the Deployment details page, click the **Create Environment** button: ![Create environment](../assets/use-cases/deploy-application-from-feature-branch/add-environment.png "Create environment") 2. Fill in the required fields: - **Basic configuration**: - Cluster: `in-cluster` - Environment name: `dev` - Deploy namespace: `krci-fl-dev` - Description: `Personal environment for Firstname Lastname user` ![Basic configuration step](../assets/use-cases/deploy-application-from-feature-branch/create-environment-basic-configuration.png "Basic configuration step") - **Pipeline configuration**: - Trigger type: `Manual` - Deploy Pipeline template: `deploy` - Clean Pipeline template: `clean` ![Pipeline configuration step](../assets/use-cases/deploy-application-from-feature-branch/create-environment-pipeline-configuration.png "Pipeline configuration step") - **Quality Gates**: Leave everything as is: ![Quality gates step](../assets/use-cases/deploy-application-from-feature-branch/create-environment-quality-gates.png "Quality gates step") - **Review**: Review the specified values and click the **Create environment** button: ![Review step](../assets/use-cases/deploy-application-from-feature-branch/create-environment-review.png "Review step") 3. In the congratulations window, click **Go to environment**. The feature Environment is now ready for deployment. #### Deploy Application To deploy an application, follow the steps below: 1. Click **Configure deploy**, select the image tag to be deployed, and proceed with the **Start deploy** button: ![Application deployment](../assets/use-cases/deploy-application-from-feature-branch/deploy-application.png "Application deployment") 2. Verify the deployment status in the **Pipelines** section: ![Check deploy pipeline run](../assets/use-cases/deploy-application-from-feature-branch/check-deploy-pipeline.png "Check deploy pipeline run") 3. Click the pipeline run name to view its details: ![View deploy pipeline details](../assets/use-cases/deploy-application-from-feature-branch/deploy-pipeline-details.png "View deploy pipeline details") 4. Open the application in Argo CD using the **Argo CD** button: ![Check deploy pipeline run](../assets/use-cases/deploy-application-from-feature-branch/open-in-argo-cd.png "Check deploy pipeline run") 5. View enhanced insights and log analysis: ![Argo CD application state](../assets/use-cases/deploy-application-from-feature-branch/argo-cd-state.png "Argo CD application state") ### Deploy With Custom Parameters The platform utilizes a Helm chart found in the **deploy-templates** folder of each Project repository. To deploy a feature branch with custom variables, select the **Values override** option on the Environment page. The **Values override** option allows you to override default parameters in the **deploy-templates** folder of the Helm chart. :::note Before enabling this option, you must provide custom parameters for the application. To do this, please add the required values to the `values.yaml` file in the GitOps repository, ensuring you follow the expected structure. ::: For the `inventory-service` application, no Helm chart variables are defined yet. So, we will modify the Helm chart to add a parameter. If your Helm chart already has variables to override, you can proceed to step 4. 1. In KubeRocketCI, navigate back to the application and open the branch source code: ![Open application source code](../assets/use-cases/deploy-application-from-feature-branch/application-source-code.png "Open application source code") 2. Clone the application repository to the local machine. Check out the feature branch and paste the contents below to the `deploy-templates/templates/deployment.yaml` and `deploy-templates/values.yaml` files: ```yaml title="values.yaml" extraEnv: {} ``` ```yaml title="deploy-templates/deployment.yaml" ... ports: - name: http containerPort: {{ .Values.service.port }} protocol: TCP {{- if .Values.extraEnv }} env: {{- toYaml .Values.extraEnv | nindent 12 }} {{- end }} livenessProbe: {{- toYaml .Values.livenessProbe | nindent 12 }} readinessProbe: {{- toYaml .Values.readinessProbe | nindent 12 }} ... ``` The resulting difference should look this way: ![Open application source code](../assets/use-cases/deploy-application-from-feature-branch/application-diff.png "Open application source code") 3. Commit your changes, push your branch to the remote repository, and build the application (feature branch) once again. 4. Open the GitOps repository using the **Go to the source code** button: ![Go to the source code button](../assets/use-cases/deploy-application-from-feature-branch/go-to-the-source-code.png "Go to the source code button") 5. Clone the GitOps repository to the local machine. Check out another branch and create a file by the `//-values.yaml` pattern (in our case, `fl/dev/inventory-service-values.yaml`). 6. Open the `//-values.yaml` file and paste the contents below: ```yaml title="values.yaml" extraEnv: - name: NAME value: "Hello from KubeRocketCI" ``` 7. Commit your changes, push your branch to the remote repository and create a pull request: ![Pull request details](../assets/use-cases/deploy-application-from-feature-branch/gitops-difference.png "Pull request details") 8. Merge the pull request. 9. On the Environment page, click the **Configure deploy** button. When deploying an application, enable the **Values override** option in the Environment settings: ![Enable values override option](../assets/use-cases/deploy-application-from-feature-branch/enable-values-override.png "Enable values override option") When the **Values override** option is enabled, the platform navigates to the GitOps repository to pull the parameters from the `-values.yaml` file and then overrides them in an Argo CD application accordingly. 10. If the application is deployed in the "in-cluster", open the pod terminal using a dedicated button on the Environment details page: ![Show logs button](../assets/use-cases/deploy-application-from-feature-branch/show-terminal-button.png "Show logs button") 11. Run the following command to verify that the variable has been applied to the pod: ```bash env | grep NAME ``` ![Show logs button](../assets/use-cases/deploy-application-from-feature-branch/application-terminal-krci.png "Show logs button") 12. (Optional) If you deploy the application in the [remote cluster](../user-guide/add-cluster.md), you can open the deployed application in Argo CD and select the pod block: ![Enable values override option](../assets/use-cases/deploy-application-from-feature-branch/application-pod.png "Enable values override option") 13. (Optional) In the **Terminal** tab, run the following command to verify that the variable has been applied to the pod: ```bash env | grep NAME ``` ![Application terminal (Argo CD)](../assets/use-cases/deploy-application-from-feature-branch/application-terminal-argo-cd.png "Application terminal (Argo CD)") ### Cleanup After merging the feature branch, please delete the branch and Environment. #### Delete Feature Environment 1. In the KubeRocketCI portal, return to the Environment. 2. Select the application from the **Applications** tab and click the **Delete** button to remove the application from the Environment: ![Delete application from environment](../assets/use-cases/deploy-application-from-feature-branch/delete-application.png "Delete application from environment") 3. Navigate to the **Deployments** section. Click the actions button and select **Delete**: ![Delete deployment](../assets/use-cases/deploy-application-from-feature-branch/delete-deployment.png "Delete deployment") #### Delete Feature Branch The last step is to delete a feature branch for the application: 1. Navigate to the **Projects** section. 2. Open the Project that contains the feature branch. 3. Delete the branch from the **Branches** tab: ![Delete branch in KubeRocketCI](../assets/use-cases/deploy-application-from-feature-branch/delete-feature-branch-portal.png "Delete branch in KubeRocketCI") 4. Delete the feature branch from GitHub: ![Delete branch in GitHub](../assets/use-cases/deploy-application-from-feature-branch/delete-feature-branch-github.png "Delete branch in GitHub") ## Related Articles - [Use Cases](./index.md) - [Add Application](../user-guide/add-application.md) - [Add Deployment](../user-guide/add-cd-pipeline.md) - [Manage Deployments](../user-guide/manage-environments.md) - [Manage GitOps](../user-guide/gitops.md) --- ## Secured Secrets for Application Deployment # Secured Secrets Management for Application Deployment This use case demonstrates the secure management of sensitive data, such as passwords, API keys, and other credentials used by an application during development or production runtime. The process involves storing sensitive data in an external secret store within a `krci-vault` namespace. The confidential information is then transmitted from the `krci-vault` namespace to the namespace where the application is deployed. This ensures that the application can utilize these credentials to establish a secure connection to the database. In this scenario, the KubeRocketCI platform leverages capabilities of the `External Secret Operator`. Developers can use another external secret store (Hashicorp Vault, AWS Secret Store, or another provider) to ensure that confidential information is securely stored and accessed only when necessary. To follow this approach, the next steps are involved: 1. Configure the KubeRocketCI platform with the external secret store provider, [namespace will be used](../operator-guide/secrets-management/external-secrets-operator-integration.md#kubernetes-provider) for the below scenario. 2. Create a separate namespace, with the name `krci-vault`, to store the sensitive data. 3. Keep sensitive data, such as passwords, API keys, and credentials, in the `krci-vault` namespace. 4. Establish a connection between the deployed namespace and the vault namespace to securely access the sensitive data when required. By following these steps, developers can ensure that sensitive data is protected and accessed securely within the KubeRocketCI platform. This approach enhances the overall security of the application and reduces the risk of unauthorized access to confidential information. ## Roles This documentation is tailored for the Platform Team, Developers and Team Leads. ## Goals - Ensure secure handling of confidential information within the Deployment Environment. ## Preconditions - KubeRocketCI instance is [configured](../operator-guide/prerequisites.md) with [GitOps](../user-guide/gitops.md) repo (to be able to create Projects); - External Secrets Operator is [installed](../operator-guide/secrets-management/install-external-secrets-operator.md); - Developer has access to the KubeRocketCI instances using the Single-Sign-On approach; - Developer has merge permissions in the one of the [Git Server](../user-guide/git-server-overview.md) repository, e.g. GitHub; - Developer has permissions to create resources such as namespace, roles, and role bindings. ## Scenario To utilize External Secrets in the KubeRocketCI platform, follow the steps outlined below: ### Add Application To begin, you will need an application first. Here are the steps to create it: 1. Log in to the KubeRocketCI portal: ![Logging Page](../assets/use-cases/general/login-1.png "Logging screen") 2. On the main menu, click the **Manage Namespaces** button: ![Manage Namespaces](../assets/use-cases/general/settings-1.png "Manage Namespaces button") 3. Ensure the `Namespace` value points to the namespace with the KubeRocketCI installation: ![Default and allowed namespaces](../assets/use-cases/fastapi-scaffolding/default-allowed-namespace.png "Default and allowed namespaces") :::note Don't forget to press Enter to add the namespace to the allowed namespaces list. ::: 4. Create a new Project with the `Application` type using the `Create` strategy. Select the **Projects** section and click **+ Create project**: ![Projects overview](../assets/use-cases/general/components-1.png "Projects overview") 5. Select the **Custom configuration** option, choose the `Application` Codebase type as we intend to deliver our application as a container and deploy it within the Kubernetes cluster. Select the **Create** strategy to scaffold our application from the template provided by the KubeRocketCI and click **Continue**: ![Initial setup](../assets/use-cases/general/component-type-1.png "Initial setup") 6. On the **Git & project info** tab, define the following values and click **Continue**: - Git server: `github` - Owner: `github_account_name` - Repository name: `es-usage` - Default branch: `main` - Project name: `es-usage` - Description: `External-secrets usage` ![Application Info](../assets/use-cases/external-secrets/create-application.png "Step application info") 7. On the **Build config** tab, define the values and click **Continue**: - Code language: `Java` - Language version/framework: `Java 17` - Build tool: `Maven` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` - Deployment options: `helm-chart` ![Advanced Settings](../assets/use-cases/external-secrets/create-application-advance.png "Step advanced settings") 8. On the **Review** tab, verify the project configuration and click **Create project**: ![Review and create](../assets/use-cases/external-secrets/review-and-create.png "Review and create") 9. Check the application status. It should be green: ![Application Status](../assets/use-cases/external-secrets/application-status.png "Application status") ### Create Deployment This section outlines the process of establishing a CD pipeline within UI Portal. There are two fundamental steps in this procedure: - Create a `Deployment`; - Within the `Deployment`, configure the `Environment`. :::note Ensure [GitOps repository](../user-guide/gitops.md) is connected to the KubeRocketCI instance. ::: Follow the instructions below to complete the process successfully: 1. In the KubeRocketCI portal, navigate to **Deployments** tab and push the **Create deployment** button to create pipeline: ![Deployment overview](../assets/use-cases/general/create-cd-pipeline-1.png "Deployment overview") 2. In the **Create Deployment** dialog, define the below values: - **Applications**: - Applications: Add `es-usage` application - Branch: Select the `main` branch ![Applications tab](../assets/use-cases/external-secrets/create-deployment-applications.png "Applications tab") - **Pipeline configuration**: - Pipeline name: `deploy` - Description: `Deployment for application that uses External Secrets Operator` - Deployment type: Select `Container` - Promote applications: Leave unchecked ![Pipeline configurations tab](../assets/use-cases/external-secrets/create-deployment-pipeline-configuration.png "Pipeline configurations tab") - **Review**: Verify the Deployment configuration and click **Create deployment**: ![Pipeline configurations tab](../assets/use-cases/external-secrets/create-deployment-review.png "Pipeline configurations tab") 3. Click the `Open deployment` button to continue creating the environment: ![Deployment created](../assets/use-cases/fastapi-scaffolding/deployment-created.png "Deployment created") 4. Click the **+ Create environment** button: ![Create Stage Button](../assets/use-cases/external-secrets/create-environment-button.png "Create stage button") 5. In the **Create new environment** dialog add the **sit** environment with the values below: - **Basic configuration**: - Cluster: `in-cluster` - Environment name: `sit` - Deploy namespace: `krci-deploy-sit` - Description: `System integration testing` ![Pipeline tab](../assets/use-cases/external-secrets/create-environment-basic-configuration.png "Pipeline tab with parameters") - **Pipeline configuration**: - Trigger type: `Manual` - Deploy Pipeline template: `deploy` - Clean Pipeline template: `clean` ![Pipeline tab](../assets/use-cases/external-secrets/create-environment-pipeline-configuration.png "Pipeline tab with parameters") - **Quality Gates**: Leave everything as is: ![Quality gates step](../assets/use-cases/external-secrets/create-environment-quality-gates.png "Quality gates step") - **Review**: Review the specified values and click the **Create environment** button: ![Review step](../assets/use-cases/external-secrets/create-environment-review.png "Review step") 6. As soon as the Environment is created, click **Open environment**: ![Open environment](../assets/use-cases/external-secrets/open-environment.png "Open environment") ### Configure RBAC for External Secret Store :::note Three namespaces are used: - `krci`, the namespace where KubeRocketCI is deployed; - `krci-vault`, the vault namespace where sensitive data is stored; - `krci-deploy-sit`, the **Deploy namespace** for the application (same value as in **Create environment** → **Basic configuration** → **Deploy namespace**). The application deploy namespace is not arbitrary: the UI suggests a name derived from the KubeRocketCI installation namespace, environment name, and stage name (see [Create Environment](../user-guide/add-cd-pipeline.md#create-environment)). In this guide the platform namespace is **`krci`**, the Deployment is named **`deploy`**, the Environment name is **`sit`**, and the deploy target is **`krci-deploy-sit`** — use the same string everywhere below (RoleBinding subject namespace, Helm templates, `kubectl -n`, and allowed namespaces in the portal). ::: To ensure the proper functioning of the system, it is crucial to create the following resources: 1. Create namespace `krci-vault` to store secrets: ```bash kubectl create namespace krci-vault ``` 2. Create Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: mongo namespace: krci-vault stringData: password: pass username: user type: Opaque ``` 3. Create Role to access the Secret: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: krci-vault name: external-secret-store rules: - apiGroups: [""] resources: - secrets verbs: - get - list - watch - apiGroups: - authorization.k8s.io resources: - selfsubjectrulesreviews verbs: - create ``` 4. Create RoleBinding: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: eso-from-krci namespace: krci-vault subjects: - kind: ServiceAccount name: secret-manager namespace: krci-deploy-sit roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: external-secret-store ``` ### Add External Secret to Helm Chart Now that RBAC is configured properly, it is time to add external secrets templates to application Helm chart. Follow the instructions provided below: 1. Navigate to one of the `Git Servers` where the `es-usage` application was created during the [Add Application](#add-application) step. In this example, it is `GitHub`: ![Github Repository](../assets/use-cases/external-secrets/github_repo.png "Github repository") 2. Create a commit in the `es-usage` repository and add the following configuration files to the Helm chart: 1. `deploy-templates/templates/sa.yaml`: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: secret-manager namespace: krci-deploy-sit ``` 2. `deploy-templates/templates/secret-store.yaml`: ```yaml apiVersion: external-secrets.io/v1 kind: SecretStore metadata: name: krci-vault-store namespace: krci-deploy-sit spec: provider: kubernetes: remoteNamespace: krci-vault auth: serviceAccount: name: secret-manager server: caProvider: type: ConfigMap name: kube-root-ca.crt key: ca.crt ``` 3. `deploy-templates/templates/external-secret.yaml`: ```yaml apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: mongo # target secret name namespace: krci-deploy-sit # target namespace spec: refreshInterval: 1h secretStoreRef: kind: SecretStore name: krci-vault-store data: - secretKey: username # target value property remoteRef: key: mongo # remote secret key property: username # value will be fetched from this field - secretKey: password # target value property remoteRef: key: mongo # remote secret key property: password # value will be fetched from this field ``` 4. `deploy-templates/templates/deployment.yaml`. Add the environment variable for mongodb to the existing deployment configuration that used the secret: ```yaml env: - name: MONGO_USERNAME valueFrom: secretKeyRef: name: mongo key: username - name: MONGO_PASSWORD valueFrom: secretKeyRef: name: mongo key: password ``` 3. Push the changes made to the `es-usage` repository. ### Deploy Application Deploy the application by following the steps below: 1. Build Container from the latest branch commit. To build the initial version of the application's `main` branch, go to the **Projects** -> **es-usage** -> **Branches** -> **main** and press the **Trigger build pipeline run** button: ![Build Pipeline](../assets/use-cases/external-secrets/project_build.png "Build pipeline") 2. Build pipeline for the `es-usage` application starts: ![Build Status](../assets/use-cases/external-secrets/build_status.png "Build status") 3. Once the build pipeline has successfully completed, navigate to the **Deployments** tab and select the `deploy` pipeline: Choose the **sit** environment and click on the **Configure deploy** button: ![Configure Deploy](../assets/use-cases/external-secrets/configure_deploy.png "Configure deploy") 4. In the `Image stream version`, select latest version and push the **Start deploy** button: ![Start Deploy](../assets/use-cases/external-secrets/start_deploy.png "Start deploy") 5. Ensure application status is `Healthy` and `Synced`: ![App Synced](../assets/use-cases/external-secrets/app_synced.png "App synced") ### Check Application Status To ensure the application is deployed successfully, use `kubectl` with a context that can read the deploy namespace (replace `krci-deploy-sit` if your **Deploy namespace** differs). 1. Check that External Secrets resources are reconciled: ```bash kubectl get secretstore -n krci-deploy-sit NAME AGE STATUS CAPABILITIES READY krci-vault-store 4m38s Valid ReadWrite True ``` ```bash kubectl get externalsecret -n krci-deploy-sit NAME STORE REFRESH INTERVAL STATUS READY mongo krci-vault-store 1h SecretSynced True ``` If `READY` is not `True`, inspect events: ```bash kubectl describe externalsecret mongo -n krci-deploy-sit kubectl describe secretstore krci-vault-store -n krci-deploy-sit ``` 2. Confirm the Kubernetes `Secret` named `mongo` exists in the deploy namespace (created by ESO from the vault): ```bash kubectl get secret mongo -n krci-deploy-sit ``` Confirm keys exist (values stay hidden in this output): ```bash kubectl describe secret mongo -n krci-deploy-sit ``` Under **Data**, expect keys `username` and `password`. ![Secrets](../assets/use-cases/external-secrets/secret_list.png "Secrets") 3. See which **Secrets** the workload is wired to (desired state from the `Deployment`; values are never printed here): `kubectl get deployment` only shows high-level fields (replicas, image, etc.), not secret references. Use **`describe`** or **`get -o yaml`** on the `Deployment` to inspect the Pod template. ```bash kubectl get deployment -n krci-deploy-sit -l app.kubernetes.io/name=es-usage kubectl describe deployment -n krci-deploy-sit -l app.kubernetes.io/name=es-usage ``` ```text Environment: MONGO_USERNAME: Optional: false MONGO_PASSWORD: Optional: false ``` That output means the Deployment injects those keys from the `mongo` Secret into every Pod it creates. Optional — compact list of env vars that use `secretKeyRef` (requires a single Deployment in the selector result): ```bash kubectl get deployment -n krci-deploy-sit -l app.kubernetes.io/name=es-usage \ -o jsonpath='{range .items[0].spec.template.spec.containers[0].env[*]}{.name}{"\t"}{.valueFrom.secretKeyRef.name}{"\t"}{.valueFrom.secretKeyRef.key}{"\n"}{end}' ``` Each line is: `VARIABLE_NAME`, Secret name, Secret key. Empty lines can appear for env entries that are not backed by a Secret (ignore them). :::note The steps above confirm External Secrets, the synced `mongo` Secret, and the **desired** Pod template on the `Deployment`. If the application misbehaves or Pods are not healthy anyway, inspect **runtime** state with `kubectl describe pod` (and Pod events) in the deploy namespace, and review the **Argo CD Application** resource that owns this release (sync status, rendered manifests, and events in the Argo CD UI or via `kubectl get application -n ` / `kubectl describe application` as appropriate for your installation). ::: ## Related Articles - [Use Cases](./index.md) - [Add Application](../user-guide/add-application.md) - [CD Pipeline](../user-guide/add-cd-pipeline.md) --- ## KubeRocketCI Use Cases: Practical Scenarios and Solutions The Use Cases section provides useful recommendations of how to operate with the KubeRocketCI tools and manage the custom resources. Get acquainted with the description of technical scenarios and solutions. ## All Use Cases * [Scaffold and Deploy FastAPI Application](./application-scaffolding.md) * [Autotest as a Quality Gate](./autotest-as-quality-gate.md) * [Test Environment Using Environment Variables](./cd-autotests-run-with-env-variables.md) * [Create and Use Custom Tekton Pipelines](./custom-pipelines-flow.md) * [Deploy Application From a Feature Branch](./deploy-application-from-feature-branch.md) * [Secured Secrets for Application Deployment](./external-secrets.md) * [Application With Custom Build Tool/Framework](./tekton-custom-pipelines.md) --- ## Application With Custom Build Tool/Framework # Deploy Application With Custom Build Tool/Framework This use case demonstrates how to implement custom CI/CD pipelines within the KubeRocketCI platform to address project-specific requirements. KubeRocketCI extends this capability by supporting the **customization** of Applications (Projects) and Tekton pipelines, allowing teams to integrate and develop functionalities or services not readily available on the platform. ## Goals The goal of this is to configure KubeRocketCI in a way to: - Incorporate and manage custom Tekton pipeline libraries that addresses project requirements. - Modify existing pipelines and tasks within these libraries to align with specific developmental goals and introduce novel functionalities. - Facilitate a swift setup and implementation process, enabling teams to focus on development without being hindered by platform limitations. ## Preconditions Before starting this use case, ensure that: - KubeRocketCI instance with GitHub and Tekton is [configured](../operator-guide/prerequisites.md); - Developer has access to the KubeRocketCI instances using the Single-Sign-On approach; - Developer has `Write` permissions for GitHub repository to merge the code; - Argo CD instance is configured with the [Add-ons repository](../operator-guide/add-ons-overview.md) added. ## Scenario To streamline the process of implementing custom logic within the KubeRocketCI platform, follow this scenario: - **Add Custom Application**: Begin by adding your application to the KubeRocketCI platform. - **Add Custom Tekton Library**: Create and add a custom Tekton library designed to implement the CI/CD logic required by your application. - **Validate the Implementation**: After setting up your Application and Tekton library, conduct testing to ensure that the pipelines execute as intended. ### Add Custom Application to KubeRocketCI :::tip A `Custom Application` refers to any application that utilizes a programming language, framework, or build tool not natively supported by the platform, or an existing application that necessitates specific customizations to its pipeline. This encompasses scenarios where the default pipeline configurations and toolchains are unable to meet the unique requirements of the application, thereby requiring the development and integration of tailored solutions. ::: 1. Log in to the KubeRocketCI portal: ![Logging Page](../assets/use-cases/general/login-1.png "Logging screen") 2. On the main menu, click the **Manage Namespaces** button: ![Manage Namespaces](../assets/use-cases/general/settings-1.png "Manage Namespaces button") 3. Ensure the `Namespace` value points to the namespace with the KubeRocketCI installation: ![Default and allowed namespaces](../assets/use-cases/fastapi-scaffolding/default-allowed-namespace.png "Default and allowed namespaces") :::note Don't forget to press Enter to add the namespace to the allowed namespaces list. ::: 4. Select the **Projects** section and click **+ Create project**: ![Projects tab](../assets/use-cases/general/components-1.png "Projects tab") 5. Create a new Project of the `Application` type using the `Clone` strategy since we are cloning the application from the existing repository: ![Clone Project](../assets/use-cases/tekton-custom/tekton-custom02.png "Clone Project") 6. In the **Git & project info** tab, define the following values and click the **Continue** button: - Repository URL: `https://github.com/epmd-edp/go-go-beego.git` - Git Server: select the available GitHub server - Owner: `github_account_name` - Repository name: `tekton-hello-world` - Project name: `tekton-hello-world` - Default branch: `main` - Description: `Custom tekton-hello-world application` ![Git & project info](../assets/use-cases/tekton-custom/tekton-custom03.png "Git & project info") 7. In the **Build config** tab, define the values and click the **Continue** button: - Deployment option: `helm-chart` - Application code language: `Other` - Language version/framework: `go` - Build tool: `shell` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Build config](../assets/use-cases/tekton-custom/tekton-custom04.png "Build config") 8. On the **Review** tab, verify the project configuration and click **Create project**: ![Review and create](../assets/use-cases/tekton-custom/review-and-create.png "Review and create") 9. On the congratulations page, click **View all projects**: ![Ready project](../assets/use-cases/tekton-custom/ready-component.png "Ready project") 10. Check the application status. It should be green: Now that the application is successfully created, proceed to adding the Tekton library to the KubeRocketCI platform. :::warning It's important to align with the Tekton Pipeline name to ensure correct pipeline execution for review and build events. The name for PipelineRun is dynamically generated via `TriggerTemplates` located in the [pipelines-library](https://github.com/epam/edp-tekton/tree/master/charts/pipelines-library/templates/triggers) and aligned to the structure provided below: ```yaml pipelineRef: name: github-$(tt.params.buildtool)-$(tt.params.framework)-$(tt.params.cbtype)-build-$(tt.params.versioning-type) ``` ```yaml pipelineRef: name: github-$(tt.params.buildtool)-$(tt.params.framework)-$(tt.params.cbtype)-review ``` This naming convention facilitates the automatic creation of PipelineRun instances in response to payload from GitHub: the review pipeline runs during Merge Request events, and the build pipeline runs after the changes are merged. Ensure that the Pipeline name matches this structure to enable the correct triggering of pipeline executions. In our case, the build pipeline name should be `github-shell-go-app-build-semver` and the review pipeline name should be `github-shell-go-app-review`. ::: ### Add Tekton Library KubeRocketCI allows for the creation of custom Tekton libraries to address specific project requirements. This feature enables the modification of existing pipelines and tasks to align with the unique needs of the application. 1. Select the **Projects** tab and push the create **+ Create Project** button: ![Projects tab](../assets/use-cases/tekton-custom/tekton-custom05.png "Projects tab") 2. Select **Custom configuration**, **Library**, **Create** and click **Continue**: ![Create Library](../assets/use-cases/tekton-custom/tekton-custom06.png "Create Library") :::note The KubeRocketCI `Create` strategy will automatically pull the code for the Tekton Helm application from [the template](https://github.com/epmd-edp/helm-helm-pipeline.git). ::: 3. In the **Git & project info** tab, define the following values and click the **Continue** button: - Git Server: select the available GitHub server - Owner: `github_account_name` - Repository name: `custom-tekton-chart` - Project name: `custom-tekton-chart` - Description: `Repository for storing and managing custom Tekton resources` ![Create From Template](../assets/use-cases/tekton-custom/tekton-custom07.png "Create From Template") 4. In the **Build config** tab, define the values and click the **Continue** button: - Application code language: `Helm` - Language version/framework: `Pipeline` - Build tool: `Helm` - Default branch: `main` - Codebase versioning type: `semver` - Start version from: `0.1.0` - Suffix: `SNAPSHOT` ![Codebase Info](../assets/use-cases/tekton-custom/tekton-custom08.png "Step codebase info") 5. On the **Review** tab, verify the project configuration and click **Create project**: ![Advanced Settings](../assets/use-cases/tekton-custom/tekton-custom09.png "Advanced settings") 6. On the congratulations page, click **View all projects**. 7. Check the codebases' status: ![Projects overview page](../assets/use-cases/tekton-custom/tekton-custom10.png "Projects overview page") You have successfully added the custom Tekton library to the KubeRocketCI platform and are now ready to modify the Tekton pipeline to align with the specific requirements of your custom application. ### Modify Tekton Pipeline :::info We strongly advise against altering the platform's default Tekton resources. To ensure seamless upgrades and maintain custom functionality, it is recommended to develop and manage your custom pipelines within a dedicated Custom Tekton Library. This approach safeguards your customizations and facilitates a smoother update process. ::: Now that the Tekton Helm library is created, it is time to clone, modify and then apply it to the Kubernetes cluster. 1. In the Projects tab, click one of the `custom-tekton-chart` to enter the application menu: ![Select overview page](../assets/use-cases/tekton-custom/tekton-custom11.png "Codebase status") 2. Navigate to **Branches** and click to the **Git** button: ![Branches tab](../assets/use-cases/tekton-custom/tekton-custom12.png "Branches tab") 3. Clone the repository with `SSH` using **Code** button: ![Create branch in GitHub](../assets/use-cases/tekton-custom/tekton-custom13.png "Create branch in GitHub") 4. Examine the repository structure. It should look this way by default: ```plaintext tekton-custom-pipelines ├── Chart.yaml ├── templates │   ├── pipelines │   │   ├── deploy │   │   │   ├── custom-clean.yaml │   │   │   └── custom-deploy.yaml │   │   ├── bitbucket-build-default.yaml │   │   ├── bitbucket-build-semver.yaml │   │   ├── bitbucket-build-lib-default.yaml │   │   ├── bitbucket-build-lib-semver.yaml │   │   ├── bitbucket-review.yaml │   │   ├── gerrit-build-default.yaml │   │   ├── gerrit-build-semver.yaml │   │   ├── gerrit-build-lib-default.yaml │   │   ├── gerrit-build-lib-semver.yaml │   │   ├── gerrit-review.yaml │   │   ├── github-build-default.yaml │   │   ├── github-build-semver.yaml │   │   ├── github-build-lib-default.yaml │   │   ├── github-build-lib-semver.yaml │   │   ├── github-review.yaml │   │   ├── gitlab-build-default.yaml │   │   ├── gitlab-build-semver.yaml │   │   ├── gitlab-build-lib-default.yaml │   │   ├── gitlab-build-lib-semver.yaml │   │   └── gitlab-review.yaml │   ├── resources │   │   └── npm-settings.yaml │   ├── tasks │   │   ├── deploy │   │   │   └── hello-world-deploy.yaml │   │   └── hello-world.yaml │   └── triggers │   ├── custom-clean.yaml │   └── custom-deploy.yaml └── values.yaml ``` For more detailed information and explanations regarding the repository structure, please refer to [Creating and Using Custom Tekton Pipelines](./custom-pipelines-flow.md) use case. :::note Update the values in the `values.yaml` file. The [dnsWildCard](https://github.com/epam/edp-install/blob/v3.9.0/deploy-templates/values.yaml#L10) parameter is the platform address. ::: Below is a sample configuration for the `values.yaml` file: ```yaml dnsWildCard: "example.domain.com" ``` 5. Modify and add tasks or pipelines. Consider the scenario where it's necessary to incorporate the `helm-lint` task into the review pipeline. To achieve this, set the `metadata.name` field to `github-shell-go-app-review` and append the following code snippet to the `templates/pipelines/github-review.yaml` file, specifically below the `hello-world` task, located here: [github-review.yaml](https://github.com/epmd-edp/helm-helm-pipeline/blob/master/templates/pipelines/github-review.yaml#L114): ```yaml apiVersion: tekton.dev/v1 kind: Pipeline metadata: name: github-shell-go-app-review ... - name: hello-world taskRef: name: hello-world runAfter: - init-values params: - name: BASE_IMAGE value: "$(params.image-version)" - name: username value: "$(params.username)" workspaces: - name: source workspace: shared-workspace subPath: source - name: helm-lint taskRef: kind: Task name: helm-lint runAfter: - hello-world params: - name: EXTRA_COMMANDS value: | ct lint --validate-maintainers=false --charts deploy-templates/ workspaces: - name: source workspace: shared-workspace subPath: source ``` :::note The `helm-lint` task references to the default `pipeline-library` Helm chart which is deployed to the cluster as part of the KubeRocketCI setup process. The `runAfter` parameter indicates the execution sequence, specifying that the `helm-lint` task is scheduled to run subsequent to the completion of the `hello-world` task. ::: Also, in the `templates/pipelines/github-build-semver.yaml` file, specify the name `github-shell-go-app-build-semver` for the custom build pipeline: ```yaml apiVersion: tekton.dev/v1 kind: Pipeline metadata: name: github-shell-go-app-build-semver ... ``` 6. Commit and push the modified Tekton Helm chart to GitHub: ```bash git checkout -b helm git add . git commit -m "Add Helm chart testing for go-shell application" git push -u origin helm ``` 7. Navigate Github -> **Pull requests** -> **Compare & pull request** -> **Create pull request**. ![Create pull request](../assets/use-cases/tekton-custom/tekton-custom15.png "Create pull request") 8. Check the GitHub code review for the custom Helm chart pipelines repository in KubeRocketCI. Navigate to **Projects** -> **Project name** and click to the review pipeline run: ![KubeRocketCI pipelines overview](../assets/use-cases/tekton-custom/tekton-custom14.png "KubeRocketCI pipelines overview") 9. Explore the pipeline status and steps: ![Explore pipeline](../assets/use-cases/tekton-custom/tekton-custom16.png "Explore pipeline") 10. Go to the GitHub **Pull requests** -> **Add Helm chart testing for go-shell application** and click **Merge pull request**: ![Merge PR](../assets/use-cases/tekton-custom/tekton-custom17.png "Merge PR") ### Deliver Custom Tekton Pipelines to the cluster To deploy custom pipelines to the cluster, you can use Argo CD, which includes a repository with [add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main). To deliver custom Tekton pipelines to the cluster, follow these steps: 1. Clone the forked repository with add-ons and make the following changes. In the `clusters/core/apps/values.yaml` file, set the `repoUrl` and `namespace` fields to specify the Git URL of the `custom-tekton-chart` repository and the namespace where the KubeRocketCI platform is deployed. Also, set the `kuberocketci-pipelines.enable` field to `true` to enable the deployment of the `kuberocketci-pipelines` Argo CD Application: ```yaml kuberocketci-pipelines: enable: true namespace: repoUrl: ssh://git@github.com:22//custom-tekton-chart.git ``` 2. Commit and push the changes to the repository. After making the necessary changes, commit and push the changes to the repository. Use the following commands to commit and push the changes: ```bash git add . git commit -m "Enable custom Tekton pipelines deployment" git push origin main ``` 3. After pushing the changes, access **Argo CD**, navigate to the Application that corresponds to the repository with the **add-ons**, and initiate the `Sync` process. This will apply the `custom-tekton-chart` Helm Chart to the cluster within the specified namespace: ![Argo CD Sync](../assets/use-cases/tekton-custom/argocd-sync.png "Argo CD Sync") 4. Open the **kuberocketci-pipelines** component by clicking the link button: ![KubeRocketCI pipelines component](../assets/use-cases/tekton-custom/open-krci-pipelines.png "KubeRocketCI pipelines component") 5. Find the **github-shell-go-app-build-semver** pipeline and select **Details**: ![Select pipeline](../assets/use-cases/tekton-custom/select-pipeline.png "Select pipeline") 6. On the resource window, click **Sync** -> **Synchronize**: ![Synchronize resource](../assets/use-cases/tekton-custom/synchronize-resource.png "Synchronize resource") 7. Repeat the steps 5-6 for the **github-shell-go-app-review** pipeline: ![Synced pipelines](../assets/use-cases/tekton-custom/synced-pipelines.png "Synced pipelines") ### Create Application Merge Request Since we applied the Tekton library to the Kubernetes cluster in the previous step, let's test the review and build pipelines for our `tekton-hello-world` application. Perform the below steps to merge new code (Merge Request) that passes the Code Review flow. For the steps below, we use GitHub UI but the same actions can be performed using the command line and Git tool: 1. In the Projects tab, click on the `tekton-hello-world` name to enter the application details page. 2. Navigate to **Branches** and click to the branch name button: ![Project's branch tab](../assets/use-cases/tekton-custom/tekton-custom18.png "Project's branch tab") 3. Navigate to the **Branches** menu, create a new branch from the **New branch** menu with name `test`, and click on the test branch: ![Create Branch](../assets/use-cases/tekton-custom/tekton-custom19.png "Create Branch") 4. Change file `deploy-templates/values.yaml` by changing the `replicaCount` value: ```yaml ... replicaCount: 2 ... ``` 5. Click **Pull requests** -> **Compare & pull request** -> and click **Create pull request**. 6. Check the Review Pipeline status. Navigate KubeRocketCI -> **Projects** -> `tekton-hello-world` and click on review pipeline. The `helm-lint` task should be displayed there: ![Explore Lint Step](../assets/use-cases/tekton-custom/tekton-custom21.png "Explore Lint Step") 7. After review procedure successfully ended - approve merge request. Navigate Github -> **Pull requests** -> `test` and click on **Merge pull request** button. Then, your code is merged to the main branch, triggering the Build Pipeline: ![Merge Pull Request](../assets/use-cases/tekton-custom/tekton-custom22.png) 8. Check the pipelines in the KubeRocketCI dashboard: ![Explore Build Pipeline](../assets/use-cases/tekton-custom/tekton-custom23.png "Explore Build Pipeline") Under the hood, the following process takes place: 1. GitHub sends a payload to the Tekton EventListener when a Merge Request event occurs. 2. The EventListener captures the payload with the assistance of an Interceptor. 3. The TriggerTemplate creates a PipelineRun. The detailed scheme is shown below: ```mermaid graph LR; A[GitHub events] --> |Payload| B(Tekton EventListener) --> C(Tekton Interceptor CEL filter) --> D(TriggerTemplate)--> E(PipelineRun) ``` This chart will be using the core of `common-library` and `pipelines-library` and custom resources on the top of them. ## Related Articles - [Tekton Overview](../operator-guide/ci/tekton-overview.md) - [Add Application using KubeRocketCI Portal](../user-guide/add-application.md) --- ## Add Application # Add Application KubeRocketCI portal allows you to create an application, clone an existing repository with the application to your Version Control System (VCS), or using an external repository and importing an application to the environment. When an application is created or cloned, the system automatically generates a corresponding repository within the integrated Version Control System. Before following these guidelines, be sure to [add Git Server](./add-git-server.md) to the platform. The **Create Application** wizard contains four steps: * Initial Setup * Git & Project Info * Build Config * Review To add an application, navigate to the **Projects** section on the navigation bar and click **+ Create Project**. ## Initial Setup Once clicked, the **Create new project** wizard will appear. In this dialog, you can make a choice: * **Select Ready Template** - this option allows you to select a preconfigured, ready-to-go application (e.g., Antora documentation or Echo server). * **Custom Configuration** - this option allows you create any of the supported Project type. In this case, you should select **Application**. Choose one of the strategies and click **Continue**: * **Create** – creates a sample Project on the pattern in accordance with an application language, a build tool, and a framework. This strategy is recommended for projects that start developing their applications from scratch. * **Import** - allows using existing VCS repository to integrate with KubeRocketCI. While importing the existing repository, select the Git server from the drop-down list and define the relative path to the repository, such as `epmd-edp/python-python-flask`. * **Clone** – clones the indicated repository into KubeRocketCI. While cloning the existing repository, it is required to fill in the **Repository URL** field and specify the credentials if needed: ## Git & Project Info In our example, we will use the **Create** strategy: Select all the settings that define how the application will be added to Git server: * **Git server** - the pre-configured server where the component will be hosted. Select one from the drop-down list. Please refer to the [Manage Git Servers](git-server-overview.md) page to learn how to create the one. * **Owner** - the relative path to the Git repository where Project will be created (e.g., `MyGithubUsername123` or `my-github-username`). * **Repository name** - the name of the repository that will store code for this Project. Must be at least two characters using the lower-case letters, numbers and inner dashes. * **Default branch** - the default branch the Project will be created with. The default branch cannot be deleted. * **Project name** - the name of the Project. Must be at least two characters using the lower-case letters, numbers and inner dashes. * **Description** - brief and concise description that explains the purpose of the application. * **Private** - by default, all the created Projects have private visibility settings in your Git account. Uncheck this option to create a public Git repository. * **Empty project** - check this box to create an application with an empty repository. The empty repository option is available only for the **Create** strategy. ## Build Config Specify the application language and versioning properties: * **Code Language** - defines the code language with its supported frameworks: * Java – selecting a specific Java version (17, 21, and 25 are available; Java 8 and 11 have been deprecated starting from KubeRocketCI version 3.12). * JavaScript - selecting JavaScript allows using React, Vue, Angular, Express, Next.js and Antora frameworks. * Python - selecting Python allows using the Python v.3.8, FastAPI, Flask frameworks. * Go - selecting Go allows using the Beego, Gin and Operator SDK frameworks. * C# - selecting C# allows using the .Net v.3.1 and .Net v.6.0 frameworks. * Helm - selecting Helm allows using the Helm framework. * Other - selecting Other allows extending the default code languages when creating a codebase with the clone/import strategy. :::note The **Create from template** strategy does not allow to customize the default code language set. ::: * **Language version/framework** - defines the specific framework or language version of the application. The field depends on the selected code language. * **Build Tool** - allows to choose the build tool to use. A set tools and can be changed in accordance with the selected code language. * Java - selecting Java allows using the Gradle or Maven tool. * JavaScript - selecting JavaScript allows using the NPM or PNPM tool. * C# - selecting C# allows using the .Net tool. * Python - selecting Python allows using Python tool. * Go - selecting Go allows using Go tool. * Helm - selecting Helm allows using Helm tool. :::note The **Select Build Tool** field disposes of the default tools and can be changed in accordance with the selected code language. ::: :::info Tekton pipelines offer built-in support for Java Maven Multi-Module projects. These pipelines are capable of recognizing Java deployable modules based on the information in the **pom.xml** file and performing relevant deployment actions. It's important to note that although the Dockerfile is typically located in the root directory, Kaniko, the tool used for building container images, uses the targets folder within the deployable module's context. For a clear illustration of a Multi-Module project structure, please refer to this [example](https://github.com/epmd-edp/java-maven-java17-multimodule.git) on GitHub, which showcases a commonly used structure for Java Maven Multi-Module projects. ::: * **Codebase versioning type** - defines how will the application tag be changed once the new image version is built. There are two versioning types: * **default**: Using the `default` versioning type, in order to specify the version of the current artifacts, images, and tags in the Version Control System, a developer should navigate to the corresponding file and change the version **manually**. * **semver**: Using the `semver` versioning type, a developer indicates the version number from which all the artifacts will be versioned and, as a result, **automatically** registered in the corresponding file (e.g. pom.xml). When selecting the `semver` versioning type, the extra fields will appear, type the version number from which you want the artifacts to be versioned. This versioning type is recommended and selected by default. :::note The **Start Version From** field should be filled out in compliance with the semantic versioning rules, e.g. 1.2.3 or 10.10.10 (0.1.0 by default). The **Suffix** field is also mandatory and has the **SNAPSHOT** value by default. Please refer to the [Semantic Versioning](https://semver.org/) page for details. ::: * **Deployment Options** - select the deployment option available. * **helm-chart**: Application will be deployed as a Helm chart using the Argo CD tool. * **rpm-package**: Application will be deployed as an rpm package using the Ansible tool. For more details, please refer to the [Deploy RPM Packages](../operator-guide/cd/deploy-rpm.md) page. * **CI Pipelines** - select the available CI pipelines provider. Tekton is used by default. * **Specify the pattern to validate a commit message** - the regular expression used to indicate the pattern that is followed on the project to validate a commit message in the code review pipeline. An example of the pattern: `^[PROJECT_NAME-d{4}]:.*$`. * **Integrate with Jira server** - this check box is used in case it is required to connect Jira tickets with the commits and have a respective label in the **Fix Version** field. :::note To adjust the Jira integration functionality, first apply the necessary changes described on the [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) page. ::: * **Jira Server** - the integrated Jira server with related Jira tasks. * **Specify the pattern to find a Jira ticket number in a commit message** - based on this pattern, the value from KubeRocketCI will be displayed in Jira. * **Mapping field name** - the section where the additional Jira fields are specified the names of the Jira fields that should be filled in with attributes from KubeRocketCI: * Select the name of the field in a Jira ticket. The available fields are the following: _Fix Version/s_, _Component/s_ and _Labels_. * Click the **Add** button to add the mapping field name. * Enter Jira pattern for the field name: * For the **Fix Version/s** field, select the **EDP_VERSION** variable that represents an EDP upgrade version, as in _2.7.0-SNAPSHOT_.Combine variables to make the value more informative. For example, the pattern **EDP_VERSION-EDP_COMPONENT** will be displayed as _2.7.0-SNAPSHOT-nexus-operator_ in Jira. * For the **Component/s** field select the **EDP_COMPONENT** variable that defines the name of the existing repository. For example, _nexus-operator_. * For the **Labels** field select the **EDP_GITTAG** variable that defines a tag assigned to the commit in Git Hub. For example, _build/2.7.0-SNAPSHOT.59_. * Click the bin icon to remove the Jira field name. ## Review and Create The **Review an Create** window allows to ensure the application configuration suits your needs and verify you entered specifications correctly. :::note After the complete adding of the application, inspect the [Manage Applications](application.md) page to learn how you can operate applications. ::: ## Related Articles * [Manage Applications](application.md) * [Add CD Pipeline](add-cd-pipeline.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) * [Manage Git Providers](../user-guide/add-git-server.md) --- ## Add Autotest # Add Autotest KubeRocketCI portal allows you to clone an existing repository with the autotest to your Version Control System (VCS), or use an external repository and import an autotest to the environment for running in stages or using as quality gates for applications. When an autotest is cloned or imported, the system automatically generates a corresponding repository within the integrated Version Control System. Before following these guidelines, be sure to [add Git Server](./add-git-server.md) to the platform. The **Create Autotest** wizard contains four steps: * Initial Setup * Git & Project Info * Build Config * Review To add an autotest, navigate to the **Projects** section on the navigation bar and click **+ Create Project**. ## Initial Setup Once clicked, the **Create new project** wizard will appear. In this dialog, you can make a choice: * **Select Ready Template** – this option allows you to select a preconfigured, ready-to-go autotest template. * **Custom Configuration** – this option allows you to create any of the supported Project types. In this case, you should select **Autotest**. Choose one of the strategies and click **Continue**: * **Import** – allows using an existing VCS repository to integrate with KubeRocketCI. While importing the existing repository, select the Git server from the drop-down list and define the relative path to the repository, such as `epmd-edp/examples/basic/edp-auto-tests-simple-example`. * **Clone** – clones the indicated repository into KubeRocketCI. While cloning the existing repository, it is required to fill in the **Repository URL** field and specify the credentials if needed. :::note Autotest does not support the **Create** strategy. Use **Import** or **Clone** to add an autotest. ::: ## Git & Project Info In our example, we will use the **Import** strategy: Select all the settings that define how the autotest will be added to the Git server: * **Git server** – the pre-configured server where the component will be hosted. Select one from the drop-down list. Please refer to the [Manage Git Servers](git-server-overview.md) page to learn how to create one. * **Owner** - the relative path to the Git repository where Project will be created (e.g., `MyGithubUsername123` or `my-github-username`). * **Repository name** - the name of the repository that will store code for this project. Must be at least two characters using the lower-case letters, numbers and inner dashes. * **Default branch** – the default branch the Project will be created with. The default branch cannot be deleted. * **Project name** – the name of the autotest. Must be at least two characters using lower-case letters, numbers, and inner dashes. * **Description** – brief and concise description that explains the purpose of the autotest. * **Private** – by default, all created Projects have private visibility settings in your Git account. Uncheck this option to create a public Git repository. ## Build Config Specify the autotest language and versioning properties: * **Code Language** – defines the code language with its supported frameworks. For autotests, **Other** is often used to extend the default code languages when using the Clone/Import strategy. * **Language version/framework** – defines the specific framework or language version of the autotest. The field depends on the selected code language. Specify Java 17, Java 21, or Java 25 to be used. * **Build Tool** – allows you to choose the build tool to use. For autotests, Gradle and Maven are available. * **Autotest report framework** – all autotest reports are created in the Allure framework by default. * **Codebase versioning type** – defines how the autotest tag will be changed once the new image version is built. There are two versioning types: * **default**: Using the `default` versioning type, to specify the version of the current artifacts, images, and tags in the Version Control System, a developer should navigate to the corresponding file and change the version **manually**. * **semver**: Using the `semver` versioning type, a developer indicates the version number from which all artifacts will be versioned and, as a result, **automatically** registered in the corresponding file (e.g. pom.xml). When selecting the `semver` versioning type, extra fields will appear; type the version number from which you want the artifacts to be versioned. This versioning type is recommended and selected by default. :::note The **Start Version From** field should be filled out in compliance with the semantic versioning rules, e.g. 1.2.3 or 10.10.10 (0.1.0 by default). The **Suffix** field is also mandatory and has the **SNAPSHOT** value by default. Please refer to the [Semantic Versioning](https://semver.org/) page for details. ::: * **Specify the pattern to validate a commit message** – the regular expression used to indicate the pattern that is followed on the project to validate a commit message in the code review pipeline. An example of the pattern: `^[PROJECT_NAME-d{4}]:.*$`. * **Integrate with Jira server** – this check box is used in case it is required to connect Jira tickets with the commits and have a respective label in the **Fix Version** field. :::note To adjust the Jira integration functionality, first apply the necessary changes described on the [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) page. ::: * **Jira Server** – the integrated Jira server with related Jira tasks. * **Specify the pattern to find a Jira ticket number in a commit message** – based on this pattern, the value from KubeRocketCI will be displayed in Jira. * **Mapping field name** – the section where the additional Jira fields are specified, i.e. the names of the Jira fields that should be filled in with attributes from KubeRocketCI: * Select the name of the field in a Jira ticket. The available fields are: _Fix Version/s_, _Component/s_, and _Labels_. * Click the **Add** button to add the mapping field name. * Enter the Jira pattern for the field name: * For the **Fix Version/s** field, select the **EDP_VERSION** variable that represents a KubeRocketCI upgrade version, as in _2.7.0-SNAPSHOT_. Combine variables to make the value more informative. For example, the pattern **EDP_VERSION-EDP_COMPONENT** will be displayed as _2.7.0-SNAPSHOT-nexus-operator_ in Jira. * For the **Component/s** field, select the **EDP_COMPONENT** variable that defines the name of the existing repository. For example, _nexus-operator_. * For the **Labels** field, select the **EDP_GITTAG** variable that defines a tag assigned to the commit in GitHub. For example, _build/2.7.0-SNAPSHOT.59_. * Click the bin icon to remove the Jira field name. ## Review and Create The **Review and Create** window allows you to ensure the autotest configuration suits your needs and verify you entered the specifications correctly. :::note After the complete adding of the autotest, inspect the [Manage Autotests](autotest.md) page to learn how you can operate autotests. ::: ## Related Articles * [Manage Autotests](autotest.md) * [Add Application](add-application.md) * [Add CD Pipeline](add-cd-pipeline.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) * [Manage Git Servers](git-server-overview.md) --- ## Add Deployment # Add Deployment KubeRocketCI uses Deployments — a mechanism that enables GitOps, automated deployment, promotion within pipelines, and multi-environment support. You can create a Deployment once you have at least one application with a provisioned branch. If you don't have one yet, follow the [Add Application](./add-application.md) guide. Additionally, ensure to familiarize yourself with the [Manage GitOps](gitops.md) page as it is required to add a GitOps repository before creating a Deployment. ## Create Deployment The **Create new deployment** dialog has three steps: **Applications**, **Pipeline Configuration**, and **Review**. Navigate to the **Deployments** section on the navigation bar and click **+ Create deployment**. Once clicked, the **Create new deployment** dialog will appear. ### Select Applications In the **Applications** stage: 1. Select the application(s) you are going to deploy 2. Define target branches for the application(s). 3. Click **Continue**. ### Configure Pipeline Settings In the **Pipeline Configuration** stage, specify the following configuration: 1. Enter the deployment name that will be displayed in the Deployments list. The name must be at least two characters long and may contain lowercase letters, numbers, and dashes. 2. Enter an informative but concise description. 3. Select the necessary application from the **Mapping field name** drop-down menu and click on its name. 4. Specify the application parameters: * **Branch** - Select the application branch from the drop-down menu. * **Promote applications** - When enabled, applications will be promoted through stages using the latest successful build from each previous stage. When disabled, all stages will deploy the same version that was initially selected for the pipeline, regardless of any newer builds. :::note If there is another deployed environment stage with the respective codebase stream (equivalent to an image stream in OpenShift terminology), the pattern combination will be as follows: `[pipeline name]-[stage name]-[application name]-[verified]`. ::: 5. Click **Continue**. ### Review and Create Verify the Deployment configuration and click the **Create** button to finish the setup and proceed with configuring the environment. :::note The namespace created by the environment has the following pattern combination: **[KubeRocketCI namespace]-[environment name]-[stage name]**. Please be aware that the namespace length should not exceed 63 symbols. ::: ## Create Environment After creating a Deployment, select **Open Deployment** on the congratulations page. If you skipped this step, open the Deployment details page by clicking on its name. ### Basic Configuration Environments are created the following way: 1. On the **Deployment details** menu, click the **Create Environment** button. 2. On the **Configure Environment** tab of the **Create Environment** dialog, fill in the following fields: * **Cluster** - Choose the cluster to deploy the stage in; * **Environment name** - Enter the Environment name; * **Deploy Namespace** - Specify the Kubernetes namespace where the resources will be deployed in. By default, this field is pre-populated automatically but keep in mind that the namespace name must be no longer than 63 symbols; * **Description** - Enter the description for this stage; 3. Click **Continue**. ### Pipeline Configuration In this tab, you need to configure the deployment approach: * **Trigger type** - Select the trigger type: * _Auto_ - The environment will deploy automatically once the image is built. This keeps environments up-to-date without manual intervention. * _Manual_ - The user has to trigger deployment manually by clicking the **Deploy** button in the environment menu. Refer to the [Architecture Scheme of CD Pipeline Operator](https://github.com/epam/edp-cd-pipeline-operator/blob/master/docs/arch.md) for additional details. * **Deploy pipeline template** - Choose a predefined template outlining the deployment process for your application. You can select one of the built-in options (with or without autotests), or create a custom PipelineTemplate resource to add your own. * **Clean pipeline template** - Choose one of the pre-defined pipelines offered by KubeRocketCI to define the cleanup logic. If you have specific requirements for the environment cleanup procedure, you can create your own cleanup pipeline, which will be added to the **Clean Pipeline template** drop-down list. ### Quality Gates On the **Add quality gates** tab, you define the quality control for the Deployment: * **Quality gate type** - Select the quality gate type: * Manual - Promotion must be confirmed manually in Tekton; * Autotests - Promotion proceeds automatically when the autotests pass successfully; * **Autotest** - Select the previously created [autotest](add-autotest.md) name; * **Autotest branch** - Specify a branch for the autotest; * **Step name** - Type the step name, which will be displayed in Tekton, for every quality gate. :::info Execution sequence The image promotion and execution of the pipelines depend on the sequence in which the environments are added. ::: ### Review and Create Verify the specified parameters and click the **Create environment** button to start the provisioning of the pipeline. ## Related Articles * [Manage Deployments](../user-guide/manage-environments.md) * [Add Quality Gate](../user-guide/add-quality-gate.md) --- ## Add Cluster # Add Cluster This page provides comprehensive instructions on how to integrate an external cluster into the KubeRocketCI workloads. By doing so, it creates an opportunity for users to employ multi-cluster deployment, thereby facilitating the segregation of different environments across various clusters. ## Prerequisites Before moving ahead, ensure you have already performed the guidelines outlined in the [Argo CD Integration](../operator-guide/cd/argocd-integration.md#deploy-argo-cd-application-to-remote-cluster-optional) page. Besides, user needs to have a cluster admin role to add clusters. ## Integrate External Cluster To deploy an application to a remote cluster, follow the steps below: 1. Navigate to **Configuration** -> **Deployment** -> **Clusters** and click the **+ Add cluster** button. 2. In the **Add cluster** window, choose the credentials type and specify the required fields. Click the **Save** button to add the cluster: * **Cluster Name**: a unique and descriptive name for the external cluster (e.g., dev2). The platform stores the cluster as a Kubernetes Secret named `-cluster` — this Secret name is the identifier used in all further configuration steps; * **Cluster Host**: the cluster’s Kubernetes API endpoint. The value must be a full HTTPS URL (e.g., `https://example-cluster-domain.com:6443`) — a bare host name without the `https://` scheme does not pass the form validation; * **Cluster Token**: a [Kubernetes token](../operator-guide/cd/deploy-application-in-remote-cluster-via-token.md#get-kubernetes-token) with permissions to access the cluster. This token is required for proper authorization; * **Skip TLS verification**: allows connect to cluster without cluster certificate verification; * **Cluster Certificate**: a Kubernetes certificate essential for authentication. Obtain this certificate from the configuration file of the user account you intend to use for accessing the cluster. :::note The `Cluster Certificate` field is hidden if the `skip TLS verification` option is enabled. ::: * **Cluster Name**: a unique and descriptive name for the external cluster (e.g., prod); * **Cluster Host**: the cluster’s Kubernetes API endpoint. The value must be a full HTTPS URL (e.g., `https://example-cluster-domain.com:6443`) — a bare host name without the `https://` scheme does not pass the form validation; * **Certificate Authority Data**: base64-encoded Kubernetes certificate essential for authentication. Obtain this certificate from the configuration file of the user account you intend to use for accessing the cluster; * **Role ARN**: arn:aws:iam::\:role/AWSIRSA_\{cluster_name\}_CDPipelineAgent. :::note For more details on how to work with clusters integrated using IRSA approach, please refer to the [Deploy Application In Remote Cluster via IRSA](../operator-guide/cd/deploy-application-in-remote-cluster-via-irsa.md) page. ::: 3. Wait for the platform to confirm the connection. Clicking **Save** only creates a Kubernetes Secret named `-cluster` in the platform namespace — the actual connectivity check runs asynchronously in the cd-pipeline-operator (usually under a minute). The cluster card then shows a **Connected** or **Disconnected** badge together with the error message, based on the `app.edp.epam.com/cluster-connected` and `app.edp.epam.com/cluster-error` annotations the operator sets on the Secret. A "Secret has been created" notification does **not** mean the credentials are valid — always wait for the **Connected** badge before proceeding: ![Cluster connection status](../assets/user-guide/clusters/cluster-connection-status.png "Cluster connection status") :::note Add the cluster while the KubeRocketCI platform namespace (e.g., `krci`) is selected in the Portal. The Secret is created in the currently selected namespace, and the operator only processes cluster Secrets in the platform namespace. ::: 4. As soon as the cluster is connected, open the terminal which has access to the cluster that runs the KubeRocketCI deployment. 5. Open the `krci-config` ConfigMap edit menu using the `kubectl edit` command: ```bash kubectl edit ConfigMap krci-config -n krci ``` 6. In the YAML file, add the `available_clusters` parameter and insert the **Secret name** of the cluster — the cluster name you entered in the Portal with the `-cluster` suffix appended: ```yaml title="krci-config ConfigMap" data: available_clusters: -cluster ``` :::warning The value must exactly match the name of the cluster Secret (`-cluster` for Bearer clusters), because it becomes the `clusterName` of the Environment (Stage) and is resolved as a Secret name by the cd-pipeline-operator. Using the plain cluster name without the suffix results in an Environment that fails with `failed to get cluster secret: secrets "" not found`. For clusters added with the IRSA credentials type, use the name **without** the `-cluster` suffix — the operator derives a kubeconfig Secret named `` from the IRSA configuration. To list several clusters, separate the values with a comma followed by a space, e.g. `available_clusters: dev2-cluster, prod-cluster`. ::: 7. Ensure the `available_clusters` parameter is added into the config map: ``` kubectl get ConfigMap krci-config -n krci -o yaml ``` ## Integrate ArgoCD with External Cluster To integrate ArgoCD with an external cluster, you need to register the target cluster’s credentials with ArgoCD so that ArgoCD can securely connect to and manage resources in that cluster. This process involves creating a Kubernetes Secret **in the ArgoCD namespace** containing the cluster connection configuration — Argo CD only discovers cluster Secrets in its own control-plane namespace (typically `argocd`); the apps-in-any-namespace feature does not apply to cluster Secrets. Depending on your platform and security requirements, authentication can be configured via a static token or by using an AWS IAM Role for Service Accounts (IRSA). :::tip For clusters added with the Bearer credentials type, the cd-pipeline-operator already generates a ready-to-use Argo CD cluster Secret next to the source Secret in the platform namespace as soon as the connectivity check succeeds. Its name is the cluster Secret name with the `-argocd-cluster` suffix appended: a cluster added as `dev2` is stored as Secret `dev2-cluster` and produces `dev2-cluster-argocd-cluster`. Instead of composing the Secret manually, you can copy that generated Secret into the ArgoCD namespace (keep the `argocd.argoproj.io/secret-type: cluster` label). ::: :::warning The `name` field inside the Secret must exactly match the value you added to `available_clusters` (e.g., `-cluster`). The Argo CD Applications generated by the platform reference the destination cluster **by name**, taken from the Environment's `clusterName` — if the names differ, deployments fail with a "cluster not found" error. Also make sure the `krci` AppProject allows the new destination, as described in the [Argo CD Integration](../operator-guide/cd/argocd-integration.md#deploy-argo-cd-application-to-remote-cluster-optional) page; otherwise deployments fail with `InvalidSpecError: application destination server '-cluster' ... do not match any of the allowed destinations`. ::: Choose the configuration method that matches your external cluster's authentication mechanism and follow the relevant steps below to prepare your cluster for use with ArgoCD. ```yaml apiVersion: v1 kind: Secret metadata: name: -cluster namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: "-cluster" server: "https://EXAMPLED539D4633E53DE1B71EXAMPLE.gr7..eks.amazonaws.com" config: | { "tlsClientConfig": { "insecure": false, "caData": "" }, "bearerToken": "" } ``` ```yaml apiVersion: v1 kind: Secret metadata: name: -cluster namespace: argocd labels: argocd.argoproj.io/secret-type: cluster stringData: config: | { "awsAuthConfig": { "clusterName": "", "roleARN": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" }, "tlsClientConfig": { "insecure": false, "caData": "" } } name: "" server: "https://EXAMPLED539D4633E53DE1B71EXAMPLE.gr7..eks.amazonaws.com" ``` ```json "-cluster": { "config": { "awsAuthConfig": { "clusterName": "", "roleARN": "arn:aws:iam:::role/AWSIRSA_{cluster_name}_ArgoCDMaster" }, "tlsClientConfig": { "insecure": false, "caData": "" } }, "name": "", "server": "https://EXAMPLED539D4633E53DE1B71EXAMPLE.gr7..eks.amazonaws.com" } ``` After applying the configuration, you can verify the cluster connection `ArgoCD` -> `Settings` -> `Clusters` -> ``. ## Deploy application on new cluster To create a Deployment with an Environment, follow the instructions specified in the [Add Deployment](./add-cd-pipeline.md) page. When creating an Environment, specify your new cluster name in the **Cluster** field. ## Remove External Cluster To disconnect an external cluster from the platform, remove the resources in the following order: 1. Delete (or migrate) all Environments that reference the cluster. 2. Remove the cluster name from the `available_clusters` parameter in the `krci-config` ConfigMap. 3. Delete the cluster in the Portal (**Configuration** -> **Deployment** -> **Clusters**) — this deletes the cluster Secret together with the generated Argo CD Secret owned by it. 4. Delete the Argo CD cluster Secret you created in the ArgoCD namespace and remove the destination from the `krci` AppProject. :::warning Always delete Environments **before** the cluster Secret. If the Secret is removed first, the cd-pipeline-operator can no longer reach the cluster during Environment deletion and skips the remote cleanup, leaving orphaned namespaces and RBAC objects on the external cluster. ::: ## Related Articles * [Argo CD Integration](../operator-guide/cd/argocd-integration.md) * [Add Application](add-application.md) * [Add Library](add-library.md) * [Add Autotest](add-autotest.md) * [Add Deployment Flow](add-cd-pipeline.md) --- ## Add Git Server # Add Git Server This guide provides a step-by-step procedure for integrating KubeRocketCI with Version Control Systems (VCS) such as GitHub, GitLab, or Bitbucket. This is the mandatory step to enable KubeRocketCI to interact with code repositories, allowing users to automate workflows, manage components, and streamline the CI/CD process. ## Integration Procedure To integrate KubeRocketCI with appropriate VCS, follow the steps below: 1. Generate an SSH key pair and add a public key to your [GitHub](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent), [GitLab](https://docs.gitlab.com/ee/user/ssh.html), or [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification) account. ```bash ssh-keygen -t ed25519 -C "email@example.com" ``` 2. Generate access token for [GitLab](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) or [GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) account with read/write access to the API. Both personal and project access tokens are applicable. For [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/create-an-api-token/), generate an API token with the required permissions. To create access token in GitHub, follow the steps below: * Log in to GitHub. * Click the profile account and navigate to **Settings** -> **Developer Settings**. * Select *Personal access tokens (classic)* and generate a new token with the following scopes: - `repo` - full control of private repositories, including cloning, commit statuses, and posting pipeline result comments on pull requests. - `admin:repo_hook` - to create and manage repository webhooks. - `admin:org_hook` - to create and manage organization webhooks. - `user` - to read user account and email information (Pull Request author details). ![Repo permissions](../assets/operator-guide/github-scopes-1.png "Repo permissions") :::note The access above is required for the codebase operator to setup hooks. The `repo` scope also lets KubeRocketCI post pipeline result comments on pull requests, so no additional scope is needed for that feature. ::: ![Admin:repo permission](../assets/operator-guide/github-scopes-2.png "Admin:repo permission") ![Admin:org permission](../assets/operator-guide/github-scopes-4.png "Admin:org permission") ![User permission](../assets/operator-guide/github-scopes-3.png "User permission") :::warning Make sure to save a new personal access token because it won't be displayed later. ::: To create access token in GitLab, follow the steps below: * Log in to GitLab. * In the top-right corner, click the avatar and select **Settings**. * On the **User Settings** menu, select **Access Tokens**. * Choose a name and an optional expiry date for the token. * In the **Scopes** block, select the **api** scope for the token. :::note The `api` scope grants full API access, which also lets KubeRocketCI post pipeline result comments as merge request notes. No additional scope is needed for that feature. ::: ![Personal access tokens](../assets/operator-guide/scopes.png "Personal access tokens") * Click the **Create personal access token** button. :::note Make sure to save the access token as there will not be any ability to access it once again. ::: In case you want to create a project access token instead of a personal one, take the following steps: * Log in to GitLab and navigate to the project. * On the **User Settings** menu, select *Access Tokens*. * Choose a name and an optional expiry date for the token. * Choose a role: *Owner* or *Maintainer*. * In the **Scopes** block, select the *api* scope for the token. ![Project access tokens](../assets/operator-guide/scopes-project.png "Project access tokens") * Click the **Create project access token** button. To create an API token in Bitbucket, follow the steps below: * Log in to Bitbucket. * In the top navigation bar, click the **Settings** icon located in the upper-right corner. * Navigate to **Personal settings** and select **Atlassian account settings**. * Select the **Security** tab on the top navigation bar. * In the **API tokens** section, click the **Create and manage API tokens** button. * In the opened **API tokens** page, click the **Create API token with scopes** button. ![Create API token](../assets/operator-guide/git-servers/bitbucket-create-api-token.png "Create API token") * In the **Name and expiry** section, provide a name for the token and set the desired expiration period. ![Name and expiry](../assets/operator-guide/git-servers/bitbucket-name-and-expiry.png "Name and expiry") * In the **Select app** section, choose the "Bitbucket" option as API token app. ![Select app](../assets/operator-guide/git-servers/bitbucket-select-app.png "Select app") * In the **Select scopes** section, select the required scopes for the token. The following scopes are required for KubeRocketCI integration: - `read:account` - to read user account information. - `admin:repository:bitbucket` - to manage repositories. - `read:repository:bitbucket` - to read repository data. - `write:repository:bitbucket` - to write repository data, including commit build statuses. - `read:pullrequest:bitbucket` - to read pull request information. - `write:pullrequest:bitbucket` - to post pipeline result comments on pull requests. - `read:webhook:bitbucket` - to read webhook configurations. - `read:workspace:bitbucket` - to enable Git resource discovery in the KubeRocketCI portal. - `write:webhook:bitbucket` - to create and manage webhooks. ![Select scopes](../assets/operator-guide/git-servers/bitbucket-select-scopes.png "Select scopes") * In the **Create token** section, verify the provided information and click the **Create token** button. ![Create token](../assets/operator-guide/git-servers/bitbucket-create-token.png "Create token") * Copy the generated token and store it securely, as it will not be displayed again. ![Copy token](../assets/operator-guide/git-servers/bitbucket-copy-token.png "Copy token") 3. Enable integration in KubeRocketCI: :::warning For integration with **Bitbucket**, the **token** field must contain the base64-encoded string `:`. Note the following: * Use your **Atlassian account email** as the username part (not your Bitbucket nickname). A wrong identifier results in a silent `401 Unauthorized`. * Encode the value with the `-n` flag — it is **mandatory**. A trailing newline makes the credential invalid and produces an `Invalid Authorization header` error. ```bash echo -n "jane.doe@example.com:ATATT3xFf...your-api-token" | base64 ``` ::: To enable integration with the selected VCS, it is necessary to add a new Git Server in KubeRocketCI portal. Navigate to the **Configuration** section and select the **Git Servers** under **Version Control System**. Click the **Add Git Server** button and fill in the following fields in the opened dialog: * **Git provider** – select the Git hosting service: GitHub, GitLab, or Bitbucket. * **Name** – a unique name for this Git Server integration (e.g., `my-github` or `company-gitlab`). Used to identify the server when creating codebases. * **Host** – the base URL of your Git server (e.g., `https://github.com`, `https://gitlab.com`, or your self-hosted GitLab/GitHub/Bitbucket URL). * **User** – the username or account name used to access the Git server (e.g., your GitHub username or GitLab user name). * **SSH port** – the port used for SSH connections to the Git server. Default is usually `22`; change only if your server uses a different SSH port. * **HTTPS port** – the port used for HTTPS connections. Default is usually `443`; change only if your server uses a different HTTPS port. * **Override WebHook URL** – (optional) a custom URL where the Git server will send webhook events. Leave empty to use the default KubeRocketCI webhook endpoint. See [Advanced Configuration: Using a Custom Webhook URL](#advanced-configuration-using-a-custom-webhook-url) for details. * **Skip Webhook SSL Verification** – (optional) enable this to skip TLS/SSL verification for webhook requests (e.g., for self-signed certificates in development). Not recommended for production. * **Disable Tekton Resources** – (optional) enable this to prevent KubeRocketCI from creating Tekton pipeline resources (e.g., Pipelines, Tasks) for this Git server. Use when you manage Tekton resources externally. * **Private SSH Key** – the content of the private SSH key (the one that corresponds to the public key you added to GitHub/GitLab/Bitbucket). Paste the full key including the `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----` lines. * **Token** – the token you generated in step 2. Click **Save** to create the Git Server. As a result, you will be able to create codebases using an integrated Version Control System. ## Bitbucket Default Branch Management When onboarding components via KubeRocketCI with Bitbucket as the Git Server, Bitbucket automatically creates a `master` branch as the default, regardless of the branch specified during component creation (e.g., `main`). This may result in inconsistencies with the expected default branch. To change the default branch from `master` to desired branch in Bitbucket, follow these steps: * Log in to Bitbucket. * Navigate to the repository where the default branch needs to be changed. * In the left sidebar menu, select **Repository Settings**. * Proceed to the **Advanced** section. Locate the **Main branch** field and select your desired branch to set it as the default. * Click **Save changes** to apply your modifications. ## Advanced Configuration: Using a Custom Webhook URL Custom Webhook URL configuration is effectively utilized in Kubernetes clusters facing specific restrictions on traffic routing or requiring customized webhook event handling. Beyond the conventional Ingress, alternative traffic routing solutions such as Service Mesh (e.g., Istio), NodePort services, or external Load Balancers can be employed to manage traffic into the cluster, offering flexibility in addressing diverse networking policies and security requirements. User is responsible for the following action: * **Ingress Traffic Routing:** The configuration and management of Ingress for custom webhook URL. This entails ensuring that the custom URL is configured to accept incoming traffic and webhook events from your VCS (e.g., GitHub, GitLab). It's essential that your networking setup, including firewalls and DNS, allows your git hosting service to reach the specified URL endpoint. * **Tekton EventListener Setup:** The creation and configuration of the Tekton EventListener, which processes webhook events directed to the custom URL. This setup involves defining the EventListener to capture and handle events from your VCS, triggering the necessary CI/CD pipelines within KubeRocketCI based on these events. For more information on setting up a Ingress and Tekton EventListener for custom webhook URL, refer to the [Helm Chart](https://github.com/epam/edp-tekton/tree/release/0.12/charts/pipelines-library/templates/resources/gitservers). ## Related Articles * [Add Application](add-application.md) * [Manage Git Servers](git-server-overview.md) --- ## Add Infrastructure # Add Infrastructure KubeRocketCI portal allows you to create an infrastructure project, clone an existing repository with the infrastructure to your Version Control System (VCS), or use an external repository and import an infrastructure to the environment. The Infrastructure codebase type is used to create and manage resources in a cloud provider. When an infrastructure project is created or cloned, the system automatically generates a corresponding repository within the integrated Version Control System. Before following these guidelines, be sure to [add Git Server](./add-git-server.md) to the platform. The **Create Infrastructure** wizard contains four steps: * Initial Setup * Git & Project Info * Build Config * Review To add an infrastructure, navigate to the **Projects** section on the navigation bar and click **+ Create Project**. ## Initial Setup Once clicked, the **Create new project** wizard will appear. In this dialog, you can make a choice: * **Select Ready Template** – this option allows you to select a preconfigured, ready-to-go infrastructure template (e.g., Terraform). * **Custom Configuration** – this option allows you to create any of the supported Project types. In this case, you should select **Infrastructure**. Choose one of the strategies and click **Continue**: * **Create** – creates a sample Project on the pattern in accordance with an infrastructure language, a build tool, and a framework. This strategy is recommended for projects that start defining their infrastructure from scratch. * **Import** – allows using an existing VCS repository to integrate with KubeRocketCI. While importing the existing repository, select the Git server from the drop-down list and define the relative path to the repository, such as `epmd-edp/terraform-aws-example`. * **Clone** – clones the indicated repository into KubeRocketCI. While cloning the existing repository, it is required to fill in the **Repository URL** field and specify the credentials if needed. ## Git & Project Info In our example, we will use the **Create** strategy: Select all the settings that define how the infrastructure will be added to the Git server: * **Git server** – the pre-configured server where the component will be hosted. Select one from the drop-down list. Please refer to the [Manage Git Servers](git-server-overview.md) page to learn how to create one. * **Owner** - the relative path to the Git repository where Project will be created (e.g., `MyGithubUsername123` or `my-github-username`). * **Repository name** - the name of the repository that will store code for this Project. Must be at least two characters using the lower-case letters, numbers and inner dashes. * **Default branch** – the default branch the Project will be created with. The default branch cannot be deleted. * **Project name** – the name of the infrastructure. Must be at least two characters using lower-case letters, numbers, and inner dashes. * **Description** – brief and concise description that explains the purpose of the infrastructure. * **Private** – by default, all created Projects have private visibility settings in your Git account. Uncheck this option to create a public Git repository. * **Empty project** – check this box to create an infrastructure with an empty repository. The empty repository option is available only for the **Create** strategy. ## Build Config Specify the infrastructure language and versioning properties: * **Code Language** – defines the code language with its supported frameworks. For infrastructure, **Terraform** is commonly used; other options may be available depending on the platform configuration. * **Language version/framework** – defines the specific framework or language version of the infrastructure. The field depends on the selected code language. * **Build Tool** – allows you to choose the build tool to use. The set of tools can be changed in accordance with the selected code language. * **Codebase versioning type** – defines how the infrastructure tag will be changed once the new image version is built. There are two versioning types: * **default**: Using the `default` versioning type, to specify the version of the current artifacts, images, and tags in the Version Control System, a developer should navigate to the corresponding file and change the version **manually**. * **semver**: Using the `semver` versioning type, a developer indicates the version number from which all artifacts will be versioned and, as a result, **automatically** registered in the corresponding file. When selecting the `semver` versioning type, extra fields will appear; type the version number from which you want the artifacts to be versioned. This versioning type is recommended and selected by default. :::note The **Start Version From** field should be filled out in compliance with the semantic versioning rules, e.g. 1.2.3 or 10.10.10 (0.1.0 by default). The **Suffix** field is also mandatory and has the **SNAPSHOT** value by default. Please refer to the [Semantic Versioning](https://semver.org/) page for details. ::: * **Specify the pattern to validate a commit message** – the regular expression used to indicate the pattern that is followed on the project to validate a commit message in the code review pipeline. An example of the pattern: `^[PROJECT_NAME-d{4}]:.*$`. * **Integrate with Jira server** – this check box is used in case it is required to connect Jira tickets with the commits and have a respective label in the **Fix Version** field. :::note To adjust the Jira integration functionality, first apply the necessary changes described on the [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) page. ::: * **Jira Server** – the integrated Jira server with related Jira tasks. * **Specify the pattern to find a Jira ticket number in a commit message** – based on this pattern, the value from KubeRocketCI will be displayed in Jira. * **Mapping field name** – the section where the additional Jira fields are specified, i.e. the names of the Jira fields that should be filled in with attributes from KubeRocketCI: * Select the name of the field in a Jira ticket. The available fields are: _Fix Version/s_, _Component/s_, and _Labels_. * Click the **Add** button to add the mapping field name. * Enter the Jira pattern for the field name: * For the **Fix Version/s** field, select the **EDP_VERSION** variable that represents a KubeRocketCI upgrade version, as in _2.7.0-SNAPSHOT_. Combine variables to make the value more informative. For example, the pattern **EDP_VERSION-EDP_COMPONENT** will be displayed as _2.7.0-SNAPSHOT-nexus-operator_ in Jira. * For the **Component/s** field, select the **EDP_COMPONENT** variable that defines the name of the existing repository. For example, _nexus-operator_. * For the **Labels** field, select the **EDP_GITTAG** variable that defines a tag assigned to the commit in GitHub. For example, _build/2.7.0-SNAPSHOT.59_. * Click the bin icon to remove the Jira field name. ## Review and Create The **Review and Create** window allows you to ensure the infrastructure configuration suits your needs and verify you entered the specifications correctly. :::note After the complete adding of the infrastructure, inspect the [Manage Infrastructures](infrastructure.md) page to learn how you can operate infrastructure codebase types. ::: ## Related Articles * [Manage Infrastructures](infrastructure.md) * [Add CD Pipeline](add-cd-pipeline.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) * [Manage Git Servers](git-server-overview.md) --- ## Add Library # Add Library KubeRocketCI portal allows you to create a library, clone an existing repository with the library to your Version Control System (VCS), or use an external repository and import a library to the environment. When a library is created or cloned, the system automatically generates a corresponding repository within the integrated Version Control System. Before following these guidelines, be sure to [add Git Server](./add-git-server.md) to the platform. The **Create Library** wizard contains four steps: * Initial Setup * Git & Project Info * Build Config * Review To add a library, navigate to the **Projects** section on the navigation bar and click **+ Create Project**. ## Initial Setup Once clicked, the **Create new project** wizard will appear. In this dialog, you can make a choice: * **Select Ready Template** – this option allows you to select a preconfigured, ready-to-go library (e.g., shared pipeline or Terraform module). * **Custom Configuration** – this option allows you to create any of the supported Project types. In this case, you should select **Library**. Choose one of the strategies and click **Continue**: * **Create** – creates a sample Project on the pattern in accordance with a library language, a build tool, and a framework. This strategy is recommended for projects that start developing their libraries from scratch. * **Import** – allows using an existing VCS repository to integrate with KubeRocketCI. While importing the existing repository, select the Git server from the drop-down list and define the relative path to the repository, such as `epmd-edp/python-python-flask`. * **Clone** – clones the indicated repository into KubeRocketCI. While cloning the existing repository, it is required to fill in the **Repository URL** field and specify the credentials if needed. ## Git & Project Info In our example, we will use the **Create** strategy: Select all the settings that define how the library will be added to the Git server: * **Git server** – the pre-configured server where the component will be hosted. Select one from the drop-down list. Please refer to the [Manage Git Servers](git-server-overview.md) page to learn how to create one. * **Owner** - the relative path to the Git repository where Project will be created (e.g., `MyGithubUsername123` or `my-github-username`). * **Repository name** - the name of the repository that will store code for this Project. Must be at least two characters using the lower-case letters, numbers and inner dashes. * **Default branch** – the default branch the Project will be created with. The default branch cannot be deleted. * **Project name** – the name of the library. Must be at least two characters using lower-case letters, numbers, and inner dashes. * **Description** – brief and concise description that explains the purpose of the library. * **Private** – by default, all created Projects have private visibility settings in your Git account. Uncheck this option to create a public Git repository. * **Empty project** – check this box to create a library with an empty repository. The empty repository option is available only for the **Create** strategy. ## Build Config Specify the library language and versioning properties: * **Code Language** – defines the code language with its supported frameworks: * Java – selecting a specific Java version (17, 21, and 25 are available; Java 8 and 11 have been deprecated starting from KubeRocketCI version 3.12). * JavaScript – selecting JavaScript allows using React, Vue, Angular, Express, and Next.js frameworks. * Python – selecting Python allows using Python v.3.8, FastAPI, and Flask frameworks. * Groovy-pipeline – selecting Groovy-pipeline allows customizing stage logic. * Terraform – selecting Terraform allows using different Terraform versions via the **Terraform version manager** ([tfenv](https://github.com/tfutils/tfenv#usage)). KubeRocketCI supports all actions available in Terraform. For details, please refer to the [Use Terraform Library in KubeRocketCI](../operator-guide/ci/ci-pipeline-terraform.md) page. * Rego – this option allows using Rego code language with an Open Policy Agent (OPA) Library. * Container – this option allows using the Kaniko tool for building container images from a Dockerfile. * Helm – this option allows using [chart testing lint](https://github.com/helm/chart-testing) for Helm charts or using a Helm chart as a set of other Helm charts. * C# – selecting C# allows using .Net v.3.1 and .Net v.6.0 frameworks. * Other – selecting Other allows extending the default code languages when creating a codebase with the Clone/Import strategy. :::note The **Create from template** strategy does not allow customizing the default code language set. ::: * **Language version/framework** – defines the specific framework or language version of the library. The field depends on the selected code language. * **Build Tool** – allows you to choose the build tool to use. The set of tools can be changed in accordance with the selected code language. :::note The **Select Build Tool** field disposes of the default tools and can be changed in accordance with the selected code language. ::: * **Codebase versioning type** – defines how the library tag will be changed once the new image version is built. There are two versioning types: * **default**: Using the `default` versioning type, to specify the version of the current artifacts, images, and tags in the Version Control System, a developer should navigate to the corresponding file and change the version **manually**. * **semver**: Using the `semver` versioning type, a developer indicates the version number from which all artifacts will be versioned and, as a result, **automatically** registered in the corresponding file (e.g. pom.xml). When selecting the `semver` versioning type, extra fields will appear; type the version number from which you want the artifacts to be versioned. This versioning type is recommended and selected by default. :::note The **Start Version From** field should be filled out in compliance with the semantic versioning rules, e.g. 1.2.3 or 10.10.10 (0.1.0 by default). The **Suffix** field is also mandatory and has the **SNAPSHOT** value by default. Please refer to the [Semantic Versioning](https://semver.org/) page for details. ::: * **Specify the pattern to validate a commit message** – the regular expression used to indicate the pattern that is followed on the project to validate a commit message in the code review pipeline. An example of the pattern: `^[PROJECT_NAME-d{4}]:.*$`. * **Integrate with Jira server** – this check box is used in case it is required to connect Jira tickets with the commits and have a respective label in the **Fix Version** field. :::note To adjust the Jira integration functionality, first apply the necessary changes described on the [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) page. ::: * **Jira Server** – the integrated Jira server with related Jira tasks. * **Specify the pattern to find a Jira ticket number in a commit message** – based on this pattern, the value from KubeRocketCI will be displayed in Jira. * **Mapping field name** – the section where the additional Jira fields are specified, i.e. the names of the Jira fields that should be filled in with attributes from KubeRocketCI: * Select the name of the field in a Jira ticket. The available fields are: _Fix Version/s_, _Component/s_, and _Labels_. * Click the **Add** button to add the mapping field name. * Enter the Jira pattern for the field name: * For the **Fix Version/s** field, select the **EDP_VERSION** variable that represents a KubeRocketCI upgrade version, as in _2.7.0-SNAPSHOT_. Combine variables to make the value more informative. For example, the pattern **EDP_VERSION-EDP_COMPONENT** will be displayed as _2.7.0-SNAPSHOT-nexus-operator_ in Jira. * For the **Component/s** field, select the **EDP_COMPONENT** variable that defines the name of the existing repository. For example, _nexus-operator_. * For the **Labels** field, select the **EDP_GITTAG** variable that defines a tag assigned to the commit in GitHub. For example, _build/2.7.0-SNAPSHOT.59_. * Click the bin icon to remove the Jira field name. ## Review and Create The **Review and Create** window allows you to ensure the library configuration suits your needs and verify you entered the specifications correctly. :::note After the complete adding of the library, inspect the [Manage Libraries](library.md) page to learn how you can operate libraries. ::: ## Related Articles * [Manage Libraries](library.md) * [Add CD Pipeline](add-cd-pipeline.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) * [Manage Git Servers](git-server-overview.md) --- ## Create Component via Marketplace # Create Component via Marketplace With the built-in Marketplace, users can easily create a new application by clicking several buttons. This page contains detailed guidelines on how to create a new component with the help of the Marketplace feature. ## Marketplace Benefits The KubeRocketCI Marketplace streamlines component creation by providing predefined templates with built-in best practices. These templates offer several key advantages: - **Accelerated Development**: Pre-configured templates eliminate the need to set up projects from scratch, significantly reducing initial development time - **Standardization**: Ensures consistency across projects by providing standardized project structures, naming conventions, and configuration patterns - **Built-in CI/CD**: Templates come with preconfigured Tekton pipelines for automated building, testing, and deployment workflows - **Quality Assurance**: Integration with SonarQube for code quality analysis and SAST tools for security scanning - **Framework Diversity**: Support for multiple programming languages and frameworks including Java, Node.js, .NET, Python, Go, and more - **Custom Templates**: Organizations can create and share custom templates tailored to their specific requirements ## Prerequisites Before creating a component via Marketplace, ensure you have: - Access to the KubeRocketCI portal with appropriate permissions - A configured Git server integration (GitHub, GitLab, Bitbucket, or Gerrit) - Basic understanding of your target application requirements and technology stack - Repository naming conventions and versioning strategy defined for your organization ## Template Categories The Marketplace offers various template categories to support different development needs: - **Applications**: Full-stack web applications, microservices, and API services - **Libraries**: Reusable code libraries and shared components - **Infrastructure**: Terraform modules and infrastructure-as-code templates - **Automated Tests**: Testing frameworks and test suites for quality assurance Each template includes comprehensive documentation, dependency management, and deployment configurations specific to its category. ## Add Component To create a component from template, follow the instructions below: 1. Navigate to the **Marketplace** section on the navigation bar to see the Marketplace overview page. 2. Choose the component and click **Use template**. 3. View the template details and click **Proceed**: ![Create from template](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-create-from-template.png "Create from template") 4. Fill in the required fields and click **Create**: ![Creating from template window](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-creating-from-template.png "Creating from template window") 5. Click the **Go to application** button to open the application details: ![Go to application button](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-go-to-application.png "Go to application button") 6. As a result, new component will appear in the **Components** section: ![Creating from template window](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-created-application.png "Creating from template window") ## Post-Creation Steps After successfully creating a component from a marketplace template, several automated processes begin: ### Automated Setup - **Repository Creation**: A new Git repository is automatically created in your configured Git server with the template's initial codebase - **CI/CD Pipeline Configuration**: Tekton pipelines are automatically set up for building, testing, and deploying your component - **Integration Setup**: Webhooks and integrations with quality gates (SonarQube, security scanning) are automatically configured ### Next Steps 1. **Code Customization**: Clone the repository and begin customizing the template code to meet your specific requirements 2. **Environment Configuration**: Configure environment-specific settings and secrets through the KubeRocketCI portal 3. **CI/CD Validation**: Trigger the first build to validate that all pipelines are working correctly 4. **Quality Gates Setup**: Review and adjust code quality thresholds and security scanning rules as needed 5. **Deployment Strategy**: Configure your deployment strategy and target environments ## Troubleshooting Common Issues ### Template Creation Failures If component creation fails, check the following: - **Repository Permissions**: Ensure your Git server integration has sufficient permissions to create repositories - **Naming Conflicts**: Verify that the repository name doesn't already exist in your Git server - **Network Connectivity**: Confirm that KubeRocketCI can reach your Git server and external dependencies - **Resource Quotas**: Check if there are any resource limitations in your Kubernetes cluster ### Missing Templates If expected templates don't appear in the Marketplace: - **Template Availability**: Verify that templates are properly registered in your KubeRocketCI instance - **Permissions**: Ensure your user account has access to the required template categories - **Template Repository**: Check if custom template repositories are correctly configured For additional support, consult the KubeRocketCI logs or contact your platform administrator. ## Related Articles - [Marketplace Overview](marketplace.md) - [Add Application](add-application.md) - [Add Library](add-library.md) - [Add Infrastructure](add-infrastructure.md) --- ## Add Quality Gate This section describes how to use quality gate in KubeRocketCI and how to customize the quality gate for the CD pipeline with the selected build version of the promoted application between stages. ## Apply New Quality Gate to Pipelines Quality gate pipeline is a usual Tekton pipeline but with a specific label: `app.edp.epam.com/pipelinetype: deploy`. To add and apply the quality gate to your pipelines, follow the steps below: 1. To use the Tekton pipeline as a quality gate pipeline, add this label to the pipelines: ```yaml metadata: labels: app.edp.epam.com/pipelinetype: deploy ``` 2. Insert the \ value that is the quality gate name displayed in the quality gate drop-down list of the CD pipeline menu: ```yaml metadata: name: ``` 3. Ensure the task `promote-images` contains steps and logic to apply to the project. Also ensure that the last task is `promote-images` which parameters are mandatory: ```yaml spec: params: - default: '' description: Codebases with a tag separated with a space. name: CODEBASE_TAG type: string - default: '' name: CDPIPELINE_CR type: string - default: '' name: CDPIPELINE_STAGE type: string tasks: - name: promote-images params: - name: CODEBASE_TAG value: $(params.CODEBASE_TAG) - name: CDPIPELINE_STAGE value: $(params.CDPIPELINE_STAGE) - name: CDPIPELINE_CR value: $(params.CDPIPELINE_CR) runAfter: - taskRef: kind: Task name: promote-images ``` 4. Create a new pipeline with a unique name or modify your created pipeline with the command below. Please be aware that the `edp` value is the name of the KubeRocketCI tenant: ```bash kubectl apply -f .yaml --namespace edp ```
Example: Pipeline.yaml ```yaml apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: labels: app.edp.epam.com/pipelinetype: deploy name: namespace: edp spec: params: - default: >- https://-edp..aws.main.edp.projects.epam.com/#/namespaces/$(context.pipelineRun.namespace)/pipelineruns/$(context.pipelineRun.name) name: pipelineUrl type: string - default: '' description: Codebases with a tag separated with a space. name: CODEBASE_TAG type: string - default: '' name: CDPIPELINE_CR type: string - default: '' name: CDPIPELINE_STAGE type: string tasks: - name: autotests params: - name: BASE_IMAGE value: bitnami/kubectl:1.25.4 - name: EXTRA_COMMANDS value: echo "Hello World" taskRef: kind: Task name: run-quality-gate - name: promote-images params: - name: CODEBASE_TAG value: $(params.CODEBASE_TAG) - name: CDPIPELINE_STAGE value: $(params.CDPIPELINE_STAGE) - name: CDPIPELINE_CR value: $(params.CDPIPELINE_CR) runAfter: - autotests taskRef: kind: Task name: promote-images ```
## Run Quality Gate Before running the quality gate, first of all, ensure that the environment has deployed the [created CD pipeline](add-cd-pipeline.md) and then ensure that the application is successfully deployed and ready to run the quality gate. To run quality gate, please follow the steps below: 1. Check the CD pipeline status. To do this, open the created CD pipeline, select `Image stream version`, click `DEPLOY` button and wait until `Applications`, `Health` and `Sync` statuses become `green`. This implies that the application is successfully deployed and ready to run the quality gate. 2. Select the name-of-quality-gate of `Quality gates` from the drop-down list and click the `RUN` button.The execution process should be started in the `Pipelines` menu: ## Add Stage for Quality Gate For a better understanding of this section, please read the documentation about how to [add a new stage for quality gate](add-cd-pipeline.md). The scheme below illustrates two approaches of adding quality gates: * The first type of adding a quality gate is about adding the specific quality gate to the specific pipeline stage. * The second type is rather optional and implies activating the `Promote in pipelines` option while creating a CD Pipeline to pass the quality gate in a certain sequence. As a result, after the quality gate is successfully passed, the projected image is promoted to the next stage. ## Related Articles * [Manage Environments](../user-guide/manage-environments.md) * [Add CD Pipeline](../user-guide/add-cd-pipeline.md) --- ## Application and Pipeline Statuses # Application and Pipeline Statuses KubeRocketCI uses Argo CD to manage the deployment of applications, and Tekton Pipelines to manage the CI/CD process. So, it's important to understand the differences between the statuses of Argo CD applications and KubeRocketCI pipelines, their meanings, purposes, and how to interpret them. This guide explains all types of Argo CD application and KubeRocketCI pipeline statuses, providing various use cases and troubleshooting recommendations. ## Argo CD Application Statuses When deploying applications through the KubeRocketCI portal, each deployable application is represented as an Argo CD application. For each application, Argo CD provides a status that indicates the current state of the application. There are two main types of statuses: - **Application Health Status**: Indicates the overall health of the application. - **Sync Status**: Indicates the synchronization status of the application with desired state in the Git repository. ### Application Health Status The application health status represents the overall health of the application. The following table lists the possible health statuses and their meanings: | Status | Description | |:-----------:|-------------------------------------------------------------------------------------------------------------------| | Healthy | Application and its resources are successfully deployed and synced. | | Progressing | Application is in the process of synchronizing its resources but has not yet reached a stable state. | | Degraded | One or more resources in application have issues or failed to reach a healthy state. | | Unknown | Health status of application cannot be determined, possibly due to lack of information from the cluster. | | Missing | Application exists in Argo CD but has not been deployed yet, or its resources have been deleted from the cluster. | | Suspended | Application is paused and awaiting an external event or manual intervention to resume. | ### Sync Status The sync status represents the synchronization status of the application with the desired state in the Git repository. The following table lists the possible sync statuses and their meanings: | Status | Description | |:---------:|---------------------------------------------------------------------------------------------------------------------------------------------------| | Synced | Resources in the cluster match the expected state in the Git repository. | | OutOfSync | Resources in the cluster differ from the expected state in the Git repository. Changes may have been made manually or a new version is available. | | Unknown | Sync status cannot be determined. Resources might not be deployed yet, or the application has issues preventing status retrieval. | ## KubeRocketCI Statuses KubeRocketCI uses Tekton Pipelines to manage the CI/CD process. For each pipeline, KubeRocketCI provides a status that indicates the current state of the pipeline. ### Pipeline Status The pipeline status represents the current state of the pipeline. The following table lists the possible pipeline statuses and their meanings: | Status | Description | |:-------------------:|---------------------------------------------------------------------------------------------------------------------------------| | Completed/Succeeded | The pipeline has completed successfully, and all tasks have been executed without errors. | | Failed | The pipeline has encountered an error, causing execution to stop before completion. Some tasks may have failed or been skipped. | | Unknown | The pipeline status cannot be determined. In most cases this status indicates some errors with starting pipeline. | ### Application Status :::note Pipeline status is not directly related to the application status. The application status can change independently of the pipeline status after the deployment process is completed. ::: After the deployment of an application, KubeRocketCI provides an application status that indicates the current state of the application. These statuses are related to the Argo CD application health status and sync status. The application status can be found under the **Applications** tab for the appropriate environment. These statuses are updated in real-time and provide a quick overview of the application health and sync status regarding the Argo CD application. ## Use Cases :::note In the following use cases, `fast-api` is used as an example application for deployment scenarios. For more information on creating and deploying `fast-api` application, refer to the [Scaffold and Deploy FastAPI Application](../use-cases/application-scaffolding.md) use case. ::: To better understand the Argo CD application and KubeRocketCI pipelines statuses, there are several use cases that showcase the statuses in different scenarios. ### Use Case 1: Healthy Application In this scenario, we will cover the status of a healthy application that has been successfully deployed and synced. Suppose we have `fast-api` application to be deployed in the `demo-dev` environment. Follow the steps below to deploy the application: 1. Navigate to the KubeRocketCI portal. 2. In the left sidebar, navigate to the **Deployments** tab and select the **demo** (or any other appropriate) environment where the application should be deployed. 3. Select the **dev** environment and navigate to the **Applications** tab. Click on the **Configure Deploy** button to deploy the application. 4. Choose the appropriate application version and click on the **Start Deploy** button to deploy the application. 5. After the deployment process is started, navigate to the **Pipelines** tab to monitor the pipeline status. Ensure that the pipeline status is `Succeeded`. This status indicates that the pipeline has completed successfully. All tasks have been executed without errors. Application is now deployed and synced. 6. To ensure that the application is healthy and synced, navigate back to the **dev** environment of the **demo** deployment. Under the **Applications** tab, the application status should be `Healthy` and `Synced`. These statuses indicate that the Argo CD application is healthy and all resources are successfully deployed and synced. 7. Also, to verify the application status in Argo CD, navigate to the Argo CD UI and check the application status. The application should be `Healthy` and `Synced`. ![Argo CD Application Status](../assets/user-guide/application-statuses/argo-cd-application-status.png "Argo CD Application Status") This use case demonstrates the statuses of a healthy application that has been successfully deployed and synced. ### Use Case 2: OutOfSync Application In this scenario, we will cover the status of an application that is out of sync with the desired state in the Git repository. Suppose we have successfully deployed the `fast-api` application in the `demo-dev` environment. Follow the steps below to make the application out of sync: 1. In KubeRocketCI portal, navigate to the **Applications** tab under the **dev** environment of the **demo** deployment. Ensure that the application status is `Healthy` and `Synced`. Also, verify the application status in Argo CD. The application should be `Healthy` and `Synced`. ![Argo CD Application Status](../assets/user-guide/application-statuses/argo-cd-application-status.png "Argo CD Application Status") 2. Make changes to the application resources directly in the cluster. This action will make the application out of sync. :::note Ensure that the correct `Default namespace` and `Allowed namespaces` values are defined in the **Account setting** of the KubeRocketCI portal. ::: For example, navigate to the Kubernetes section of the KubeRocketCI portal and scale the deployment of the `fast-api` application to 0 replicas. 3. After making the changes, navigate back to the **Applications** tab under the **dev** environment of the **demo** deployment. The application status should now be `Healthy`, but `OutOfSync`. This status indicates that the application is healthy, but the resources in the cluster differ from the expected state in the Git repository. 4. Also, verify the application status in Argo CD. The application status should be `Healthy`, but the sync status should be `OutOfSync`. ![Argo CD OutOfSync Application](../assets/user-guide/application-statuses/argo-cd-out-of-sync-application.png "Argo CD OutOfSync Application") Argo CD highlights the resources that are out of sync, making it easier to identify the changes that need to be synchronized. To identify the differences between the cluster and the Git repository states, click on the **Diff** button at the top of the screen. ![Argo CD Diff](../assets/user-guide/application-statuses/argo-cd-diff.png "Argo CD Diff") This use case demonstrates the statuses of an application that is out of sync with the desired state in the Git repository. ### Use Case 3: Degraded Application In this scenario, we will cover the status of an application that has one or more resources with issues or failed to reach a healthy state. Suppose we have successfully deployed the `fast-api` application in the `demo-dev` environment. Follow the steps below to make the application degraded: 1. In KubeRocketCI portal, navigate to the **Applications** tab under the **dev** environment of the **demo** deployment. Ensure that the application status is `Healthy` and `Synced`. Also, verify the application status in Argo CD. The application should be `Healthy` and `Synced`. ![Argo CD Application Status](../assets/user-guide/application-statuses/argo-cd-application-status.png "Argo CD Application Status") 2. Make changes to the application resources directly in the cluster that will cause one or more resources to fail. For example, navigate to the Kubernetes section of the KubeRocketCI portal and edit the `fast-api` deployment with incorrect service account name (e.g., `fast-api-degraded`). Save the changes to apply the configuration. 3. After making the changes, navigate back to the **Applications** tab under the **dev** environment of the **demo** deployment. The application status should now be `Progressing` and `OutOfSync`. 4. Also, verify the application status in Argo CD. The application status should be `Progressing` and `OutOfSync`. ![Argo CD Progressing Application](../assets/user-guide/application-statuses/argo-cd-progressing-application.png "Argo CD Progressing Application") In this case, the application is in the process of synchronizing its resources but has not yet reached a stable state. 5. After a few minutes, the application status should change to `Degraded`, indicating that one or more resources in the application have issues or failed to reach a healthy state. In the Argo CD, the application status will also change to `Degraded`. The resources that have issues will be highlighted in red. ![Argo CD Degraded Application](../assets/user-guide/application-statuses/argo-cd-degraded-application.png "Argo CD Degraded Application") This use case demonstrates the statuses of an application that has one or more resources with issues or failed to reach a healthy state. ## Troubleshooting In case of issues with the application or pipeline statuses, follow the troubleshooting recommendations below: ### Application OutOfSync Status If the application status is `OutOfSync`, it means that the deployed resources in the cluster differ from the expected state in the Git repository. To resolve this issue, follow the steps below: 1. Navigate to the Argo CD and click on the **Diff** button to identify the differences between the cluster and the Git repository states. ![Argo CD Diff](../assets/user-guide/application-statuses/argo-cd-diff-button.png "Argo CD Diff") 2. Review the differences to understand what changes need to be synchronized. Re-run the deployment pipeline to sync the resources with the desired state, or manually click on the **Sync** button in the Argo CD UI. ![Argo CD Sync](../assets/user-guide/application-statuses/argo-cd-sync-button.png "Argo CD Sync") ### Application Progressing/Degraded Status If the application status is `Progressing` or `Degraded`, it means that one or more resources in the application have issues or failed to reach a healthy state. To resolve this issue, follow the steps below: 1. Navigate to the Argo CD and check the application status. If the application is `Progressing` or `Degraded`, the deployment process is not completed successfully. 2. At the top of the screen, click on the **Sync Status** button to view the synchronization status of the application. Identify the resources that are causing the issue. ![Argo CD Sync Status](../assets/user-guide/application-statuses/argo-cd-sync-status.png "Argo CD Sync Status") 3. Find and navigate to the resource that is causing the issue. Click on the **Events** tab to view the events related to the resource. Review the events to identify the root cause of the issue. ![Argo CD Events](../assets/user-guide/application-statuses/argo-cd-events.png "Argo CD Events") 4. Resolve the issue by updating the resource configuration or fixing the underlying problem. After resolving the issue, re-run the deployment pipeline to sync the resources with the desired state. ## Related Articles - [Install Argo CD](../operator-guide/install-argocd.md) - [Install Tekton Pipelines](../operator-guide/install-tekton.md) --- ## SonarQube Project Properties for Application # SonarQube Project Properties for Application KubeRocketCI simplifies the process of managing SonarQube projects by supporting SonarQube integration with predefined configurations. However, specific use cases may require customization. Below is a guide on setting the SonarQube project properties to meet all application requirements. ## Application repo structure When an application is added to the KubeRocketCI, a default `sonar-project.properties` file is automatically generated during the codebase onboarding process. However, depending on the specific needs of the application, a separate configuration file can be created, placed in the root directory of the application repository, and used for SonarQube code quality during the pipeline:
example repository structure ```text go-example-project ├── deploy-templates │ └── .. ├── Dockerfile ├── Makefile ├── controllers │ ├── hello.go │ └── hello_test.go ├── go.mod ├── go.sum ├── main.go ├── routers │ └── router.go └── sonar-project.properties ```
## Default project parameters Below are the default SonarQube project properties used in the platform: * **sonar.projectKey:** A unique identifier for the SonarQube project. This is also set to the application **codebase name** by the KubeRocketCI pipelines. * **sonar.projectName:** A readable name for the project. This is also set to the application **codebase name** by the KubeRocketCI pipelines. * **sonar.qualitygate.wait:** Enables waiting for Quality Gate results before proceeding (e.g., **true** or **false**) by default KubeRocketCI pipelines set it **true**. * **sonar.host.url:** by default set from KuberocketCI pipelines from [KuberocketCI sonarqube integration **url**](../operator-guide/code-quality/sonarqube.md#configuration). * **sonar.login:** by default set from KuberocketCI pipelines from [KuberocketCI sonarqube integration **token**](../operator-guide/code-quality/sonarqube.md#configuration). * **sonar.organization:** by default not set from KuberocketCI pipelines and used only for [SonarQube Cloud integration](../quick-start/integrate-sonarcloud.md).
sonar-project.properties: file example of configuration ```text sonar.projectKey=go-example-project sonar.projectName=go-example-project sonar.exclusions=**/cmd/**,**/deploy/**,**/deploy-templates/**,**/*.groovy,**/config/** sonar.language=go ```
For additional assistance with configuring SonarQube properties, please refer to the [SonarQube documentation](https://docs.sonarsource.com/sonarqube-server/10.6/analyzing-source-code/scanners/sonarscanner/). ## Related Articles * [Add Application](add-application.md) * [Integrate SonarQube](../quick-start/integrate-sonarcloud.md) * [Create Application](../quick-start/create-application.md) --- ## Manage Applications # Manage Applications This section describes the subsequent possible actions that can be performed with the newly added or existing applications. ## Check Application As soon as the application is successfully provisioned, the following will be created: * An Application Codebase type will appear in the Projects list. * With the **Create** strategy, a new project will be generated on GitHub or another integrated VCS. When **Clone** is chosen, the repository will be forked from the original and copied to the KubeRocketCI-integrated repository. If **Import** is selected, the platform connects to the chosen repository. The added application will be listed in the Projects list allowing you to do the following: * Observe it in the Projects list; * Manage branches; * Run build/review pipelines; * View SonarQube/DefectDojo metrics; * View Merge Requests; * View Deployments this application is a part of. The Projects lists allows you to see the following information about your applications: * **Page guide** - Runs the tour that briefly explains the page. * **Component name (clickable)** - Displays the application name set during the application creation. * **Actions menu** - Provides additional options for each individual application, such as **Edit** and **Delete**. The application that is used in a Deployment cannot be removed. * **Status** - Displays the application status. Can be red or green depending on if the KubeRocketCI portal managed to connect to the Git Server with the specified credentials or not. There are also options to sort the applications: * **Filters** - Filter codebases by their name and type. Additionally, sort the existing applications in a table by clicking the sorting icons in the table header. Sort the applications alphabetically by their name, language, build tool, framework, and CI tool. You can also sort the applications by their status: Created, Failed, or In progress. * **Selector** - Allows you to select multiple applications for bulk delete. * **Columns (clickable)** - Sort the existing applications in a list by clicking the sorting icons in the list header. ## Edit Existing Application There are two options available to edit in the application after its creation: * A pattern to validate a commit message; * Jira integration. You can edit an application directly from the Project overview page or in the Projects list using the **Actions** button. To enable Jira integration, in the **Edit Project** dialog do the following: 1. Mark the **Integrate with Jira server** check box and fill in the necessary fields. Please see steps d-h of the [Add Application](add-application.md) page. 2. Select the **Apply** button to apply the changes. 3. To disable Jira integration, in the **Edit Project** dialog do the following: 1. Clear the **Integrate with Jira server** check box. 2. Select the **Apply** button to apply the changes. 4. To create, edit and delete application branches, please refer to the [Manage Branches](../user-guide/manage-branches.md) page. ## Related Articles * [Add Application](add-application.md) * [Manage Branches](../user-guide/manage-branches.md) --- ## Argo CD Deployment Preview: Visualize Changes Before Deployment # Preview Argo CD Diff in Deploy Pipelines Comprehensive oversight of deployment procedures is critical when introducing changes to environments, particularly in production contexts. The **deploy-diff-approve** pipeline addresses this requirement by enabling a controlled deployment review process. Prior to approving or rejecting a deployment, the pipeline presents a link to the corresponding Argo CD Application resource in the logs. This integration allows operators to examine the precise Kubernetes resource changes that will be applied during deployment using Argo CD's native diff visualization capabilities. This GitOps-aligned approach significantly enhances deployment control, minimizing the risk of failures and improving overall application stability through visual verification of changes before they reach production environments. ## Set Argo CD Diff Pipeline To leverage the **deploy-diff-approve** pipeline, select it when creating or editing an [Environment](./add-cd-pipeline.md#create-environment). For advanced use cases where you need to incorporate the Argo CD diff functionality into a custom deploy pipeline, please refer to the [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) documentation. The [deploy-diff-approve](https://github.com/epam/edp-tekton/blob/master/charts/pipelines-library/templates/pipelines/cd/deploy-diff-approve.yaml) pipeline template provides a reference implementation that can be adapted to your specific requirements. ## Pipeline Usage To utilize the pipeline with Argo CD preview functionality, follow this sequence: 1. Configure the **deploy-diff-approve** pipeline in your [Environment](./add-cd-pipeline.md#create-environment) as detailed in the [previous section](#set-argo-cd-diff-pipeline). 2. Initiate the deployment of your application. For comprehensive deployment instructions, consult the [Deploy Application](../quick-start/deploy-application.md#application-deployment) documentation. 3. Navigate to the Environment details page, select the **Pipelines** tab, and access the pipeline by clicking its name. 4. Within the Pipeline details page, locate the **approve-diff** Step within the **preview-changes** Task. The Argo CD application link will be displayed in the logs. Select this link to open the corresponding Argo CD application. 5. Authenticate with Argo CD if prompted. 6. In the Argo CD Application details interface, select the **Diff** button to view the comparison: ![Open diff](../assets/user-guide/argo-cd-preview/open-diff.png "Open diff") 7. Review the Application diff visualization that displays: - Resources being added (highlighted in green) - Resources being removed (highlighted in red) - Resources being modified (showing both previous and new configurations) Verify that these changes align with your expected deployment modifications: ![View diff](../assets/user-guide/argo-cd-preview/view-diff.png "View diff") 8. Return to the KubeRocketCI pipeline interface to approve or reject the deployment based on your review. ## Related Articles * [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) * [Deployment Strategies in KubeRocketCI](../user-guide/auto-stable-trigger-type.md) * [Argo CD Integration](../operator-guide/cd/argocd-integration.md) --- ## Artifact Versioning in KubeRocketCI # Artifact Versioning in KubeRocketCI This page describes artifact versioning types in KubeRocketCI, outlining their differences and versioning patterns. Artifact versioning in KubeRocketCI is designed to ensure each build and deployment can be uniquely identified, managed, and traced back to its source. Artifact versioning is defined for every codebase individually when creating a codebase. A new application version appears when a build pipeline run completes successfully. A successfully built artifact is marked with the new version once the `git-tag`, `update-cbb`, and `update-cbis` steps complete successfully. Application version can also be seen in the pipeline that built the version. ## Versioning Types KubeRocketCI supports two versioning types: default and semver. They offer different patterns for tagging codebases. ### Default Versioning Default versioning generates versions based on the branch name and datetime, e.g. (`BRANCH-DATETIME`): ```bash 0.0.1-20260304-145929 ``` ### Semantic Versioning Semantic versioning (semver) structures versions as `MAJOR.MINOR.PATCH-BUILD_ID`, based on the [semantic versioning standards](https://semver.org/): ``` build/0.1.0-SNAPSHOT.1 ``` Several other resources are also involved in managing semantic versioning. The first resource is **CodebaseBranch**. It contains data major version and build number for a specific branch. Besides, it contains codebase versioning-related data, including: - **Version History**: A record of all versions generated from the branch. - **Build Information**: Details of the current and most recent successful builds, which may include version identifiers. **CodebaseBranch** data is displayed in the codebase details page. The second resource is **CodebaseImageStream**. It contains application container versions built for container registry. The available container versions are displayed in the environment details page when deploying an application. ## Custom Versioning User can adjust artifact versioning pattern by updating [`get-version`](https://github.com/epam/edp-tekton/tree/master/charts/pipelines-library/templates/tasks/getversion) CI step to address your personal needs. ## Related Articles * [Add Application](add-application.md) * [Add Cluster](add-cluster.md) * [Manage GitOps](gitops.md) * [Manage Registries](manage-container-registries.md) --- ## Deployment Strategies Overview # Deployment Strategies Overview Deploying applications to production environments involves risks of downtime and service disruptions. To mitigate such issues and enhance stability, KubeRocketCI provides several deployment strategies. ## Deployment Strategies in KubeRocketCI In KubeRocketCI, application deployment can be triggered once the new application version is released. Application deployment strategies are managed by the trigger type resource. There are three trigger types: * **Manual**: Requires a user to manually initiate the deployment process after a new application version is ready. * **Auto-deploy**: Automatically triggers the deployment pipeline as soon as a new application version is built, all the applications will deployed with latest version available. * **Auto-stable**: This one is similar to **Auto-deploy** but features more complicated logic. In this trigger type, the newly released application version is deployed, whereas the rest of applications in the Environment will use the **Stable** image tag, even if they have an application image version marked as **Latest**. If there are no application images with the **Stable** tag available, the **Latest** application version will be used. ### Auto-Deploy As soon as a new application artifact is built, it is immediately deployed to the environment along with the rest of the applications using their latest versions. In this strategy, if several application versions were built in a short period of time, only the latest of them will be deployed. ![Auto-deploy trigger type](../assets/operator-guide/autodeploy-trigger-type-scheme.png "Auto-deploy trigger type") ### Auto-Stable In contrast to the **Auto-deploy** trigger type, where deploy pipelines always deploy the latest application version, **Auto-stable** can safeguard application stability by preventing deployment of latest application version if this version wasn't unstable. If there are several build pipelines running at a time, they are placed in a queue and deployed consequently. ![Auto-stable trigger type](../assets/operator-guide/auto-stable-trigger-type-scheme.png "Auto-stable trigger type") ### Latest vs Stable Versions Understanding the difference between **Latest** and **Stable** tags is essential: * **Latest**: Assigned to an application image as soon as a new version is built by developers and deployment process was initiated with this new application version. Application can have no tags if a new version is built but build pipeline wasn't completed successfully. * **Stable**: The latest successfully deployed application. Application image gets the **Stable** tag when a new application image passes project's [quality gates](../user-guide/autotest.md#add-autotest-as-a-quality-gate). Only one application image can be stable at the same time. :::note Each application can have only one **Latest** and one **Stable** version at a time. However, a single version can be marked as **Latest** and **Stable** at the same time. This typically occurs when a newly built version of the application passes all quality checks immediately after its release, making it both the most recent (**Latest**) and the verified stable (**Stable**) version. ::: ## Enable Auto-Stable Strategy Enabling the **Auto-stable** trigger type necessitates proper Environment configuration. ### Prerequisites Before proceeding, ensure you already have at least two [applications](../user-guide/add-application.md) created and added into one [Environment](../user-guide/manage-environments.md). ## Configure Deployment Flow To enable the auto-stable deployment strategy, follow the steps below: 1. When creating/editing Environment, set the **auto-stable** trigger type. 2. Make sure you have an application with both **latest** and **stable** image tags. 3. Build one of your application that are included in the Environment. 4. View the deploy pipeline details to see that it deploys only stable application version. Note that **notifications-service** application, which has both both **Latest** and **Stable** images, was deployed with the stable version. While the application version 2 (latest) exists, it was deployed with the version 1 (stable). ## Related Articles * [Manage Deployments](../user-guide/manage-environments.md) * [Add Application](../user-guide/add-application.md) --- ## Manage Autotests # Manage Autotests This section describes the subsequent possible actions that can be performed with the newly added or existing autotests. ## Check Autotest As soon as the autotest is successfully provisioned, the following will be created: * An Autotest Codebase type will appear in the Projects list. * With the **Clone** or **Import** strategy, the repository will be copied or connected to the KubeRocketCI-integrated Version Control System. Autotests do not support the **Create** strategy. The added autotest will be listed in the Projects list allowing you to do the following: * Observe it in the Projects list; * Manage branches; * Run build/review pipelines; * View SonarQube/DefectDojo metrics; * View Merge Requests. :::info Autotests cannot be deployed via Deployments; only applications can. You can use autotests as quality gates in Deployments—see [Add Autotest as a Quality Gate](#add-autotest-as-a-quality-gate). ::: The Projects list allows you to see the following information about your autotests: * **Page guide** - Runs the tour that briefly explains the page. * **Component name (clickable)** - Displays the autotest name set during the autotest creation. * **Actions menu** - Provides additional options for each individual autotest, such as **Edit** and **Delete**. An autotest that is used as a quality gate in a Deployment cannot be removed. * **Status** - Displays the autotest status. Can be red or green depending on if the KubeRocketCI portal managed to connect to the Git Server with the specified credentials or not. There are also options to sort the autotests: * **Filters** - Filter codebases by their name and type. Additionally, sort the existing autotests in a table by clicking the sorting icons in the table header. Sort the autotests alphabetically by their name, language, build tool, framework, and CI tool. You can also sort the autotests by their status: Created, Failed, or In progress. * **Selector** - Allows you to select multiple autotests for bulk delete. * **Columns (clickable)** - Sort the existing autotests in a list by clicking the sorting icons in the list header. ## Edit Existing Autotest There are two options available to edit in the autotest after its creation: * A pattern to validate a commit message; * Jira integration. You can edit an autotest directly from the Project overview page or in the Projects list using the **Actions** button. To enable Jira integration, in the **Edit Project** dialog do the following: 1. Mark the **Integrate with Jira server** check box and fill in the necessary fields. Please see steps d-h of the [Add Autotest](add-autotest.md) page. 2. Select the **Apply** button to apply the changes. To disable Jira integration, in the **Edit Project** dialog do the following: 1. Clear the **Integrate with Jira server** check box. 2. Select the **Apply** button to apply the changes. To create, edit and delete autotest branches, please refer to the [Manage Branches](manage-branches.md) page. ## Add Autotest as a Quality Gate In order to add an autotest as a quality gate to a Deployment, do the following: 1. Create a Deployment with the necessary parameters. Please refer to the [Add CD Pipeline](add-cd-pipeline.md) section for the details. 2. In the **Stages** menu, select the **Autotest** quality gate type. The promoting process will then require the autotest to pass successfully. 3. In the additional fields, select the previously created autotest name and specify its branch. 4. After filling in all the necessary fields, click **Create** to start the provisioning of the pipeline. ## Configure Autotest Launch at Specific Stage To configure the autotest to run at a specific stage with the necessary parameters: 1. Add the necessary stage to the Deployment. Please refer to the [Add CD Pipeline](add-cd-pipeline.md) documentation for the details. 2. Navigate to the **run.json** file and add the stage name and the specific parameters. ## Launch Autotest Locally You can run autotests locally using an IDE (such as IntelliJ or NetBeans). To run the autotest project for local verification: 1. Clone the project to your local machine. 2. Open the project in the IDE and find the **run.json** file to copy the necessary command value. 3. Paste the copied command value into the Command line field and run it with the necessary values and namespace. 4. All the launched tests will be executed. ## Related Articles * [Add Autotest](add-autotest.md) * [Manage Branches](manage-branches.md) * [Add CD Pipeline](add-cd-pipeline.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) --- ## Build Pipeline This section provides details on the Build pipeline of the KubeRocket CI/CD pipeline framework. Explore below the pipeline purpose, stages and possible actions to perform. ## Build Pipeline Purpose The purpose of the Build pipeline contains the following points: * Check out, test, tag and build an image from the mainstream branch after a patch set is submitted in order to inspect whether the integrated with the mainstream code fits all quality gates, can be built and tested; * Be triggered if any new patch set is submitted; * Tag a specific commit in Gerrit in case the build is successful; * Build a Docker image with an application that can be afterward deployed using the Tekton/Argo CD Deploy pipeline. Find below the functional diagram of the Build pipeline with the default stages: ```mermaid flowchart TD build --> get-nexus-repository-url compile --> test start([fa:fa-circle]) --> fetch-repository fetch-repository --> init-values get-nexus-repository-url --> push get-version --> update-build-number git-tag --> update-cbis init-values --> get-version kaniko-build --> git-tag push --> kaniko-build sast --> compile sonar --> build test --> sonar update-build-number --> sast update-cbis --> stop([fa:fa-circle]) ``` ## Build Pipeline for Application and Library The Build pipeline is triggered automatically after the Code Review pipeline is completed and the changes are submitted. To review the Build pipeline, take the following steps: 1. Open Tekton Dashboard via Overview page of the UI Portal. 2. Review stages for the application and library codebases: * Init - initialization of the Code Review pipeline inputs; * Checkout - checkout of the application code; * Get-version - get the version from the pom.XML file and add the build number; * Compile - code compilation; * Tests - tests execution; * Sonar - Sonar launch that checks the whole code; * Build - artifact building and adding to Nexus; * Build-image - docker image building and adding to Docker Registry. The Build pipeline for the library has the same stages as the application except the **Build-image** stage, i.e. the Docker image is not building. * Push - artifact docker image pushing to Nexus and Docker Registry; * Git-tag - adding of the corresponding Git tag of the current commit to relate with the image, artifact, and build version. After the Build pipeline runs all the stages successfully, the corresponding tag numbers will be created in Kubernetes/OpenShift and Nexus. ## Check the Tag in Kubernetes/OpenShift and Nexus 1. After the Build pipeline is completed, check the tag name and the same with the commit revision. Simply navigate to Gerrit → Projects → List → select the project → Tags. :::note For the Import strategy, navigate to the repository from which a codebase is imported → Tags. It is actual both for GitHub and GitLab. ::: 2. Open the Kubernetes/OpenShift Overview page and click the link to Nexus and check the build of a new version. 3. Switch to Kubernetes → CodebaseImageStream (or OpenShift → Builds → Images) → click the image stream that will be used for deployment. 4. Check the corresponding tag. ## Configure and Start Pipeline Manually The Build pipeline can be started manually. To set the necessary stages and trigger the pipeline manually, take the following steps: 1. Open the Build pipeline for the created library. 2. Click the **Build with parameters** option from the left-side menu. Modify the stages by removing the whole objects massive:`{"name". "tests"}` where _name_ is a key and _tests_ is a stage name that should be executed. 3. Open Tekton Dashboard or Component details page and check the successful execution of all stages. ### Related Articles * [Add Application](add-application.md) * [Add Autotest](add-autotest.md) * [Add Library](add-library.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) * [Autotest as Quality Gate](../use-cases/autotest-as-quality-gate.md) --- ## CD Pipeline Details **CD Pipeline (Continuous Delivery Pipeline)** - KubeRocketCI business entity that describes the whole delivery process of the selected application set via the respective stages. The main idea of the CD pipeline is to promote the application build version between the stages by applying the sequential verification (i.e. the second stage will be available if the verification on the first stage is successfully completed). The CD pipeline can include the essential set of applications with its specific stages as well. In other words, the CD pipeline allows the selected image stream (Docker container in Kubernetes terms) to pass a set of stages for the verification process (SIT - system integration testing with the automatic type of a quality gate, QA - quality assurance, UAT - user acceptance testing with the manual testing). :::note It is possible to change the image stream for the application in the CD pipeline. Please refer to the [Edit CD Pipeline](add-cd-pipeline.md) section for the details. ::: A CI/CD pipeline helps to automate steps in a software delivery process, such as the code build initialization, automated tests running, and deploying to a staging or production environment. Automated pipelines remove manual errors, provide standardized development feedback cycle, and enable the fast product iterations. To get more information on the CI pipeline, please refer to the [CI Pipeline Details](ci-pipeline-details.md) chapter. The codebase stream is used as a holder for the output of the stage, i.e. after the Docker container (or an image stream in OpenShift terms) passes the stage verification, it will be placed to the new codebase stream. Every codebase has a branch that has its own codebase stream - a Docker container that is an output of the build for the corresponding branch. :::note For more information on the main terms used in KubeRocketCI, please refer to the [KubeRocketCI Glossary](../glossary.md) ::: Explore the details of the CD pipeline below. ## Deploy Pipeline The Deploy pipeline is used by default on any stage of the Continuous Delivery pipeline. It addresses the following concerns: * Deploying the application(s) to the main STAGE (SIT, QA, UAT) environment in order to run autotests and to promote image build versions to the next environments afterwards. * Deploying the application(s) to a custom STAGE environment in order to run autotests and check manually that everything is ok with the application. * Deploying the latest or a stable and some particular numeric version of an image build that exists in Docker registry. * Promoting the image build versions from the main STAGE (SIT, QA, UAT) environment. * Auto deploying the application(s) version from the passed payload (using the CODEBASE_VERSION job parameter). Find below the functional diagram of the Deploy pipeline with the default stages: :::note The input for a CD pipeline depends on the Trigger Type for a deploy stage and can be either Manual or Auto. ::: ### Related Articles * [Add Application](add-application.md) * [Add Autotest](add-autotest.md) * [Add CD Pipeline](add-cd-pipeline.md) * [Add Library](add-library.md) * [CI Pipeline Details](ci-pipeline-details.md) * [KubeRocketCI Glossary](../glossary.md) --- ## Inject Variables Into Deploy Pipelines # Inject Variables Into Deploy Pipelines KubeRocketCI offers a robust set of features to build highly customizable CI/CD workflows. One of the key features is variable injection, which enables the parametrization of deploy pipelines. This capability is particularly valuable for running multiple automated tests simultaneously and efficiently. Below are some key use cases where variable injection can be beneficial: 1. Redefining environment-specific configurations. 2. Securely managing sensitive information such as API keys, tokens, and passwords. 3. Implementing feature toggles to enable or disable various tasks and checks. 4. Dynamically configuring application behavior based on deployment contexts. By leveraging variable injection, developers can ensure smoother deployments, enhance security, and improve operational efficiency. ## Prerequisites Before proceeding, ensure you have created the following resources: 1. [Codebase](../user-guide/add-application.md); 2. [Deployment Flow](../user-guide/add-cd-pipeline.md); 3. [Environment](../user-guide/manage-environments.md#add-a-new-environment). Additionally, customize your deploy pipeline to utilize parameters efficiently. To learn more about creating custom deploy pipelines, read the [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) page. ## Add Key-Value Variable To add a key-value variable in deploy pipelines, follow the steps below: 1. Navigate to the **KubeRocketCI portal** -> **Deployments**. 2. Enter your Deployment and then enter the Environment you need to add a variable to. In the environment details page, select the **Variables** tab. Click the **+** button to add a variable. If there are no variables yet, click the **Click here to add a new variable** message. 3. Specify the variables and click **Save**. 4. Verify that the deploy pipeline uses your variables. ## Related Articles * [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) * [Add Deployment Flow](../user-guide/add-cd-pipeline.md) * [Manage Deployments](../user-guide/manage-environments.md) --- ## Change Container Registry # Change Container Registry Modifying container registry settings may be necessary. This section provides clear guidelines for transitioning to a different registry. :::warning Removing registry settings may disrupt your CI/CD process. New components created after changing the registry, including Components and Environments, will seamlessly function. However, existing 'Components' require additional steps, as outlined below. ::: ## Reset Container Registry To reset container registry integration from the KubeRocketCI, follow the steps below: 1. In the KubeRocketCI main menu, navigate to **Configuration** -> **Artifacts storage** -> **Registry**. 2. Click the **Reset registry** button, type the `confirm` word and then click **Confirm**. ## Update Registry for the Existing Components and Environments KubeRocketCI uses `CodebaseImageStream` custom resource to define Container Registry settings for the codebases. To update the registry for the existing codebases, follow the steps below: 1. List all the existing `CodebaseImageStream` CR(s) and copy their `` and `` fields: ```bash kubectl get codebaseimagestream -n krci ``` 2. Patch the `CodebaseImageStream` CR(s) using the commands for the registry you switched to: * AWS ECR: ```bash kubectl patch codebaseimagestream -n krci --type='json' -p='[{"op": "replace", "path": "/spec/imageName", "value": "//"}]' ``` * DockerHub ```bash kubectl patch codebaseimagestream -n krci --type='json' -p='[{"op": "replace", "path": "/spec/imageName", "value": "dockerhub.io//"}]' ``` * Harbor ```bash kubectl patch codebaseimagestream -n krci --type='json' -p='[{"op": "replace", "path": "/spec/imageName", "value": "//}]' ``` * Nexus ```bash kubectl patch codebaseimagestream -n krci --type='json' -p='[{"op": "replace", "path": "/spec/imageName", "value": "//}]' ``` If necessary, update the registry credentials for the existing `CD pipelines` by copying the `regcred` secret from the `krci` namespace to all the namespaces managed by the platform. To get the list of the namespaces, run the following command: ```bash kubectl get stages -n krci -o jsonpath='{range .items[*]}{.spec.namespace}{"\n"}{end}' ``` ## Related Articles * [Manage Registries](./manage-container-registries.md) * [Integrate Harbor With KubeRocketCI Pipelines](../operator-guide/artifacts-management/harbor-integration.md) * [Integrate Docker](../quick-start/integrate-container-registry.md) --- ## CI Pipeline Details **CI Pipeline (Continuous Integration Pipeline)** - KubeRocketCI business entity that describes the integration of changes made to a codebase into a single project. The main idea of the CI pipeline is to review the changes in the code submitted through a Version Control System (VCS) and build a new codebase version so that it can be transmitted to the Continuous Delivery Pipeline for the rest of the delivery process. There are three codebase types in KubeRocketCI: 1. **Applications** - a codebase that is developed in the Version Control System, has the full lifecycle starting from the Code Review stage to its deployment to the environment; 2. **Libraries** - this codebase is similar to the Application type, but it is not deployed and stored in the Artifactory. The library can be connected to other applications/libraries; 3. **Autotests** - a codebase that inspects the code and can be used as a quality gate for the CD pipeline stage. The autotest only has the Code Review pipeline and is launched for the stage verification. :::note For more information on the above mentioned codebase types, please refer to the [Add Application](add-application.md), [Add Library](add-library.md), [Add Autotests](add-autotest.md) and [Autotest as Quality Gate](../use-cases/autotest-as-quality-gate.md) pages. ::: ## Related Articles * [Add Application](add-application.md) * [Add Autotest](add-autotest.md) * [Add Library](add-library.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) * [Autotest as Quality Gate](../use-cases/autotest-as-quality-gate.md) * [Build Pipeline](build-pipeline.md) * [Code Review Pipeline](code-review-pipeline.md) --- ## Manage Clusters # Manage Clusters This section describes the subsequent possible actions that can be performed with the newly added or existing clusters. In a nutshell, cluster in KubeRocketCI Portal is a Kubernetes secret that stores credentials and endpoint to connect to the another cluster. Adding new clusters allows users to deploy applications in several clusters, thus improving flexibility of your infrastructure. The added cluster will be listed in the clusters list allowing you to do the following: * **Learn more** - opens the cluster related documentation page. * **+ Add cluster** - displays the cluster creation form. * **Cluster properties** - shows the specified cluster properties. * **Delete cluster** - remove cluster by clicking the recycle bin icon. * **Undo/Save changes** - these buttons apply or revert changes made to the cluster configuration. ## View Authentication Data To view authentication data that is used to connect to the cluster, click the **eye** icon in the corresponding field or use the `kubectl describe` command: ```bash kubectl get secret cluster_name -n krci -o yaml ``` ## Delete Cluster To delete cluster, click the bin icon, enter cluster name, and confirm deletion. Alternatively, use the `kubectl delete` command as follows: ```bash kubectl delete secret cluster_name -n krci ``` ## Related Articles * [Add Cluster](../user-guide/add-cluster.md) * [Add Application](../user-guide/add-application.md) --- ## Code Review Pipeline This section provides details on the Code Review pipeline of the KubeRocketCI CI/CD framework. Explore below the pipeline purpose, stages and possible actions to perform. ## Code Review Pipeline Purpose The purpose of the Code Review pipeline contains the following points: * Check out and test a particular developer's change (Patch Set) in order to inspect whether the code fits all the quality gates and can be built and tested; * Be triggered if any new Patch Set appears in GitHub, GitLab, Bitbucket, or Gerrit; * Send feedback about the build process in Tekton to review the card in Gerrit; * Send feedback about Sonar violations that have been found during the Sonar stage. Find below the functional diagram of the Code Review pipeline with the default stages: ```mermaid flowchart TD build --> dockerbuild-verify compile --> test dockerbuild-verify --> stop([fa:fa-circle]) dockerfile-lint --> dockerbuild-verify fetch-repository --> init-values fetch-repository --> helm-docs fetch-repository --> dockerfile-lint fetch-repository --> helm-lint helm-docs --> stop([fa:fa-circle]) helm-lint --> stop([fa:fa-circle]) init-values --> compile start([fa:fa-circle]) --> report-pipeline-start-to-gitlab report-pipeline-start-to-gitlab --> fetch-repository sonar --> build test --> sonar ``` --- ## Components: Codebases and Onboarding Strategies # Components: Codebases and Onboarding Strategies In this section, we will introduce you to the different types of codebases and strategies for onboarding codebases onto the KubeRocketCI. ## Component and Codebase From a business perspective, `Components` represent the functional building blocks of software projects. They define the purpose and functionality of different parts of a business application, such as core applications, libraries, automated tests, and infrastructure settings. Components are about what software does and how it aligns with business goals. From a technical implementation perspective, `Codebases` are the Kubernetes custom resources that manage the technical aspects of these Components. They serve as the bridge between the business logic represented by Components and the underlying Git repositories. Codebases are responsible for the technical implementation, ensuring that the Components are efficiently stored, versioned, and synchronized with the version control system. They represent the state of Components from a technical standpoint. The created codebase will be stored in the Version Control System. This is why it is required to add a [Git server](./add-git-server.md) first to create components. ### Components Components are the building blocks of software projects. They come in different types, such as `Applications`, `Libraries`, `Autotests`, and `Infrastructure`. Each component type serves a specific purpose in the development process. Applications are the deployable unit of projects, libraries contain reusable code, autotests facilitate automated testing, and infrastructure defines a project's infrastructure settings. ### Codebases Codebases are Kubernetes custom resources (CR) that represent the state of the components. They are a crucial link between component's state and underlying Git repositories. In essence, each codebase corresponds to a specific component and reflects its current state within a single Git repository. This one-to-one mapping ensures that the component's state is efficiently managed and versioned. ## Types KubeRocketCI accommodates a variety of codebase types, each serving a specific purpose in the development process. The codebase types available in KubeRocketCI are: - **Application**: the codebase that contains the source code and manifests of application that can be deployed to Kubernetes clusters. One can use different languages, frameworks, and build tools to develop application. - **Library**: the codebase that contains reusable code components that can be shared across multiple projects. They are an essential resource for efficient and consistent development. - **Autotest**: the codebase that facilitates the implementation of automated tests, helping ensure the quality, performance and reliability of applications. - **Infrastructure**: Infrastructure codebases are used to define and manage the underlying infrastructure of projects using the `Infrastructure as Code` approach, ensuring consistency and reproducibility. ## Onboarding Strategies The platform supports the following strategies to onboard codebases on the platform: - **Create** from template - This strategy allows to create a new codebase from a predefined template that matches application language, build tool, and framework. One can choose from a variety of templates that cover different technologies and use cases. This strategy is recommended for projects that start developing their applications from scratch or want to follow the best practices of KubeRocketCI. - **Import** project - This strategy allows to import an existing codebase from a Git server that is integrated with KubeRocketCI. One can select the Git server and the repository to import, and KubeRocketCI will replicate it to the platform and perform configure. This strategy is suitable for projects that already have a codebase on a Git server and want to leverage the benefits of KubeRocketCI. - **Clone** project – This strategy allows to clone an existing codebase from any Git repository that is accessible via HTTPS. One can provide the repository URL and KubeRocketCI will clone it to the platform and configure it. This strategy is useful for projects that want to copy a codebase from an external source and customize it for their needs. ## Codebase Operator The [codebase-operator](https://github.com/epam/edp-codebase-operator) is responsible for creating and managing the codebase custom resource on the KubeRocketCI. The codebase CR defines the metadata and configuration of the codebase, such as the name, description, type, repository URL, branch, path, CD tool, etc. The codebase-operator watches for changes in the codebase CR and synchronizes them with the corresponding Git repository and KubeRocketCI components. [Learn more](https://github.com/epam/edp-codebase-operator/blob/master/docs/api.md) about the codebase-operator and the custom resource (CR) API. --- ## Configuration Overview: Integrations and Features # Configuration Overview: Integrations and Features The Configuration tab of the KubeRocketCI portal is designed to integrate KubeRocketCI with third-party components. In this page you will get a brief overview of all the sections presented in the Configuration tab. ## Configuration Features Here is the list of all the sections provided in the Configuration tab. Familiarize yourself with purposes of each section: | Configuration Section | Description | |:-|:-| | [Quick Access](quick-links.md) | Configure quick links for quick access to required tools that will be displayed on the Overview page or in specific resource details, such as application or stage details. Additionally, this section is used to configure widgets, such as SonarQube and Dependency-Track. | | [Artifacts Storage](../user-guide/manage-container-registries.md) | This section contains settings on related to application deployment. There are two tabs in the section:Nexus: Integrate platform with Nexus as an application artifact storage solution.Registry: Integrate with a container registry to store container artifacts. The supported container registry solutions are AWS ECR, DockerHub, Harbor, Nexus, GitHub CR, and OpenShift CR. | | [Deployment](../quick-start/integrate-container-registry.md) | This section contains settings on integrating both artifact and container registries. There are three tabs in the section:Clusters: Connect the platform to a remote cluster to deploy application into them.GitOps: Create a GitOps repository to redefine parameters for application Helm charts.Argo CD: Configure Argo CD integration as a mandatory component for deploying applications. | | [Security](../operator-guide/devsecops/overview.md) | Integrate platform with [DefectDojo](../operator-guide/devsecops/defectdojo.md) and [Dependency-Track](../operator-guide/devsecops/dependency-track.md) to scan your codebases for security vulnerabilities.| | [Code Quality](../operator-guide/code-quality/sonarqube.md) | Integrate platform with SonarQube as a mandatory component to scan your application when executing build and review pipelines. This integration also enables users to view code quality metrics and dependency scan results of your application directly in the KubeRocketCI portal. | | [Version Control System](../user-guide/add-git-server.md) | Connect platform to a Git Server as a mandatory component to enable a CI/CD workflow. | | [Management Tool](../operator-guide/project-management-and-reporting/jira-integration.md) | Integrate platform with a Jira server to add pull the the useful codebase metadata into Jira tickets. | ## Related Articles * [Add Quick Link](quick-links.md) * [Add Git Server](add-git-server.md) * [Add Cluster](add-cluster.md) * [Manage GitOps](gitops.md) * [Manage Container Registries](manage-container-registries.md) --- ## Add Custom Tekton Triggers To run a custom pipeline on a webhook event, you have two options: attach a labeled Tekton Trigger to the platform-managed EventListener of your Git server, or run a dedicated EventListener with its own webhook. The first option needs no webhook management at all; the second gives you full control over which events your Git provider sends. ## Choose an Approach | | Option 1: Platform EventListener | Option 2: Dedicated EventListener | |---|---|---| | Webhook | Managed by the platform — nothing to create | Created manually in the Git provider | | Available events | Only the event types the platform webhook subscribes to (merge request, comment, push) | Any events the Git provider offers — you pick them in the webhook | | Setup effort | Label one Trigger | EventListener + Ingress + webhook | | Isolation | Shares the Git server's EventListener | Fully separate pipeline entry point | Pick option 1 when your pipeline reacts to the standard code-review events. Pick option 2 when you need other event types (tags, releases, pipeline events) or want your custom CI fully isolated from the platform flow. ## Option 1: Attach to the Platform EventListener :::note Declarative trigger registration requires codebase-operator v2.35.0 or later. On earlier versions, custom Triggers must be added to the EventListener `spec.triggers` list manually — see [Legacy approach](#legacy-approach). ::: For every [Git server](add-git-server.md), KubeRocketCI creates an EventListener named `edp-` that receives all webhook events for that VCS. The EventListener serves the union of two sources: * the built-in `spec.triggers` list — the stock review and build triggers; * any Trigger in the namespace matching `spec.labelSelector` — your custom triggers. ```yaml # Managed by codebase-operator — shown for reference, do not edit spec: labelSelector: matchLabels: app.edp.epam.com/gitServer: gitlab # value = GitServer resource name triggers: - triggerRef: gitlab-build - triggerRef: gitlab-review ``` The label selector is reconciled by the operator: existing EventListeners gain it automatically on operator upgrade, and it is restored if removed. ### Create a Custom Trigger The example below runs a pipeline when a Merge Request comment contains `/deploy-preview` (GitLab). Adapt the interceptors to your VCS and event type. ```yaml apiVersion: triggers.tekton.dev/v1beta1 kind: Trigger metadata: name: gitlab-deploy-preview labels: app.edp.epam.com/gitServer: gitlab # registers this Trigger with EventListener edp-gitlab spec: interceptors: - ref: name: "gitlab" kind: ClusterInterceptor params: - name: "secretRef" value: secretName: ci-gitlab secretKey: secretString - name: "eventTypes" value: ["Note Hook"] - ref: name: "cel" kind: ClusterInterceptor params: - name: "filter" value: "body.object_attributes.note.startsWith('/deploy-preview')" bindings: - ref: gitlab-binding-preview template: ref: gitlab-deploy-preview-template ``` The label **value must be the GitServer resource name** (`kubectl get gitservers`), which matches the provider name on default installations. ### Verify the Trigger Is Registered The EventListener evaluates the same label query on every incoming event, so you can preview exactly what it will serve: ```bash kubectl get triggers -l app.edp.epam.com/gitServer=gitlab -n ``` If your Trigger is listed, it is registered. After sending a matching event, confirm the fired pipeline: ```bash kubectl get pipelineruns -l triggers.tekton.dev/trigger=gitlab-deploy-preview -n ``` ## Option 2: Run a Dedicated EventListener A dedicated EventListener runs alongside the platform one and is not touched by the operator, so you own its full lifecycle: the listener, its exposure, and the webhook. ### Create the EventListener Use your own label key so the listener serves only your triggers: ```yaml apiVersion: triggers.tekton.dev/v1beta1 kind: EventListener metadata: name: my-custom-el spec: serviceAccountName: tekton-triggers-sa- labelSelector: matchLabels: app.edp.epam.com/eventlistener: my-custom-el ``` Label your Triggers with the same key/value (`app.edp.epam.com/eventlistener: my-custom-el`) to register them — same mechanism as option 1, but against your own listener. ### Expose the EventListener Tekton creates a Service named `el-my-custom-el` on port `8080`. Publish it with an Ingress (or Route on OpenShift): ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-custom-el spec: rules: - host: el-my-custom-el. http: paths: - path: / pathType: Prefix backend: service: name: el-my-custom-el port: number: 8080 ``` ### Create the Webhook Manually In your Git provider, add a webhook on the repository (or group/organization): 1. **URL**: the Ingress host, e.g. `https://el-my-custom-el.`. 2. **Secret token**: the value your Trigger's interceptor validates (for the platform secrets: `secretString` key of the `ci-gitlab` / `ci-github` secret, or a secret you create yourself). 3. **Events**: select exactly the event types your pipeline needs — this is the main advantage of a dedicated listener: tag pushes, releases, or any other events the platform webhook does not subscribe to. For GitLab: **Settings → Webhooks**; for GitHub: **Settings → Webhooks → Add webhook** (content type `application/json`). ### Verify Send a matching event (or use the provider's webhook test button) and check the listener pod and the fired pipeline: ```bash kubectl logs deploy/el-my-custom-el -n kubectl get pipelineruns -l triggers.tekton.dev/eventlistener=my-custom-el -n ``` ## Legacy Approach Before declarative registration, custom Triggers were added by patching the EventListener `spec.triggers` list directly. This still works — the operator never modifies that list — but patched entries are lost if the EventListener is recreated. Prefer labels for anything permanent. :::warning Do not combine both mechanisms for the same Trigger. A Trigger that is listed in `spec.triggers` **and** carries the label is served twice, producing two PipelineRuns per event. ::: ## Related Articles * [KubeRocketCI: Tekton Overview](tekton-pipelines.md) * [Add Git Server](add-git-server.md) * [Annotations and Labels](../developer-guide/annotations-and-labels.md) --- ## Manage Git Servers # Manage Git Servers Git Server is responsible for integration with Version Control System, whether it is GitHub, GitLab, Bitbucket, or Gerrit. The Git Server is set via the **global.gitProviders** parameter of the [values.yaml](https://github.com/epam/edp-install/blob/release/3.9/deploy-templates/values.yaml#L12) file. To view the current Git Server, you can open Portal **Configuration** -> **Git Servers** and inspect the following properties: * **Git Server status and name** - displays the Git Server status, which depends on the Git Server integration status (Success/Failed). * **Git Server properties** - displays the Git Server type, its host address, username, SSH/HTTPS port, public and private SSH keys. * **Learn more** - opens the "Manage Git Servers" documentation page. * **Undo/Save changes** - these buttons apply or revert changes made to the Git Server. * **+ Add Git Server** - add a new blank to specify the new Git Server's parameters. * **Delete a Git Server** - deletes the Git Server after confirmation. ## View Authentication Data To view authentication data that is used to connect to the Git server, click the **eye** icon in the corresponding field or use the `kubectl describe` command as follows: ```bash kubectl get secret ci- -n krci -o yaml ``` ## Delete Git Server There are two ways for deleting Git server: using KubeRocketCI portal or CLI. To delete a Git server using KubeRocketCI portal, follow the steps below: 1. Navigate to **KubeRocketCI portal** -> **Configuration** -> **Version Control System**. 2. In the **Version Control System** tab, click the bin icon. 3. In the confirmation window, enter **confirm** and click **Delete**. To remove a Git Server from the Git Servers list, utilize the `kubectl delete` command as follows: ```bash kubectl delete GitServer git_server_name -n krci ``` ## Related Articles * [Add Git Server](add-git-server.md) * [Add Application](add-application.md) --- ## Manage GitOps # Manage GitOps This page is dedicated to the `GitOps` section of the `Configuration` tab, the process of establishing the GitOps repository, outline benefits it extends to users within the platform capabilities. GitOps, short for "Git Operations", is a modern approach to managing and automating infrastructure and application deployments. In GitOps, the desired state of your environment is declared and stored in a Git repository. With GitOps, you can ensure that your infrastructure and applications are always in sync with your intended configurations and readily adapt to changing requirements. ## Overview The purpose of the `GitOps` section is to provide users with the ability to customize the state of their environments with the help of GitOps approach that enables you to store your entire deployment configuration in a Git repository, providing version control for changes, consistent collaboration, and automated deployments. Enforcing GitOps allows you to declaratively define and automate your configurations, ensuring consistency, version control, and collaboration within your team. ## Add GitOps Repository GitOps repository is added in two steps: 1. Navigate to **Configuration** -> **Deployment**-> **GitOps** and click the **+ Add GitOps repository** button. 2. Select the creation strategy, fill in the required fields, and click **Save**: The **Create** strategy allows you to create a new repository from scratch: * **Git server**: Select the [added Git server](./add-git-server.md) where the GitOps repository will be created. * **Git repo relative path**: Enter your Git account. * **Repository name**: Specify the name for the new repository. Select the **Import** strategy if you already have a repository that stores your application Helm charts' configurations: * **Git server**: Select the [added Git server](./add-git-server.md) that has credentials to access the GitOps repository. * **Git repo relative path**: Enter your Git account. * **Repository name**: Specify the name of the repository to import. 3. Check the GitOps repository connected to the platform: As the result, the `Codebase` of `system` type will be added to the Codebase list of the **Components** section: :::warning The platform allows only one GitOps repository at a time. ::: ## GitOps Usage Once the GitOps repository is added to the platform, you can set custom parameters for the deployed Helm Chart. To redefine the parameters, follow the steps below: 1. Within the GitOps repository, follow the naming convention `//-values.yaml` to create the chart values file, e.g. `mypipe/dev/fastapi-demo-values.yaml`. 2. Add the necessary custom values to the `mypipe/dev/fastapi-demo-values.yaml` file, and then commit the file to the default branch of the GitOps repository. 3. Navigate to the **Environments** section. Open the created environment, open its stage and deploy it with the **Values override** checkbox selected as it is shown below: :::note Ensure to add the [credentials template](https://argo-cd.readthedocs.io/en/latest/user-guide/private-repositories/#credential-templates) in the Argo CD settings to grant Argo CD access to both your GitOps and codebase repositories. ::: ## Customize Yamllint Scan Rules :::note Ability to customize yamllint rules is available starting from platform version 3.11. ::: Review and build pipelines for a GitOps repository use `yamllint` to scan YAML files. By default, predefined rules are applied automatically. However, it is possible to customize and extend the default rules by adding a `.yamllint` configuration file to the root of the GitOps repository. This file should define the required rules and parameters. For example: ```yaml title=".yamllint" # yamllint configuration file # Extends the default configuration: # https://yamllint.readthedocs.io/en/stable/configuration.html#default-configuration extends: default ignore: - '.yamllint' rules: line-length: disable # Disable line-length rule document-start: disable # Disable rule for requiring '---' at the document start ``` In this example, the `line-length` and `document-start` rules are disabled. For more details on available rules and their configurations, refer to the yamllint [documentation](https://yamllint.readthedocs.io/en/stable/rules.html). After creating the `.yamllint` file, commit it to the main branch of the GitOps repository. Once the changes are applied, the `yamllint` scan in review and build pipelines will use the custom rules defined in the `.yamllint` file. ## Delete GitOps Repository In case you need to delete the GitOps repository, do the following: 1. Delete the GitOps repository in the Git provider. 2. Delete the Codebase custom resource using the `kubectl delete` command: ```bash kubectl delete Codebase -n krci ``` ## Related Articles * [Add CD Pipeline](../user-guide/add-cd-pipeline.md) * [Cluster Add-Ons Overview](../operator-guide/add-ons-overview.md) * [Manage Clusters](cluster.md) --- ## User Guide: Portal Features and CI/CD Flow # User Guide: Portal Features and CI/CD Flow The KubeRocketCI portal user guide is intended for developers and provides details on working with the KubeRocketCI portal, different codebase types, and the KubeRocketCI CI/CD flow. ## KubeRocketCI Portal The KubeRocketCI portal is a central management tool in the KubeRocketCI ecosystem that provides the ability to define pipelines, project resources and new technologies in a simple way. Using the KubeRocketCI portal enables to manage business entities: * Create such codebase types as Applications, Libraries, Autotests and Infrastructures; * Manage CI/CD Pipelines; * Add Git Servers and external Clusters; * Integrate tools, such as SonarQube, Jira, DefectDojo, etc. Choose a platform section below to open its overview and related documentation: ## Related Articles * [Security Overview (SCA, SAST, Container Scanning)](./security/sca-overview.md) --- ## Manage Infrastructures # Manage Infrastructures This section describes the subsequent possible actions that can be performed with the newly added or existing infrastructures. ## Check Infrastructure As soon as the infrastructure is successfully provisioned, the following will be created: * An Infrastructure Codebase type will appear in the Projects list. * With the **Create** strategy, a new project will be generated on GitHub or another integrated VCS. When **Clone** is chosen, the repository will be forked from the original and copied to the KubeRocketCI-integrated repository. If **Import** is selected, the platform connects to the chosen repository. The added infrastructure will be listed in the Projects list allowing you to do the following: * Observe it in the Projects list; * Manage branches; * Run build/review pipelines; * View SonarQube/DefectDojo metrics; * View Merge Requests. :::info Infrastructures cannot be deployed via Deployments; only applications can. Infrastructure codebases are used to define and manage cloud or platform resources (e.g. via Terraform). ::: The Projects list allows you to see the following information about your infrastructures: * **Page guide** - Runs the tour that briefly explains the page. * **Component name (clickable)** - Displays the infrastructure name set during the infrastructure creation. * **Actions menu** - Provides additional options for each individual infrastructure, such as **Edit** and **Delete**. * **Status** - Displays the infrastructure status. Can be red or green depending on if the KubeRocketCI portal managed to connect to the Git Server with the specified credentials or not. There are also options to sort the infrastructures: * **Filters** - Filter codebases by their name and type. Additionally, sort the existing infrastructures in a table by clicking the sorting icons in the table header. Sort the infrastructures alphabetically by their name, language, build tool, framework, and CI tool. You can also sort the infrastructures by their status: Created, Failed, or In progress. * **Selector** - Allows you to select multiple infrastructures for bulk delete. * **Columns (clickable)** - Select columns to display and sort the existing infrastructures in a list by clicking the sorting icons in the list header. ## Edit Existing Infrastructure There are two options available to edit in the infrastructure after its creation: * A pattern to validate a commit message; * Jira integration. You can edit an infrastructure directly from the Project overview page or in the Projects list using the **Actions** button. To enable Jira integration, in the **Edit Project** dialog do the following: 1. Mark the **Integrate with Jira server** check box and fill in the necessary fields. Please see steps d-h of the [Add Infrastructure](add-infrastructure.md) page. 2. Select the **Apply** button to apply the changes. To disable Jira integration, in the **Edit Project** dialog do the following: 1. Clear the **Integrate with Jira server** check box. 2. Select the **Apply** button to apply the changes. To create, edit and delete infrastructure branches, please refer to the [Manage Branches](manage-branches.md) page. ## Related Articles * [Add Infrastructure](add-infrastructure.md) * [Manage Branches](manage-branches.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) --- ## Manage Libraries # Manage Libraries This section describes the subsequent possible actions that can be performed with the newly added or existing libraries. ## Check Library As soon as the library is successfully provisioned, the following will be created: * A Library Codebase type will appear in the Projects list. * With the **Create** strategy, a new project will be generated on GitHub or another integrated VCS. When **Clone** is chosen, the repository will be forked from the original and copied to the KubeRocketCI-integrated repository. If **Import** is selected, the platform connects to the chosen repository. The added library will be listed in the Projects list allowing you to do the following: * Observe it in the Projects list; * Manage branches; * Run build/review pipelines; * View SonarQube/DefectDojo metrics; * View Merge Requests. :::info Libraries cannot be deployed via Deployments; only applications can. Libraries are used as shared code or pipelines consumed by applications and other project types. ::: The Projects list allows you to see the following information about your libraries: * **Page guide** - Runs the tour that briefly explains the page. * **Component name (clickable)** - Displays the library name set during the library creation. * **Actions menu** - Provides additional options for each individual library, such as **Edit** and **Delete**. * **Status** - Displays the library status. Can be red or green depending on if the KubeRocketCI portal managed to connect to the Git Server with the specified credentials or not. There are also options to sort the libraries: * **Filters** - Filter codebases by their name and type. Additionally, sort the existing libraries in a table by clicking the sorting icons in the table header. Sort the libraries alphabetically by their name, language, build tool, framework, and CI tool. You can also sort the libraries by their status: Created, Failed, or In progress. * **Selector** - Allows you to select multiple libraries for bulk delete. * **Columns (clickable)** - Sort the existing libraries in a list by clicking the sorting icons in the list header. ## Edit Existing Library There are two options available to edit in the library after its creation: * A pattern to validate a commit message; * Jira integration. You can edit a library directly from the Project overview page or in the Projects list using the **Actions** button. To enable Jira integration, in the **Edit Project** dialog do the following: 1. Mark the **Integrate with Jira server** check box and fill in the necessary fields. Please see steps d-h of the [Add Library](add-library.md) page. 2. Select the **Apply** button to apply the changes. To disable Jira integration, in the **Edit Project** dialog do the following: 1. Clear the **Integrate with Jira server** check box. 2. Select the **Apply** button to apply the changes. To create, edit and delete library branches, please refer to the [Manage Branches](manage-branches.md) page. ## Related Articles * [Add Library](add-library.md) * [Manage Branches](manage-branches.md) * [Adjust Jira Integration](../operator-guide/project-management-and-reporting/jira-integration.md) --- ## Manage Branches # Manage Branches This page describes how to manage branches in the created Project, whether it is an application, library, autotest or infrastructure. It also briefly explains two approaches of managing custom pipelines for codebases. ## Pipeline Management There are two approaches to define codebases with Tekton pipelines: * Using [build tool and framework](../use-cases/tekton-custom-pipelines.md); * Using codebase [branch settings](../use-cases/custom-pipelines-flow.md). ### Build Tool/Framework Approach This approach involves [adding your application](../use-cases/tekton-custom-pipelines.md) operating with language type and framework. In this approach, it is essential to create a pipeline name using the pattern: `git_server_name-build_tool-framework-codebase_type-pipeline_type-codebase_versioning_type`. Follow this approach if you need to define multiple similar applications with identical settings, as this approach would be more beneficial. ### Codebase Branch Settings This approach involves creating a Project with Tekton pipelines and selecting it in the codebase branch settings. In contrast to the approach based on [build tool and framework](../use-cases/tekton-custom-pipelines.md), this one offers two main advantages: * **Simplified management**: This method eliminates the requirement for strict resource naming conventions. With this approach, pipelines can be created without following any predefined pattern. * **Branch-based logic**: With this method, pipelines can be defined not just for an application but also for specific branches within the application. If you need to frequently and quickly redefine a build or review pipeline, this approach is highly recommended. ## Add New Branch :::note When working with libraries, pay attention to specifying the branch name: the branch name is involved in the formation of the library version, so it must comply with the [semantic versioning](https://semver.org/) rules for the library. ::: When adding a Project, the default branch is **main**. To add a new branch, follow the steps below: 1. Navigate to the **Branches** block by clicking the Project name link in the Projects list and click the **+ Create branch** button: 2. Fill in the required fields to create a new branch: **Release branch** - select this option to make a release branch. Codebases from release branches are marked with a different tag. :::note When working with release branches, keep in mind that only the **semver** versioning type supports release branches. In the **default** versioning type, this option is unavailable. Additionally, the SNAPSHOT version is reset each time you create a release branch. ::: When the **Release branch** option is selected, you will also need to specify release tag the Codebase will be marked: **Branch name** - type the branch name. :::note The **Branch name** field remains static if you create a release branch. If you want to onboard an existing branch, enter the name of the existing branch in the **Branch name** fields. ::: **From** - by default, the KubeRocketCI portal will suggest creating a branch from the main branch. If you want to create a branch from another branch or commit hash, change this field: In the **From** field, select the **Commit hash** option. In the field to the right, specify the full SHA of the commit you want to create a branch from. Note that if the **From commit hash** field is empty, the latest commit from the branch name will be used. In the **From** field, select the **Branch** option. In the field to the right, specify the name of the source branch you want to create a branch from: **Branch version** - specify the application version and tag (SNAPSHOT by default). **Build pipeline** - select the build pipeline you want to use to build the application or leave the default one. **Review pipeline** - select the review pipeline you want to use to review the application or leave the default one. 3. Click **Create**. Use additional tools to get guidance for the process: - **View diagram**: Visualizes the selected pipeline to make sure it matches your needs. - **Form Guide**: Shows details about each field of the form. :::note To get the most out of managing build/review pipelines via KubeRocketCI portal, you should follow the **Add-Ons** approach for pipeline management. ::: :::info Adding of a new branch is indicated in the context of the `semver` versioning type. ::: ## Add Existing Branch Onboarding a branch that has already been created in Git to the platform follows the same steps as when you [add a new branch](../user-guide/manage-branches.md#add-new-branch). When adding an existing branch, specify its name in the **Branch name** fields. ## Edit Existing Branch To edit branch properties, follow the steps below: 1. Navigate to the **Branches** block by clicking the Project name link in the Project list. 2. Click the actions button and select **Edit**. 3. Change the branch options and click **Apply**. ## Build Branch In order to build branch from the latest commit, do the following: 1. Navigate to the **Branches** block by clicking the library name link in the Libraries list. 2. Click the **Build** button. The pipeline run status is displayed near the branch name in the **Branches** block.The corresponding item appears in the **Pipelines** section. As an alternative way, click the **tree diagram** icon to observe the real-time status of the pipeline run. ## Delete Branch :::note The default **master/main** branch cannot be removed. Additionally, deleting a branch in the portal doesn’t remove it from Git. It is required to delete it there as well if necessary. ::: In order to delete the added branch with the corresponding record in the KubeRocketCI portal database, do the following: 1. Click the Project name link in the Projects list. 2. Select the **Branches** tab. 3. Define the branch you want to delete. 4. On the branch block, click the actions button and select **Delete**. 5. Confirm deletion of the branch by typing its name and clicking **Confirm**. ## Related Articles * [Add Application](../user-guide/add-application.md) * [Add Library](../user-guide/add-library.md) * [Add Autotest](../user-guide/add-autotest.md) --- ## Manage Container Registries # Manage Container Registries This guide provides instructions on integrating the container registry with the KubeRocketCI. ## Supported Registry Providers The following table displays the registry services supported for both OpenShift and Kubernetes clusters. |Container Registry|OpenShift|Kubernetes| |:-:|:-:|:-:| |AWS ECR|:white_check_mark:|:white_check_mark:| |DockerHub|:white_check_mark:|:white_check_mark:| |Harbor|:white_check_mark:|:white_check_mark:| |Nexus|:white_check_mark:|:white_check_mark:| |GitHub (GHCR)|:white_check_mark:|:white_check_mark:| ## Add Container Registry Follow a three-step process to integrate a container registry in KubeRocketCI: 1. In the **Configuration** -> **Artifacts storage** -> **Registry** click the **+ Add Registry** button. 2. Select **Registry Provider** and enter the required details. 3. Confirm settings by clicking the **Save** button. The required fields vary depending on which container registry is chosen: |Fields|Description| |:-|:-| |Registry Endpoint|Format for the (AWS) Elastic Container Registry endpoint: xxxxxxxxxxxx.dkr.ecr.\.amazonaws.com. Where `xxxxxxxxxxxx` is your AWS account ID and `\` is where your ECR is hosted.| |Registry Space|The suffix project name in registry.| |AWS Region| The geographic area where the (AWS) Elastic Container Registry repository is hosted.| |Authentication/IRSA Role ARN|Specify the [IAM role](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policies.html) with a policy for interacting with ECR with your Kubernetes cluster. |Fields|Description| |:-|:-| |Registry Space|The unique identifier/name of the user or company linked to your DockerHub account.| |User|The user account ID or community user account ID with push permission.| |Password/Token|Provide the [Password/Token](https://docs.docker.com/security/for-developers/access-tokens/) corresponding to your DockerHub account. It is recommended to use Token for security purposes.| |Checkbox/Use the Push Account's credentials|Check this to use the same account for pulling and pushing operations. If unchecked, please enter the user account ID and Password/Token for your DockerHub account or community user account ID with pull permission.| |Fields|Description| |:-|:-| |Registry Endpoint|Enter Harbor registry endpoint URL, for example, registry.example.com.| |Registry Space|The project name in registry.| |User|Provide the [robot account](../operator-guide/artifacts-management/harbor-integration.md#set-up-robot-account) name with push permissions.| |Password/Token|Provide the [secret](../operator-guide/artifacts-management/harbor-integration.md#set-up-robot-account) corresponding to your harbor account.| |Checkbox/Use the Push Account's credentials|Check this to use the same account for pulling and pushing operations. Provide the account name with pull permissions.| |Fields|Description| |:-|:-| |Registry Endpoint|Nexus service registry endpoint URL (e.g., image-registry.nexus-image-registry.svc:5000).| |Repository|Specify the Nexus repository that corresponds to your project.| |User|Provide the username with push permissions.| |Password/Token|Enter the confidential combination used for authenticating your access to the container registry.| |Fields|Description| |:-|:-| |Registry Space|The unique identifier/name of the user or company linked to your GitHub account.| |User|The user account ID or community user account ID with push permission.| |Password/Token|Provide the [Token](https://docs.github.com/en/enterprise-server@3.9/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) corresponding to your GitHub account. The minimal set of permissions required for the token is described in the [Manage Git Providers](../user-guide/add-git-server.md).| |Checkbox/Use the Push Account's credentials|Check this to use the same account for pulling and pushing operations. If unchecked, please enter the user account ID and Token for your GitHub account or community user account ID with pull permission.| ## Related Articles * [Install KubeRocketCI](../operator-guide/install-kuberocketci.md) * [Install Harbor](../operator-guide/artifacts-management/harbor-installation.md) * [Change Container Registry](change-container-registry.md) --- ## Manage Deployments # Manage Deployments This page describes actions that can be performed to an already created Deployment. If no Deployments are created yet, navigate to the [Add Deployment](add-cd-pipeline.md) page. The added Deployment will be listed in the Projects list allowing you to do the following: * Observe it in the Projects list; * Manage branches; * Run build/review pipelines; * View SonarQube/DefectDojo metrics; * View Merge Requests; * View Deployments this application is a part of. The added Deployment will be listed in the Deployments list allowing you to do the following: * **Deployment status** - displays the deployment status. Can be red or green depending on if the KubeRocketCI portal managed to connect to the Git Server with the specified credentials or not; * **Deployment name** (clickable) - displays the Deployment name set during the Deployment creation; * **Page guide** - runs the tour that briefly explains the page; * **Enable filtering** - enables filtering by Deployment name and namespace where this deployment is located in; * **+ Create new deployment** - opens the **Create new deployment** menu; * **Configure** - edit the Deployment by clicking the vertical ellipsis button and then selecting **Configure**. For details see the [Edit Existing Deployment](#edit-existing-deployment) section; * **Delete** - remove Deployment by clicking the vertical ellipsis button and then selecting **Delete**; :::note Please keep in mind that after deleting the deployment, all the created resources within the deployment will be deleted. ::: ## View Deployment Details To view deployment details, click the deployment name in the deployments list. Once clicked, the following data will be displayed: * **Open deployment in Argo CD** - opens the corresponding resource in Argo CD; * **Actions** - allows to edit some parameters of the deployment or delete it; * **+ Create environment** - displays the **Create new environment** menu; * **Environment name (clickable)** - opens the Environment details page; * **Environment status** - displays the status of the created Environment; * **Application name (clickable)** - opens the details of the application that is deployed within the stage; * **Application deployment status** - displays the deployed application; * **Open application logs** - opens the the application container logs; * **Open application terminal** - opens the container terminal window; * **Argo CD** - opens a new tab with Argo CD resources related to the application; * **Logging** - allows to view the stage in Grafana or Kibana. ### Edit Existing Deployment Edit the deployment directly from the deployment overview page or when viewing the deployment data: 1. Select **Edit** in the options icon menu next to the deployment name. 2. Apply the necessary changes (edit the list of applications for deploy, application branches, and promotion in the pipeline). Add new extra stages by clicking the plus sign icon and filling in the application branch and promotion in the pipeline. 3. Click the **Apply** button to confirm the changes. ### Add a New Environment In order to create a new environment for the existing deployment, follow the steps below: 1. Navigate to the **Environments** block by clicking the deployment name link in the deployments list. 2. Click the **+ Create environment** button. 3. Fill in the required fields in the dialog. Please see the [Stages Menu](../user-guide/add-cd-pipeline.md#create-environment) section for details. 4. Click the **Apply** button. ### Edit Environment In order to edit an environment for the existing deployment, follow the steps below: 1. Click the Environment name in the Deployment details page. 2. In the upper-right corner of the page, click the **Actions** button and select **Edit**. 3. In the **Edit environment** dialog, change the environment trigger type and deploy pipeline template. 4. Click the **Apply** button. ### Delete Environment :::warning You cannot remove the last Environment, as the Deployment does not exist without at least one. :::: In order to delete an Environment for the existing Deployment, follow the steps below: 1. Navigate to the **Environments** block by clicking the Deployment name link in the Deployments list. 2. Click the name of the Environment that needs to be deleted. 3. In the upper-right corner of the page, click the **Actions** button and select **Edit**. 4. Enter the Environment name and click **Confirm**. ### View Environment Data To view the environment data for the existing environment, follow the steps below: 1. Navigate to the **Environments** block by clicking the Deployment name link in the Deployments list. 2. Click the environment name. The following blocks will be displayed: a. **Overview** - general information and configuration of current environment. b. **Applications** - displays the status of the applications related to the environment and allows for [deploying applications](#deploy-application). Applications health and sync statuses are returned from the Argo CD tool. c. **Pipelines** - displays all the deploy pipeline runs launched for this environment. d. **Variables** - adds personalized variables that will be used in the deploy pipelines. e. **Monitoring** - opens the Grafana window that allows for watching various metrics. ### Deploy Application To deploy an application, follow the steps below: 1. Navigate to the **Applications** block and click the **Configure deploy** button. 2. Set deployment properties you need: a. Select the image stream version from the drop-down list. b. (Optional) Enable setting custom values for Helm Charts. For more details, please refer to the [Manage GitOps](gitops.md) page. c. Click the **Start Deploy** button to start a pipeline with the deploy script. :::info In case of using OpenShift internal registry, if the deployment fails with the ImagePullBackOff error, delete the pod that was created for this application. ::: To uninstall the application, click the **Delete** button. As a result, the application will be uninstalled in the Argo CD tool as well. Alternatively, you can use the **Clean** button. This way will be appropriate when you have some specific requirements to the environment cleanup procedure. Note that you need to make up your own logic in the cleanup pipeline to use the button or choose one of the pre-defined pipelines offered by KubeRocketCI. ### Troubleshoot Application There is a couple of KubeRocketCI portal capabilities that will help in monitoring and troubleshooting deployed applications, namely, terminal and logs. To inspect the deployed application in KubeRocketCI portal, take the following steps: 1. Open the application logs by clicking the `Show Logs` button. 2. Inspect the shown logs. 3. Open the application terminal by clicking the `Show Terminal` button. 4. Operate the terminal to fix the problem if any. ### Monitor Application KubeRocketCI portal offers monitoring capabilities powered by Grafana and Prometheus Operator. :::info To view the deployment monitoring data, you need to have Grafana and Prometheus Operator installed. We recommend installing it using our [add-ons](https://github.com/epam/edp-cluster-add-ons/blob/3fc8a60ccfbca1c7aa757cb36226a0daf2c8a224/clusters/core/addons/prometheus-operator/values.yaml). ::: To monitor an application using Grafana, follow the steps below: 1. Navigate to the Environment details page. 2. In the Environment details page, open the **Monitoring** tab. 3. In the **Monitoring** tab, view the Deployment metrics. The **Monitoring** tab provides the visual representation of the basic application deployment metrics, such as CPU and Memory requests and limits. ### Delete Application There are two buttons on the environment page that can be utilized to delete application in the portal: * **Delete** - Deletes selected applications and associated resources. * **Clean** - Manages custom cleanup actions, such as deleting cloud resources and databases, rolling back transactions, etc. The **Delete** button is optimized for applications based on a single, simple helm chart that can be deployed independently without any specific dependencies. The **Clean** button activates a deletion process (triggers a delete pipeline) that includes any custom logic you have defined. It runs the Clean Pipeline template you chose when [adding a Deployment](./add-cd-pipeline.md). This option is most suitable for applications that require complex configurations to function properly. It ensures that any associated resources with the same lifecycle are deleted when the application is no longer needed. KubeRocketCI provides an intuitive and streamlined pipeline by default. Initially, there is no distinction between the **Delete** and **Clean** buttons, as both perform identical actions. This default behavior encompasses the straightforward deletion of applications, which involves the uninstallation of the associated Helm chart. However, this functionality evolves once a custom delete pipeline is established, enabling manual configuration of the logic behind the **Clean** button. This customization allows for a more tailored approach to managing application lifecycles within KubeRocketCI. For more details on how to create a custom delete pipeline, navigate to the [Customize Environment Cleanup](../operator-guide/cd/customize-environment-deletion.md) ## Related Articles * [Customize Environment Cleanup](../operator-guide/cd/customize-environment-deletion.md) * [Add Deployment Flow](add-cd-pipeline.md) * [Add Quality Gate](../user-guide/add-quality-gate.md) --- ## Manual Approval in Pipelines # Manual Approval in Pipelines The manual approval feature gives users a smooth and controlled process of promoting applications from lower, non-critical environments, such as development or QA, to mission-critical environments like production. Additionally, it ensures that only thoroughly tested and verified changes are deployed to production, minimizing the risk of causing errors or instability. Manual approval, at its simplest, is an integrable Tekton [Task](https://github.com/epam/edp-tekton/blob/v0.13.0/charts/pipelines-library/templates/pipelines/cd/deploy-with-approve.yaml#L61) that can be integrated into common Tekton pipelines. Its purpose is to pause pipeline execution until a user approves or rejects the task. ## Prerequisites Please note that using the approval task requires you to install the [tekton-custom-task](https://github.com/KubeRocketCI/tekton-custom-task/tree/main) Helm chart. ## Enable Pipeline With Manual Approval KubeRocketCI offers a pre-defined deploy pipeline called **deploy-with-approve**. It contains an approval task and can be used for most application deployment cases. If you leverage the default deploy pipeline, you can easily switch it to **deploy-with-approve**. You can apply this pipeline to both existing and new environments. ### New Environments To apply the **deploy-with-approve** pipeline template to your environments, follow the steps below: 1. In the deployment details page, click the **Create environment** button. 2. When creating an Environment, select the **deploy-with-approve** pipeline in the [Pipeline Configuration](./add-cd-pipeline.md#pipeline-configuration) stage. ### Existing Environments If you need to change the default deploy pipeline template in an already existing environment, follow the steps below: 1. Navigate to the environment details page and click the **Edit** button. 2. In the **Edit environment** window, select another deploy pipeline. ## Approve/Reject Deployment Once the deploy pipeline has been launched and reached the approval step, you will see a corresponding notification on the pipeline details page: If the approval is rejected, the pipeline status will be failed. If you don’t make a selection within the pipeline processing time, which is **60 minutes** by default, you’ll see a crossed clock icon as the task run status, indicating that the pipeline has timed out. :::note You can set a custom timeout duration in the relevant [TriggerTemplate](https://github.com/epam/edp-tekton/blob/v0.13.0/charts/pipelines-library/templates/triggers/cd/deploy-with-approve.yaml#L46). ::: If you choose the **Approve** option, the pipeline will proceed running. ## Create Pipeline With Approval Task The custom deploy pipeline usage flow involves the following steps: ```mermaid graph LR; A(Create TriggerTemplate resource) --> B(Create Pipeline resource) --> C(Apply changes to environment) --> D(Run deploy pipeline) --> E(Approve/reject deployment) ``` To create a deploy pipeline with a manual approval task, follow the steps below: 1. Create and apply the **TriggerTemplate** custom resource. Refer to the example [TriggerTemplate](https://github.com/epam/edp-tekton/blob/v0.13.0/charts/pipelines-library/templates/triggers/cd/deploy-with-approve.yaml). Make sure to specify the correct pipeline type in the labels and provide the pipeline reference in the specifications: ```bash ... labels: app.edp.epam.com/pipelinetype: deploy ... ``` ```bash ... spec: taskRunTemplate: serviceAccountName: tekton pipelineRef: name: deploy-with-promote-approval ... ``` 2. Create and apply the **Pipeline** custom resource. Refer to the example [Pipeline](https://github.com/epam/edp-tekton/blob/v0.13.0/charts/pipelines-library/templates/pipelines/cd/deploy-with-approve.yaml). Ensure you set the concrete approval task to run: ```bash - name: approve params: - name: description value: Promote versions to the next Environment? runAfter: - deploy-app taskRef: apiVersion: edp.epam.com/v1alpha1s kind: ApprovalTask name: approve ``` 3. When [creating](#new-environments) or [modifying](#existing-environments) the deploy pipeline, select your newly added custom deploy pipeline. ## Related Articles * [Customize Environment Cleanup](../operator-guide/cd/customize-environment-deletion.md) * [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) * [Add Deployment Flow](../user-guide/add-cd-pipeline.md) * [Manage Deployments](../user-guide/manage-environments.md) --- ## Marketplace Overview # Marketplace Overview The KubeRocketCI marketplace offers a range of Templates, predefined tools and settings for creating software. These Templates speed up development, minimize errors, and ensure consistency. A key KubeRocketCI marketplace feature is customization. Organizations can create and share their own Templates, finely tuned to their needs. Each Template serves as a tailored blueprint of tools and settings. These tailored Templates include preset CI/CD pipelines, automating your development workflows. From initial integration to final deployment, these processes are efficiently managed. Whether for new applications or existing ones, these Templates enhance processes, save time, and ensure consistency. To see the Marketplace section, navigate to the **Marketplace** tab. General look of the Marketplace section is described below: ![Marketplace section](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-overview-listed.png "Marketplace section (listed view)") * **Marketplace templates** - all the components marketplace can offer; * **Template properties** - the item summary that shows the type, category, language, framework, build tool and maturity; * **Filters Enable/disable** - allows users to toggle search by item name or available namespace; * **View options** - switches between the listed view and the tiled view. See the view options for details. There is also a possibility to switch into the tiled view instead of the listed one: ![Marketplace section](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-overview-tiled.png "Marketplace section (tiled view)") To view the details of a marketplace item, simply click its name: ![Item details](../assets/user-guide/marketplace/kuberocketci-portal-marketplace-item-details.png "Item details") The details window shows supplemental information, such as item's author, keywords, release version and the link to the repository it is located in. The window also contains the **Create from template** button that allows users to create the component by the chosen template. The procedure of creating new components is described in the [Create Component via Marketplace](add-marketplace.md) page. ## Related Articles * [Add Component via Marketplace](add-marketplace.md) * [Add Application](add-application.md) * [Add Library](add-library.md) * [Add Infrastructure](add-infrastructure.md) --- ## Observability # Observability The Observability section provides monitoring, logging, and visibility into pipeline runs, application statuses, and platform metrics. This page describes the main views and dashboards available under Observability. ## Pipeline Metrics The **Pipeline Metrics** dashboard shows aggregated pipeline statistics for a chosen namespace. Use it to track run counts, success rates, failures, and duration over time. ### Filters * **Select codebase** — Limit metrics to a specific codebase. * **Time range** — Choose **Today**, **7 Days**, **30 Days**, or **90 Days** to scope the data. ### Key Metrics * **Total Runs** — Number of pipeline runs in the selected period. * **Success Rate** — Percentage of runs that succeeded (with count of succeeded runs). * **Failed** — Number of failed runs (e.g. “All green” when there are no failures). * **Avg Duration** — Average pipeline duration and the min–max range (e.g. 1m 40s – 1h 0m). ### Pipeline Type Breakdown Metrics are split by pipeline type: * **Build Pipelines** — Source code builds: total runs, success rate, succeeded and failed counts. * **Review Pipelines** — PR/MR reviews: same metrics. * **Deploy Pipelines** — Environment deployments: same metrics. Use this section to see which type (build, review, or deploy) contributes to overall success or failure. ### Pipeline Activity A timeline chart shows when pipelines ran and whether they succeeded or failed. The legend indicates **Succeeded** (e.g. teal) and **Failed** (e.g. red). Use it to spot busy hours and failure patterns over the selected time range. ## Related Articles * [KubeRocketCI Widgets](./widgets.md) * [CI/CD Pipelines](./pipelines.md) --- ## Overview(User-guide) # Overview The **Overview** page under **Platform** is the main dashboard for the selected cluster and namespace. It summarizes pipeline activity, resource health, recent work, usage, security and quality signals, integrations, and links to external tools. For a deeper description of individual widgets, see [KubeRocketCI Widgets](./widgets.md). ## Summary Cards (Header) The top row gives a quick read on **today’s** pipeline health. Four cards show pipeline metrics for the current context: * **Total Pipeline Runs** — Number of runs today. * **Success Rate** — Share of successful runs (or “no data” when there is nothing to show). * **Failed Pipelines** — Count of failed runs. * **Average Duration** — Average run duration (or “no data” when not applicable). ## Pipeline Activity The **Pipeline Activity (7 days)** widget helps you see trends and spikes over the last week. It is a bar chart: each day is a bar split by outcome — **Succeeded** (e.g. green) and **Failed** (e.g. red) — so you can spot busy days and failure spikes at a glance. ## Resource Health **Resource Health** is a strip of circular indicators so you can scan counts and status without opening other sections. Each circle summarizes one entity type: * **Codebases** — Number of codebases. * **Branches** — Total branches across codebases. * **Pipelines** — Pipeline-related count for the context. * **CD Pipelines** — CD pipelines. * **Stages** — Deployment stages. Use this strip to see whether core objects look healthy before drilling into **Projects**, **CI/CD Pipelines**, or **Deployments**. ## Activity and Git The central column includes widgets for **what ran recently**, **your own runs**, and **Git-side activity**. They typically work as follows: * **Recent Pipeline Activity** — Latest pipeline runs; often includes **View All** to open the full list. Empty when there are no runs. * **My Activity** — Runs associated with your account; **View All** when available. * **Git Activity** — Pull request activity when connected; may show “no pull request data” if none is available. ## Resource Usage On the right side, **Resource Usage** summarizes how much cluster capacity the platform workload is using and which pods stand out. You will usually see: * Totals or gauges for **CPU**, **Memory**, and **Pods**. * A **top pods by CPU** (or similar) list with names and usage so you can spot heavy consumers. ## Security and Quality Two widgets condense security and static-analysis posture for the current scope. They show: * **Vulnerability Summary** — Totals by severity (Critical, High, Medium, Low) from integrated vulnerability data (e.g. Dependency-Track). * **SonarQube Quality** — Aggregated quality gate results for SonarQube projects linked to the platform. ## Integrations **Integrations** lists external systems the portal talks to (Jira, registries, observability, and others). It is useful for spotting misconfiguration or downtime before you rely on a tool in a workflow. The widget may show counts such as how many connections need attention, short error text, and actions like **Fix** next to a failing integration. ## DORA Metrics **DORA** widgets summarize delivery behavior over a fixed window (often the **last 30 days**). They support conversations about speed and stability. Typical fields are: * **Deployment Frequency** — Deploy count over the period, successes, and a simple cadence (e.g. average days between deploys). * **Change Failure Rate** — Share of failed deploys vs total, sometimes with a status such as “Needs attention” when above a threshold. ## Links The **Links** area at the bottom is a grid of shortcuts to integrated tools (Argo CD, Git providers, container registry, DefectDojo, Dependency-Track, SonarQube, Prometheus, Grafana, Kibana, Jaeger, Jira, Keycloak, Nexus, and others depending on your setup). Use **+ Add Link** to add custom entries; see [Manage Quick Links](./quick-links.md). ## Related Articles These pages go deeper into areas that overlap with the Overview dashboard: * [KubeRocketCI Widgets](./widgets.md) * [Observability](./observability.md) * [Security](./security/sca-overview.md) --- ## Pipelines Overview # Pipelines Overview Pipelines are an integral part of any CI/CD. They are involved in code build, review, and deployment procedures. Such a big variety of pipelines results in a demand for having a single place where the pipelines can be managed. For this purpose, the Pipelines section was added to the KubeRocketCI portal. ## Pipelines Page Overview To see the CI/CD Pipelines section, open the KubeRocketCI portal and select the **CI/CD Pipelines** section: * **PipelineRuns** - Shows the recent PipelineRuns. You can search PipelineRuns and filter them by status, type, and Codebase they launched for; * **Pipelines** - This tab allows to view and edit all the Tekton Pipelines created in the platform; * **Tasks** - This tab allows to view and edit all the Tekton Tasks created in the platform. ![PipelineRuns list](../assets/user-guide/pipelines/pipelineruns-list.png "PipelineRuns list") ## Pipeline Overview To inspect pipeline details, click the PipelineRun name: * The **Details** tab displays the PipelineRun status and logs; * The **View YAML** tab displays the YAML configuration of your pipeline. It also allows to view the general information and resources that the pipeline is connected with; * The **Diagram** tab displays the pipeline's real-time status. Click the task name to navigate to the corresponding task in the **Details** tab; * The **Results** tab shows the resulting artifact that was built in the pipeline. Note that this tab is accessible only in Build Pipelines that produce artifacts. ![KubeRocketCI PipelineRun details page showing the Triggered By field](../assets/user-guide/pipelines/pipelinerun-details.png "PipelineRun details showing the Triggered By field") The PipelineRun header also shows the **Triggered By** field — the identity that started the run. For runs launched by automation (for example, a Git webhook or a scheduled build) this is the service account that created the run; for runs started from the portal it is the corresponding user. The field is resolved from the [platform audit trail](../operator-guide/monitoring-and-observability/audit-trails-setup.md) and is distinct from the **Author** field, which shows the Git commit author. When the initiator cannot be resolved — for example, historical runs created before audit capture was enabled — the field shows **N/A**. You can also view the build pipeline results from the codebase details page and the Pipelines section. ## Operate With Pipelines The Pipelines section allows you to track, restart, and delete pipelines. To restart the PipelineRun, click the actions button and select **Run again**. To restart the PipelineRun using different parameters, click the actions button and select **Run with params**. To delete the PipelineRun, click the actions button and select **Delete**. Alternatively, you can enter the PipelineRun and delete/restart the pipeline using the corresponding buttons. :::note There is also an option to store the long-term logs using the OpenSearch tool. You can install OpenSearch using our [cluster add-ons](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/apps/values.yaml#L227) repository. ::: ## Edit Pipelines/Tasks KubeRocketCI portal allows to edit the existing Tekton Pipelines and Tasks directly in the Pipelines section. In our example, we will show how to edit a Task but the same procedure applies to Pipelines: 1. Navigate to the **Tasks** tab. 2. Enter a Task by clicking its name. 3. On the Task details page, in the top right corner of the screen, click **Edit**. 4. On the edit window, make your changes and click **Saved**. Now you know how to view and manage pipelines in KubeRocketCI. ## Related Articles * [Marketplace Overview](./marketplace.md) * [Manage Deployment Flows](./manage-environments.md) * [Manage Applications](./application.md) --- ## Platform Cleanup Guide # Platform Cleanup Guide Throughout your project's lifecycle, various resources are created within the KubeRocketCI platform. Over time, some of these resources may become deprecated. It is crucial to clean up any associated artifacts to ensure the platform functions correctly, especially if you intend to re-add a component with the same name. This page provides comprehensive guidelines for completely deleting resources in KubeRocketCI, from applications to third-party resources. ## Prerequisites To follow the guidelines below, ensure you have: * Admin access to the KubeRocketCI portal * Admin access to third-party tools (Nexus, SonarQube, container registry, Git, etc.) ## Platform Resource Cleanup This section covers platform resource removal, such as components, deployments, and pipeline runs. ### Delete Deployments Platform cleanup starts with deleting [deployments](add-cd-pipeline.md), a core resource that allows the platform to build an automated workflow. Deleting a deployment will delete applications from its associated cluster. We recommend cleaning environments before deleting deployments to ensure graceful resource deletion and avoid potential issues. There are two strategies for cleaning environments: * **Using the delete option**: This strategy involves using the **Delete** button to remove an environment. It’s suitable for environments that don’t rely on specific workload setups to deploy. * **Using a clean pipeline**: This strategy involves creating a [clean pipeline](../operator-guide/cd/customize-environment-deletion.md) to handle the deletion of third-party dependencies, such as cloud resources and databases, that share the same lifecycle as your environment. We recommend this strategy for applications with complex third-party interactions. To delete an environment using the **Delete** button, read the [Manage Environments](manage-environments.md#delete-environment) page. :::note A deployment must include at least one environment. ::: Once the environments are deleted, proceed with deleting a deployment. To delete a [deployment](./manage-environments.md), follow these steps: 1. Navigate to **KubeRocketCI portal** -> **Deployments**. 2. Choose a deployment to delete. Click the actions button and select **Delete**. 3. Enter the deployment name and click **Delete**. ### Delete Codebases After deleting deployments, you can delete the associated resources they operate with. These resources, collectively referred to as codebases, include [applications](application.md), [libraries](library.md), [autotests](autotest.md), and [infrastructures](infrastructure.md). In this example, we will demonstrate how to delete an application. The same procedure applies to other types of codebases: 1. Navigate to **KubeRocketCI portal** -> **Projects**. 2. Choose an application to delete. Click the actions button and select **Delete**. 3. Enter the application name and click **Delete**. ### Delete Pipeline Runs The last platform resource remaining is [pipeline runs](./pipelines.md). A pipeline run refers to an execution instance of a pipeline. Each time you trigger a build, review, deploy, or clean pipeline, a new pipeline run emerges. To delete pipeline runs, follow these steps: 1. Navigate to **KubeRocketCI portal** -> **CI/CD Pipelines**. 2. Choose pipeline runs and click **Delete**. 3. On the confirmation window, enter **confirm** and click **Delete**. ## Third-Party Resource Cleanup KubeRocketCI supports integrations with a wide range of [third-party tools](../operator-guide/devsecops/overview.md#tool-descriptions), many of which are involved in the codebase management process. Therefore, it is crucial to clean up resources associated with these integrations. ### Version Control Systems If you want to re-add a component with the same name, cleaning up artifacts is mandatory for correct platform operation. When using a create or clone strategy, [codebase operator](https://github.com/epam/edp-codebase-operator) provisions a repository in the Version Control System (VCS) each time you create a new codebase. :::note When re-adding a repository instead of deleting it, clear repository tags to prevent versioning conflicts. For re-adding a Helm chart, clean the **deploy-templates** folder. ::: Below are links to the official documentation for deleting code repositories: To delete a GitHub repository, read the [Deleting a repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/deleting-a-repository) page. To delete a GitLab repository, read the [Manage projects](https://docs.gitlab.com/ee/user/project/working_with_projects.html#delete-a-project) page. To delete a Bitbucket repository, read the [Delete a repository](https://support.atlassian.com/bitbucket-cloud/docs/delete-a-repository/) page. To delete a Gerrit repository, refer to the [Project Deletion](https://gerrit-review.googlesource.com/Documentation/intro-project-owner.html#project-deletion) section. ### Delete Nexus Artifacts KubeRocketCI uses Nexus Repository Manager as an artifact registry for storing application binaries. If the Nexus instance is installed using [add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main), a predefined [cleanup policy](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/nexus-operator/templates/scripts/create-tasks.yaml) is deployed along with it. This policy automatically deletes unused images after [30 days](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/nexus-operator/templates/scripts/create-tasks.yaml#L132). Ensure that the policy is activated to enable successful automatic cleanup. #### Repository Types in Nexus A Nexus repository is a collection of components such as JAR files, Docker images, and other artifacts. KubeRocketCI uses the following repository types: * **Container Repository**: Stores container images and Helm charts. * **Maven Repository**: Handles Maven artifacts like JARs and POMs. * **Dotnet Repository**: Manages NuGet packages. * **NPM Repository**: Stores NPM packages. * **Python Repository**: Hosts Pip packages. * **Yum Repository**: Stores RPM packages. #### Repository Structure Repositories are categorized into the following types: * **Hosted**: Stores locally uploaded artifacts (e.g., krci-npm-hosted). * **Group**: Aggregates multiple repositories for unified access (e.g., krci-maven-group). * **Proxy**: Mirrors artifacts from remote repositories (e.g., krci-maven-proxy). #### Manual Deletion Procedure To clean up unnecessary application binaries manually, follow these steps: 1. Navigate to Nexus main menu and select the **Browse** tab: ![Nexus main menu](../assets/user-guide/nexus-main-menu.png "Nexus main menu") 2. On the **Browse** tab, select the folder with the name that represents the application type: For example, to delete a .NET application, navigate to **krci-dotnet-releases** or **krci-dotnet-snapshots** folder. Understanding the difference between the release and snapshot folders is essential: * **Releases**: This repository type stores applications built in release branches. These applications are usually tagged as release candidates (RC). * **Snapshots**: This type of repository stores other applications and their versions, usually tagged as snapshots. ![Browse Nexus repository snapshots](../assets/user-guide/browse-nexus-repository-snapshots.png "Browse Nexus repository snapshots") 3. In the Nexus repository, select the package folder and click **Delete folder**: ![Delete Nexus folder](../assets/user-guide/delete-nexus-folder.png "Delete Nexus folder") 4. (Optional) Repeat steps 1-3 for all the packages that need to be cleaned up. ### Delete Container Images KubeRocketCI uses container images to deploy the applications to [deployments](../user-guide/add-cd-pipeline.md). To clean up a container registry, follow the corresponding guidelines: To delete container images from the AWS Elastic Container Registry, read the [Deleting an image in Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/delete_image.html#:~:text=To%20delete%20an%20image%20(AWS%20Management%20Console)&text=In%20the%20navigation%20pane%2C%20choose,to%20delete%20and%20choose%20Delete.) page. To delete container images from DockerHub, read the [Delete a repository](https://docs.docker.com/docker-hub/repos/delete/) page. To delete container images from Harbor, read the [Deleting Artifacts](https://goharbor.io/docs/2.3.0/working-with-projects/working-with-images/deleting-artifact/) page. 1. Navigate to Nexus main menu and select the **Browse** tab: ![Nexus main menu](../assets/user-guide/nexus-main-menu.png "Nexus main menu") 2. On the **Browse** tab, select the repository with the **krci-container-hosted** name: ![krci-container-hosted repository](../assets/user-guide/nexus-edp-container-hosted.png "krci-container-hosted repository") 3. On the repository details page, select the folder name and click **Delete folder**: ![krci-container-hosted repository](../assets/user-guide/nexus-delete-container-image.png "krci-container-hosted repository") To delete container images from GitHub Container Registry, read the [Working with the Container registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) page. ### Delete SonarQube Projects KubeRocketCI generates a SonarQube project for every new codebase. To clean up application projects, follow the guidelines described in the [official documentation](https://docs.sonarsource.com/sonarqube-server/8.9/project-administration/project-settings/#deleting-a-project). Deleting a codebase without removing the corresponding SonarQube project may lead to issues in review pipelines. ### Observability Tools KubeRocketCI supports various observability tools, including DefectDojo, Dependency-Track, Datadog, and more. While these tools don't directly impact the platform's core functionality, they can generate significant data during operation. Some tools, such as Dependency-Track and SonarQube, offer deep integration with KubeRocketCI through dedicated widgets. These widgets allow users to view code quality metrics and dependency scan results directly in the codebase details menu. #### DefectDojo There are two methods to delete observability data from DefectDojo: * **By deleting a product**: This method lets you delete observability data for the entire KubeRocketCI platform simultaneously. * **By deleting engagements**: This method lets you selectively delete data for specific codebases. Deleting a DefectDojo product is preferable if you no longer need your product's security scans. Typically, the product name matches the Kubernetes namespace. A product is deleted on the product's details page: ![Delete DefectDojo product](../assets/user-guide/delete-defectdojo-product.png "Delete DefectDojo product") Deleting engagements will clean up security scan runs for specific codebases. This method is preferable when retaining certain parts of the product's data. To find the security scans for specific codebases, use search filters: ![DefectDojo filters](../assets/user-guide/defectdojo-filters.png "DefectDojo filters") To delete an engagement, click the three dots icon and select **Delete**: ![Delete DefectDojo engagement](../assets/user-guide/delete-defectdojo-engagement.png "Delete DefectDojo engagement") #### Dependency-Track At the moment, Dependency-Track has fewer options for selectively deleting data. Therefore, it is often preferable to delete an entire Dependency-Track project. To delete a Dependency-Track project, follow these steps: 1. Log into your Dependency-Track instance. 2. Open the Projects list and find the project you want to delete. 3. Enter the project details by clicking its name: ![Search project](../assets/user-guide/deptrack-search-project.png "Search project") 4. In the project's details page, click the **View Details** button: ![View project details](../assets/user-guide/deptrack-delete-project.png "View project details") 5. On the project details menu, click **Delete**: ![Delete project](../assets/user-guide/deptrack-view-details.png "Delete project") ## Related Articles * [Manage Applications](./application.md) * [Add Application](./add-application.md) * [Manage Deployments](./manage-environments.md) --- ## Portal Settings # Portal Settings This page describes all the general settings related to KubeRocketCI portal. It explains how to adjust resource view, namespace visibility, and use the kubeconfig file provided by the portal. ## Overview KubeRocketCI portal has a number of settings that can help you define resource visibility and provide you with system information. The cluster name is displayed in the top left corner of the screen. By clicking on the cluster name, you can set default and allowed namespaces or get the kubeconfig file. You can also find helpful information in the **Quick Actions** section of the Main Menu, at the top right corner of the screen. ## Manage Namespaces In the **Manage Namespaces** window, you can view the namespace accessibility configuration you are supposed to specify at your first login. Here you can also edit the default and allowed namespaces: * **Default namespace** – when you create a resource within the portal, it will be deployed in this namespace. You can set only one default namespace. By default, the namespace where the KubeRocketCI instance is deployed will be set here. * **Allowed namespace** – resources from these namespaces will be visible in the portal. :::note You can't view resources in the namespaces that you are not allowed to access according to your RBAC configuration. KubeRocketCI portal doesn't grant you extra access rights. ::: ## Get Kubeconfig You can see the kubeconfig file used by the portal by clicking the **Get Kubeconfig** button: Here you can view the current kubeconfig file and its expiry time. Additionally, you can also copy this kubeconfig file to use it locally. To access KubeRocketCI resources from your local machine, follow the steps below: 1. Copy the kubeconfig file from the portal. 2. Open your terminal and navigate to the **./ssh** directory. 3. Create a file and paste the kubeconfig data there. 4. Switch to this kubeconfig file: ```bash export KUBECONFIG=/Users//.ssh/ ``` 5. Verify you can access KubeRocketCI resources. ## User Details On the **User Details** window, you can view your personal information: * **Name**: your Keycloak username; * **Email**: your Keycloak user email; * **Subject**: your Keycloak user ID; * **Issuer URL**: the URL of the Keycloak realm that handles authorization; * **Groups**: your Keycloak group membership. ## User Profile Click the User profile button in the bottom left corner of the screen to access helpful resources and support options. When clicked, the following options are displayed: * **Documentation**: opens the product documentation site; * **Join discussions**: opens the GitHub Discussions page for the product, where you can participate in community conversations; * **Open an issue/request**: Opens the GitHub page where you can create a new issue or feature request for the product; * **Log out**: terminates the session and redirects you to the login page. :::note The profile picture is also pulled from Keycloak. ::: ## Related Articles * [Authentication and Authorization: Overview](../operator-guide/auth/platform-auth-model#keycloak-roles-and-groups) * [Install Keycloak](../operator-guide/auth/keycloak#configuration) --- ## Protect Resources From Deletion/Modification # Protect Resources From Deletion/Modification KubeRocketCI allows to prevent accidental resource deletion and modification by applying a protected label. This page describes the protected label feature, its purpose, types, and usage. ## Overview Protected label is a Kubernetes label that tells KubeRocketCI to block any delete or edit operations with the resource. The platform checks the label first before performing an operation with the resource. Protected labels support the following resources: * [Codebases](../api/codebase.md#codebase); * [Codebase branches](../api/codebase.md#codebasebranch); * [Deployments](../api/cd-pipeline.md#cdpipeline); * [Environments](../api/cd-pipeline.md#stage). ## Protection Types Protection label can optionally block specific operations over a resource: * **app.edp.epam.com/edit-protection: delete-update** - Blocks both the delete and modify operations; * **app.edp.epam.com/edit-protection: delete** - Blocks only delete operations; * **app.edp.epam.com/edit-protection: update** - Blocks only modify operations. ## Apply Protected Label To apply the resource deletion and modification block, follow the steps below. We will use a codebase resource as an example, but the procedure applies to all the supported resources: 1. Open the terminal which has access to the cluster that runs KubeRocketCI. 2. Get the codebase list: ```bash kubectl get codebase -n krci ``` 3. Define the codebase you want to protect from accidental deletion and/or modification. 4. Enter the resource edit menu: ```bash kubectl edit codebase -n krci ``` 5. In the labels section of the codebase specifications, add the label from the [Protection Types](#protection-types) list. 6. Save and quit the menu. ## Remove Protected Label To remove a label, navigate back to the resource, edit the resource by removing the label, and save the changes. ## Related Articles * [Manage Applications](../user-guide/application.md) * [Customize Deploy Pipeline](../operator-guide/cd/customize-deploy-pipeline.md) * [Add Deployment Flow](../user-guide/add-cd-pipeline.md) * [Manage Deployments](../user-guide/manage-environments.md) --- ## Manage Quick Links # Manage Quick Links The Links section is designed to store all external references in one place. These links are displayed directly on the [Overview](index.md) page of the KubeRocketCI portal. Additionally, some links are also shown on the Component and Environment sections. ## Add Quick Link There are two methods to add Quick Links: - [Add Quick Link via KubeRocketCI portal](#add-quick-link-via-kuberocketci-portal) - [Add Quick Link via Helm chart](#add-quick-link-via-helm-chart) ### Add Quick Link via KubeRocketCI Portal To add a Quick Link via the KubeRocketCI portal, follow the steps below: 1. Navigate to **Configuration** -> **Quick Access** -> **Links** and click the **+ Add Link** button. 2. In the appeared window, insert the link name, URL, and SVG icon in base 64 format. 3. If the **Show on Overview Page** option is selected, the link will be displayed on the **Overview** page. ### Add Quick Link via Helm Chart The Quick Links can be specified in the [edp-install](https://github.com/epam/edp-install) Helm chart during platform installation or upgrade process. For some components, the Quick Link section is already present in the `values.yaml` file. In this case, it is enough to specify the necessary URL for the appropriate component. An example of the `quickLinks` section in the `values.yaml` file, with all supported components, provided below: ```yaml title="values.yaml" quickLinks: # Base URL value for the argocd Quick Link argocd: "" # Base URL value for the codemie Quick Link codemie: "" # Base URL value for the defectdojo Quick Link defectdojo: "" # Base URL value for the dependency track Quick Link dependency_track: "" # Base URL value for the container registry Quick Link (e.g. Nexus, DockerHub, ECR, etc.) docker_registry: "" # Configuration for the logging Quick Link logging: # -- Define the provider name for correct URL generation. # Available providers: "opensearch", "datadog". # If the provider name is not specified, the base URL will be used. provider: "" # Base URL value for the logging Quick Link url: "" # Configuration for the monitoring Quick Link monitoring: # -- Define the provider name for correct URL generation. # Available providers: "grafana", "datadog". # If the provider name is not specified, the base URL will be used. provider: "" # Base URL value for the monitoring Quick Link url: "" # Base URL value for the nexus Quick Link nexus: "" # Base URL value for the sonar Quick Link sonar: "" ``` If the required component is not listed in the `quickLinks` section, it is possible to add the Quick Link using the `extraQuickLinks` section. An example of specifying the Quick Link through the `extraQuickLinks` section in the `values.yaml` file is provided below: ```yaml title="values.yaml" extraQuickLinks: # Name of the Quick Link - documentation: # Base URL value for the documentation Quick Link url: "https://docs.kuberocketci.io/" # Visibility option for the Quick Link on the Overview page. visible: true # SVG icon in base 64 format. icon: PHN2ZyB4b... # Name of the Quick Link - jaeger: # Base URL value for the jaeger Quick Link url: "https://jaeger.example.com/" # Visibility option for the Quick Link on the Overview page. visible: true # SVG icon in base 64 format. icon: PHN2ZyBoZ... ``` After specifying the necessary Quick Links in the `values.yaml` file, the Quick Links will be displayed on the **Overview** page after the platform installation. ## Edit Quick Link To edit a Quick Link, follow the steps below: 1. Navigate to **Configuration** -> **Quick Access** -> **Links**. Click the Actions menu and select **Edit**. 2. Edit the necessary fields and click **Apply**. ## Delete Quick Link :::note Quick Links of type **system** cannot be deleted as they are crucial for the platform's functionality. ::: To delete a Quick Link, follow the steps below: 1. Navigate to **Configuration** -> **Quick Access** -> **Links**. Click the Actions menu and select **Delete**. 2. In the **Confirm deletion** window, enter the name of the link and click **Confirm**. ## Related Articles * [Add Git Server](add-git-server.md) * [Add Cluster](add-cluster.md) * [Manage GitOps](gitops.md) * [Manage Registries](manage-container-registries.md) --- ## Cluster Compliance # Cluster Compliance **Cluster Compliance Reports** under **Security > Cluster Security > Compliance** show Kubernetes security compliance benchmarks (CIS, NSA, PSS) for the selected cluster. You can open each report to see the pass rate, severity breakdown, and individual controls. ## Cluster Compliance Reports The page title is **Cluster Compliance Reports**, with a short description: *Kubernetes security compliance benchmarks (CIS, NSA, PSS)*. ### Table Use the **Columns** control (e.g. "Columns 7") to choose which columns are visible. The table includes: | Column | Description | |--------------|-------------| | **Title** | Full name of the compliance standard (e.g. National Security Agency - Kubernetes Hardening Guidance v1.0, Kubernetes Pod Security Standards - Restricted/Baseline, AWS EKS CIS Foundations v1.4). | | **Type** | Short type label with a colored badge: **NSA**, **PSS-RESTRICTED**, **PSS-BASELINE**, **CIS**. | | **Version** | Version of the standard (e.g. 1.0, 0.1). | | **Pass** | Number of passed checks (e.g. 27, 17, 11, 48). | | **Fail** | Number of failed checks (dash when zero). | | **Pass Rate**| Percentage of passed checks (e.g. 100%, 94%). | | **Actions** | **View Details** button to open the full report. | **Pagination** at the bottom (e.g. "Rows per page: 25", "1–4 of 4") and navigation arrows let you move through the list. Click **View Details** on a row to open the report for that benchmark. ## Report Details When you open a report, the breadcrumbs show **Security > Cluster Security > Compliance > <benchmark name>** (e.g. *National Security Agency - Kubernetes Hardening Guidance v1.0*). ### Header Header contains the following information: * **Report title** — Full benchmark name and optional subtitle; often a link to the source (e.g. nsa.gov). * **Overall status** — **Pass Rate** (e.g. 100%) in a bar, and summary **Passed** / **Failed** counts (e.g. "27 Passed", "0 Failed") with icons. ### Severity Breakdown Four cards summarize controls by severity: * **Critical** — Count and status (e.g. "8", "All passed"). * **High** — Count and status (e.g. "5", "All passed"). * **Medium** — Count and status (e.g. "11", "All passed"). * **Low** — Count and status (e.g. "3", "All passed"). ### Compliance Controls Table Above the table you can filter by **Severity** (e.g. "All severities") and **Status** (e.g. "All"). The **Columns** button customizes visible columns. The table lists each control: | Column | Description | |--------------|-------------| | **Control ID** | Identifier (e.g. 3.0, 5.0, 5.1, 6.0, 1.2, 8.0). | | **Name** | Short name of the control (e.g. "Use CNI plugin that supports NetworkPolicy API (Manual)", "Encrypt etcd communication", "Make sure -authorization-mode=RBAC"). | | **Severity** | Critical, High, Medium, or Low. | | **Status** | Pass or Fail. | | **Failures** | Details of failures (dash when passed). | Use this view to see which controls passed or failed and to plan remediation. **Pagination** at the bottom (e.g. "Rows per page: 25") applies when there are many controls. ## Related Articles * [Cluster Configuration Audits](./cluster-configuration-audits.md) * [Cluster RBAC Assessments](./cluster-rbac-assessments.md) * [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) * [Cluster Vulnerability Reports](./cluster-vulnerability-reports.md) --- ## Cluster Configuration Audits # Cluster Configuration Audits **Cluster Configuration Audit Reports** under **Security > Cluster Security > Configuration Audits** show cluster-wide configuration audit security assessments. Unlike [Namespace Configuration Audits](namespace-configuration-audits), the scope is the whole cluster, not a single namespace. You can open each report to see failed checks, descriptions, and remediation steps. Reports are generated by **Trivy**; they appear in the list once cluster resources have been scanned. ## Cluster Configuration Audit Reports The page title is **Cluster Configuration Audit Reports**, with a short description: *Cluster-wide configuration audit security assessments*. There is no namespace filter — the list covers all audited entities at cluster level. ### Table Use the **Columns** control (e.g. "Columns 7") above the table to choose which columns are visible. The table includes: | Column | Description | |-----------------|-------------| | **Name** | Name of the audited cluster resource or entity. | | **Critical** | Number of critical findings. | | **High** | Number of high-severity findings. | | **Medium** | Number of medium-severity findings. | | **Low** | Number of low-severity findings. | | **Total Checks**| Total number of checks run. | If no cluster configuration audit reports exist yet, the page shows a message that no reports were found and that **Trivy cluster configuration audit reports will appear here once cluster resources are scanned**. **Pagination** at the bottom (e.g. "Rows per page: 25", "0 of 0") and navigation arrows let you move through the list when reports are present. Each row typically has an action (e.g. eye icon) to open **Audit Details**. ## Audit Details When you open a report, the breadcrumbs show **Security > Cluster Security > Configuration Audits > Audit Details**. The detail view follows the same pattern as [Namespace Configuration Audits](namespace-configuration-audits#audit-details): a header with the audited resource, scan summary (checks passed/failed, scanner version, last scan), severity counts, and a table of findings. You can filter by **Severity** and **Status**, and expand each row to see **Description**, **Messages**, and **Remediation**. The only difference is that the audit applies to cluster-level resources rather than a single namespace. ## Related Articles * [Namespace Configuration Audits](./namespace-configuration-audits.md) * [Cluster RBAC Assessments](./cluster-rbac-assessments.md) * [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) * [Cluster Vulnerability Reports](./cluster-vulnerability-reports.md) --- ## Cluster Infrastructure Assessments # Cluster Infrastructure Assessments **Cluster Infrastructure Assessment Reports** under **Security > Cluster Security > Infrastructure Assessments** show cluster-wide infrastructure security assessments for cluster resources (e.g. nodes). Unlike [Namespace Infrastructure Assessments](namespace-infrastructure-assessments), the scope is the whole cluster. You can open each report to see failed checks, descriptions, and remediation steps. Reports are generated by **Trivy**. ## Cluster Infrastructure Assessment Reports The page title is **Cluster Infrastructure Assessment Reports**, with a short description: *Cluster-wide infrastructure security assessments for cluster resources*. ### Table Use the **Columns** control above the table to choose which columns are visible. The table includes: | Column | Description | |-----------------|-------------| | **Name** | Name of the assessed cluster resource (e.g. a node such as `node-…compute.internal`). | | **Critical** | Number of critical findings. | | **High** | Number of high-severity findings. | | **Medium** | Number of medium-severity findings. | | **Low** | Number of low-severity findings. | | **Total Checks**| Total number of checks run. | | **Last Updated**| Date and time of the last assessment (may be empty). | Each row has an **eye icon** (or similar) at the end to open **Assessment Details**. **Pagination** at the bottom (e.g. "Rows per page: 25", "1–11 of 11") and navigation arrows let you move through the list. ## Assessment Details When you open an assessment from the list, the breadcrumbs show **Security > Cluster Security > Cluster Infrastructure Assessments > Assessment Details**. ### Header * **Resource** — The assessed cluster resource (e.g. a node identified as `…compute.internal` or similar) with a **Cluster** scope indicator. * **Scan summary** — **Scope: Cluster**, **Checks** (total), **Passed**, **Failed** (e.g. "Checks: 4 Passed: 0 Failed: 4"), **Scanner** and version (e.g. Trivy v0.29.0). * **Severity counts** — Badges for Critical, High, Medium, Low (e.g. "0 Critical", "4 High"). ### Security Checks Table Above the table you can filter by **Severity** (e.g. "All severities") and **Status** (e.g. "All"). The **Columns** button (e.g. "Columns 5") customizes visible columns. The table lists each finding with: | Column | Description | |-------------|-------------| | **Check ID** | Identifier of the check (e.g. KCV0069, KCV0073, KCV0086, KCV0077). Rows can be expanded to show full details. | | **Title** | Short title (e.g. "Ensure that the kubelet service file permissions are set to 600 or more restrictive", "Ensure that the -kubeconfig kubelet.conf file permissions are set to 600 or more restrictive"). | | **Category** | Category of the check (e.g. "Kubernetes Security Check"). | | **Severity** | Severity level (e.g. High), often with color. | | **Status** | Result (e.g. Fail), often with an icon. | **Expand a row** to see: * **Description** — Why the check matters (e.g. the kubelet service file should have restrictive permissions). * **Messages** — Concrete instance (e.g. which file or setting is involved). * **Remediation** — What to change, including specific paths where applicable (e.g. set permissions of `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` to 600 or more restrictive, or adjust kubelet config file permissions). Use this view to understand each finding and apply the suggested remediation on the cluster nodes. **Pagination** at the bottom (e.g. "1–4 of 4", "Rows per page: 25") applies when there are many findings. ## Related Articles * [Namespace Infrastructure Assessments](./namespace-infrastructure-assessments.md) * [Cluster Configuration Audits](./cluster-configuration-audits.md) * [Cluster RBAC Assessments](./cluster-rbac-assessments.md) * [Cluster Vulnerability Reports](./cluster-vulnerability-reports.md) --- ## Cluster RBAC Assessments # Cluster RBAC Assessments **Cluster RBAC Assessment Reports** under **Security > Cluster Security > RBAC Assessments** show cluster-wide RBAC policy security assessments for **ClusterRoles**. Unlike [Namespace RBAC Assessments](namespace-rbac-assessments), the scope is the whole cluster and the assessed resources are ClusterRoles, not namespace-scoped Roles. You can open each report to see failed checks, descriptions, and remediation steps. ## Cluster RBAC Assessment Reports The page title is **Cluster RBAC Assessment Reports**, with a short description: *Cluster-wide RBAC policy security assessments for ClusterRoles*. ### Table Use the **Columns** control (e.g. "Columns 7") above the table to choose which columns are visible. The table includes: | Column | Description | |-----------------|-------------| | **Name** | Name of the ClusterRole being assessed (e.g. `clusterrole-575df9df4d`). | | **Critical** | Number of critical findings. | | **High** | Number of high-severity findings. | | **Medium** | Number of medium-severity findings. | | **Low** | Number of low-severity findings. | | **Total Checks**| Total number of checks run. | Each row has an **eye icon** (or similar) at the end to open **Assessment Details**. **Pagination** at the bottom (e.g. "Rows per page: 25") and navigation arrows let you move through the list. ## Assessment Details When you open an assessment from the list, the breadcrumbs show **Security > Cluster Security > Cluster RBAC Assessments > Assessment Details**. ### Header * **Resource** — Name and kind (e.g. `clusterrole-575df9df4d`, **ClusterRole**) with a shield icon in a highlighted block. * **Scan summary** — **Scope: Cluster**, **Checks** (total), **Passed**, **Failed** (e.g. "Checks: 1 Passed: 0 Failed: 1"), **Scanner** and version (e.g. Trivy v0.29.0). * **Severity counts** — Badges for Critical, High, Medium, Low (e.g. "1 Critical", "0 High"). ### Security Checks Table Above the table you can filter by **Severity** (e.g. "All severities") and **Status** (e.g. "All"). The **Columns** button (e.g. "Columns 5") customizes visible columns. The table lists each finding with: | Column | Description | |-------------|-------------| | **Check ID** | Identifier of the check (e.g. KSV114). Rows can be expanded to show full details. | | **Title** | Short title (e.g. "Manage webhookconfigurations"). | | **Category** | Category of the check (e.g. "Kubernetes Security Check"). | | **Severity** | Severity level (e.g. Critical), often with color. | | **Status** | Result (e.g. Fail), often with an icon. | **Expand a row** to see: * **Description** — Why the check matters (e.g. webhooks can intercept or mutate resources, including secrets and pod specs). * **Messages** — Concrete instance (e.g. which ClusterRole has which verbs on which resources, such as `mutatingwebhookconfigurations` and `validatingwebhookconfigurations`). * **Remediation** — What to change (e.g. remove write/impersonate verbs for webhook configuration resources; acceptable verbs may be get, list, watch). Use this view to understand each finding and apply the suggested remediation. **Pagination** at the bottom (e.g. "1–1 of 1", "Rows per page: 25") applies when there are many findings. ## Related Articles * [Namespace RBAC Assessments](./namespace-rbac-assessments.md) * [Cluster Configuration Audits](./cluster-configuration-audits.md) * [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) * [Cluster Vulnerability Reports](./cluster-vulnerability-reports.md) --- ## Cluster Vulnerability Reports # Cluster Vulnerability Reports **Cluster Vulnerability Reports** under **Security > Cluster Security > Vulnerability Reports** list cluster-wide container image vulnerability reports. Unlike [Container Vulnerability Reports](container-vulnerability-reports) (namespace-scoped), the scope is the whole cluster. You can open each report to see scan metadata and the full list of CVEs. ## Reports List The page title is **Cluster Vulnerability Reports**, with a short description: *Cluster-wide container image vulnerability reports*. There is no namespace filter — the list covers all reported images at cluster level. ### Table Use the **Columns** control (e.g. "Columns 9") to choose which columns are visible. The table includes: | Column | Description | |-------------|-------------| | **Image** | Container image identifier (e.g. `kubernetes.1.34.2-eks-b312614`). | | **Critical** | Count of critical vulnerabilities. | | **High** | Count of high-severity vulnerabilities. | | **Medium** | Count of medium-severity. | | **Low** | Count of low-severity. | | **Unknown** | Count of unknown-severity. | | **OS Family**| Operating system family of the image (e.g. `amazon`). | | **Last Updated** | Date and time when the report was last updated. | Each row has an **eye icon** (or similar) at the end to open **Report Details**. **Pagination** at the bottom (e.g. "Rows per page: 25", "1–1 of 1") lets you move through the list. ## Report Details When you open a report, the breadcrumbs show **Security > Cluster Security > Cluster Vulnerability Reports > Report Details**. At the top you see: * **Image identifier** — Full image reference (e.g. `k8s.io/kubernetes:1.34.2-eks-b3126f4`) with a shield icon. A **No Vulnerabilities** badge appears when no issues were found. * **Context** — **Container** name (e.g. k8s-cluster), **Resource** (e.g. ClusterSbomReport/55967dc54c), **Scope: Cluster**, and base **OS** (e.g. amazon 2023.8.20250908). * **Severity summary** — Badges with counts: Critical, High, Medium, Low, Unknown (e.g. "0 Critical", "0 High"). Two tabs are available: **Overview** and **Vulnerabilities** (with the total count when there are findings). ### Overview Tab The Overview tab shows structured information: * **Scan Information** — **Scanner** (e.g. Trivy), **Scanner Version** (e.g. 0.66.0), **Vendor** (e.g. Aqua Security), **Last Scan** (date and time). * **Image Information** — **Registry** (e.g. k8s.io), **Repository** (e.g. kubernetes), **Tag** (e.g. 1.34.2-eks-b3126f4), **Digest** (if present). * **Operating System** — **Family** and **Version** of the base image (e.g. amazon 2023.8.20250908). * **Resource Information** — **Scope** (Cluster), **Resource Kind** (e.g. ClusterSbomReport), **Resource Name**, **Container** name. Use this to understand how and when the image was scanned and which cluster resource it belongs to. ### Vulnerabilities Tab The **Vulnerabilities** tab lists all findings for this image. The table has: | Column | Description | |---------------------|-------------| | **CVE ID** | CVE identifier (e.g. CVE-2026-0861). **Each CVE ID is a link** that opens the vulnerability details in the [Aqua Vulnerability Database](https://avd.aquasec.com/) (e.g. `https://avd.aquasec.com/nvd/2026/cve-2026-0861/`), where you can read description, CVSS, affected software, and mitigations. | | **Severity** | Severity level (e.g. High), often with a colored tag. | | **Resource** | Affected package or component. | | **Installed Version** | Version currently in the image. | | **Fixed Version** | Version that fixes the issue (may be empty if no fix is known). | | **Score** | Severity score (e.g. 8.1). | | **Title** | Short description of the vulnerability. | Use **Columns** (e.g. "Columns 7") to customize the table. **Pagination** (e.g. "Rows per page: 25") lets you move through the list when there are many vulnerabilities. If the image has no known vulnerabilities, the tab shows an empty state: **No vulnerabilities found** and a short message such as *This container image has no known vulnerabilities*, with "0 of 0" in the pagination. ## Related Articles * [Container Vulnerability Reports](./container-vulnerability-reports.md) * [Cluster Configuration Audits](./cluster-configuration-audits.md) * [Cluster RBAC Assessments](./cluster-rbac-assessments.md) * [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) --- ## Compliance Configuration Audits # Compliance Configuration Audits **Security > Compliance > Configuration Audits** shows **Kubernetes configuration** findings **framed for compliance**: unsafe defaults, weak workload settings, missing hardening, and similar checks that map to **benchmarks or policies** your organization tracks (for example CIS-style controls, Pod Security expectations, or other bundled profiles—exact labels depend on your deployment). On this page you can expect: * **Resources or report rows** with **failed** or **passed** checks tied to compliance-relevant configuration. * **Severity** and **status** per check, plus **remediation** text you can hand to platform or application teams. * Optional **framework** or **policy** context (which standard or chapter the control belongs to), when the product surfaces it. This is the **Compliance** entry point for **configuration audit** detail. For a **cluster-wide summary** of benchmark pass rates (multiple standards in one list), see [Cluster Compliance](./cluster-compliance.md). For **namespace-scoped operational** configuration audits, see [Namespace Configuration Audits](./namespace-configuration-audits.md); for **cluster-wide operational** audits, see [Cluster Configuration Audits](./cluster-configuration-audits.md). ## Related Articles * [Cluster Compliance](./cluster-compliance.md) * [Namespace Configuration Audits](./namespace-configuration-audits.md) * [Cluster Configuration Audits](./cluster-configuration-audits.md) * [Compliance RBAC Assessments](./compliance-rbac-assessments.md) * [Compliance Infrastructure Assessments](./compliance-infrastructure-assessments.md) * [Compliance Vulnerability Reports](./compliance-vulnerability-reports.md) --- ## Compliance Infrastructure Assessments # Compliance Infrastructure Assessments **Security > Compliance > Infrastructure Assessments** covers **infrastructure-level** posture for **compliance**: nodes and cluster components, **kubelet**-related settings, file permissions, and other checks that belong in **hardening benchmarks** rather than only in “YAML for one namespace.” On this page you can see: * **Reports per node or per scope** (as your deployment defines), with **severity breakdowns** and **failed vs passed** checks. * **Drill-down** listing individual checks with **description**, **message**, and **remediation** that often points to **specific paths or configuration keys** on the node. Until infrastructure scanning has run and data is synced, the page may stay **empty** or show a **no reports** state. For the **operational** (non-Compliance menu) counterparts, see [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) and [Namespace Infrastructure Assessments](./namespace-infrastructure-assessments.md). ## Related Articles * [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) * [Namespace Infrastructure Assessments](./namespace-infrastructure-assessments.md) * [Compliance Configuration Audits](./compliance-configuration-audits.md) * [Compliance RBAC Assessments](./compliance-rbac-assessments.md) * [Compliance Vulnerability Reports](./compliance-vulnerability-reports.md) --- ## Compliance RBAC Assessments # Compliance RBAC Assessments **Security > Compliance > RBAC Assessments** presents **role-based access control** risk with **compliance and audit** in mind: least privilege, separation of duties, and evidence that dangerous **Roles** or **ClusterRoles** are flagged and tracked. On this page you can see: * **One row per assessed Role or ClusterRole** (or equivalent grouping), with **severity counts** and **total checks**. * **Assessment details** with a filterable list of checks: **ID**, **title**, **category**, **severity**, **status**. * **Expanded rows** show **why** the permission is problematic, a **concrete message** (which resource/verbs are involved), and **remediation** (narrow verbs, remove wildcards, split roles). The same class of RBAC findings may also appear under [Namespace RBAC Assessments](./namespace-rbac-assessments.md) or [Cluster RBAC Assessments](./cluster-rbac-assessments.md); **Compliance** here stresses **control-oriented** language and reporting for reviewers. ## Related Articles * [Namespace RBAC Assessments](./namespace-rbac-assessments.md) * [Cluster RBAC Assessments](./cluster-rbac-assessments.md) * [Compliance Configuration Audits](./compliance-configuration-audits.md) * [Compliance Infrastructure Assessments](./compliance-infrastructure-assessments.md) * [Compliance Vulnerability Reports](./compliance-vulnerability-reports.md) --- ## Compliance Vulnerability Reports # Compliance Vulnerability Reports **Security > Compliance > Vulnerability Reports** is aimed at **compliance and audit** use cases: **CVE-style** findings on **container images** (packages and dependencies), with **severity**, **scores**, **affected component**, **installed** and **fixed** versions, and links to external advisory pages where the product provides them. On this page you can expect the same **shape of data** as in other vulnerability report screens — **CVE ID**, **severity**, **resource** (package), **installed version**, **fixed version**, **score**, **title** — with navigation and filters tuned for **evidence collection** and **remediation tracking** (which workload, which image, SLA by severity). This is how menu differs from others: | Location | Scope / use | |----------|-------------| | [Container Vulnerability Reports](./container-vulnerability-reports.md) | **Namespace** — images in use in a namespace. | | [Cluster Vulnerability Reports](./cluster-vulnerability-reports.md) | **Cluster** — cluster-scoped image or SBOM-style reports. | | **Compliance > Vulnerability Reports** (this page) | Same **CVE-oriented** findings, presented under **Compliance** for **policy and reporting** workflows. | Exact grouping depends on your product configuration; the **types of rows** (CVEs, severities, remediation) stay consistent across these views. ## Related Articles * [Container Vulnerability Reports](./container-vulnerability-reports.md) * [Cluster Vulnerability Reports](./cluster-vulnerability-reports.md) * [Compliance Configuration Audits](./compliance-configuration-audits.md) * [Compliance RBAC Assessments](./compliance-rbac-assessments.md) * [Compliance Infrastructure Assessments](./compliance-infrastructure-assessments.md) --- ## Exposed Secrets # Exposed Secrets **Security > Container Scanning > Exposed Secrets** lists **credentials and other sensitive material** found inside **container images** (or related artifacts the scanner inspects), such as API keys, tokens, passwords, or private keys embedded in layers or files. This is **separate from vulnerability (CVE) reporting**: an image can have no critical CVEs and still contain a leaked secret. Depending on how your environment is configured, you may get: * **Findings** tied to a scanned image or workload, with **severity** or risk indicators. * **Location hints** (path, layer, or context) so you can trace the issue back to a build step, copied config, or baked-in file. * Actions or links to **open details** and follow **remediation** guidance. Use this view to **rotate** any exposed credential and **change the build** so secrets are never committed into images (runtime `Secret` objects, external secrets, CI variables, `.dockerignore`, multi-stage builds without copying `.env`, etc.). This is how this fits Container Scanning: * [Container Scanning Overview](./container-scanning-overview.md) — high-level image risk for the namespace. * [Container Vulnerability Reports](./container-vulnerability-reports.md) — CVE-style drill-down per image. ## Related Articles * [Container Scanning Overview](./container-scanning-overview.md) * [Container Vulnerability Reports](./container-vulnerability-reports.md) --- ## Container Scanning Overview # Container Scanning Overview The **Container Scanning > Overview** page shows the **Trivy Security Overview** dashboard: a summary of container image vulnerabilities for the selected namespace (and cluster) in the portal header. ## Key metrics At the top, four cards summarize the current state: * **Total Vulnerabilities** — Total number of vulnerabilities across all scanned images. * **Critical Vulnerabilities** — Count of critical issues (with a note that they require immediate attention). * **Images Scanned** — Number of container images scanned. * **Fixable Vulnerabilities** — How many vulnerabilities have a fix available (with a note "Fix available"). Use these to quickly see exposure and remediation potential. ## Severity Breakdown A table lists vulnerability distribution by severity: | Severity | Count | Fixable | Distribution | |-----------|-------|---------|--------------| | Critical | … | … | …% (bar) | | High | … | … | …% (bar) | | Medium | … | … | …% (bar) | | Low | … | … | …% (bar) | | Unknown | … | … | …% (bar) | | **Total** | … | … | 100% (bar) | Each row shows the count, how many are fixable, and a horizontal bar for the share of that severity in the total. ## Severity Distribution A **donut chart** shows the same breakdown by severity (e.g. Critical, High, Medium, Low, Unknown) with labels and percentages, so you can see at a glance where most issues sit. ## Top Vulnerable Images A table lists images with the highest vulnerability counts. For each image you typically see: * Image name (e.g. `library/nginx:latest`). * Namespace (e.g. the current namespace). * Counts by severity: Critical, High, Medium, Low. * Total vulnerability count. Use this to prioritize which images to update or replace. ## Vulnerability Reports At the bottom of the page, a **Vulnerability Reports** block is shown: *View all container vulnerability scan results*, with an arrow. **Clicking this block opens the [Vulnerability Reports](container-vulnerability-reports) section**, where you can see the full list of container vulnerability scan results and drill into details. ## Related Articles * [Container Vulnerability Reports](./container-vulnerability-reports.md) * [Exposed Secrets](./container-exposed-secrets.md) --- ## Container Vulnerability Reports # Container Vulnerability Reports **Container Vulnerability Reports** lists unique container images that have vulnerabilities in the selected namespace. You can open each report to see scan metadata and the full list of CVEs. This page is reachable from **Security > Container Scanning > Vulnerability Reports** or by clicking the **Vulnerability Reports** block on the [Container Scanning Overview](container-scanning-overview) page. ## Reports List The page shows a short description such as *Unique container images with vulnerabilities in namespace <namespace>*. At the top right you can change the **Namespace** to filter by namespace. ### Table Use the **Columns** control (e.g. "Columns 9") to choose which columns are visible. The table includes: | Column | Description | |-------------|-------------| | **Image** | Container image (e.g. `library/nginx:latest`). Rows can be expanded (arrow) for more detail. | | **Namespace** | Namespace where the image is used. | | **Resources** | Number of resources using this image (e.g. "1 resource"); the value is clickable and leads to the report details. | | **Critical** | Count of critical vulnerabilities. | | **High** | Count of high-severity vulnerabilities. | | **Medium** | Count of medium-severity. | | **Low** | Count of low-severity. | | **Total** | Total vulnerability count. | | **Last Scan**| Date and time of the last scan. | **Pagination** at the bottom (e.g. "Rows per page 25", "1–1 of 1") lets you move through the list. Click a row or the **Resources** link to open **Report Details** for that image. ## Report Details When you open a report, the breadcrumbs show **Security > Container Scanning > Vulnerability Reports > Report Details**. At the top you see: * **Severity summary** — Badges with counts: Critical, High, Medium, Low, Unknown (e.g. "0 Critical", "5 High"). * **Container block** — Image name (e.g. `index.docker.io/library/nginx:latest`), **Container** name, **Resource** (e.g. ReplicaSet/webserver-bc6dbb848), **Namespace**, and base **OS** (e.g. debian 13.3). Two tabs are available: **Overview** and **Vulnerabilities** (with the total count, e.g. "Vulnerabilities 5"). ### Overview Tab The Overview tab shows structured information: * **Scan Information** — **Scanner** (e.g. Trivy), **Scanner Version** (e.g. 0.66.0), **Vendor** (e.g. Aqua Security), **Last Scan** (date and time). * **Image Information** — **Registry** (e.g. index.docker.io), **Repository** (e.g. library/nginx), **Tag** (e.g. latest), **Digest** (truncated hash). * **Operating System** — **Family** and **Version** of the base image (e.g. debian 13.3). * **Resource Information** — **Namespace**, **Resource Kind** (e.g. ReplicaSet), **Resource Name**, **Container** name. Use this to understand how and when the image was scanned and where it runs. ### Vulnerabilities Tab The **Vulnerabilities** tab lists all findings for this image. The table has: | Column | Description | |---------------------|-------------| | **CVE ID** | CVE identifier (e.g. CVE-2026-0861, CVE-2026-22695). **Each CVE ID is a link** that opens the vulnerability details in the [Aqua Vulnerability Database](https://avd.aquasec.com/) (e.g. `https://avd.aquasec.com/nvd/2026/cve-2026-0861/`), where you can read description, CVSS, affected software, and mitigations. | | **Severity** | Severity level (e.g. High), often with a colored tag. | | **Resource** | Affected package or component. | | **Installed Version** | Version currently in the image. | | **Fixed Version** | Version that fixes the issue (may be empty if no fix is known). | | **Score** | Severity score (e.g. 8.1, 7.1). | | **Title** | Short description of the vulnerability. | Use **Columns** (e.g. "Columns 7") to customize the table. **Pagination** (e.g. "Rows per page: 25", "1–5 of 5") lets you move through the list when there are many vulnerabilities. ## Related Articles * [Container Scanning Overview](./container-scanning-overview.md) * [Exposed Secrets](./container-exposed-secrets.md) --- ## Namespace Configuration Audits # Namespace Configuration Audits **Configuration Audit Reports** under **Security > Namespace Security > Configuration Audits** show Kubernetes resource misconfigurations in the selected namespace. You can open each resource to see failed checks, descriptions, and remediation steps. ## Configuration Audit Reports The page title is **Configuration Audit Reports**, with a short description such as *Kubernetes resource misconfigurations in namespace <namespace>*. In the top right you can change the **Namespace** to filter reports. ### Table Use the **Columns** control above the table to choose which columns are visible. The table includes: | Column | Description | |------------------|-------------| | **Resource Name** | Name of the Kubernetes resource (e.g. ReplicaSet name). | | **Resource Kind** | Kind of resource (e.g. ReplicaSet). | | **Critical** | Number of critical findings. | | **High** | Number of high-severity findings. | | **Medium** | Number of medium-severity findings. | | **Low** | Number of low-severity findings. | | **Total Checks** | Total number of checks run. | | **Last Updated** | Date and time of the last audit. | Each row has an **eye icon** (or similar) at the end to open **Audit Details**. **Pagination** at the bottom (e.g. "Rows per page: 25", "1–1 of 1") and navigation arrows let you move through the list. ## Audit Details When you open a resource, the breadcrumbs show **Security > Namespace Security > Configuration Audits > Audit Details**. ### Header * **Resource** — Name and kind (e.g. `webserver-bc6dbb848`, ReplicaSet) in a highlighted block. * **Scan summary** — **Namespace**, **Checks** (total), **Passed**, **Failed** (e.g. "Checks: 3 Passed: 0 Failed: 3"), **Scanner** and version (e.g. Trivy v0.29.0), **Last scan** (date and time). * **Severity counts** — Badges for Critical, High, Medium, Low (e.g. "0 Critical", "3 High"). ### Audit Findings Table Above the table you can filter by **Severity** (e.g. "All severities") and **Status** (e.g. "All"). The **Columns** button customizes visible columns. The table lists each finding with: | Column | Description | |-------------|-------------| | **Check ID** | Identifier of the check (e.g. KSV014, KSV118). Rows can be expanded to show full details. | | **Title** | Short title (e.g. "Root file system is not read-only", "Default security context configured"). | | **Category** | Category of the check (e.g. "Kubernetes Security Check"). | | **Severity** | Severity level (e.g. High), often with color. | | **Status** | Result (e.g. Fail), often with an icon. | **Expand a row** to see: * **Description** — Why the check matters (e.g. an immutable root filesystem limits what attackers can write to disk). * **Messages** — Concrete instance of the issue (e.g. which container and resource should be changed). * **Remediation** — What to change to fix it (e.g. set `containers[].securityContext.readOnlyRootFilesystem` to `true`). Use this view to understand each finding and apply the suggested remediation. **Pagination** at the bottom (e.g. "1–3 of 3", "Rows per page: 25") applies when there are many findings. ## Related Articles * [Namespace RBAC Assessments](./namespace-rbac-assessments.md) * [Namespace Infrastructure Assessments](./namespace-infrastructure-assessments.md) --- ## Namespace Infrastructure Assessments # Namespace Infrastructure Assessments **Infrastructure Assessment Reports** under **Security > Namespace Security > Infrastructure Assessments** show **infrastructure-oriented security checks** for resources in the **selected namespace**. The focus is on **misconfigurations and hardening gaps** in how platform and supporting resources are set up—not on “CVE inside application package” style issues (those belong under container vulnerability flows). After scans have run, you typically see: * A **table** of assessed targets with: **severity counts** (Critical, High, Medium, Low), **total checks**, and **last updated** (when shown). * **Drill-down** into a row: **check ID**, **title**, **category**, **severity**, **status**, and expandable **description**, **message**, and **remediation** text. If nothing has been scanned yet, the page may show an **empty state** explaining that reports will appear once infrastructure assessment data is available for the namespace. ## How this differs from sibling views | View | What it emphasizes | |------|---------------------| | [Namespace Configuration Audits](./namespace-configuration-audits.md) | **Workload-oriented** configuration (e.g. pod `securityContext`, read-only root filesystem, defaults). | | [Namespace RBAC Assessments](./namespace-rbac-assessments.md) | **RBAC**—Roles, risky verbs, overly broad grants. | | **Infrastructure Assessments** (this page) | **Infrastructure / platform wiring** in the namespace—controllers, configs, and related surfaces that are not purely “this pod’s securityContext” or “this Role’s verbs.” | ## Related Articles * [Namespace Configuration Audits](./namespace-configuration-audits.md) * [Namespace RBAC Assessments](./namespace-rbac-assessments.md) * [Cluster Infrastructure Assessments](./cluster-infrastructure-assessments.md) --- ## Namespace RBAC Assessments # Namespace RBAC Assessments **RBAC Assessment Reports** under **Security > Namespace Security > RBAC Assessments** show RBAC policy security assessments in the selected namespace. You can open each resource (e.g. Role) to see failed checks, descriptions, and remediation steps. ## RBAC Assessment Reports The page title is **RBAC Assessment Reports**, with a short description such as *RBAC policy security assessments in namespace <namespace>*. In the top right you can change the **Namespace** to filter reports. ### Table Use the **Columns** control above the table to choose which columns are visible. The table includes: | Column | Description | |------------------|-------------| | **Resource Name** | Name of the RBAC resource (e.g. Role or RoleBinding name such as `crossplane-admin`, `crossplane-edit`). | | **Resource Kind** | Kind of resource (e.g. **Role**). | | **Critical** | Number of critical findings. | | **High** | Number of high-severity findings. | | **Medium** | Number of medium-severity findings. | | **Low** | Number of low-severity findings. | | **Total Checks** | Total number of checks run. | Each row has an **eye icon** (or similar) at the end to open **Assessment Details**. **Pagination** at the bottom (e.g. "Rows per page: 25", "1–3 of 3") and navigation arrows let you move through the list. ## Assessment Details When you open an assessment from the list, the breadcrumbs show **Security > Namespace Security > RBAC Assessments > Assessment Details**. ### Header * **Resource** — Name and kind (e.g. `crossplane-admin`, **Role**) with a shield icon in a highlighted block. * **Scan summary** — **Namespace**, **Checks** (total), **Passed**, **Failed** (e.g. "Checks: 2 Passed: 0 Failed: 2"), **Scanner** and version (e.g. Trivy v0.29.0). * **Severity counts** — Badges for Critical, High, Medium, Low (e.g. "2 Critical", "0 High"). ### Security Checks Table Above the table you can filter by **Severity** (e.g. "All severities") and **Status** (e.g. "All"). The **Columns** button customizes visible columns. The table lists each finding with: | Column | Description | |-------------|-------------| | **Check ID** | Identifier of the check (e.g. KSV050, KSV045). Rows can be expanded to show full details. | | **Title** | Short title (e.g. "Manage Kubernetes RBAC resources", "No wildcard verb roles"). | | **Category** | Category of the check (e.g. "Kubernetes Security Check"). | | **Severity** | Severity level (e.g. Critical), often with color. | | **Status** | Result (e.g. Fail), often with an icon. | **Expand a row** to see: * **Description** — Why the check matters (e.g. an effective level of access equivalent to cluster-admin should not be provided). * **Messages** — Concrete instance (e.g. which Role has which verbs on which resources, such as `roles` and `rolebindings`). * **Remediation** — What to change (e.g. remove write permission verbs for `roles` and `rolebindings`, or avoid wildcard verbs). Use this view to understand each RBAC finding and apply the suggested remediation. **Pagination** at the bottom (e.g. "1–2 of 2", "Rows per page: 25") applies when there are many findings. ## Related Articles * [Namespace Configuration Audits](./namespace-configuration-audits.md) * [Namespace Infrastructure Assessments](./namespace-infrastructure-assessments.md) --- ## SCA Overview # SCA Overview The Security section covers security settings, access control, secrets management, and protected resources. It is organized into the following sub-sections: :::note * The **Trivy** tool manages **scanning and hardening** views: **Container Scanning**, **Namespace Security**, **Cluster Security** (configuration audits, RBAC, infrastructure, cluster vulnerability reports, compliance benchmarks and related drill-downs), and the **Compliance** entries that mirror those scan types. If **Trivy** is not integrated and running against your cluster/namespaces, **those screens stay empty or without fresh data**. * **SCA** (this portfolio and [Projects](./sca-projects)) comes from **Dependency Track** (BOMs, component vulnerabilities, policies). * **SAST** ([SAST](./sca-sast)) comes from **SonarQube**. ::: * **SCA** (Software Composition Analysis) — [Overview](./sca-overview), [Projects](./sca-projects) * **SAST** — [SAST](./sca-sast) * **Container Scanning** — [Overview](./container-scanning-overview), [Vulnerability Reports](./container-vulnerability-reports), [Exposed Secrets](./container-exposed-secrets) * **Namespace Security** — [Configuration Audits](./namespace-configuration-audits), [RBAC Assessments](./namespace-rbac-assessments), [Infrastructure Assessments](./namespace-infrastructure-assessments) * **Cluster Security** — [Compliance](./cluster-compliance), [Configuration Audits](./cluster-configuration-audits), [RBAC Assessments](./cluster-rbac-assessments), [Infrastructure Assessments](./cluster-infrastructure-assessments), [Vulnerability Reports](./cluster-vulnerability-reports) * **Compliance** — [Configuration Audits](./compliance-configuration-audits), [RBAC Assessments](./compliance-rbac-assessments), [Infrastructure Assessments](./compliance-infrastructure-assessments), [Vulnerability Reports](./compliance-vulnerability-reports) The **Software Composition Analysis** portfolio dashboard shows portfolio-level vulnerability metrics and policy violations across all projects. Use it to track vulnerabilities, projects at risk, and policy compliance over time. ## Key Metrics Four KPI cards with trend lines summarize: * **Portfolio Vulnerabilities** — Total vulnerability count for the portfolio. * **Projects at Risk** — Number of projects that have active risk (e.g. vulnerabilities or policy violations). * **Vulnerable Components** — Number of components (dependencies) with known vulnerabilities. * **Inherited Risk Score** — Aggregate risk score derived from vulnerabilities and policy state. At the top of the page you can select **30 Days**, **60 Days**, **90 Days**, or **1 Year** to scope all metrics and charts. ## Portfolio Statistics A summary block shows current counts: * **Projects** and **Vulnerable Projects** * **Components** and **Vulnerable Components** * **Policy Violations** — total, plus split by **License**, **Operational**, and **Security** * **Portfolio Vulnerabilities** and **Suppressed** (suppressed findings) Use this to see the overall posture at a glance. ## Charts * **Portfolio Vulnerabilities** — Stacked area chart of vulnerabilities over time by severity: Critical, High, Medium, Low, Unassigned. Legend shows current counts and percentages. * **Policy Violations by State** — Stacked area chart of violations by state: Fail, Warn, Info. * **Policy Violations by Classification** — Stacked area chart by type: Security, License, Operational. * **Auditing Progress (Findings)** — Trend of **Audited** vs **Unaudited** findings over the selected period. * **Auditing Progress (Violations)** — Trend of **Audited** vs **Unaudited** violations. * **Projects** — Stacked area chart of **Non-Vulnerable** vs **Vulnerable** projects over time (with total project count). * **Components** — Stacked area chart of **Non-Vulnerable** vs **Vulnerable** components over time (with total component count). Charts use the same time range as the page filter and show a last-measurement timestamp. ## Related Articles * [SCA Projects](./sca-projects.md) * [Observability](../observability.md) * [KubeRocketCI Widgets](../widgets.md) --- ## SCA Projects # SCA Projects The **SCA Projects** view lists software composition analysis projects and their security metrics. Use it to find projects by name or version and see risk scores, vulnerability counts, and policy violations at a glance. ## Projects Table A search bar at the top lets you **search projects by name or version**. The table shows one row per project (or per project version) with these columns: * **Project Name** — Name of the project (e.g. codebase or application). * **Version** — Version or branch (e.g. `master`, `main`, `1.0.0`). * **Latest** — Indicates whether this version is the latest. * **Classifier** — Type of project: `APPLICATION`, `LIBRARY`, or other. * **Last BOM Import** — Date and time of the last Bill of Materials (BOM) import. * **BOM Format** — Format of the BOM (e.g. `CycloneDX 1.6`). * **Risk Score** — Numerical risk score for the project. * **Active** — Whether the project is active (e.g. `Active`). * **Vulnerabilities** — Count of vulnerabilities; often shown with a horizontal bar (e.g. green when zero). * **Policy Violations** — Count of policy violations; same visual style as vulnerabilities. You can change which columns are visible using the **Columns** control (e.g. "Columns 10") and use pagination at the bottom (rows per page, e.g. 25; navigation for pages). ## Project Details When you click a project row, you open **Project Details** for that project. The breadcrumbs show **Security > SCA > Projects > Project Details**. At the top you see the project name (e.g. `edp-codebase-operator`), a branch/version dropdown (e.g. `master`), and a **View in Dependency Track** link. A summary row shows vulnerability counts by criticality: **Critical**, **High**, **Medium**, **Low**, **Unassigned** (each with a count). Tabs let you switch between: * **Overview** — Vulnerability trend graph, summary cards (Critical, High, Medium, Low, Unassigned, Risk Score), and policy violations by state and by classification. * **Components** — List of components (dependencies) with counts. * **Services** — Services associated with the project. * **Dependency Graph** — Graph view of dependencies. * **Audit Vulnerabilities** — Vulnerabilities pending or audited. * **Exploit Predictions** — Predicted exploits. * **Policy Violations** — Policy violations with breakdown by criticality. ### Overview Tab * **Project Vulnerabilities** — Metadata (Last BOM Import, Last Vulnerability Analysis, Last Measurement), a **vulnerability trend** line chart over time by severity, and **summary cards** for Critical, High, Medium, Low, Unassigned, and Risk Score. * **Policy Violations by State** — Chart or list of violations by state (e.g. Fail, Warn, Info). * **Policy Violations by Classification** — Breakdown by classification (e.g. Security Risk, License Risk, Operational Risk) with counts and percentages. ## Related Articles * [SCA Overview](./sca-overview.md) * [Security](../security.md) --- ## SAST # SAST **Static Application Security Testing (SAST)** in KubeRocketCI shows SonarQube projects and their code quality metrics. Use **Security > SAST > Projects** to see all projects and open details for any of them. ## SAST Projects The **SAST > Projects** page lists projects analyzed by SonarQube. The subtitle describes it as *SonarQube projects and their code quality metrics*. ### Search and Table * **Search** — Use **Search projects by name...** at the top to filter the list. * **Columns** — The **Columns** control (e.g. "Columns 9") lets you choose which columns are shown. The table includes: | Column | Description | |--------|-------------| | **Project Name** | Name of the codebase or application. | | **Quality Gate** | Status: **Passed** (green checkmark) or **N/A** (gray). | | **Visibility** | Project visibility (e.g. **PUBLIC**). | | **Bugs** | Number of bugs (e.g. `0 A`; the letter is a grade). | | **Vulnerabilities** | Number of vulnerabilities (e.g. `0 A`). | | **Code Smells** | Number of code smells (e.g. `2 A`, `1 A`). | | **Coverage** | Test coverage percentage (e.g. `0.0%`, `77.8%`). | | **Duplications** | Duplicated lines percentage (e.g. `0.0%`). | | **Last Analysis** | Date and time of the last analysis. | Use this view to monitor security and code quality across projects at a glance. ## Project Details When you click a project in the list, you open **Project Details**. The breadcrumbs show **Security > SAST > Projects > Project Details**. At the top you see: * **Project name** (e.g. `example-sast-project`) and a **PUBLIC** badge. * **Key** — SonarQube project key (e.g. `Key: example-sast-project`). * **Last analysis** — Timestamp of the last run. * **View in SonarQube** — Link to open the project in SonarQube. A **metrics bar** shows summary grades and values: **Vulnerabilities**, **Bugs**, **Code Smells**, **Hotspots Reviewed**, **Coverage**, **Duplications** (with letter grades like A and color indicators). Two main tabs are available: **Overview** and **Issues**. ### Overview Tab The Overview tab shows card-based metrics: * **Reliability** — Number of bugs (with grade badge, e.g. A). * **Security** — Number of vulnerabilities (with grade badge). * **Maintainability** — Code smells and technical debt (with grade badge). * **Security Review** — Security hotspots to review (with grade badge). * **Coverage** — Test coverage percentage (with color dot, e.g. red for 0%). * **Duplications** — Duplicated lines density (with color dot). * **Size** — Lines of code. * **Quality Gate** — Quality gate status (e.g. **Passed** with OK badge). **Quality Gate Details** is a table with columns: **Metric**, **Operator**, **Threshold**, **Actual**, **Status**. It lists each quality gate condition (e.g. `blocker_violations`, `critical_violations`) and whether the project meets it (OK or not). You can change visible columns via the **Columns** control (e.g. "Columns 5"). ### Issues Tab The **Issues** tab shows findings for the project (e.g. "Issues 2" when there are 2 issues). * **Issue type filters** — **All Issues**, **Bugs**, **Vulnerabilities**, **Code Smells** to filter by kind. * **Severity filters** — Buttons for **BLOCKER**, **CRITICAL**, **MAJOR**, **MINOR**, **INFO** to filter by severity. The issues table has: | Column | Description | |--------|-------------| | **Severity** | Severity with icon (e.g. **MAJOR**). | | **Type** | Issue type (e.g. **CODE_SMELL**). | | **Message** | Short description of the issue. | | **File** | File path and line (e.g. `src/app/app.component.css:1`); often clickable. | | **Effort** | Estimated fix effort (e.g. "1 min", "5 mins"). | | **Created** | When the issue was created or last seen (e.g. analysis date). | Use **Columns** to customize the table. **Pagination** at the bottom (e.g. "Rows per page: 25", "1–2 of 2") lets you move through the list when there are many issues. ## Related Articles * [SCA Overview](./sca-overview.md) * [SCA Projects](./sca-projects.md) --- ## Security # Security The Security section is documented under its sub-sections. For the main overview and SCA portfolio dashboard, see **[SCA Overview](security/sca-overview)**. --- ## KubeRocketCI: Tekton Overview # KubeRocketCI: Tekton Overview KubeRocketCI is a platform that utilizes the Tekton stack to implement continuous integration and continuous delivery (CI/CD) pipelines, offering access to all Tekton features. It comes pre-configured with various Tekton resources, including pipelines, tasks, triggers, and other components. This documentation outlines how to interact with these Tekton resources within the KubeRocketCI platform, addressing common use cases, pipeline configuration, and execution processes. ## Prerequisites Before executing pipelines, ensure that the project is onboarded to the KubeRocketCI platform and that the relevant branches are properly configured. ## Pipeline Types in the Platform KubeRocketCI supports seven main types of pipelines: * **Review Pipeline**: This pipeline is initiated when a pull request is created. It aims to check code quality and run tests before the code can be merged. * **Build Pipeline**: This pipeline is triggered when a pull request is merged. It is responsible for building the application and generating deployable artifacts. * **Deploy Pipeline**: This pipeline handles the deployment of the application to a target environment, applying specific configurations. * **Clean Pipeline**: This pipeline is responsible for cleaning up any resources created during the deployment, ensuring a fresh environment for future deployments. * **Test Pipeline**: This pipeline is designed to execute automated tests against deployed environments. It ensures that the application functions as expected. * **Security Pipeline**: This pipeline performs security scanning and vulnerability analysis. It is used to identify vulnerabilities, misconfigurations, and compliance issues. * **Release Pipeline**: Manages the process of approving and publishing new releases, ensuring that all compliance and governance standards are met. Each pipeline type is identified using specific labels in Kubernetes objects: * `app.edp.epam.com/pipelinetype: review` — for review pipelines; * `app.edp.epam.com/pipelinetype: build`— for build pipelines; * `app.edp.epam.com/pipelinetype: deploy` — for deploy pipelines; * `app.edp.epam.com/pipelinetype: clean`— for clean pipelines; * `app.edp.epam.com/pipelinetype: tests`— for test pipelines; * `app.edp.epam.com/pipelinetype: security`— for security pipelines; * `app.edp.epam.com/pipelinetype: release`— for release pipelines. ## Creating a Pipeline To create a pipeline, you need to define and apply Tekton resources such as tasks, pipelines, and triggers in the cluster. KubeRocketCI includes a library component for managing Tekton resources declaratively. This library integrates with Argo CD, enabling version control and automatic synchronization of pipeline definitions. Below is the recommended flow for creating comprehensive and efficient Tekton pipelines: 1. Start by modifying an existing pipeline example from the library to suit your needs. 2. Test changes by applying resources directly to the cluster. 3. Once verified, commit changes to the repository to ensure they are tracked and version-controlled. 4. Check/apply changes in Argo CD. ## Configuring Pipelines for Projects Proper pipeline configuration depends on its purpose. Pipeline configuration is different for CI and CD pipelines. ### Review and Build Pipelines (CI) Two approaches are available for defining review and build pipelines: 1. [Custom Framework/Build Tool](/docs/use-cases/tekton-custom-pipelines): * Useful for onboarding multiple projects with the same pipeline structure. * Pipelines follow a naming pattern and are mapped to projects via project settings. 2. [Branch-Specific Pipelines](/docs/use-cases/custom-pipelines-flow#replace-pipelines-with-personalized-versions): * Ideal for enabling different pipelines for different branches of a project. * Pipelines are explicitly defined for specific branches. Beyond the stock review and build flows, custom Tekton Triggers can be registered with a Git server's EventListener declaratively by labeling them with `app.edp.epam.com/gitServer: `, or served by a dedicated EventListener with its own manually created webhook — see [Add Custom Tekton Triggers](customize-ci-triggers.md). ### Deploy and Clean Pipelines (CD) Deploy and Clean pipelines require the creation of a TriggerTemplate resource with appropriate labels: * `app.edp.epam.com/pipelinetype: deploy` — for deploy pipelines; * `app.edp.epam.com/pipelinetype: clean` — for clean pipelines. The TriggerTemplate defines parameters (e.g., service account name, timeout) and references the deploy pipeline in its pipelineRef field. ### Security Pipeline Security pipeline conducts security scans and vulnerability assessments as a standalone process, decoupled from the build pipelines. Moving security checks into a separate pipeline enables more frequent, targeted scans, reduces build pipeline time, and allows to manage and evolve scanning logic independently of application delivery. The scan result summary can be found in the **Results** tab. It refers you to the DefectDojo tool. ### Test Pipeline Test pipelines execute automated tests for environments independently of deployments. This allows teams to validate changes quickly without the overhead of full environment deployment. Previously, autotests could only be triggered after the application deploy pipeline, but with test pipelines, tests are now fully independent of deployment. This separation allows teams to validate application functionality on demand, run tests at any stage, and improve feedback cycles by decoupling testing from application delivery. ### Release Pipelines Release pipeline orchestrates the approval and publishing workflow for new releases, supporting organizational compliance and governance requirements. By isolating release logic in its own pipeline, teams can implement custom approval steps, integrate with external systems, and ensure that release processes are auditable and consistent. :::note KubeRocketCI does not offer pre-built release pipelines. You can create custom release pipelines tailored to your project's needs. ::: ## Trigger Pipelines A specific event triggers each pipeline type. ### Review Pipeline Trigger a review pipeline using one of the four methods: 1. Create a pull request in the repository targeting a configured branch. The review pipeline will be triggered automatically once the pull request is created: ![Pull request in Bitbucket](../assets/user-guide/tekton-pipelines/bitbucket-pull-request.png "Pull request in Bitbucket") 2. Use the **Run Again** button on the PipelineRun details page in the KubeRocketCI portal (if a PipelineRun exists). 3. Use the **Rerun** button on the PipelineRun details page in the Tekton dashboard (if a PipelineRun exists): ![Rerun review pipeline in Tekton](../assets/user-guide/tekton-pipelines/review-pipeline-tekton-dashboard-run-again.png "Rerun review pipeline in Tekton") 4. Add a comment that begins with the `/recheck` or `/ok-to-test` command on the pull request page: :::note The `/recheck` and `/ok-to-test` commands must be at the beginning of your comment to be recognized. These commands work for all the available Version Control Systems. ::: ![Recheck pull request](../assets/user-guide/tekton-pipelines/bitbucket-pull-request-recheck.png "Recheck pull request") ### Build Pipeline Overall, there are four methods of triggering a build pipeline: 1. Merge a pull request into a configured branch. 2. Use the **Run Again** button in the KubeRocketCI portal (if a PipelineRun exists). 3. Use the **Rerun** button in the Tekton dashboard (if a PipelineRun exists): ![Rerun build pipeline in Tekton](../assets/user-guide/tekton-pipelines/build-pipeline-tekton-dashboard-rerun.png "Rerun build pipeline in Tekton") 4. Use the **Build** button in the KubeRocketCI portal within the branches section. ### Deploy Pipeline Deploy pipelines can be triggered manually or automatically. Automatic triggers are implemented using the **TriggerType** custom resource. There are three ways to trigger a deploy pipeline: 1. Use **Configure Deploy** and **Start Deploy** buttons in the KubeRocketCI portal. 2. Configure the pipeline with the `Auto` TriggerType to deploy automatically after the build pipeline finishes and a new artifact version is created. 3. Use the `Auto-stable` TriggerType for deploying stable component versions while updating only the rebuilt component. ### Clean Pipeline Trigger the clean pipeline using the **Clean** button in the KubeRocketCI portal within the appropriate stage section. ### Security, Test, and Release Pipeline To trigger a security, test, and release pipeline, follow the steps below: 1. Navigate to the **Pipelines** section. 2. Open the **Pipelines** tab. 3. On the **Pipelines** tab, use the filter to select a security, test, or release pipeline. 4. In the pipelines list, click the actions button and select **Run with parameters**. 5. On the create resource window, specify the required parameters and click **Save & Apply**: ```bash params: - name: git-source-url value: git@github.com:/orders-processing.git - name: git-source-revision value: main - name: CODEBASE_NAME value: orders-processing ``` * **git-source-url**: Git or HTTPS address of the Git repository where the application code is stored. * **git-source-revision**: Git branch of the repository. * **CODEBASE_NAME**: Enter the name of the codebase you want to scan. ```bash params: - name: git-source-url value: git@github.com:/orders-processing.git - name: git-source-revision value: main - name: makefile-target value: dev - name: base-image value: epamedp/maven-java21-make:0.1.3 ``` * **git-source-url**: Git or HTTPS address of the Git repository where the autotests are stored. * **git-source-revision**: Git branch of the repository. * **makefile-target**: Enter one of the targets specified in the make file of your autotests. * **base-image**: Specify the container image that will be used to run your autotests. ## Related Articles * [Pipelines Overview](../user-guide/pipelines.md) * [Add Library](../user-guide/add-library.md) * [Add Autotest](../user-guide/add-autotest.md) --- ## KubeRocketCI Widgets # KubeRocketCI Widgets KubeRocketCI offers widgets as data visualization tools to observe the status of both the platform and its resources. This page describes all the widgets presented in KubeRocketCI. ## Widgets Overview The first widget users view when using KubeRocketCI portal is **Platform Dashboard**. It shows the amount of pipeline runs, amount of failed pipeline runs and percentage of successful pipeline runs. When clicking on this small widget, you will be redirected to the Overview page which will show you the following widget groups: * **Header**: Shows you the total amount of PipelineRuns, their success rate, failed pipelines, and average duration. * **Pipeline Activity (7 days)**: Shows you 7 columns that represent the amount of triggered pipeline runs and their success rate. * **Resource Health**: these widgets reflect all the resources in the **default namespace**: * **Codebases**: Displays all the created codebases; * **Branches**: Shows the total amount of codebase branches; * **Pipelines**: Shows all the initiated pipeline runs; * **Deployments**: Displays all the created deployments; * **Environments**: Shows the total amount of created environments; * **+ Add Widget**: Opens the **Add new widget** window. * **Resource Usage**: These widgets show you a summary of cluster resources consumed by the platform. * **Vulnerability Summary**: This is the total amount of vulnerabilities of the Projects found by Dependency-Track. * **SonarQube Quality**: This is the summary for all the SonarQube Projects. * **Recent Pipeline Activity**: Shows recent PipelineRuns, including their name (clickable), status, duration, and start time. * **DORA Metrics**: These widgets provide insights into software delivery performance using DORA metrics, including Deployment Frequency and Change Failure Rate, covering the past 30 days. * **Links**: Clickable links that can lead you to other services integrated with the platform. You can manually create Quick Links by following the instructions in the [Manage Quick Links](./quick-links.md) guide. The status of the resources is displayed real-time and doesn't require any preliminary configuration to for widgets to work. Not only these widgets display the amount of resources but also their statuses (success, fail, in progress). Additionally, users can track any specific application status and deployed versions using custom widgets. To add a custom widget, follow the steps below: 1. On the **Overview** section, click the **+ Add Widget** button. 2. On the **Add new widget** window, select the **Application deployed versions** widget type. 3. Select the application to track and click **Add**. 4. View the application widget to get the information about Deployments and Environments where application is deployed. 5. To delete the widget, hover your mouse cursor over the widget block and click the bin icon that appears. ## SonarQube & Dependency-Track Widgets KubeRocketCI also offers widgets to track codebases' code quality directly from the KubeRocketCI portal. These widgets pull codebase-related data from SonarQube and Dependency-Track tools. To enable these widgets, you need to pass the following steps: 1. Integrate platform with [SonarQube](../operator-guide/code-quality/sonarqube.md) and/or [Dependency-Track](../operator-guide/devsecops/dependency-track.md). 2. Trigger the review pipeline in a codebase, allowing SonarQube and Dependency-Track to scan your code. :::note At least one build pipeline must be run for the codebase to activate the widgets. ::: 3. Verify the widgets started working for the codebase. :::note The SonarQube and Dependency-Track widgets only track the default branch. ::: ## Resource Quota Widget The last available widget is a resource quota widget. To open the widget, click the circle icon in the top-right corner of the screen. This widget shows resource requests and limits (CPU, Memory, Namespace) of both the deployments and the entire platform. To enable the widget, you need to deploy KubeRocketCI in a [Capsule](../operator-guide/advanced-installation/capsule.md) tenant. ## Related Articles * [SonarQube Integration](../operator-guide/code-quality/sonarqube.md) * [Integrate Dependency-Track](../operator-guide/devsecops/dependency-track.md) --- ## Advanced AWS EKS Management: Implementing SSO via OIDC and Keycloak In today's cloud-first world, ensuring seamless and secure access to Amazon Elastic Kubernetes Service (EKS) is essential for IT teams. Our guide helps you enhance EKS security by integrating Single Sign-On (SSO) with OpenID Connect (OIDC) and Keycloak. This integration simplifies authentication and strengthens security measures. We aim to provide you with effective strategies to implement a robust SSO solution that meets your organization's standards, making your EKS environment more secure and compliant. KubeRocketCI leverages this integration to provide Role-Based Access Control (RBAC) for your EKS clusters, ensuring that only authorized users can access platform resources. ## Prerequisites Before you begin, ensure you have the following: - A running [AWS EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) cluster with the necessary permissions for access and management. - Forked and cloned the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. - The [kubelogin](https://github.com/int128/kubelogin) plugin installed for authenticating to the EKS cluster using OIDC. - The [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) cli tool installed. - The [aws cli](https://aws.amazon.com/cli/) tool installed. - [Keycloak](../../docs/operator-guide/auth/keycloak) installed and configured with the [kuberocketci-rbac](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) Helm chart, which can be found in the [add-ons repository](https://github.com/epam/edp-cluster-add-ons). - The [Keycloak-operator](https://github.com/epam/edp-keycloak-operator) installed. ## Understanding SSO, OIDC, and Keycloak In the context of enhancing digital security and user experience, we prioritize the integration of three key elements: Single Sign-On (SSO), OpenID Connect (OIDC), and the Keycloak solution. Here’s how they connect: - **Single Sign-On (SSO)** serves as the foundation, enabling users to access multiple applications with one set of login credentials, significantly simplifying the authentication process. - **OpenID Connect (OIDC)** builds on the SSO framework by providing an authentication layer, which uses straightforward identity verification to ensure secure and seamless access across services. - **Keycloak** acts as the orchestrator, implementing both SSO and OIDC to manage user identities and security protocols efficiently. It provides a comprehensive platform for securing applications and services with minimal hassle for end-users. Together, these technologies streamline the login process, reinforce security, and enhance the user experience by allowing secure, seamless navigation across our digital ecosystem. ### What is SSO? Single sign-on (SSO) is a user authentication method that lets you use one set of login credentials (such as a username and password) to access multiple applications. The primary benefits of SSO include an improved user experience by eliminating the need for multiple passwords and logins, and enhanced security through centralized management of user access. Organizations widely adopt SSO to streamline their authentication processes and reduce the likelihood of password fatigue among users, thereby decreasing the risk of security breaches. For more information, see [Single sign-on on Wikipedia](https://en.wikipedia.org/wiki/Single_sign-on). ```mermaid sequenceDiagram participant U as User participant SSO as SSO Gateway participant A1 as Application 1 participant A2 as Application 2 participant A3 as Application 3 U->>SSO: Log in once SSO->>U: Authentication & Session creation Note over U, A3: User accesses applications without re-logging in U->>A1: Access request A1->>SSO: Verify user session SSO->>A1: Session valid A1->>U: Access granted U->>A2: Access request A2->>SSO: Verify user session SSO->>A2: Session valid A2->>U: Access granted U->>A3: Access request A3->>SSO: Verify user session SSO->>A3: Session valid A3->>U: Access granted ``` This diagram shows the following steps: 1. User logs in once at the single sign-on (SSO) gateway by providing their credentials. 2. The SSO gateway authenticates the user, creates a session, and then allows the user to access multiple applications. 3. When the user attempts to access Application 1, the application verifies the user's session with the SSO gateway. 4. The SSO gateway confirms that the session is valid, and Application 1 grants the user access. The same process is repeated for Application 2 and Application 3. Since the user's session is already established with the SSO gateway, they do not need to log in again to access these applications. ### Understanding OIDC OpenID Connect (OIDC) is an authentication layer on top of the OAuth 2.0 protocol. It lets clients verify the identity of the end user based on authentication by an authorization server and get basic profile information about the end user in an interoperable and REST-like manner. OIDC uses JSON Web Tokens (JWTs) to securely transmit information about an end user from the identity provider to the client. This protocol is essential for modern web applications, providing a more secure and streamlined method for user authentication and authorization. Reference: [OIDC Specification](https://openid.net/specs/openid-connect-core-1_0.html) OIDC enables single sign-on (SSO) functionality, simplifying the user experience by allowing individuals to use a single set of credentials across multiple applications. The protocol also supports robust security features, including token revocation and introspection, enhancing overall application security. This diagram simplifies the OIDC flow into its core components: ```mermaid sequenceDiagram participant U as User participant C as Client Application participant AS as Authorization Server participant RS as Resource Server U->>C: Requests access C->>AS: Redirects to Authorization Server for authentication AS->>U: Prompts user for authentication (login) U->>AS: Submits credentials AS->>C: Redirects with authorization code C->>AS: Exchanges authorization code for ID token & access token AS->>C: Returns ID token & access token C->>U: Grants access C->>RS: Accesses resources using access token RS->>C: Returns requested resources ``` 1. **User (U)**: The end user who wants to access the client application. 2. **Client Application (C)**: The application requiring authentication from the user. 3. **Authorization Server (AS)**: The server that authenticates the user and issues tokens to the client application. 4. **Resource Server (RS)**: The server hosting protected resources that the client application wants to access on behalf of the user. The sequence starts with the user requesting access to the client application, moving through authentication with the authorization server, and ending with the client application accessing protected resources. ### Keycloak Overview Keycloak is an open-source identity and access management solution for modern applications and services. It offers features like single sign-on (SSO), social login, and identity brokering, making it a comprehensive solution for managing user identities. Keycloak integrates seamlessly with LDAP (Lightweight Directory Access Protocol) and Active Directory and supports OpenID Connect (OIDC), OAuth 2.0, and Security Assertion Markup Language (SAML) 2.0. By using Keycloak, organizations can enhance their security and provide a better user experience without building complex identity management features from scratch. For more details, see the Keycloak [official documentation](https://www.keycloak.org/documentation.html). ## Keycloak Configuration The first step to enable OIDC authentication to the AWS EKS cluster using Keycloak is to set up Keycloak with the necessary configurations, such as realm, client, groups, and other settings. For this purpose, we will use the [kuberocketci-rbac](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/kuberocketci-rbac) Helm chart available in the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository. 1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository and navigate to the `clusters/core/addons/kuberocketci-rbac` directory. 2. In the `values.yaml` file, set the `kubernetes.enabled` field to `true` to enable the creation of the necessary Keycloak resources: ```yaml title="values.yaml" kubernetes: enabled: true ``` 3. If you use the External Secrets Operator to manage secrets, ensure the AWS Parameter Store object contains the correct Client Secret value for the **keycloak-client-eks-secret** secret: ```json title="AWS Parameter Store object" { ... "keycloak-client-eks-secret": { "clientSecret": "" }, ... } ``` If you are not using the External Secrets Operator, you can create the **keycloak-client-eks-secret** secret manually: ```bash kubectl create secret generic keycloak-client-eks-secret \ --from-literal=clientSecret= ``` 4. Install the **kuberocketci-rbac** Helm chart. You can use the **kubectl** cli tool or **Argo CD** for this purpose. - **kubectl** Ensure you are in the `clusters/core/addons/kuberocketci-rbac` directory if you want to install the chart with the kubectl command. For example: ```bash kubectl upgrade --install kuberocketci-rbac -n security . ``` - **Argo CD** If you are using Argo CD to deploy charts in the **edp-cluster-add-ons** repository, ensure that the following fields for the **kuberocketci-rbac** Helm chart are correctly set in the `values.yaml` file in the `clusters/core/apps` directory: ```yaml title="values.yaml" kuberocketci-rbac: createNamespace: false enable: true ``` After the installation is complete, check that the Keycloak resources, such as the realm, client, and groups, have been created successfully. ## Adding Users to Groups in Keycloak To manage user access to the AWS EKS cluster, you need to assign users to specific groups in Keycloak. These groups will define the permissions for users accessing the AWS EKS cluster. 1. Log in to the Keycloak admin console and navigate to the **shared** realm. 2. Select the **Users** section from the left sidebar menu and select the user you want to add to a group. Navigate to the **Groups** tab and click on the **Join Group** button to add the user to a group. ![Keycloak Add User to Group](../assets/aws-eks-oidc-keycloak/keycloak-add-user-to-group.png) 3. Select the group you want to add the user to (e.g., **oidc-cluster-admins**). Click on the **Join** button to add the user to the selected group. ![Keycloak Join Group](../assets/aws-eks-oidc-keycloak/keycloak-join-group.png) 4. Repeat the process of adding users to groups for all users who need access to the AWS EKS cluster. ## Configuring Keycloak as an Identity Provider in AWS EKS :::note Updated (2026-07-05) This section now also sets **Username Claim**/`username_claim` to `preferred_username` and **Username Prefix**/`username_prefix` to `-`. Without a username claim, Kubernetes falls back to the `sub` claim prefixed with the issuer URL (e.g. `https:///auth/realms/shared#`), which is hard to attribute to a real user in `kubectl` output and audit logs. `preferred_username` is recommended over `email`, since Kubernetes' OIDC authenticator rejects tokens where `email_verified` is `false` — common for users brokered from a SAML upstream IdP (e.g. Azure AD), which has no `email_verified` equivalent to assert. RBAC is unaffected either way, since this setup authorizes by group membership, not username. ::: There are two methods to configure Keycloak as an identity provider in AWS EKS: using the AWS Management Console or Terraform. ### Method 1: Using the AWS Management Console 1. Log in to the [AWS Management Console](https://aws.amazon.com/console/) and navigate to the Amazon EKS service. Select the EKS cluster you want to configure and click on the **Access** tab. ![EKS Cluster Access Tab](../assets/aws-eks-microsoft-entra-oidc-integration/eks-cluster-access-tab.png) 2. In the **OIDC identity providers** section, click on the **Associate identity provider** button. ![Associate Identity Provider](../assets/aws-eks-microsoft-entra-oidc-integration/associate-identity-provider.png) 3. Fill in the following details for the Keycloak Identity Provider: - **Name**: `Keycloak` - **Issuer URL**: `https:///auth/realms/shared`, where `` is the URL of your Keycloak instance. - **Client ID**: `eks`. - **Groups Claim**: `groups`. - **Username Claim**: `preferred_username`. - **Username Prefix**: `-`. ![Identity Provider Details](../assets/aws-eks-microsoft-entra-oidc-integration/identity-provider-details.png) 4. The process of applying the changes may take a few minutes. Once completed, you will see the Keycloak Identity Provider associated with your EKS cluster. ### Method 2: Using Terraform To configure Keycloak as an Identity Provider in AWS EKS cluster using Terraform, you can use the [AWS EKS Terraform module](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/20.30.1). Here's an example of how to define the Keycloak Identity Provider in Terraform configuration files: - **variables.tf**: ```hcl variable "cluster_identity_providers" { description = "Configuration for OIDC identity provider" type = any default = {} } ``` - **terraform.tfvars**: ```hcl cluster_identity_providers = { keycloak = { client_id = "eks" issuer_url = "https:///auth/realms/shared" groups_claim = "groups" username_claim = "preferred_username" username_prefix = "-" } } ``` - **main.tf**: ```hcl module "eks" { source = "terraform-aws-modules/eks/aws" version = "20.14.0" ... # OIDC Identity provider cluster_identity_providers = var.cluster_identity_providers } ``` After applying the Terraform configuration, the Keycloak identity provider will be associated with your EKS cluster. ## Authenticating to AWS EKS cluster using kubectl 1. Configure **kubeconfig** file to use the Keycloak Identity Provider for authentication to the AWS EKS cluster. You can use the following template: ```yaml title="kubeconfig" apiVersion: v1 preferences: {} kind: Config clusters: - cluster: server: https://.eks.amazonaws.com certificate-authority-data: name: eks contexts: - context: cluster: eks user: name: eks current-context: eks users: - name: user: exec: apiVersion: client.authentication.k8s.io/v1beta1 command: kubectl args: - oidc-login - get-token - -v1 - --oidc-issuer-url=https:///auth/realms/shared - --oidc-client-id=eks - --oidc-client-secret= ``` Replace the placeholders with the actual values: - ``: The endpoint of your AWS EKS cluster. - ``: The CA certificate of your AWS EKS cluster. - ``: The email address of the Keycloak user. - ``: The URL of your Keycloak instance. - ``: The Client secret of the **eks** Keycloak client (provided during the **Keycloak Configuration** step). 2. Save the kubeconfig file and set the `KUBECONFIG` environment variable to point to the file: ```bash export KUBECONFIG= ``` 3. Test the authentication to the AWS EKS cluster by running the following command: ```bash kubectl get nodes ``` After the first command execution, you will be prompted to log in to Keycloak. Enter your credentials to authenticate and access the EKS cluster. If the authentication is successful, you will see the list of nodes in the EKS cluster. ## Configuring KubeRocketCI Portal with Keycloak OIDC Authentication The KubeRocketCI platform natively supports Keycloak as an Identity Provider for OIDC authentication. 1. To configure the KubeRocketCI Portal with Keycloak OIDC authentication, navigate to the [edp-install](https://github.com/epam/edp-install) Helm chart and set the following values in the `values.yaml` file: ```yaml title="values.yaml" edp-headlamp: enabled: true config: oidc: enabled: true issuerUrl: "https:///auth/realms/shared" clientID: "eks" clientSecretName: "keycloak-client-headlamp-secret" clientSecretKey: "clientSecret" ``` Replace the **keycloak_url** with the URL of your Keycloak instance. 2. Ensure the AWS Parameter Store object contains the correct Client Secret value for the **keycloak-client-headlamp-secret** secret: ```json title="AWS Parameter Store object" { ... "keycloak-client-headlamp-secret": "" ... } ``` 3. After setting the values, install the **edp-install** Helm chart to apply the changes: ```bash helm upgrade --install krci --namespace krci . ``` 4. After applying the changes, the KubeRocketCI Portal will be configured to use Keycloak OIDC authentication. Users will be able to log in to the Portal using **Sign In** option. ![Sign In](../assets/aws-eks-microsoft-entra-oidc-integration/sign-in.png) ## Conclusion Integrating OpenID Connect (OIDC) authentication with Keycloak in AWS EKS enhances security and simplifies user access management. By leveraging Keycloak's capabilities, you can implement a reliable Single Sign-On (SSO) solution that meets your organization's security standards. This guide has provided step-by-step instructions to configure Keycloak as an Identity Provider, set up necessary Keycloak resources, and enable OIDC authentication for the KubeRocketCI Portal. By following these steps, you can ensure secure and seamless access to your EKS clusters and KubeRocketCI Portal, improving both security and user experience. --- ## Integrating OIDC Authentication with Microsoft Entra in AWS EKS # Integrating OIDC Authentication With Microsoft Entra in AWS EKS In modern cloud environments, secure and efficient access management is essential, especially for platforms like Amazon EKS. This blog will guide you through integrating OpenID Connect (OIDC) authentication using Microsoft Entra, making it easier to manage access to your EKS clusters and KubeRocketCI Portal. By implementing this approach, you can simplify user authentication while ensuring strong security controls. Whether you're improving compliance or streamlining access for your team, this integration is a practical solution to enhance your cloud-native workflows. :::info **Updated: July 2026** — Links verified and integration steps re-checked against the current Microsoft Entra admin center. ::: ## Prerequisites Before you begin, ensure you have the following: - Access to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) with administrative privileges. - A running [AWS EKS](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) cluster with the necessary permissions for access and management. - The [kubelogin](https://github.com/int128/kubelogin) plugin installed for authenticating to the EKS cluster using OIDC. - The [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) CLI tool installed. - The [aws cli](https://aws.amazon.com/cli/) tool installed. If you use Keycloak instead of Microsoft Entra, see [Keycloak OIDC for EKS](/docs/operator-guide/auth/configure-keycloak-oidc-eks). ## Understanding SSO, OIDC, and Microsoft Entra In the context of enhancing digital security and user experience, we prioritize the integration of three key elements: Single Sign-On (SSO), OpenID Connect (OIDC), and Microsoft Entra. Here’s how they connect: - **Single Sign-On (SSO)** serves as the foundation, enabling users to access multiple applications with one set of login credentials, significantly simplifying the authentication process. - **OpenID Connect (OIDC)** builds on the SSO framework by providing an authentication layer, which uses straightforward identity verification to ensure secure and seamless access across services. - **Microsoft Entra** (formerly known as **Azure Active Directory**) is Microsoft's comprehensive identity and access management solution. It supports the implementation of both Single Sign-On (SSO) and OpenID Connect (OIDC), enabling organizations to securely manage user identities and enforce access controls. With its reliable set of tools, Microsoft Entra simplifies authentication, enhances security, and ensures seamless access to applications and services, making it an essential platform for modern identity management. Together, these technologies streamline the login process, reinforce security, and enhance the user experience by allowing secure, seamless navigation across our digital ecosystem. ### Microsoft Entra Overview Microsoft Entra, formerly known as Azure Active Directory (Azure AD), is a modern identity and access management solution designed for secure access to applications and services. It provides features like Single Sign-On (SSO), identity federation, and seamless integration with on-premises directories such as LDAP and Active Directory. Microsoft Entra supports industry-standard protocols, including OpenID Connect (OIDC), OAuth 2.0, and Security Assertion Markup Language (SAML) 2.0, making it a versatile solution for managing user identities. By leveraging Microsoft Entra, organizations can enhance security, simplify user access, and avoid the complexities of building identity management features from scratch. For more details, see the [Microsoft Entra official documentation](https://learn.microsoft.com/en-gb/entra/identity/). ## Create a new Microsoft Entra Tenant To get started with Microsoft Entra, you need to create a new tenant in the Microsoft Entra Admin Center. Follow these steps: 1. Log in to the [Microsoft Entra Admin Center](https://entra.microsoft.com/) using your Microsoft account. ![Microsoft Entra Admin Center](../assets/aws-eks-microsoft-entra-oidc-integration/microsoft-entra-admin-center.png) 2. In the left sidebar menu, select **Overview** section and then navigate to **Manage tenants** tab. ![Manage Tenants](../assets/aws-eks-microsoft-entra-oidc-integration/manage-tenants.png) 3. Click on **Create** button to create a new tenant. ![Create Tenant](../assets/aws-eks-microsoft-entra-oidc-integration/create-tenant.png) 4. Select the configuration type **Workforce**. ![Configuration Type](../assets/aws-eks-microsoft-entra-oidc-integration/configuration-type.png) 5. Fill in the fields for **Tenant Name**, **Domain Name**, and **Location**. :::note The **Tenant Name** and **Domain Name** `kuberocketci` are used as a demonstration examples. In your case, it is recommended to choose names that align with your organization's specific needs and naming conventions. ::: ![Tenant Details](../assets/aws-eks-microsoft-entra-oidc-integration/tenant-details.png) 6. The new tenant will be created, and you can start configuring it for OIDC integration. Ensure you have switched to the new tenant. ## Creating and Configuring OIDC Application in Microsoft Entra After creating the tenant, you need to set up an OIDC Application in Microsoft Entra. Here's how you can do it: 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Applications** and then click on **App registrations**. ![App Registrations](../assets/aws-eks-microsoft-entra-oidc-integration/app-registrations.png) 2. Click on the **New registration** button to create a new application. ![New Registration](../assets/aws-eks-microsoft-entra-oidc-integration/new-registration.png) 3. Fill in the details for the application, such as **Name**, **Supported account types**, and **Redirect URI** (`http://localhost:8000/`). :::note The **Name** `kuberocketci` is used as a demonstration example. In your case, it is recommended to choose a name that aligns with your application's specific needs and naming conventions (e.g. your AWS EKS cluster name). ::: ![Application Details](../assets/aws-eks-microsoft-entra-oidc-integration/application-details.png) 4. In the created application, navigate to the **Authentication** section from the left sidebar menu. In the **Implicit grant and hybrid flows** section, select **ID tokens** for the token type. In the **Allow public client flows** section, set the value to **No**. ![Authentication Settings](../assets/aws-eks-microsoft-entra-oidc-integration/authentication-settings.png) 5. Navigate to the **Certificates & secrets** section from the left sidebar menu. In the **Client secrets** tab, click on the **New client secret** button to create a new secret. ![Client Secret](../assets/aws-eks-microsoft-entra-oidc-integration/client-secret.png) 6. Copy the generated client secret value and store it securely. ![Client Secret Value](../assets/aws-eks-microsoft-entra-oidc-integration/client-secret-value.png) 7. Navigate to the **Token configuration** section and click on **Add group claim** button. Choose the group type as **Security Groups** and for the ID token type, select **Group ID**. ![Token Configuration](../assets/aws-eks-microsoft-entra-oidc-integration/token-configuration.png) 8. (Optional) Additionally, add an optional **upn** claim in the **Token configuration** section. :::note This step is optional and should only be performed if external users (i.e., users with **User type: Guest**) will be added to the Microsoft Entra Tenant and require access to the Application. ::: ![upn Claim](../assets/aws-eks-microsoft-entra-oidc-integration/upn-claim.png) 9. (Optional) After adding the **upn** claim, click on the three dots next to it, select **Edit**, and turn on the **Externally authenticated** toggle to **Yes** value. :::note This step is optional and should only be performed if external users (i.e., users with **User type: Guest**) will be added to the Microsoft Entra Tenant and require access to the Application. ::: ![Externally Authenticated](../assets/aws-eks-microsoft-entra-oidc-integration/externally-authenticated.png) 10. Navigate to the **API permissions** section. Ensure that the **User.Read** permission is added under the **Microsoft Graph** API. If not, click on the **Add a permission** button, select **Microsoft Graph**, and add the **User.Read** permission. After adding the permission, click on the **Grant admin consent for 'Tenant name'** button to grant the required permissions. ![API Permissions](../assets/aws-eks-microsoft-entra-oidc-integration/api-permissions.png) 11. (Optional) Additionally, for **Microsoft Graph** API, add the **OpenId** permissions, such as **openid**, **email**, and **profile**. :::note This step is optional and should only be performed if external users (i.e., users with **User type: Guest**) will be added to the Microsoft Entra Tenant and require access to the Application. ::: ![OpenID Permissions](../assets/aws-eks-microsoft-entra-oidc-integration/openid-permissions.png) 12. The OIDC Application in Microsoft Entra is now configured and ready for integration with AWS EKS. ## Creating Users and Groups in Microsoft Entra :::note Only users who are part of the groups configured in the Microsoft Entra Admin Center will be able to authenticate to the AWS EKS cluster using OIDC. ::: To create users and groups in Microsoft Entra, follow these steps: ### Creating a Group 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Groups** and then **All groups**. Click on **New group** button to create a new group(s) for users who will have access to the AWS EKS cluster. ![New Group](../assets/aws-eks-microsoft-entra-oidc-integration/new-group.png) 2. Fill in the details for the group, such as **Group type** and **Group name**. Click on the **Create** button to create the group. ![Group Details](../assets/aws-eks-microsoft-entra-oidc-integration/group-details.png) 3. The group will be created, and you can start adding users to it. ### Adding Users to the Group 1. In the Microsoft Entra Admin Center, in the left sidebar menu, select **Users** and then click on **All users**. In the **New user** tab, click on the **Create new user** button to create a new user. ![New User](../assets/aws-eks-microsoft-entra-oidc-integration/new-user.png) 2. Fill in the details for the user, such as **User principal name**, **Mail nickname**, **Display name**, and temporary password. In the **Properties** tab you can set the **First name**, **Last name**, and other details. ![User Details](../assets/aws-eks-microsoft-entra-oidc-integration/user-details.png) 3. In the **Assignment** tab, click on the **Add group** button. In the **Select Group** window, choose the group(s) you created earlier (e.g., `oidc-cluster-admins`) and click on the **Select** button. ![Add Group](../assets/aws-eks-microsoft-entra-oidc-integration/add-group.png) 4. Click on the **Review + create** button to create the user. The user will be created and added to the group(s) you selected. ## Configuring Microsoft Entra as an Identity Provider in AWS EKS There are two methods to configure Microsoft Entra as an Identity Provider in AWS EKS: through the AWS Management Console and using Terraform. :::note The Application data, such as **Directory (tenant) ID**, **Application (client) ID**, and **Issuer URL**, can be found in the **Overview** section of the OIDC Application in the Microsoft Entra Admin Center. ![Application Data](../assets/aws-eks-microsoft-entra-oidc-integration/application-data.png) ::: ### Method 1: Using the AWS Management Console 1. Log in to the [AWS Management Console](https://aws.amazon.com/console/) and navigate to the [Amazon EKS console](https://console.aws.amazon.com/eks). Select the EKS cluster you want to configure and click on the **Access** tab. ![EKS Cluster Access Tab](../assets/aws-eks-microsoft-entra-oidc-integration/eks-cluster-access-tab.png) 2. In the **OIDC identity providers** section, click on the **Associate identity provider** button. ![Associate Identity Provider](../assets/aws-eks-microsoft-entra-oidc-integration/associate-identity-provider.png) 3. Fill in the following details: - **Name**: `Entra` - **Issuer URL**: `https://login.microsoftonline.com//`, where `` is the **Directory** (tenant) **ID**. Ensure that the URL ends with `/`. - **Client ID**: ``, which corresponds to the **Application** (client) **ID** of the OIDC Application. - **Username Claim**: `upn`. - **Groups Claim**: `groups`. ![Identity Provider Details](../assets/aws-eks-microsoft-entra-oidc-integration/identity-provider-details.png) 4. The process of applying the changes may take a few minutes. Once completed, the Microsoft Entra OIDC identity provider will be associated with the AWS EKS cluster. ### Method 2: Using Terraform To configure Microsoft Entra as an Identity Provider in AWS EKS using Terraform, you can use [AWS EKS Terraform module](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/20.30.1). Here's an example of how you can do it: - **variables.tf**: ```hcl variable "cluster_identity_providers" { description = "Configuration for OIDC identity provider" type = any default = {} } ``` - **terraform.tfvars**: ```hcl cluster_identity_providers = { entra = { client_id = "" issuer_url = "https://sts.windows.net//" groups_claim = "groups" username_claim = "upn" } } ``` - **main.tf**: ```hcl module "eks" { source = "terraform-aws-modules/eks/aws" version = "20.14.0" ... # OIDC Identity provider cluster_identity_providers = var.cluster_identity_providers } ``` :::note You may notice the issuer URL format differs between methods: the AWS Console example above uses `https://login.microsoftonline.com//` (Entra v2), while this Terraform example uses `https://sts.windows.net//` (the legacy Entra v1 format). Both are valid, but the issuer you configure must match the `iss` claim of the tokens your app registration actually issues, so check the app's manifest `accessTokenAcceptedVersion` setting before choosing one. ::: After applying the Terraform configuration, the Microsoft Entra OIDC identity provider will be associated with the AWS EKS cluster. ## Configuring RBAC Resources in AWS EKS cluster for Microsoft Entra User Groups In this section, user authorization will be configured using Kubernetes Role-Based Access Control (RBAC). Microsoft Entra groups will be linked to Kubernetes ClusterRoles through ClusterRoleBinding resources, enabling precise control over resource access within the EKS cluster. Additionally, Roles and RoleBindings can be used for more granular access control within specific namespaces. :::note The **Object ID** of the Microsoft Entra group can be found in the **Overview** section of the group in the Microsoft Entra Admin Center. ![Group Object ID](../assets/aws-eks-microsoft-entra-oidc-integration/group-object-id.png) ::: 1. Log in to the AWS EKS cluster and create the following **ClusterRoleBinding** resource, which associates the Microsoft Entra group `oidc-cluster-admins` with the `cluster-admin` Kubernetes Cluster Role. Replace `` with the Object ID of the `oidc-cluster-admins` group, which can be found on the `Group` overview page in the Microsoft Entra admin center. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: oidc-cluster-admins roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: Group name: apiGroup: rbac.authorization.k8s.io ``` Save the above YAML to a file, for example, `clusterrolebinding.yaml`, and apply it to the EKS cluster using the following command: ```bash kubectl apply -f clusterrolebinding.yaml ``` The `oidc-cluster-admins` group will now have `cluster-admin` permissions within the EKS cluster. ## Authenticating to AWS EKS using Microsoft Entra with kubectl To authenticate to the AWS EKS cluster using Microsoft Entra, you can use the `kubectl` CLI tool with the `kubelogin` plugin. The `kubelogin` plugin simplifies the OIDC authentication process by handling the token exchange and session management. Here's how you can authenticate to the EKS cluster: 1. Create or update the kubeconfig file with the OIDC configuration. Replace `` with the name of your EKS cluster and `` with the AWS region code where the cluster is located. ```bash aws eks update-kubeconfig --region --name ``` 2. Execute the following command to create a new kubeconfig context using the `kubelogin` plugin. Replace `` with the name of your EKS cluster. ```bash kubectl config set-credentials "eks" \ --exec-api-version=client.authentication.k8s.io/v1beta1 \ --exec-command=kubelogin \ --exec-arg=get-token \ --exec-arg=--oidc-issuer-url \ --exec-arg=https://sts.windows.net// \ --exec-arg=--oidc-client-id \ --exec-arg= \ --exec-arg=--oidc-client-secret \ --exec-arg= ``` Replace ``, ``, and `` with the corresponding values from the OIDC Application in the Microsoft Entra Admin Center. :::note You can test the authentication to the EKS cluster immediately by running the following command: ```bash kubectl --user=eks get nodes ``` ::: 3. Set the context for the kubeconfig file to use the `eks` user and the OIDC configuration. Replace `` with the name of your EKS cluster. ```bash kubectl config set-context eks --user=eks --cluster= ``` To switch to the `eks` context, execute the following command: ```bash kubectl config use-context eks ``` Test the authentication by running the following command: ```bash kubectl get nodes ``` ## Configuring KubeRocketCI Portal with Microsoft Entra OIDC Authentication 1. Starting from version 3.10, KubeRocketCI platform supports Microsoft Entra as an Identity Provider for OIDC authentication in the Portal UI. To configure Microsoft Entra OIDC authentication, navigate to the [edp-install](https://github.com/epam/edp-install) Helm chart repository and set the following values in the `values.yaml` file: ```yaml ... edp-headlamp: enabled: true config: oidc: enabled: true issuerUrl: "https://sts.windows.net//" clientID: "" clientSecretName: "" clientSecretKey: "clientSecret" ... ``` Replace `` and `` with the corresponding values from the OIDC Application in the Microsoft Entra Admin Center. Also, specify the name of the Kubernetes Secret containing the Application **Client secret** value in the `clientSecretName` field. 2. In the Microsoft Entra Admin Center, navigate to the created OIDC Application and select the **Authentication** section. In the **Redirect URIs** field, add the URL of the KubeRocketCI Portal, for example, `https://portal-./oidc-callback`. ![Redirect URIs](../assets/aws-eks-microsoft-entra-oidc-integration/redirect-uris.png) 3. After applying the changes, the KubeRocketCI Portal will be configured to use Microsoft Entra OIDC authentication. Users will be able to log in to the Portal using **Sign In** option. ![Sign In](../assets/aws-eks-microsoft-entra-oidc-integration/sign-in.png) If the Portal sits behind an OAuth2 proxy, see [OAuth2 Proxy](/docs/operator-guide/auth/oauth2-proxy) for the general setup and [Microsoft Entra OAuth2 Proxy Authentication](/docs/operator-guide/microsoft-entra/oauth2-proxy-authentication) for the Entra-specific configuration. ## Conclusion Integrating OpenID Connect (OIDC) authentication with Microsoft Entra in AWS EKS is a powerful way to enhance security and streamline user access management. By leveraging the capabilities of SSO, OIDC, and Microsoft Entra, organizations can simplify authentication processes, enforce access controls, and ensure secure navigation across cloud-native environments. Whether you're managing user identities, enhancing compliance, or improving user experience, this integration provides a robust solution to meet your identity and access management needs. By following the steps outlined in this guide, you can configure Microsoft Entra as an Identity Provider in AWS EKS, authenticate users using OIDC, and secure access to your EKS clusters and KubeRocketCI Portal effectively. **Related reading:** - [Advanced AWS EKS Management: Implementing SSO via OIDC and Keycloak](/blog/advanced-aws-eks-management-oidc-keycloak) - the same OIDC pattern for AWS EKS, using Keycloak as the identity provider instead of Microsoft Entra. - [Keycloak OIDC for EKS](/docs/operator-guide/auth/configure-keycloak-oidc-eks) - the docs reference for the Keycloak-based setup. - [OAuth2 Proxy](/docs/operator-guide/auth/oauth2-proxy) and [Microsoft Entra OAuth2 Proxy Authentication](/docs/operator-guide/microsoft-entra/oauth2-proxy-authentication) - securing the KubeRocketCI Portal behind an OAuth2 proxy. --- ## Kubernetes-Native CI/CD with Tekton # Kubernetes-Native CI/CD with Tekton | KubeRocketCI Building CI/CD on Kubernetes used to mean running Jenkins or GitLab CI in a pod and calling it done. [Tekton](https://tekton.dev/docs/) changed that by making pipelines first-class Kubernetes objects - Tasks and Pipelines are CRDs, PipelineRuns are namespaced resources, and every step log is a container log. KubeRocketCI goes a step further: it ships a complete, production-grade CI/CD platform on top of Tekton so your team gets sensible defaults, a portal UI, GitOps-managed pipeline definitions, and opinionated quality gates - without the months of plumbing work that comes with assembling those pieces from scratch. I've seen teams go from a bare cluster to a working build-deploy loop in under a day using this stack. Here's how the layering works in practice - from Tekton's native primitives to the Helm-templated pipeline library, the webhook-to-PipelineRun chain, each pipeline type, and the portal UI that surfaces it all. ## Why Does Kubernetes-Native CI/CD Matter? Traditional CI servers are external systems that *talk to* Kubernetes. They authenticate, schedule jobs, wait for responses, and manage their own scaling independently of the cluster. Tekton turns that model inside out - the cluster *is* the CI engine. Every pipeline run is a pod, every step is a container, and the Kubernetes control plane handles scheduling, scaling, and secrets injection natively. The practical consequences: - **No CI server to maintain.** No Jenkins controller HA setup, no separate runner fleet. Tekton scales horizontally with your cluster. - **Uniform RBAC.** Pipeline permissions use Kubernetes ServiceAccounts - no separate credential management UI. - **Pipeline as code.** Pipeline definitions are YAML manifests, version-controlled alongside application code, and applied to any conformant Kubernetes cluster identically. - **Native secret injection.** Tekton Tasks consume Kubernetes Secrets and ConfigMaps directly - no plugin abstraction required. - **Shift-left by default.** Code quality, security scanning, and test gates run inside the cluster as ordinary containers, co-located with the workloads they protect. ## KubeRocketCI and the Tekton Stack [KubeRocketCI](/docs/about-platform) is an open-source, cloud-agnostic SaaS/PaaS platform built on [Tekton](https://tekton.dev/docs/) - a CNCF Graduated project - developed and maintained by EPAM Systems. Where Tekton gives you primitives, KubeRocketCI gives you a working system. The platform deploys and manages the full Tekton component set: | Component | Role | |---|---| | **Tekton Pipelines** | Executes all CI and CD pipeline runs | | **Tekton Triggers** | Listens for Git webhook events and fires the matching pipeline | | **Tekton Interceptors** | Enriches and filters GitHub, GitLab, and Bitbucket payloads before routing | | **Tekton Chains** | Signs pipeline artifacts for supply-chain provenance | | **Tekton Results** | Stores run history for audit trails and portal display | On top of Tekton, KubeRocketCI adds: - **Pipeline library** - Helm-packaged Tasks and Pipelines for Java, Go, Node.js, Python, .NET, Helm, Terraform, and more. - **Portal UI** - full pipeline management with DAG visualization, live log streaming, and inline manual approval gates. - **Argo CD integration** - pipeline definitions are stored in Git and synchronized by Argo CD. Every change to a pipeline is a reviewed, audited commit. See [Basic Concepts](/docs/basic-concepts) and the [Tekton Overview](/docs/user-guide/tekton-pipelines) for the full capability matrix. ## How Are Tekton Pipelines Defined in KubeRocketCI? ### The Helm-Templated Library Pipelines in KubeRocketCI are not hand-written one-offs - they are generated by Helm from a structured library in [edp-tekton](https://github.com/epam/edp-tekton). The naming convention follows: ``` {gitProvider}-{buildTool}-{framework}-app-{pipelineType}-{versioning} ``` For example, a Java 17 Maven build pipeline triggered from GitHub with default versioning renders as: ``` github-maven-java17-app-build-default ``` The same pipeline for GitLab becomes `gitlab-maven-java17-app-build-default`. This pattern means every combination of git provider, language, framework, and versioning strategy gets a consistently named, independently configurable pipeline - without duplicating task logic. Shared task sequences live in common Helm templates (e.g., `_common_java_maven.yaml`) and are included by reference. ### The Real Build Task Sequence For a Java Maven application, the build pipeline executes these tasks in order: ``` get-version └─ update-build-number └─ get-cache └─ compile └─ test (JaCoCo coverage) └─ sonar (quality gate: sonar.qualitygate.wait=true) └─ build (mvn clean package -DskipTests) └─ push (mvn deploy -DskipTests) ``` The `sonar` step blocks with `sonar.qualitygate.wait=true` - the build fails fast at the quality gate rather than discovering issues downstream. In my experience, this single flag is the difference between catching technical debt in minutes versus after a release candidate is already tagged. Code that doesn't pass SonarQube never produces an artifact. Each task is a reusable Tekton `Task` CRD (`maven`, `sonarqube-maven`) - the pipeline only sets parameters and `runAfter` ordering. Swap the `maven` task image to change the JDK version; the pipeline logic stays the same. ### The Webhook-to-PipelineRun Chain When a pull request is merged on GitHub, the event travels through four Tekton resources before a PipelineRun starts: ```mermaid flowchart LR A[GitHub Webhook] --> B[EventListener] B --> C[Custom Interceptor] C --> D[TriggerBinding] D --> E[TriggerTemplate] E --> F[PipelineRun] ``` The **Custom Interceptor** is the key piece. It enriches the raw GitHub payload with platform-specific `extensions` that the webhook payload alone doesn't contain: - `extensions.codebase` - the KubeRocketCI codebase name for this repository - `extensions.codebasebranch` - the branch object with its pipeline configuration - `extensions.pipelines.build` - the exact pipeline name to run (e.g., `github-maven-java17-app-build-default`) - `extensions.pullRequest.headSha`, `extensions.pullRequest.author`, `extensions.pullRequest.url` The **TriggerBinding** then extracts these into named parameters: `gitrevision`, `targetBranch`, `gitsha`, `codebase`, `codebasebranch`, `pipelineName`, `commitMessagePattern`, and Jira integration fields. The **TriggerTemplate** stamps these into a PipelineRun spec and creates it in the cluster. This architecture means the decision of *which pipeline runs for which codebase* is stored in the platform's Kubernetes objects - not hardcoded in webhook configuration. ## Pipeline Types: From Code Review to Release KubeRocketCI defines seven pipeline types, identified by the `app.edp.epam.com/pipelinetype` label. All are implemented as Tekton Pipelines. ### Review Pipeline (`pipelinetype: review`) Triggered by a pull request. Runs static analysis, linting, unit tests, and code-quality gates before a merge is allowed. Typically completes in under five minutes. Trigger methods: open a PR against a configured branch, comment `/recheck` or `/ok-to-test`, or use the **Run Again** button in the portal. ### Build Pipeline (`pipelinetype: build`) Triggered on merge to the target branch. Executes the full task sequence (see above), builds and pushes a container image, and produces a versioned artifact. Versioning follows either `BRANCH-[DATETIME]` (default) or SemVer (MAJOR.MINOR.PATCH-BUILD_ID) depending on project configuration. ### Deploy Pipeline (`pipelinetype: deploy`) Takes a versioned artifact and applies it to a target environment. Trigger modes: manual (via portal), `Auto` (deploy automatically on new artifact), or `Auto-stable` (promote only the rebuilt component, keep others at stable versions). ### Clean Pipeline (`pipelinetype: clean`) Tears down an environment on demand. Paired with `Auto` deploys and dynamic environments, this enables full ephemeral workflows - spin up for a PR, validate, clean up on merge. ### Security Pipeline (`pipelinetype: security`) A standalone pipeline decoupled from build, purpose-built for vulnerability scanning. Scan results surface in the portal's **Results** tab and link through to DefectDojo. Decoupling security means scans can run at any cadence without affecting build cycle time. ### Test Pipeline (`pipelinetype: tests`) Executes automated test suites against already-deployed environments, independent of the deploy cycle. Teams validate application behavior on demand without triggering a full redeploy. ### Release Pipeline (`pipelinetype: release`) Orchestrates approval and publishing for new releases. KubeRocketCI does not ship a pre-built release pipeline - release governance is organization-specific. The full Tekton API surface is available to build custom approval steps, external system integrations, and auditable release flows. The complete pipeline type reference is in [Pipelines Overview](/docs/user-guide/pipelines) and [Tekton Overview](/docs/user-guide/tekton-pipelines). ## Customizing Pipelines The pre-built library covers most tech stacks, but production workloads always have exceptions. KubeRocketCI supports two customization patterns without forking the platform. ### Custom Framework / Build Tool Define a custom pipeline once with a naming pattern, and every project matching that pattern picks it up automatically. The [Tekton custom pipelines use case](/docs/use-cases/tekton-custom-pipelines) walks through the full authoring flow. ### Branch-Specific Pipelines When different branches need different logic - `main` with full integration tests, `release` with image signing - pipelines are defined explicitly per branch. See the [custom pipelines flow guide](/docs/use-cases/custom-pipelines-flow#replace-pipelines-with-personalized-versions). Both approaches follow the same Git-first workflow: write Tekton YAML, apply to the cluster to verify, commit, and let Argo CD synchronize the authoritative state. ## The Portal: Pipeline Management UI The KubeRocketCI portal ships a dedicated `tekton` module with full pipeline management across four page areas: pipeline list, pipeline details, PipelineRun list, and PipelineRun details. **Pipeline list with type filter.** The pipeline list filters by `app.edp.epam.com/pipelinetype` label - select `build`, `review`, `deploy`, or any of the seven types to narrow the view. Each pipeline shows its type, last run status, and a **Run with params** button. **Run with params.** Clicking **Run with params** fetches the `TriggerTemplate` linked to the pipeline (via the `app.edp.epam.com/triggertemplate` label), generates a PipelineRun draft pre-filled with correct parameters, opens a YAML editor for review, and creates the PipelineRun resource via the Kubernetes API on save. No `kubectl` required. **DAG visualization.** The pipeline details page renders the task graph using React Flow - nodes for each task, directed edges from `runAfter` dependencies, isolated tasks shown separately. Colors map to live status: blue (running), green (succeeded), red (failed), amber (pending). Finally-tasks are distinguished visually from the main execution chain. ![KubeRocketCI portal DAG view of a review pipeline - tasks for sonar, helm-lint, dockerfile-lint, and GitHub status reporting](../assets/dag-pipeline-visualization.png) **Live logs and history.** The PipelineRun details page streams live container logs while a run is active. After completion, logs are served from Tekton Results - the same view, no separate log aggregation setup needed. **Inline manual approval.** When a pipeline step is an `ApprovalTask` (a platform-native CRD), the portal renders Approve / Reject buttons directly in the task view - with an optional comment field. The approval decision (`spec.action`, `spec.approve.approvedBy`, `spec.approve.comment`) is written back to the `ApprovalTask` resource. This is how release gates and environment promotion approvals are implemented as Kubernetes-native objects. ## Getting Started ### Step 1 - Install the platform Follow the [Quick Start: Install KubeRocketCI](/docs/quick-start/platform-installation) guide for a local cluster. For production on AWS, see [Deploy on AWS EKS](/docs/operator-guide/deploy-aws-eks). ### Step 2 - Install Tekton components The [Install Tekton](/docs/operator-guide/install-tekton) guide covers both vanilla Kubernetes and OKD/OpenShift (Tekton Operator). For the fully automated path, see [Add-Ons Overview](/docs/operator-guide/add-ons-overview). ### Step 3 - Connect your Git provider Walk through [Integrate GitHub](/docs/quick-start/integrate-github) or configure GitLab/Bitbucket in the portal under **Git Servers**. ### Step 4 - Integrate container registry and code quality [Integrate DockerHub](/docs/quick-start/integrate-container-registry) wires up image push. [Integrate SonarQube](/docs/quick-start/integrate-sonarcloud) activates the quality gate step in the build pipeline. ### Step 5 - Onboard your first application [Create Application](/docs/quick-start/create-application) registers your codebase. The platform provisions a review and build pipeline for the default branch automatically - the Helm-templated library selects the correct pipeline name based on your tech stack. ### Step 6 - Configure a deployment environment [Deploy Application](/docs/quick-start/deploy-application) connects Argo CD and configures the first CD pipeline stage. [Integrate Argo CD](/docs/quick-start/integrate-argocd) covers the Argo CD setup. From a fresh cluster to a working build-and-deploy loop typically takes a few hours. ## What Every Onboarded App Gets Automatically Once the platform is running, every onboarded application has: - A **review pipeline** on every pull request - compile, test, SonarQube quality gate. - A **build pipeline** on every merge - versioned artifact, container image pushed, Tekton Chains provenance attestation. - A **CD pipeline** connected to Argo CD - promoting artifacts through configured stages. - **Pipeline definitions in Git** - every change to a pipeline goes through code review, synchronized by Argo CD. - **Live + archived logs** - streaming during a run, served from Tekton Results after completion. - **Manual approval gates** - inline approve/reject in the portal for any pipeline step that requires human sign-off. This is what our own dogfooding instance looks like - we run KubeRocketCI on KubeRocketCI itself. Over 90 days, the `krci` namespace logged **18,397 pipeline runs** (~200/day across build, review, and deploy types), with build pipelines hitting a **93% success rate** at an average duration of **14 minutes 38 seconds**. No external CI server involved. ![KubeRocketCI Pipeline Metrics: 18,397 runs over 90 days, 74% success rate, 93% build success rate, 14m 38s average duration](../assets/pipeline-metrics.png) The entire path - review pipeline on PR open, build pipeline on merge, artifact versioning, image push, and Argo CD sync - runs without the developer writing a single line of Jenkins Groovy or pipeline YAML. The only required input is declaring the tech stack at onboarding time. No manual wiring required. The opinionated defaults deliver a production-grade setup from day one, and the full Tekton API surface is available when you need to diverge. ## Next Steps - Explore the [Tekton Overview](/docs/user-guide/tekton-pipelines) for the complete pipeline type and trigger reference. - Try the [application scaffolding use case](/docs/use-cases/application-scaffolding) to see the platform generate a project skeleton with a working CI/CD pipeline in minutes. - Follow [autotest as quality gate](/docs/use-cases/autotest-as-quality-gate) to promote automatically only when integration tests pass. - Read [deploy application from a feature branch](/docs/use-cases/deploy-application-from-feature-branch) for ephemeral environment patterns. KubeRocketCI is open-source under Apache License 2.0. Platform source, Helm charts, and the full Tekton pipeline library are on [GitHub](https://github.com/epam/edp-tekton). --- ## krci CLI: From Terminal to AI Agents # krci CLI: Daily Platform Ops from Terminal to AI Agents | KubeRocketCI Most of my day-to-day platform work happens in a conversation. I sit in a Claude Code session - or any AI assistant with shell access - and ask plain-language questions about the state of our delivery cluster: *what's failing, what's drifting, what's vulnerable, what's stale.* The agent answers by calling the [krci CLI](https://github.com/KubeRocketCI/cli), the predictable, JSON-emitting client over the [KubeRocketCI Portal's tRPC API](/docs/developer-guide/reference-architecture). I read the answer, decide what to do, and when a question turns into a routine I drop the underlying invocation into a script and let it run on cron or `/loop`. This post is a snapshot of that workflow with one running example - operator vulnerability status - and a tour of the other questions the same pattern answers. ## Workflows, Not Sessions KubeRocketCI is built so the platform's primitives are reachable from every runtime that matters in modern engineering: the portal, the CLI, Tekton pipelines, GitLab CI, schedulers, and AI agents. A workflow I write once runs everywhere it needs to: - **Pipelines.** A `krci` invocation that answers a question in my terminal runs unchanged inside a Tekton task or a GitLab CI job. Quality gates and release governance read live platform state without bespoke API clients. - **Schedulers.** A short bash wrapper in cron, or a Kubernetes CronJob produces a daily report on a hands-off cadence - same command, same output, same trust boundary. - **Chat and webhooks.** JSON output drops straight into Slack, Teams, or PagerDuty without a translation layer. - **AI agents in the SDLC.** AI coding agents - [Claude Code](https://code.claude.com/docs), Cursor, the Anthropic and OpenAI SDKs - read terminal output natively. Giving an agent shell access to `krci` turns the platform into a tool surface it can call: list codebases, inspect SBOMs, compare branches, summarize changes since the previous run. No MCP server to write, no proprietary protocol to learn, no stale API client to maintain. The portal and the CLI are peers: the portal is the right interface for browsing, comparing, and approving; the CLI is the right interface for composing, scheduling, and handing off to agents. Both are first-class products and ship in lockstep. ## The krci CLI as a Tool Surface A short tour of the top-level surface, because this is what the agent has access to: ```text $ krci --help Available Commands: auth Authentication commands deployment Manage deployments (CDPipelines) env Inspect KRCI environments (Stages) pipelinerun Manage pipeline runs project Manage projects (Codebases) sca Inspect Dependency-Track projects, components, and vulnerability findings sonar Inspect SonarQube projects, quality gates, and issues version Print krci CLI version ``` When you run any of these commands - say, `krci deployment list` to inspect CD pipelines across your environments-the output follows the same disciplined structure. Columns are consistent, status values are predictable, and the table is human-readable *and* machine-parseable without any translation. Here's what that looks like in practice: ![Example of krci deployment list command output showing a table with deployment status, environment, sync state, and other metadata. The table displays three deployments: krci-gitfusion with healthy status in dev, and two tekton instances in dev and qa environments with varying sync states.](../assets/krci-deployment-example.png) This table shows exactly what's deployed where. The `VERSION` column tracks the build artifact (`0.5.0-SNAPSHOT.4`), the `ENV` column shows which environment it's in, and the `STATUS` column tells you health at a glance -`healthy` or `missing`. The `SYNC` column flags when an environment has drifted (`OutOfSync`) from what GitOps expects. In seconds, you can answer: *"What's actually running in qa? Is it in sync? What version?"* The agent can answer it too, and answer the next question - *"Show me all deployments older than 30 days"* - without any special prompting. Every group follows the same shape - `list`, `get`, sometimes a third verb - and the flags repeat (`--branch`, `--severity`, `-o json|table`). Predictability is the point: an agent can reason about the next command from the previous one, and so can I. Authentication is OIDC against the same Keycloak the portal uses; the agent inherits the existing session, so there is no long-lived API key embedded in any prompt. ## Asking the Agent: "What's the Vulnerability Status of Our Operators?" Here's a recent question, verbatim, from a Claude Code session: > *"Using krci, give me the current vulnerability status of every operator registered on the platform. Main branch only, critical and high counts, top affected components per operator."* What the agent returned: ```text OPERATOR CRIT HIGH TOP COMPONENTS (HIGH+) cd-pipeline-operator 0 0 clean codebase-operator 0 2 github.com/tektoncd/pipeline keycloak-operator 0 0 clean nexus-operator 0 0 clean sonar-operator 0 0 clean ``` One row is doing all the work: `codebase-operator` master carries two high-severity findings, and both trace back to a single `github.com/tektoncd/pipeline` cluster. Everything else is clean. That's the kind of answer I can act on without scrolling - one operator, one component, one upgrade to plan. The command I run by hand to verify the answer is one line: ```bash $ krci sca list --page-size 500 -o json \ | jq -r '.data.items[] | select(.name | endswith("-operator")) | select(.version == "master") | [.name, .metrics.critical, .metrics.high] | @tsv' ``` That's the whole loop. I described what I wanted in the language of the work, the agent shaped a `krci` invocation, and I can re-run any step independently to validate the result. No prior knowledge of `jq` flags, subcommand layout, or the platform's internal data model was required on my side. ### Drilling In Stays in Plain Language When a row of the answer needs follow-up, I keep the conversation going: > *"Drill into `codebase-operator` master - which components are driving those two highs?"* The agent's reply (verifiable by running the command yourself): ```text $ krci sca components codebase-operator --branch=master --severity=high COMPONENT VULNS (C/H/M/L) github.com/tektoncd/pipeline 0/2/4/0 ``` One component, both highs, two natural-language turns, two commands. I never had to remember the flag shape; the agent bridged intent to invocation and the CLI's predictable output bridged invocation to answer. ## The Same Pattern, Other Questions Vulnerability status is one of many. The agent + CLI handle the same shape of question across the platform - I describe it in plain language, the agent runs `krci`, I verify the output against the platform: - *"Which pipeline runs failed in the last 24 hours, grouped by codebase?"* → `krci pipelinerun list -o json` filtered on `.status.conditions`. - *"Which environments are running an out-of-date image of `code-assistant`?"* → `krci env list -o json` joined against the latest tag. - *"What's the SonarQube quality gate state of every Java codebase?"* → `krci sonar list -o json` filtered by language. - *"Which projects on the platform haven't produced an SBOM yet?"* → diff `krci project list` against `krci sca list`. - *"Show me every CD pipeline whose latest deploy is older than 30 days."* → `krci deployment list -o json` filtered on `.status.lastDeploy`. Same pattern every time: ask in language, get an answer the CLI can verify, move on. The agent does the orchestration; the CLI does the work. ## Making It Stick: Schedule, Script, Notify The moment a question becomes recurring, it stops belonging in chat. The same `krci` invocation the agent assembled in the conversation becomes the body of a script: ```bash #!/usr/bin/env bash # Daily operator vulnerability digest - main branches only. krci sca list --page-size 500 -o json \ | jq -r '.data.items[] | select(.name | endswith("-operator")) | select(.version == "master") | [.name, .metrics.critical, .metrics.high] | @tsv' \ | column -t ``` From there it's the standard fan-out: - **Cron or Kubernetes CronJob** for a daily markdown digest in `~/reports/operators-$(date +%F).md`. - **Slack/Teams webhook** for "first new critical on a release branch" alerts. Pipe the JSON through `jq`, post on threshold. - **Claude Code `/loop`** for a six-hour heartbeat that reports only what changed: ```text /loop 6h Run cli/scripts/operators-digest.sh and tell me ONLY what changed since the previous run. Reply in under 60 words. ``` Same script, same JSON, three runtimes. The CLI is unchanged; only the wrapper changes. ## Why the krci CLI Works Well as an Agent Tool Surface Three properties matter, and each one comes from deliberate design rather than accident: - **Stable, structured output.** `krci ... -o json` is shaped consistently across releases. The agent doesn't need a parser; it navigates JSON. - **Identity is solved upstream.** OIDC sessions are inherited from the developer's existing login. No API keys leak into prompts, transcripts, or scripts. - **Predictable verb shape.** `list`, `get`, sometimes a third verb, with the same flags everywhere. The agent generalizes from one command to the next, and so do humans. The result is that the conversation with the agent stays in the language of the work - *"vulnerability status of operators"*, *"failed runs in the last 24 hours"*, *"environments running an out-of-date image"* - and the CLI handles the translation. Both pieces are independently inspectable: I can replay any command myself, the agent can replay it tomorrow, and the script that captures a routine is short enough to review in one sitting. ## Honest Notes Before You Adopt the Pattern A few things I learned the slightly hard way: - Not every codebase produces an SBOM yet. Some appear in `project list` but are absent from `sca list` because no CycloneDX upload has happened on their build pipelines. The CLI surfaces the gap rather than hiding it, which is itself useful, but means agent answers should always cross-check both lists. - Cross-project aggregation sometimes still uses a shell loop. There's no `krci sca list --filter='critical>0'` flag yet - native flags are on the roadmap, the agent compensates in the meantime. - BOM age (`Last BOM`) shows up in `krci sca get` but not on `sca list`. If staleness matters to your routine, pin it explicitly in your script. I include this list because tools earn trust by being honest about what they don't do yet. ## Try It Yourself 1. Install krci from [the release page](https://github.com/KubeRocketCI/cli/releases) and run `krci auth login`. 2. Open your AI assistant of choice, give it shell access, and ask your own version of the question above. Anything along the lines of *"using krci, show me the vulnerability status of every operator on the platform, main branch only"* will work on day one. 3. When the answer becomes a routine, drop the underlying command into a script and run it on cron, a CronJob, or `/loop`. That's the entire loop: ask in language, get an answer the CLI can verify, schedule it when it matters. ## Further Reading - [krci CLI on GitHub](https://github.com/KubeRocketCI/cli) - command reference, authentication setup, and release binaries. - [SCA with Dependency-Track](/docs/operator-guide/devsecops/dependency-track) - how SBOMs reach the platform. - [Kubernetes-Native CI/CD with Tekton](/blog/kubernetes-native-cicd-tekton-kuberocketci) - the pipeline layer that produces those SBOMs. - [CycloneDX SBOM](https://cyclonedx.org/), [Dependency-Track](https://dependencytrack.org/), and [Claude Code](https://code.claude.com/docs) - the upstream tools used in this post. The krci CLI is open-source under Apache License 2.0. Source, issues, and release binaries live on [GitHub](https://github.com/KubeRocketCI/cli). --- ## Try KubeRocketCI Locally in 2 Commands # Try KubeRocketCI Locally in 2 Commands - No Cloud Account Required Evaluating an internal developer platform without a working instance is like buying a car from a brochure. Every KubeRocketCI install path in the official docs assumes a cluster you already have - AWS EKS, GKE, an on-prem control plane. Today I ran the [try-kuberocketci testbed](https://github.com/KubeRocketCI/try-kuberocketci) end-to-end on my Apple Silicon Mac using Docker Desktop and two commands: **`make testbed`** (approximately 18–20 minutes) and **`make e2e`** (approximately 12 minutes). The result is a fully wired KubeRocketCI local install - Tekton, Argo CD, SonarQube, self-hosted GitLab CE, Prometheus, Grafana, Tekton Results, and the Portal - running in a disposable [kind](https://kind.sigs.k8s.io) cluster. No cloud account. No `/etc/hosts` edits. No clicking through UIs to trigger pipelines. This post walks through exactly what happened, command by command, screenshot by screenshot. ## What is KubeRocketCI? [KubeRocketCI](/docs/about-platform) (KRCI) is an open-source, Kubernetes-native internal developer platform for cloud-native CI/CD, developed and maintained under Apache 2.0. It assembles [Tekton](https://tekton.dev), [Argo CD](https://argo-cd.readthedocs.io), SonarQube, and your Git provider into a cohesive, opinionated developer experience - managing the lifecycle of your Codebases (applications, libraries, autotests) from source through review, build, and GitOps-based deployment, exposed through a single Portal UI. KubeRocketCI markets itself as cutting time-from-project-initiation-to-active-development from days to hours. The testbed lets you verify that claim in 30 minutes on a laptop. On our own dogfooding cluster over a recent 90-day window, the platform logged [18,397 pipeline runs with a 93% build success rate](/blog/kubernetes-native-cicd-tekton-kuberocketci) - and it runs the same Tekton pipeline stack you are about to stand up locally. This repository is not KubeRocketCI itself - it is a local installer and test harness that brings the whole platform up on your machine, ready for evaluation, development, and demo preparation. ## Prerequisites and Hardware Requirements To run a KubeRocketCI Docker Desktop setup, you need only Docker Desktop - no cloud account, no pre-existing Kubernetes cluster, and no registry account are required. | Requirement | Minimum | Recommended | Notes | |---------------------------|--------------------|-------------|-----------------------------------------------------| | Docker Desktop | 4.x+ | Latest | Allocate RAM in Docker Settings → Resources | | RAM allocation | 8 GB (core only) | 12 GB+ | Full bed with GitLab + SonarQube needs 12 GB+ | | Disk space | ~20 GB free | ~30 GB | Container images and volumes | | OS | macOS, Linux, WSL2 | macOS/Linux | Windows via WSL2 supported | | Apple Silicon | Supported | - | amd64 images run via Docker Desktop Rosetta 2 | | `make` | Any version | - | Pre-installed on macOS/Linux | | `kind`, `helm`, `kubectl` | Latest stable | - | `make tools` installs `kind` via Homebrew | | Internet access | Required | - | Images pulled from public registries during install | :::warning RAM is the most common failure mode Docker Desktop with **at least 12 GB of RAM allocated** is required for the full bed. Below this threshold, GitLab CE or SonarQube will OOM-kill during startup. My live run used ~11.7 GB allocated to the Docker engine - right at the edge, and it completed successfully, but the README's 12 GB+ recommendation is well-founded. Check Docker Settings → Resources → Memory before running `make testbed`. ::: On Apple Silicon (M1/M2/M3), Docker Desktop's Rosetta 2 emulation runs the amd64-only images (Portal, GitLab, sonar-operator) transparently. No configuration changes are needed in the testbed - it just works. ## How to Try KubeRocketCI Locally: The Full Stack You Get The fastest way to try KubeRocketCI locally is two commands on Docker Desktop. The try-kuberocketci testbed spins up a KubeRocketCI kind cluster with the full CI/CD platform - including Tekton, Argo CD, Prometheus/Grafana, Tekton Results, SonarQube, and a self-hosted GitLab CE - in two commands: `make testbed` (approximately 18–20 minutes) and `make e2e` (approximately 12 minutes). All versions are pinned through the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) GitOps repository - the same source of truth used in production KubeRocketCI deployments - making the local testbed a faithful reproduction of a real cluster rather than a stripped-down demo. | Step | Layer | Component | Version | Namespace | |------|----------------|---------------------------------------------------|-----------------------------------------------|------------------| | 1 | Cluster | kind (Kubernetes-in-Docker) | v1.36.1 (kind default; node image not pinned) | - | | 2 | Ingress | ingress-nginx | controller-v1.11.3 | ingress-nginx | | 3 | Certs | cert-manager | v1.16.2 | cert-manager | | 4 | CI engine | [Tekton](https://tekton.dev) Pipelines / Triggers | v1.6.0 / v0.34.0 | tekton-pipelines | | 5 | CD engine | [Argo CD](https://argo-cd.readthedocs.io) | chart 9.5.17 | argocd | | 6 | Observability | kube-prometheus-stack + Grafana | 84.5.0 | monitoring | | 7 | Run storage | Tekton Results | v0.17.2 | tekton-pipelines | | 8 | Code quality | SonarQube Community + sonar-operator 3.3.0 | 2025.3.1 (chart) / 25.5-community | sonar | | 9 | SCM + Registry | GitLab CE + bundled Container Registry | 17.5.1-ce | gitlab | | 10 | Platform | KubeRocketCI (edp-install) | **3.14.0** | krci | **Why is KubeRocketCI installed last?** The edp-install chart renders provider resources - the GitServer CRD, EventListener, Ingress rules - at install time. If GitLab or Argo CD is not already running when the chart applies, the operator's SSH connection check fails and the platform never reaches a healthy state. Installing KubeRocketCI last, after every dependency is ready, is what makes the chart wire itself correctly on first reconcile. ## Step 1 - Clone the Repo and Spin Up the Testbed ```bash git clone https://github.com/KubeRocketCI/try-kuberocketci cd try-kuberocketci make testbed # ~18-20 min ``` That is the entire KubeRocketCI local install. One clone, one command. :::tip Check Docker Desktop RAM allocation first Before running `make testbed`, open Docker Desktop → Settings → Resources and verify the memory slider is set to 12 GB or higher. This is the single most common reason the install fails, and it takes 30 seconds to check. ::: `make testbed` builds in strict dependency order, installing KubeRocketCI last. The sequence: ```mermaid flowchart TD A[kind cluster] --> B[ingress-nginx] B --> C[cert-manager] C --> D[Tekton Pipelines + Triggers] D --> E[Argo CD] E --> F[Prometheus + Grafana] F --> G[Tekton Results] G --> H[SonarQube] H --> I[gitlab-up: GitLab CE + creds + CoreDNS] I --> J[krci: edp-install 3.14.0 - installed LAST] J --> K[gitlab-integrate: CA trust + task patches + GitOps repo] J --> L[argocd-integrate: repo creds + deploy patch] J --> M[sonar-integrate: token + ci-sonarqube secret] K --> N[make e2e ready] L --> N M --> N ``` GitLab CE is the slowest component to initialize - plan for it to take several minutes of the total. Once `make testbed` completes, run `make status` to see the live cluster state and print all service URLs with credentials: ```text NAME STATUS ROLES AGE VERSION krci-control-plane Ready control-plane 13m v1.36.1 --- krci pods --- NAME READY STATUS RESTARTS AGE cd-pipeline-operator-57b96f56ff-x7nlw 1/1 Running 0 84s codebase-operator-68b7599446-h8cn7 1/1 Running 0 41s el-edp-gitlab-6cfc7857d9-fnxxd 1/1 Running 0 84s gitfusion-86cb475dc9-mdbgb 1/1 Running 0 84s krci-portal-65956587d5-6tvtf 1/1 Running 0 31s tekton-cache-89846975f-86fx8 1/1 Running 0 84s tekton-interceptor-6c68789887-88z86 1/1 Running 0 84s ... ================ Tool URLs & credentials (local only) ================ Argo CD UI: http://argocd.127.0.0.1.nip.io SonarQube UI: http://sonar.127.0.0.1.nip.io GitLab UI: https://gitlab.127.0.0.1.nip.io Results API: http://tekton-results.127.0.0.1.nip.io Grafana UI: http://grafana.127.0.0.1.nip.io Portal UI: https://portal.127.0.0.1.nip.io ``` All six services are immediately reachable in a browser. No `/etc/hosts` edits - I will explain why in the next section. ## nip.io Wildcard DNS: No /etc/hosts Editing Required The try-kuberocketci testbed uses [nip.io](https://nip.io) wildcard DNS so every platform service gets a stable, browser-accessible URL - no `/etc/hosts` edits, no port-forwards, no local DNS configuration required. Any subdomain of `.nip.io` resolves to `` via a public DNS server; the testbed generates all ingress hostnames in the `*.127.0.0.1.nip.io` pattern, so every service is immediately reachable from your browser with zero local DNS configuration. Every competing local Kubernetes tutorial either requires manual `/etc/hosts` edits to map ingress hostnames to `127.0.0.1`, or skips DNS entirely and uses port-forwards. For a KubeRocketCI Docker Desktop setup that serves six platform UIs simultaneously, manual host-file management would be a non-starter - nip.io eliminates the entire problem class. | Service | URL | |---------------------|------------------------------------------| | KubeRocketCI Portal | `https://portal.127.0.0.1.nip.io` | | Argo CD | `http://argocd.127.0.0.1.nip.io` | | GitLab CE | `https://gitlab.127.0.0.1.nip.io` | | SonarQube | `http://sonar.127.0.0.1.nip.io` | | Grafana | `http://grafana.127.0.0.1.nip.io` | | Tekton Results API | `http://tekton-results.127.0.0.1.nip.io` | The DNS trick works for your browser on the host. Inside the cluster, pods resolve `gitlab.127.0.0.1.nip.io` to the in-cluster GitLab Service via CoreDNS rewrites, and the kind node's containerd uses a registry mirror config to pull images from GitLab's bundled registry. This split-horizon DNS design is one of the more interesting pieces of the testbed architecture - the full walkthrough is in [docs/architecture.md](https://github.com/KubeRocketCI/try-kuberocketci/blob/main/docs/architecture.md). ## Step 2 - Run the End-to-End Pipeline ```bash make e2e # ~12 min ``` ### How Does `make e2e` Run the Full CI/CD Pipeline Automatically? `make e2e` automates the complete CI/CD cycle - opening a GitLab Merge Request, triggering a Tekton review pipeline, merging the MR, building and pushing a container image with kaniko to GitLab's bundled registry, and asserting the Go/Gin sample application is live via Argo CD sync - in approximately 12 minutes with zero UI clicks. Here is the verbatim output from my run today: ```text ==> Applying the sample Codebase (test-go-app) ==> Waiting for the Codebase to be provisioned in GitLab (project + template push) ==> Waiting for the codebase-operator to create the project webhook ==> Waiting for the CodebaseImageStream test-go-app-main to exist ==> Creating the demo CDPipeline + dev Stage (triggerType: Auto) ==> Opening a merge request (branch ci-e2e-...) to trigger the review pipeline merge request !1 opened ==> Waiting for the webhook to create the review PipelineRun(s) triggered: pipelinerun.tekton.dev/review-test-go-app-main-2g7kd ==> Evaluating per-task results (review) review-test-go-app-main-2g7kd -> ALL_GREEN ==> Merging merge request !1 (action=merge fires the build trigger) merge request !1 merged ==> Waiting for the build run(s) to finish (kaniko build+push can take several minutes) ==> Evaluating per-task results (build) build-test-go-app-main-m8kpm -> ALL_GREEN ==> Reading the built image tag from CodebaseImageStream test-go-app-main built tag: main-20260606-132413 ==> Waiting for the Auto trigger to deploy tag main-20260606-132413 triggered: pipelinerun.tekton.dev/deploy-demo-dev-kkmqg ==> Evaluating per-task results (deploy) deploy-demo-dev-kkmqg -> ALL_GREEN ==> Verifying the application is deployed in krci-demo-dev on tag main-20260606-132413 OK test-go-app gitlab.127.0.0.1.nip.io:5050/krci/test-go-app:main-20260606-132413@sha256:a39d120b... E2E RESULT: PASS - review + build + deploy all green; test-go-app:main-20260606-132413 auto-deployed to krci-demo-dev. ``` The post-run cluster state confirms everything: ```bash $ kubectl -n krci get pipelinerun NAME TYPE SUCCEEDED REASON build-test-go-app-main-m8kpm build True Completed deploy-demo-dev-kkmqg deploy True Succeeded review-test-go-app-main-2g7kd review True Completed $ kubectl -n krci-demo-dev get deploy,pods NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/test-go-app 1/1 1 1 85s pod/test-go-app-595fd7f5dc-8sbrc 1/1 Running 0 85s $ kubectl get applications -A NAMESPACE NAME SYNC STATUS HEALTH STATUS krci demo-dev-test-go-app Synced Healthy ``` The full MR-to-deployed-workload sequence, as a pipeline diagram: ```mermaid sequenceDiagram participant E2E as make e2e script participant GL as GitLab CE participant EL as EventListener participant TK as Tekton participant CIS as CodebaseImageStream participant CD as cd-pipeline-operator participant Argo as Argo CD E2E->>GL: create test-go-app project + open MR !1 GL->>EL: webhook (MR open) EL->>TK: gitlab-review PipelineRun TK-->>E2E: review ALL_GREEN (~82s) E2E->>GL: merge MR !1 GL->>EL: webhook (MR action=merge) EL->>TK: gitlab-build PipelineRun TK->>GL: kaniko pushes image to :5050/krci/test-go-app:main-20260606-132413 TK->>CIS: update tag main-20260606-132413 CIS->>CD: Auto trigger (new tag) CD->>TK: CDStageDeploy → deploy PipelineRun TK->>Argo: sync deploy-templates Helm chart → ns krci-demo-dev Argo-->>E2E: test-go-app 1/1 Available, Synced + Healthy ``` ### Why GitLab's Bundled Container Registry? A common blocker in local CI/CD setups is the container registry: DockerHub rate limits, Harbor setup complexity, credentials management. GitLab CE ships with a built-in Container Registry served on port 5050. The testbed wires kaniko to push images directly to it - eliminating every external registry dependency. Kaniko trusts the self-signed cert via a chart flag (`edp-tekton.kaniko.customCert: true`), and a group deploy token backs both push and pull. The registry URL is `gitlab.127.0.0.1.nip.io:5050/krci/`, which maps cleanly to each codebase's GitLab project registry. No DockerHub account, no Harbor, no separate registry pod. A note on the review/build trigger split: the EventListener has two separate Trigger CRs - `gitlab-review` fires on MR `open/reopen/update`; `gitlab-build` fires on MR `action=merge`. This means **merging** the MR - not a push to the branch - kicks the build pipeline. It is an intentional design choice that avoids spurious build runs on every force-push. Also worth noting: GitLab can occasionally deliver the MR webhook twice, creating a duplicate review run that fails fast with a harmless HTTP 400 when it tries to post the same commit status context. The e2e script asserts at least one run is fully green - the duplicate does not affect the result. ## Exploring the Platform After Install After `make e2e` passes, every component has real data in it. Here is what I found navigating each UI - a practical KubeRocketCI getting started tour of each surface. ### KubeRocketCI Portal The Portal signs in with a Kubernetes bearer token rather than a username/password - run `make token` to mint a 24-hour token, then paste it into the Portal's "More options → Use Service Account Token" field. (OIDC is intentionally not wired up in the testbed - there is no identity issuer inside a kind cluster - so the service-account token is the local sign-in path. `make status` flags the in-cluster Portal's token login as still being wired up; I captured these views through the Portal using that same `make token` service-account flow.) ![KubeRocketCI Portal home dashboard - try KubeRocketCI locally on a kind cluster, showing 3 pipeline runs with 100% success rate](../assets/try-kuberocketci-locally/krci-portal-home.png) The home dashboard shows the pipeline metrics from the e2e run: **3 total runs, 100% success rate, 0 failed, average duration 2m 18s**. This is the same observability surface that surfaces 18,397 runs on the production cluster - just seeded with a single e2e cycle. ![KubeRocketCI Portal Projects page showing krci-gitops and test-go-app codebases registered on local Kubernetes](../assets/try-kuberocketci-locally/krci-portal-projects.png) The Projects page shows both registered codebases: `krci-gitops` (the GitOps/Helm system codebase that KubeRocketCI requires) and `test-go-app` (the Go/Gin application created by the e2e script). Both show status Created. ![KubeRocketCI Portal PipelineRuns list showing deploy, build, and review runs all green for test-go-app](../assets/try-kuberocketci-locally/krci-portal-pipelineruns.png) The PipelineRuns list shows all three runs - deploy, build, review - all green, all for `test-go-app` on PR #1. ![KubeRocketCI Portal Build PipelineRun DAG visualization showing all tasks green including sonar, kaniko, git-tag steps](../assets/try-kuberocketci-locally/krci-portal-build-diagram.png) The build pipeline DAG is the most visually satisfying part. Every node in the React Flow graph is green: `report-pipeline-start` → `fetch-repository` → `get-version` → `sonar` → `build` → `container-build` → `git-tag` → `update-cbis` → `save-cache`. The sonar step includes a quality gate wait - the build would have failed here if SonarQube reported issues. ![KubeRocketCI Portal Build PipelineRun details showing gitlab-go-gin-app-build-default pipeline, 1m 34s duration, all tasks completed](../assets/try-kuberocketci-locally/krci-portal-pipelinerun-dag.png) The build run details confirm the pipeline name (`gitlab-go-gin-app-build-default`) and duration (1m 34s). This is the Helm-templated pipeline library at work - the name encodes the Git provider, language, framework, and pipeline type, as described in the [Kubernetes-native CI/CD with Tekton](/blog/kubernetes-native-cicd-tekton-kuberocketci) post. ![KubeRocketCI Portal Deployments page showing demo CD pipeline with test-go-app application deployed](../assets/try-kuberocketci-locally/krci-portal-deployments.png) The Deployments page shows the `demo` CDPipeline created by the e2e script, with `test-go-app` as its application. ![KubeRocketCI Portal Pipeline Metrics showing 3 total runs, 100% success, 2m 18s average duration for build, review, and deploy pipelines](../assets/try-kuberocketci-locally/krci-portal-metrics.png) The Observability → Pipeline Metrics view breaks down the three runs: Build 1/100%, Review 1/100%, Deploy 1/100%, average 2m 18s across all types. ### Argo CD ![Argo CD UI showing demo-dev-test-go-app application Synced and Healthy with target revision main-20260606-132413 in krci-demo-dev namespace](../assets/try-kuberocketci-locally/argocd-applications.png) Argo CD shows `demo-dev-test-go-app` as Synced + Healthy, target revision `main-20260606-132413`, path `deploy-templates`, namespace `krci-demo-dev`. This is the GitOps delivery leg: after kaniko pushed the image and the CodebaseImageStream updated, the cd-pipeline-operator created a CDStageDeploy resource, which triggered a deploy PipelineRun, which applied the Helm chart through Argo CD. The full GitOps chain, locally. ### GitLab ![Self-hosted GitLab CE showing test-go-app project pipelines both Passed - merge build and e2e review Tekton commit statuses](../assets/try-kuberocketci-locally/gitlab-project.png) The self-hosted GitLab project shows both pipelines as Passed - Tekton's `gitlab-set-status` task wrote commit statuses back to GitLab after each pipeline run completed. Pipeline #1 is the review run (MR open), pipeline #2 is the build run (MR merge). The Go/Gin app is live at `test-go-app` in namespace `krci-demo-dev` - its `/` route returns 404 by design (no root handler in the Gin app), but the server is live and the deployment is `1/1 Available`. ![KubeRocketCI Portal deploy PipelineRun details showing all four deploy tasks green - pre-deploy, deploy-app, post-deploy, promote-images - for the demo-dev stage](../assets/try-kuberocketci-locally/krci-portal-deploy-dag.png) The deploy PipelineRun details confirm the full GitOps delivery chain completed in 57 seconds - all four tasks green (pre-deploy → deploy-app → post-deploy → promote-images), the Argo CD sync applied, and the workload live in `krci-demo-dev`. To explore [Tekton pipelines in KubeRocketCI](/docs/user-guide/tekton-pipelines) further, or start customizing pipelines for your own stack, the [krci CLI daily use](/blog/krci-cli-daily-use) post covers how to drive the platform from the terminal and hand off to AI agents. For the foundational [basic concepts](/docs/basic-concepts), the docs cover Codebases, CDPipelines, CodebaseImageStreams, and Stages in depth. ## Tearing Down: make down ```bash make down ``` `make down` deletes the kind cluster and all associated Kubernetes resources, leaving Docker Desktop in a clean state. Nothing persists outside the kind cluster - no volumes, no registry state, no credentials. The testbed is fully disposable and can be re-run from scratch in under 20 minutes. This disposability is a first-class design goal, not an afterthought. For full from-zero validation: `make down && make testbed && make e2e`. The [try-kuberocketci repository](https://github.com/KubeRocketCI/try-kuberocketci) runs this sequence as part of its own CI regression. :::info Safe on shared machines The testbed uses fixed local-only credentials, self-signed TLS, and broad RBAC - all intentionally, for a throwaway kind cluster bound to localhost. It is explicitly not safe to expose or use on any shared, internet-reachable, or production cluster. See [SECURITY.md](https://github.com/KubeRocketCI/try-kuberocketci/blob/main/SECURITY.md) for the details. ::: ## Use Cases: Why Try KubeRocketCI Locally? The fastest way to try KubeRocketCI locally is also the safest way to evaluate it - no cloud spend, no shared infrastructure, no consequences if something breaks. Here are the four use cases where the testbed pays off immediately. **Evaluating KubeRocketCI before adoption.** Before committing months of engineering time to an IDP, run `make testbed` and spend 30 minutes with the real platform - not a demo video or a vendor-configured sandbox. Compare hands-on against Backstage, kubriX, or other IDPs with a working instance under your own fingers. The [official platform installation guide](/docs/quick-start/platform-installation) covers the cloud path when you are ready to move from local to a real cluster. **Learning platform engineering.** The testbed is a coherent, fully-wired example of Tekton, Argo CD, GitOps, SonarQube, and Tekton Results working together. Every piece is independently inspectable with `kubectl`, `helm`, and the respective UIs. Productive learning by reading a real system, not a toy demo. **Contributing to KubeRocketCI.** The testbed is designed as a contributor sandbox. Run `make down && make testbed` to reproduce an issue from scratch without a cloud account. Each component can be rebuilt independently (`make sonar`, `make argocd`, etc.), so debugging one piece does not require tearing down the rest. **Demo preparation.** Spin up a fresh instance in under 20 minutes before a presentation; tear down afterward. The `make e2e` output gives you a verified, live end-state to present - not a screen recording. ## How This Compares to Other Local IDP Options The table below compares try-kuberocketci against the three closest local IDP alternatives across the features that matter most for local evaluation - install time, automated testing, component depth, and Apple Silicon support. | Feature | try-kuberocketci | CNOE idpBuilder | kubriX kind guide | Manual Tekton+ArgoCD | |----------------------------|--------------------------------|-------------------------|-------------------|----------------------| | Commands to full stack | 2 (`make testbed`, `make e2e`) | 1 (`idpbuilder create`) | 6 manual steps | 15+ manual steps | | Install time | ~18–20 min | ~5–10 min | ~30 min | 60+ min | | Automated E2E test | Yes (zero UI clicks) | No | No | No | | Tekton CI pipelines | Yes | No | No | Manual | | Self-hosted GitLab CE | Yes | No (Gitea) | No | No | | SonarQube | Yes | No | No | Optional/manual | | Tekton Results | Yes | No | No | No | | Prometheus + Grafana | Yes | No | No | No | | Apple Silicon support | Explicit (Rosetta) | Yes | Not documented | Not documented | | nip.io DNS (no /etc/hosts) | Yes | Partial | No | No | | GitOps-pinned versions | Yes (edp-cluster-add-ons) | Partial | No | No | | Teardown command | `make down` | Manual | Manual | Manual | CNOE idpBuilder is the closest open-source alternative - it installs Argo CD, Gitea, and ingress-nginx in approximately 5–10 minutes with a single command. It is a solid choice for a minimal GitOps sandbox. What it does not have: Tekton CI pipelines, SonarQube quality gates, Tekton Results for pipeline history, Prometheus/Grafana observability, or an automated end-to-end pipeline proof. The try-kuberocketci testbed installs a significantly larger stack and uniquely validates the full MR-to-deployed-workload cycle without any UI interaction. Also note: GitHub integration is fully supported in production KubeRocketCI ([integrate GitHub with KubeRocketCI](/docs/quick-start/integrate-github)) - the testbed uses GitLab specifically because it can be self-hosted in the kind cluster without a cloud account. ## Frequently Asked Questions ### What is KubeRocketCI? KubeRocketCI (KRCI) is an open-source internal developer platform for cloud-native CI/CD on Kubernetes, developed by EPAM and released under Apache 2.0, that integrates Tekton, Argo CD, and GitLab (or GitHub, Bitbucket) into a unified developer workflow platform. It manages Codebases from source through review, build, and GitOps-based deployment, surfacing everything through a single Portal UI and CLI. ### How do I try KubeRocketCI without a cloud account? Clone the [try-kuberocketci repository](https://github.com/KubeRocketCI/try-kuberocketci) and try KubeRocketCI locally by running `make testbed` - it installs the full KubeRocketCI platform on a local kind cluster on Docker Desktop in approximately 18–20 minutes, with no cloud account or pre-existing Kubernetes cluster required. Run `make e2e` afterward to validate the full CI/CD pipeline automatically. ### What are the hardware requirements to run KubeRocketCI locally? Docker Desktop with at least 12 GB of RAM allocated to the Docker engine is required for the full bed (8 GB suffices if you skip GitLab and SonarQube). Approximately 20 GB of free disk space is also needed for container images and volumes. No other cloud or network infrastructure is required. ### Does KubeRocketCI work on Apple Silicon (M1/M2/M3)? Yes. Apple Silicon (M1/M2/M3) is fully supported: Docker Desktop's Rosetta 2 emulation runs the amd64 container images (Portal, GitLab CE, sonar-operator) transparently on ARM64 hardware, with no configuration changes required in the testbed. I ran the entire flow on an Apple Silicon Mac for this post - the testbed explicitly documents and tests ARM64 support. ### How long does it take to install KubeRocketCI locally? **`make testbed`** installs the full platform in **approximately 18–20 minutes**; **`make e2e`** runs a complete automated pipeline proof in **approximately 12 minutes**. Total time from zero to a verified live workload is under 35 minutes. GitLab CE is typically the slowest component to initialize - the rest of the stack comes up faster. ### What components does `make testbed` install? In dependency order: a single-node kind cluster, ingress-nginx, cert-manager, Tekton Pipelines/Triggers, Argo CD, Prometheus and Grafana, Tekton Results, SonarQube Community, self-hosted GitLab CE (with bundled container registry), and KubeRocketCI edp-install 3.14.0 - 10 components in a defined sequence, with KubeRocketCI installed last so the chart can wire itself to every running dependency on first reconcile. ### How do I run an end-to-end CI/CD pipeline locally with KubeRocketCI? Run **`make e2e`** after `make testbed` completes. It automates the full cycle: applies a sample Go/Gin Codebase, waits for the codebase-operator to provision the GitLab project and webhook, opens a Merge Request to trigger the Tekton review pipeline, merges the MR to trigger the build pipeline (kaniko pushes the image to GitLab's bundled registry), waits for Argo CD to sync the deployment, and asserts the workload is `1/1 Available` - all in approximately 12 minutes with zero UI clicks. ### How do I tear down the local KubeRocketCI environment? Run **`make down`** to delete the kind cluster and all associated Kubernetes resources, leaving Docker Desktop in a clean state. Nothing persists outside the kind cluster. The testbed can be re-run from scratch in under 20 minutes and is designed to be fully disposable - suitable for evaluation, demo preparation, contributor testing, and automated platform CI regression. ### Why is there no /etc/hosts editing required? The testbed uses [nip.io](https://nip.io) wildcard DNS: any subdomain of `.nip.io` resolves to `` via public DNS, so all platform services get stable browser-accessible URLs (`*.127.0.0.1.nip.io` → `127.0.0.1`) without any local DNS configuration. Inside the cluster, CoreDNS rewrites handle pod-to-service resolution, and the kind node's containerd uses a registry mirror config for image pulls - no host-level DNS changes at any point. ### What is the difference between try-kuberocketci and CNOE idpBuilder? CNOE idpBuilder spins up a minimal IDP stack (Argo CD + Gitea + ingress-nginx) in approximately 5–10 minutes but ships no Tekton CI pipelines, no SonarQube, no GitLab CE, no Tekton Results, and no automated e2e validation. The try-kuberocketci testbed installs a full production-parity platform with all of these in ~18–20 minutes and uniquely includes a fully automated end-to-end pipeline test that validates a real MR-to-deployed-workload cycle without any UI interaction. ## Summary The fastest way to try KubeRocketCI locally is two commands on Docker Desktop. **`make testbed`** brings up a complete KubeRocketCI platform - 10 components including Tekton, Argo CD, SonarQube, GitLab CE, Prometheus/Grafana, and Tekton Results - in approximately 18–20 minutes. **`make e2e`** validates the full MR-to-deployed-workload pipeline automatically in approximately 12 minutes. Total: under 35 minutes from zero to a verified live workload, on Docker Desktop, with no cloud account. **`make down`** leaves nothing behind. I ran this end-to-end on June 6, 2026 on an Apple Silicon Mac with Docker Desktop. The e2e test passed with 3/3 green runs (review + build + deploy), 100% success rate, average duration 2m 18s, and the Go/Gin sample app deployed at tag `main-20260606-132413`. The most useful next steps from here: - Explore the [KubeRocketCI quick start overview](/docs/quick-start/quick-start-overview) to continue KubeRocketCI getting started and move from a local kind cluster to a real cloud deployment. - Read [Kubernetes-native CI/CD with Tekton](/blog/kubernetes-native-cicd-tekton-kuberocketci) for the deeper pipeline architecture and production metrics. - Use the [krci CLI](/blog/krci-cli-daily-use) to drive the platform from the terminal and AI agents after onboarding your first real application. - Check [KubeRocketCI documentation](https://docs.kuberocketci.io) for the full operator guide and integration references. KubeRocketCI is open-source under Apache License 2.0. The testbed, platform source, and Helm charts are all on [GitHub](https://github.com/KubeRocketCI/try-kuberocketci). {/* cspell:ignore fnxxd mdbgb tvtf kkmqg sbrc */} --- ## Ephemeral Preview Environments on Kubernetes An ephemeral preview environment is an isolated, temporary Kubernetes deployment created from a single feature branch and torn down when the work is done. Every branch gets its own namespace, its own image, its own URL - and zero of it lingers afterward. Ephemeral environments on Kubernetes make this pattern available on your own cluster - but a hands-on, open-source, portal-native version - feature branch to isolated namespace to one-click destroy, backed by real Tekton CI and Argo CD GitOps - is conspicuously missing from the public record. So I built one, end to end, on the same local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed from my last post: a [kind](https://kind.sigs.k8s.io) cluster running [KubeRocketCI](/docs/about-platform) 3.14.0 with Tekton, Argo CD, and self-hosted GitLab. This post is the full walkthrough - every screenshot, every line of terminal output, captured from a live run. We will take a stable `main` deployment, branch off it, ship a change that is visible *only* in the preview environment, inject per-environment config through GitOps, prove the two environments never touch each other, and then destroy the whole thing - leaving the baseline exactly as it was. ## What Is an Ephemeral Preview Environment? An **ephemeral preview environment** is a short-lived, namespace-isolated copy of your application, provisioned automatically from a feature branch or pull request and destroyed when that branch merges or is no longer needed. Unlike a shared staging environment - where one branch at a time monopolizes the slot - a preview environment is dedicated to a single branch, carries no long-term state, and releases its resources (namespace, pods, ingress) the moment you tear it down. The terms *preview environment* and *ephemeral environment* are used interchangeably. "Preview" emphasizes the use case (review a branch before merge); "ephemeral" emphasizes the lifecycle (temporary by design). In KubeRocketCI they are the same object: a Deployment **Stage** with its own isolated Kubernetes namespace, created for a branch and removed on demand. A preview environment lifecycle has five stages: ```mermaid flowchart LR A[1\. Triggerbranch / portal click] --> B[2\. Buildversioned image] B --> C[3\. Provisionnamespace + GitOps sync] C --> D[4\. Validatetest the live URL] D --> E[5\. Destroynamespace removed] ``` 1. **Trigger** - create or push a branch in the Portal. 2. **Build** - Tekton pipeline builds and pushes a versioned image. 3. **Provision** - namespace created, Argo CD syncs the application. 4. **Validate** - test the live URL. 5. **Destroy** - namespace and Argo CD application deleted. ### Ephemeral vs. Staging: Why Shared Environments Become Bottlenecks A shared staging environment serializes your team: only one branch can occupy it at a time, so parallel feature work cannot be validated simultaneously, and "who broke staging?" becomes a recurring standup question. Ephemeral preview environments give every branch its own namespace, so several features can be tested concurrently without interfering - and when a branch merges, its namespace is deleted, so there is no idle staging box quietly accruing cost. That last point matters more than it looks. Industry surveys consistently find a large share of provisioned Kubernetes capacity sits idle, and the trend is getting worse - [CAST AI's 2026 State of Kubernetes Optimization Report](https://cast.ai/blog/2026-state-of-kubernetes-resource-optimization-cpu-at-8-memory-at-20-and-getting-worse/) measured average CPU utilization at just **8%** across roughly 23,000 production clusters (down from 10% a year earlier), and [Flexera's 2026 State of the Cloud report](https://www.flexera.com/about-us/press-center/flexera-finds-cloud-value-is-rising-while-ai-waste-grows) pegs self-estimated cloud waste at **~29%**, its first rise in five years. Permanent staging environments are a textbook source of that waste. An environment that deletes itself cannot become idle. ### Namespace-per-Branch, vCluster, or Dedicated Cluster? There are three common isolation models. **Namespace-per-branch** reuses one shared control plane, provisions in seconds, and is sufficient for most teams - it is the model used here. **vCluster** adds a lightweight virtual API server per environment for stronger isolation at higher overhead. A **dedicated cluster per branch** is maximum isolation but impractical at scale. KubeRocketCI uses namespace-per-branch, with Argo CD managing each namespace's application manifests. :::note Namespace isolation is not network isolation A separate namespace scopes names, RBAC, and quotas - but by default pods in different namespaces can still reach each other over the cluster network. If your preview environments need network isolation too, add `NetworkPolicy` rules; the namespace boundary alone does not provide them. ::: ## How KubeRocketCI Delivers Preview Environments Before the walkthrough, a quick map of the moving parts. KubeRocketCI wraps a few [core concepts](/docs/basic-concepts) that turn a Git branch into a running environment: - **Codebase / CodebaseBranch** - your application (`test-go-app`) and a branch within it (`feature-tt-123`). Creating a branch in the Portal creates the `CodebaseBranch` resource and mirrors the branch into Git. - **CodebaseImageStream (CBIS)** - the record of container image tags built for a branch. The build pipeline writes new tags here; the CD side reads from it. This handoff is the key to ordering (more on that below). - **Deployment (CDPipeline) and Environment (Stage)** - the deployment flow and the individual environments within it. A Stage owns a namespace and an Argo CD Application. - **Trigger type** - `Manual`, `Auto`, or `Auto-stable`. We use **Auto**: the environment redeploys whenever a new image tag lands in the CBIS, so the preview env always tracks the branch head with no manual step. (See [Deployment Strategies](/docs/user-guide/auto-stable-trigger-type).) Here is the full path from a feature-branch build to a live, isolated workload: ```mermaid sequenceDiagram participant Dev as Developer (Portal) participant TK as Tekton (build) participant CBIS as CodebaseImageStream participant CD as cd-pipeline-operator participant Argo as Argo CD participant NS as ns krci-sk-feat Dev->>TK: Build feature-tt-123 TK->>TK: kaniko build + push image TK->>CBIS: write tag feature-tt-123-20260607-060023 CBIS->>CD: Auto trigger (new tag) CD->>Argo: CDStageDeploy -> deploy PipelineRun Argo->>NS: sync deploy-templates Helm chart NS-->>Dev: test-go-app 1/1, Synced + Healthy ``` The crucial detail is the **CBIS handoff**: Tekton writes the image tag *after* the image is actually pushed, and the Auto trigger only fires once that tag exists. The image is therefore guaranteed to be present before Argo CD tries to deploy it - which avoids the `ImagePullBackOff` race that bare deploy-only tooling is prone to. ## Baseline: The Local Testbed Starting Point The starting point is the end state of the [previous post](/blog/try-kuberocketci-locally): the sample Go/Gin app `test-go-app` built from `main` and deployed by the `demo` Deployment into namespace `krci-demo-dev`. ```bash $ kubectl get applications -n krci NAMESPACE NAME SYNC STATUS HEALTH STATUS krci demo-dev-test-go-app Synced Healthy $ kubectl -n krci-demo-dev get deploy test-go-app -o jsonpath='{.spec.template.spec.containers[0].image}' gitlab.127.0.0.1.nip.io:5050/krci/test-go-app:main-20260606-132413@sha256:a39d120b... ``` One property of `main` matters for the rest of this post: the Gin app only handles `/hello`, so the root path returns 404. That 404 is our control - the feature environment will return 200 on the same path, and `main` will keep returning 404 the entire time. ```bash # main, GET / (no root handler on main) $ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:18080/ 404 ``` ## Step 1: Create the Feature Branch via the Portal In the `test-go-app` component, **Branches → Create branch**. I name it `feature-tt-123`, leave **From** as `main`, and keep the default review and build pipelines. ![KubeRocketCI Portal Create Branch dialog for test-go-app, creating feature-tt-123 from main with default review and build pipelines](../assets/ephemeral-preview-environments-kubernetes-feature-branch/create-feature-branch.png "Creating a feature branch in the KubeRocketCI Portal") Clicking **Create** provisions a `CodebaseBranch` resource and mirrors the branch into GitLab with an (initially empty) CodebaseImageStream: ```bash $ kubectl -n krci get codebasebranch,codebaseimagestream | grep feature codebasebranch.v2.edp.epam.com/test-go-app-feature-tt-123-0afaf created feature-tt-123 codebaseimagestream.v2.edp.epam.com/test-go-app-feature-tt-123-0afaf test-go-app ``` ## Step 2: Make a Change That Only the Preview Will Show To prove isolation, the feature branch needs to behave differently from `main`. I add a root `/` handler that returns `200` and echoes a `NAME` environment variable (we will set that variable per-environment in Step 7), plus the Helm `extraEnv` plumbing the override needs. The whole change is 30 lines across four files: ```go title="main.go (feature-tt-123)" r := gin.Default() r.GET("/", func(c *gin.Context) { c.JSON(http.StatusOK, gin.H{ "status": "ok", "name": os.Getenv("NAME"), }) }) r.GET("/hello", func(c *gin.Context) { ``` ```yaml title="deploy-templates/templates/deployment.yaml (feature-tt-123)" ports: - name: http containerPort: {{ .Values.service.port }} protocol: TCP {{- if .Values.extraEnv }} env: {{- toYaml .Values.extraEnv | nindent 12 }} {{- end }} ``` The default `extraEnv: {}` goes into `values.yaml`, and a matching unit test goes into `main_test.go`. Crucially, this change lives **only on `feature-tt-123`** - `main` is never touched. ## Step 3: Build the Feature Branch Feature branches are built on demand: on the **Branches** tab, the `feature-tt-123` row has a **Build** button. One click starts the `gitlab-go-gin-app-build-default` pipeline. ![KubeRocketCI Portal Branches tab showing feature-tt-123 with a successful build next to the main branch](../assets/ephemeral-preview-environments-kubernetes-feature-branch/branches-after-build.png "The feature branch built and ready, alongside main") The build is the familiar Tekton DAG - fetch, version, SonarQube quality gate, kaniko image build, Git tag, and the all-important `update-cbis` step that records the new tag: ![KubeRocketCI build pipeline DAG for feature-tt-123, all tasks green: fetch-repository, get-version, sonar, build, container-build, git-tag, update-cbis](../assets/ephemeral-preview-environments-kubernetes-feature-branch/build-pipeline-dag.png "The feature-branch build pipeline, all green") ```bash $ kubectl -n krci get pipelinerun | grep build-test-go-app-feature-tt-123-0afaf build-test-go-app-feature-tt-123-0afaf-2d31 True Completed ``` ### Reading the New Image Tag from the CodebaseImageStream With **default** [versioning](/docs/user-guide/artifact-versioning), the tag is `BRANCH-DATETIME`. The build writes it to the CBIS: ```bash $ kubectl -n krci get codebaseimagestream test-go-app-feature-tt-123-0afaf \ -o jsonpath='{.spec.tags[*].name}' feature-tt-123-20260607-060023 ``` ## Step 4: Create the Deployment Flow and Auto-Trigger Environment Now the environment. I follow the recommended convention of naming the deployment flow after my initials - `sk` - so personal preview flows are easy to find. In **Deployments → Create Deployment**, I select `test-go-app` and set its branch to `feature-tt-123`. ![KubeRocketCI Create Deployment wizard with test-go-app selected and its branch set to feature-tt-123](../assets/ephemeral-preview-environments-kubernetes-feature-branch/deployment-select-app-branch.png "Selecting the application and the feature branch for the deployment flow") Then I add an environment named `feat` with trigger type **Auto**: ![KubeRocketCI Create Environment wizard showing the Auto trigger type selected for automatic deployment](../assets/ephemeral-preview-environments-kubernetes-feature-branch/environment-auto-trigger.png "Choosing the Auto trigger type so the environment tracks the branch") ### Naming Conventions: How KRCI Builds the Namespace The namespace is derived as `krci--`, which the form pre-fills as `krci-sk-feat`. The environment overview confirms the wiring - Auto trigger, `in-cluster`, namespace `krci-sk-feat`, deploy and clean pipeline templates, quality gate Manual: ![KubeRocketCI environment overview for feat: Auto trigger, in-cluster, namespace krci-sk-feat, deploy and clean pipelines](../assets/ephemeral-preview-environments-kubernetes-feature-branch/environment-overview.png "The feat environment overview") Creating the environment *before* the next build is deliberate: with Auto, the deploy fires on a CBIS *update*. Arm the environment first, and the build that follows will deploy itself. ## Step 5: The Feature Image Auto-Deploys Triggering one more build updates the CBIS, the Auto trigger fires, and the `cd-pipeline-operator` creates the deploy run - no clicks. The deploy pipeline runs its four tasks and goes green in 47 seconds: ![KubeRocketCI auto-triggered deploy pipeline DAG for the feat environment: pre-deploy, deploy-app, post-deploy, promote-images, all green](../assets/ephemeral-preview-environments-kubernetes-feature-branch/auto-deploy-pipeline-dag.png "The Auto-triggered deploy pipeline for the preview environment") The whole preview environment now shows in one card - the `feat` Stage, its namespace, the deploy/clean pipelines, and the application `Healthy + Synced` on the feature tag: ![KubeRocketCI Deployments view showing the sk deployment with the feat preview environment, namespace krci-sk-feat, and test-go-app Healthy and Synced](../assets/ephemeral-preview-environments-kubernetes-feature-branch/preview-environment-card.png "The live preview environment card") The cluster agrees: ```bash $ kubectl -n krci get pipelinerun | grep sk-feat deploy-sk-feat-pjsxx True Succeeded $ kubectl get applications -n krci NAMESPACE NAME SYNC STATUS HEALTH STATUS krci demo-dev-test-go-app Synced Healthy # main, untouched krci sk-feat-test-go-app Synced Healthy # the new preview env $ kubectl -n krci-sk-feat get deploy,pods NAME READY UP-TO-DATE AVAILABLE deployment.apps/test-go-app 1/1 1 1 NAME READY STATUS pod/test-go-app-666f6c6994-rltdl 1/1 Running ``` ## Step 6: Proof of Isolation - Feature Gets 200, Main Gets 404 This is the part no one publishes with real output. Two namespaces, two image tags, two versions of the code, running side by side on the same cluster: ```bash $ kubectl -n krci-demo-dev get deploy test-go-app -o jsonpath='{..image}' ...test-go-app:main-20260606-132413@sha256:a39d120b... # main $ kubectl -n krci-sk-feat get deploy test-go-app -o jsonpath='{..image}' ...test-go-app:feature-tt-123-20260607-060023@sha256:812a57de... # feature ``` And the behavioral proof - same path, two answers: ```bash # feature env (krci-sk-feat) -> GET / HTTP/1.1 200 OK {"name":"","status":"ok"} # main env (krci-demo-dev) -> GET / HTTP/1.1 404 Not Found # sanity: both still serve /hello feature /hello -> 200 {"message":"Hello, World!"} main /hello -> 200 {"message":"Hello, World!"} ``` The feature branch's new code is live in `krci-sk-feat` and answers `200`. The exact same request to `main` in `krci-demo-dev` still returns `404`. Nothing on `main` changed. That is the entire promise of a preview environment, demonstrated rather than asserted. ## Step 7: Per-Environment Config via GitOps Values Override Preview environments usually need their own configuration - a different feature flag, API endpoint, or, here, a `NAME` variable. The GitOps way to do this is a values file in the [GitOps repository](/docs/user-guide/gitops) at the path `//-values.yaml`, with **Values override** enabled for the environment. ```yaml title="sk/feat/test-go-app-values.yaml (in the GitOps repo)" extraEnv: - name: NAME value: "Hello from the sk/feat preview environment" ``` On the environment's **Applications** tab, I open **Configure Deploy**, flip the per-app **Values override** toggle, and **Start Deploy**: ![KubeRocketCI Configure Deploy view with the Values override toggle enabled for test-go-app and the Start Deploy button](../assets/ephemeral-preview-environments-kubernetes-feature-branch/values-override-enabled.png "Enabling per-environment Values override before deploying") ```bash $ kubectl -n krci get pipelinerun | grep sk-feat deploy-sk-feat-pjsxx True Succeeded deploy-sk-feat-ec0e True Succeeded ``` KubeRocketCI pulls the values file from the GitOps repo and applies it to the Argo CD Application at sync time - no image rebuild required. The pod picks up the variable, and because our root handler echoes it, the override is now visible over HTTP too: ```bash $ kubectl -n krci-sk-feat exec deploy/test-go-app -- env | grep ^NAME= NAME=Hello from the sk/feat preview environment # feature env GET / (now reflects the per-environment value) {"name":"Hello from the sk/feat preview environment","status":"ok"} # main pod has no such variable $ kubectl -n krci-demo-dev exec deploy/test-go-app -- env | grep ^NAME= (no output) ``` ## Step 8: Destroy the Environment - Namespace Gone, Zero Residual Cost Teardown is a first-class action, not a side effect of closing a PR. On the deployment, **Actions → Delete** asks me to type the name to confirm: ![KubeRocketCI confirm-deletion dialog for the sk deployment, requiring the name to be typed to delete the deployment and all its environments](../assets/ephemeral-preview-environments-kubernetes-feature-branch/destroy-environment-confirm.png "Destroying the preview environment from the Portal") The `cd-pipeline-operator` deletes the namespace and its Argo CD Application. In about 5 seconds the preview environment is gone - and the `main` baseline is exactly as it was: ```bash $ kubectl get ns krci-sk-feat Error from server (NotFound): namespaces "krci-sk-feat" not found $ kubectl get applications -n krci NAMESPACE NAME SYNC STATUS HEALTH STATUS krci demo-dev-test-go-app Synced Healthy # baseline intact $ kubectl -n krci-demo-dev get deploy test-go-app NAME READY UP-TO-DATE AVAILABLE test-go-app 1/1 1 1 ``` ### Cleanup Checklist: Stage, Branch, GitOps Entry Are Independent One thing to know: destroying the Stage does **not** delete the branch. ```bash $ kubectl -n krci get codebasebranch | grep feature test-go-app-feature-tt-123-0afaf created feature-tt-123 # still here ``` The Stage, the `CodebaseBranch`, the Git branch, and the GitOps values file are independent lifecycle objects. To fully clean up after a feature, delete the branch on the **Branches** tab and remove the `//` entry from the GitOps repo. That independence is a feature: you can spin a preview environment up and down repeatedly for the same branch without ever touching Git history. ## KubeRocketCI vs. Other Preview Environment Approaches | Capability | KubeRocketCI | Argo CD ApplicationSet (PR generator) | Uffizzi | Okteto / Qovery / Vercel (SaaS) | |---|---|---|---|---| | License | Apache-2.0 | Apache-2.0 | Proprietary (SaaS only since 2024) | Proprietary / freemium | | Self-hosted | Yes | Yes | No | Control plane is SaaS | | Builds the image | Yes (Tekton) | **No** - deploy only | Yes | Yes | | GitOps CD | Yes (Argo CD) | Yes (Argo CD) | No (Compose/Helm) | Varies | | Developer Portal UI | Yes | No (YAML) | Limited | Yes | | Per-branch namespace | Yes | Yes | Yes | Yes | | Per-env values override | Yes (GitOps) | Manual | Limited | Yes | | One-click destroy | Yes | Manual | Yes | Yes | | Cost per environment | Compute only | Compute only | Compute only | Per-env / per-seat fees | ### The Argo CD ApplicationSet PR Generator: Powerful, but Incomplete The Argo CD ApplicationSet **Pull Request generator** is the closest pure-OSS pattern, and it is genuinely useful - it watches open PRs and creates an Argo CD Application for each. But it is a *deployment* tool only: **it does not build container images**. The CI pipeline must build and push the tagged image first, and if the ApplicationSet reconciles before the image exists, the pods hit `ImagePullBackOff`. KubeRocketCI sidesteps that ordering problem with the CodebaseImageStream handoff described earlier - Tekton writes the tag, and the Auto trigger only fires once it is there. You also get a Portal, a Build button, and a Destroy button instead of hand-written `ApplicationSet` YAML. ### Why SaaS Preview Tools Don't Cover Full-Stack Kubernetes Vercel and Netlify nail preview deployments for frontend/static workloads, and Qovery/Okteto/Bunnyshell offer polished managed experiences - but they put a SaaS control plane (and a per-environment or per-seat bill) between you and your cluster. Demand is real, but the tooling still frustrates teams: in [Port's 2025 State of Internal Developer Portals survey](https://www.port.io/state-of-internal-developer-portals), just **6%** of developers were satisfied with the self-service tooling they use to spin up environments on demand. The open question that leaves is how to get that developer experience **on your own cluster, with no SaaS bill** - exactly the gap an Apache-2.0 IDP like KubeRocketCI fills. ## Frequently Asked Questions ### What is the difference between a preview environment and an ephemeral environment? They are used interchangeably. "Preview environment" emphasizes the use case - reviewing a feature branch before merge - while "ephemeral environment" emphasizes the lifecycle: it is temporary and torn down when done. In KubeRocketCI both refer to the same object: a Deployment Stage with its own isolated Kubernetes namespace, created for a branch and destroyed on demand. ### Do I need a SaaS subscription to get preview environments on Kubernetes? No. KubeRocketCI is Apache-2.0 open source and runs entirely on your own cluster - kind locally, or EKS/GKE/AKS in production. There is no per-environment fee and no SaaS control plane. You bring the cluster; KRCI provides the Portal, CI (Tekton), and GitOps CD (Argo CD) to manage the full preview-environment lifecycle. ### Does setting up ephemeral environments require YAML expertise? Not for the day-to-day flow. Creating a branch, building it, and deploying to an isolated namespace are all Portal form actions. The only YAML you write is the optional per-environment values file in the GitOps repo - and only when you actually need to override a value for a specific preview. The CDPipeline and Argo CD Application resources are managed by KRCI operators. ### How does per-environment configuration override work in a GitOps workflow? You add a file at `//-values.yaml` in the GitOps repository - for example `sk/feat/test-go-app-values.yaml` - containing Helm values that override the application's defaults. Enable the **Values override** toggle on the environment, then deploy. KubeRocketCI applies those values to the Argo CD Application at sync time, so the pod reflects them without rebuilding the image. ### What happens to the namespace when I delete the environment in KRCI? The `cd-pipeline-operator` deletes the Kubernetes namespace and the associated Argo CD Application - all pods, services, and configmaps in that namespace are removed. The `CodebaseBranch` and the Git branch are separate objects, so destroying the Stage does not delete or merge the branch; clean those up separately if you want to. ### Does the Argo CD ApplicationSet Pull Request generator build container images? No. It is a deployment tool that watches open pull requests and creates Argo CD Applications for them. Your CI pipeline (Tekton, GitHub Actions, GitLab CI) must build and push the image first. If the ApplicationSet fires before the image is ready, pods hit `ImagePullBackOff`. KubeRocketCI avoids this with the CodebaseImageStream handoff: Tekton writes the tag, and the Auto trigger deploys only after it exists. ### How much does it cost to run ephemeral environments on Kubernetes with KubeRocketCI? The software cost is zero - KubeRocketCI is Apache-2.0 with no per-environment fee. The only cost is the compute the environment consumes while it runs, and because preview environments are destroyed when the branch work is done, they do not accumulate idle compute the way permanent staging environments do. On the local kind testbed in this post, the preview environment cost nothing beyond the Docker Desktop resources already allocated. ## Summary Starting from a stable `main` deployment, we created a feature branch in the Portal, shipped a change visible only on that branch, built it through Tekton, and let an **Auto** environment deploy it to its own isolated namespace through Argo CD GitOps. We proved the isolation with real output - `feature` returns `200`, `main` keeps returning `404` - injected per-environment config through a GitOps values override, and then destroyed the environment in one action, leaving the baseline untouched and zero residual cost behind. No SaaS bill, no hand-written `ApplicationSet` YAML, no `kubectl` required for the happy path - just a feature branch, a few portal clicks, and a real isolated Kubernetes environment that cleans up after itself. The pattern - feature branch deployment on Kubernetes with isolated namespaces - is entirely self-hosted and free. I ran the entire flow on June 7, 2026 on the local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed; every screenshot and command output above is from that run. The most useful next steps from here: - Spin up the testbed yourself with [Try KubeRocketCI Locally](/blog/try-kuberocketci-locally), then follow this flow on your own machine. - Read the [Deploy Application From a Feature Branch](/docs/use-cases/deploy-application-from-feature-branch) use case and [Manage Deployments](/docs/user-guide/manage-environments) for the full reference. - Explore [Deployment Strategies](/docs/user-guide/auto-stable-trigger-type) (Manual, Auto, Auto-stable) and the [Argo CD Diff preview](/docs/user-guide/argo-cd-preview) for gated promotions. - See the deeper [Kubernetes-native CI/CD with Tekton](/blog/kubernetes-native-cicd-tekton-kuberocketci) architecture and the [krci CLI](/blog/krci-cli-daily-use) for driving all of this from the terminal. KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the testbed are all on [GitHub](https://github.com/KubeRocketCI/try-kuberocketci). {/* cspell:ignore pjsxx 0afaf rltdl */} --- ## GitLab CI Integration in KubeRocketCI # GitLab CI Integration: Run CI in GitLab Instead of Tekton with KubeRocketCI **GitLab CI integration** in KubeRocketCI lets a single application run its CI pipeline in **GitLab CI - on a GitLab Runner - instead of Tekton**, while still being managed as a first-class Codebase on the platform. You set one field on the Codebase (`spec.ciTool: gitlab`), and KubeRocketCI generates a **`.gitlab-ci.yml`** in the repository; GitLab then runs the pipeline, with no Tekton involved. From then on, every merge request runs a review pipeline and every merge runs a build pipeline - all native GitLab CI, all on your own cluster. This is part three of my hands-on series on the local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed. In [part one](/blog/try-kuberocketci-locally) I stood up the full platform in two commands; in [part two](/blog/ephemeral-preview-environments-kubernetes-feature-branch) I built ephemeral preview environments from a feature branch. Both ran their CI in **Tekton**. This post takes the same [kind](https://kind.sigs.k8s.io) cluster running [KubeRocketCI](/docs/about-platform) 3.13 and shows the **multi-CI** path: how GitLab CI integration works, the three things you must set up *before* you enable it - a Runner, the onboarded **CI/CD components**, and a **ConfigMap** - and a full review-to-build run with real output. ## What "GitLab CI Integration" Means in KubeRocketCI By default, KubeRocketCI runs your CI in Tekton. Version 3.13 added **multi-CI**: a per-Codebase choice of engine, set with one field - `spec.ciTool: tekton` (the default) or `spec.ciTool: gitlab` (see the [3.13 notes](/docs/operator-guide/upgrade/upgrade-krci-3.13) and [Codebase API](/docs/api/codebase)). Choosing `gitlab` is what this post is about. When you set `ciTool: gitlab`, KubeRocketCI handles the wiring. Instead of the Tekton `EventListener` and webhook it would normally create, it **generates a `.gitlab-ci.yml` and commits it to the repository**; GitLab CI then runs the pipeline. The file is generated rather than written by hand, and no Tekton is involved - the build runs as native GitLab CI jobs on a GitLab Runner, and the logs reside in GitLab. Other aspects are unchanged: the application remains a first-class **Codebase**, with the same Portal, branches, and GitOps deployment as a Tekton application. GitLab-CI and Tekton applications can run side by side on the same platform; only the CI execution moves to GitLab (see [the comparison below](#gitlab-ci-vs-tekton-when-to-use-which)). ```mermaid flowchart LR A[Set spec.ciTool: gitlab] --> B[KubeRocketCI generates.gitlab-ci.yml in the repo] --> C[GitLab CI runs iton the GitLab Runner] ``` On a real codebase the file lands in the repo, and there is no project webhook - GitLab CI does not need one: ```bash $ kubectl -n krci get codebase java-gitlabci-app -o jsonpath='{.spec.ciTool}' gitlab # .gitlab-ci.yml is committed to the repo; project webhooks: 0 ``` The generated `.gitlab-ci.yml` is intentionally minimal: it contains no build logic of its own and instead references reusable GitLab CI/CD components. Those components must be in place first. ## Before You Enable It: Three Things to Set Up First GitLab CI integration has three prerequisites that the Tekton path does not - a one-time platform setup, not something you repeat per app. Skip any one and the first pipeline fails to start or cannot resolve its components; complete them once, and subsequent GitLab-CI Codebases require no additional setup. ```mermaid flowchart LR A[1\. GitLab Runnerk8s executor] --> D[EnableciTool: gitlab] B[2\. Onboard CI/CDcomponents] --> D C[3\. Create thetemplate ConfigMap] --> D D --> E[Pipelines runin GitLab CI] ``` ### 1. Install a GitLab Runner (KubeRocketCI Does Not Bundle One) Tekton ships with KubeRocketCI; a **GitLab Runner does not**. GitLab CE already serves CI, but with no runner registered, jobs sit `pending` forever. On the testbed, `make gitlab-ci` installs the official `gitlab/gitlab-runner` Helm chart with the **Kubernetes executor** and registers it. ```bash $ kubectl -n gitlab-runner get pods NAME READY STATUS RESTARTS AGE gitlab-runner-f655c776c-xr9cm 1/1 Running 0 26m # the runner reports online as an instance runner runner: krci-kubernetes online=True executor=kubernetes ``` The Kubernetes executor runs each job as a pod in the cluster - the same model Tekton uses, so your CI capacity scales with the cluster. The testbed uses Helm for simplicity, but on a real cluster you install the runner the GitOps way, like every other platform component: KubeRocketCI ships a [`gitlab-runner` add-on](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/gitlab-runner) in the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons) repository that Argo CD reconciles, with the runner's registration token supplied through External Secrets. Either way you end up with one registered runner, sized to your job volume, ready to execute GitLab CI jobs. ### 2. Onboard the KubeRocketCI CI/CD Components Into Your Instance This prerequisite is easy to overlook. The injected `.gitlab-ci.yml` does not contain the build steps directly; it references a **GitLab CI/CD component** - the [`kuberocketci/ci-java17-mvn`](https://gitlab.com/kuberocketci/ci-java17-mvn) library, which exposes `review` and `build` entry points over a 7-stage flow (`prepare → test → build → verify → package → publish → release`). The catch: **GitLab CI/CD components only resolve on the same GitLab instance** (`$CI_SERVER_FQDN`). A component reference like `$CI_SERVER_FQDN/kuberocketci/ci-java17-mvn/build@0.1.1` will not reach out to `gitlab.com` - it resolves against *your* GitLab. So you must **onboard (mirror) the components into your instance before enabling GitLab CI**, and tag them at the version your template pins. ![Mirrored KubeRocketCI ci-java17-mvn CI/CD component in the self-hosted GitLab - review and build templates at tag 0.1.1](../assets/gitlab-ci-integration-kuberocketci/gitlab-cicd-component.png "The onboarded KubeRocketCI CI/CD component in the local GitLab instance") On the testbed, `make gitlab-ci` mirrors the component into the local GitLab as `kuberocketci/ci-java17-mvn` at tag `0.1.1` (with two local-only patches: native arm64 builds and pushing to the in-cluster GitLab Container Registry instead of Docker Hub). In your own environment, this is where you publish the KubeRocketCI components - or your own forked components - to your GitLab CI/CD Catalog so every project can `include` them. :::tip Onboard the components first, enable the Codebase second If you set `ciTool: gitlab` before the component exists at the pinned tag on your instance, KubeRocketCI still injects `.gitlab-ci.yml`, but the very first pipeline fails at the `include:` stage with an unresolved-component error. Mirror and tag the components first. ::: ### 3. Create the `.gitlab-ci.yml` Template ConfigMap KubeRocketCI does not invent the pipeline - it fills in a template stored in a **ConfigMap**, one per tech stack. The Codebase points at the template with an annotation (or KubeRocketCI falls back to a `gitlab-ci-{lang}-{buildtool}` naming convention), and your codebase name is substituted in before the file is committed. The ConfigMap for Java/Maven contains only orchestration - a `workflow` rule set, variables, and two `include`d component entry points (`review` and `build`) gated by rules: ```yaml title="manifests/gitlab-ci-java-maven-configmap.yaml (excerpt)" apiVersion: v1 kind: ConfigMap metadata: name: gitlab-ci-java-maven namespace: krci data: .gitlab-ci.yml: | workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_REF_PROTECTED == "true" - if: $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+$/ variables: CODEBASE_NAME: "{{.CodebaseName}}" CONTAINER_IMAGE: "maven:3.9-eclipse-temurin-17" IMAGE_REGISTRY: "$CI_REGISTRY_IMAGE" CHART_DIR: "deploy-templates" include: # REVIEW - merge-request validation (test/build/lint/sonar/dockerbuild-verify; no push) - component: $CI_SERVER_FQDN/kuberocketci/ci-java17-mvn/review@0.1.1 inputs: { codebase_name: ${CODEBASE_NAME}, container_image: ${CONTAINER_IMAGE}, chart_dir: ${CHART_DIR} } rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" # BUILD - protected/default branch: build, push image, deploy/tag - component: $CI_SERVER_FQDN/kuberocketci/ci-java17-mvn/build@0.1.1 inputs: { codebase_name: ${CODEBASE_NAME}, container_image: ${CONTAINER_IMAGE}, image_registry: ${IMAGE_REGISTRY}, chart_dir: ${CHART_DIR} } rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_REF_PROTECTED == "true" stages: [prepare, test, build, verify, package, publish, release] ``` Apply it once: ```bash $ kubectl apply -f manifests/gitlab-ci-java-maven-configmap.yaml configmap/gitlab-ci-java-maven created ``` These two rules are the entire control flow: the `review` component runs on `merge_request_event`, and the `build` component runs on the protected/default branch. That split produces the "MR → review, merge → build" behavior, with no webhook required. ## Enable GitLab CI on a Codebase (`spec.ciTool: gitlab`) With the Runner online, the components onboarded, and the ConfigMap applied, enabling GitLab CI is one Codebase manifest. The `ciTool: gitlab` field selects the engine; the annotation selects the template: ```yaml title="manifests/sample-gitlabci-codebase.yaml (excerpt)" apiVersion: v2.edp.epam.com/v1 kind: Codebase metadata: name: java-gitlabci-app namespace: krci annotations: app.edp.epam.com/gitlab-ci-template: gitlab-ci-java-maven # pick the ConfigMap spec: type: application lang: java framework: java17 buildTool: maven ciTool: gitlab # <-- run CI in GitLab CI, not Tekton strategy: import gitServer: gitlab gitUrlPath: /krci/java-gitlabci-app disablePutDeployTemplates: true ``` In the KubeRocketCI Portal, the component is a normal Codebase - same lifecycle, same branches, same deployment story as a Tekton app - but its **CI Tool reads `GitLab`**. That is the platform side of the integration: you manage the app on KubeRocketCI, even though its pipeline executes in GitLab. ![KubeRocketCI Portal overview for java-gitlabci-app: CI Tool set to GitLab, language Java, build tool Maven](../assets/gitlab-ci-integration-kuberocketci/krci-portal-gitlabci-codebase.png "The GitLab-CI Codebase as seen in the KubeRocketCI Portal") Once KubeRocketCI finishes, the injected `.gitlab-ci.yml` is visible in the GitLab project - the `{{.CodebaseName}}` placeholder now reads `java-gitlabci-app`, and the two component includes are wired and ready: ![.gitlab-ci.yml injected by the KubeRocketCI codebase-operator in GitLab, including the review and build CI/CD components](../assets/gitlab-ci-integration-kuberocketci/gitlab-ci-yml-injected.png "The .gitlab-ci.yml injected by the codebase-operator, visible in GitLab") ## The Pipeline in Action: Review on MR, Build on Merge The flow mirrors the Tekton path from the [previous posts](/blog/ephemeral-preview-environments-kubernetes-feature-branch) - open a merge request to validate, merge to build and publish - but every job runs in GitLab CI on the Runner. ```mermaid sequenceDiagram participant Dev as Developer participant GL as GitLab CE participant RN as GitLab Runner (k8s) participant REG as GitLab Container Registry Dev->>GL: open merge request GL->>RN: review pipeline (merge_request_event) RN-->>GL: test + lint + sonar + dockerbuild-verify green (no push) Dev->>GL: merge MR GL->>RN: build pipeline (protected branch) RN->>REG: buildkit builds + pushes image RN->>GL: maven-deploy + git-tag RN-->>GL: build pipeline green ``` In the GitLab project, both pipelines show green - one triggered by the merge request, one by the merge to `main`: ![GitLab Pipelines list for java-gitlabci-app: review (merge request) and build (push) pipelines passed on the GitLab Runner](../assets/gitlab-ci-integration-kuberocketci/gitlab-pipelines-list.png "Every GitLab CI pipeline passed - review on the MR, build on the merge") ### Review Pipeline (Merge Request) Opening a merge request triggers a `merge_request_event` pipeline that runs the `review` component - validation only, **no image push**. On the live run it finished green in **91 seconds**, 10 jobs across four stages: ```text review pipeline #4 (merge_request_event) - 91s - GREEN prepare init-values test test compile-check lint test helm-docs helm-lint dockerfile-lint build build sonar verify dockerbuild-verify # buildkit builds the image but does NOT push ``` The `sonar` job is a real SonarQube quality gate - the same engine the Tekton pipelines use - and `dockerbuild-verify` proves the image builds without publishing it. GitLab renders the MR with its pipeline result attached: ![GitLab merge request for java-gitlabci-app with the merge_request_event review pipeline passing before merge](../assets/gitlab-ci-integration-kuberocketci/gitlab-merge-request-pipeline.png "The review pipeline attached to the merge request in GitLab") ### Build Pipeline (Protected Branch) Merging the MR pushes to the protected `main` branch, which trips the second `workflow` rule and runs the `build` component. It is the review jobs **plus** the publishing stages - `buildkit-build` (build *and* push), `maven-deploy`, and `git-tag`. On the live run it finished green in **100 seconds**, 12 jobs: ```text build pipeline #5 (push, protected main) - 100s - GREEN prepare init-values test test compile-check lint test helm-docs helm-lint dockerfile-lint build build sonar package buildkit-build maven-deploy # build+push image, publish artifact publish git-tag # tag the release ``` The build pipeline graph in GitLab shows the stages fanning out - the same DAG model you get in Tekton, rendered by GitLab: ![GitLab CI build pipeline graph for java-gitlabci-app: prepare to publish stages, all 12 jobs green](../assets/gitlab-ci-integration-kuberocketci/gitlab-build-pipeline-graph.png "The build pipeline graph in GitLab - all jobs green") ### The Image Lands in GitLab's Container Registry The `buildkit-build` job builds the container image and pushes it to GitLab's bundled **Container Registry** using the CI job token - no external registry, no Docker Hub credentials. After the build pipeline, the image is published with a content-addressed tag, and `git-tag` has created the matching release tag: ```bash # image published by the build pipeline image: krci/java-gitlabci-app tags: [723b97d8, cee1e98e] # release tags created by the git-tag job tag: v0.1.0-cee1e98e tag: v0.1.0-723b97d8 ``` ![GitLab Container Registry for java-gitlabci-app: image pushed by the buildkit-build job of the build pipeline](../assets/gitlab-ci-integration-kuberocketci/gitlab-container-registry.png "The image published to GitLab's Container Registry by the build pipeline") From here, the [GitOps delivery story](/blog/ephemeral-preview-environments-kubernetes-feature-branch) is identical to the Tekton path - the published image feeds the same CodebaseImageStream and Argo CD deployment flow covered in part two of this series. ## GitLab CI vs Tekton: When to Use Which Both engines are first-class. The choice is about where your team already lives and what you want to own. | Dimension | Tekton (default) | GitLab CI (`ciTool: gitlab`) | |---|---|---| | Pipeline definition | Tekton `PipelineRun` (KRCI-managed) | `.gitlab-ci.yml` (operator-injected, GitLab-native) | | Trigger mechanism | Tekton webhook | GitLab `workflow` rules (no webhook) | | Where job logs live | KubeRocketCI Portal / Tekton | GitLab CI UI | | Reusable build logic | Tekton tasks / pipeline library | GitLab CI/CD **components** (must be on your instance) | | Image build | kaniko | buildkit | | Runner needed | No (Tekton bundled) | **Yes** - install a GitLab Runner | | Best when | you want a unified, K8s-native CI surface in the Portal | your org standardizes on GitLab CI and its component catalog | **Use Tekton** when you want every Codebase to share one Kubernetes-native CI surface inside the Portal, with pipeline history in [Tekton Results](/blog/kubernetes-native-cicd-tekton-kuberocketci). **Use GitLab CI** when your organization has standardized on GitLab CI/CD components and wants pipeline authoring and logs to stay in GitLab - while still getting the KubeRocketCI Codebase lifecycle, Portal, and GitOps deployment around them. And because the choice is per-Codebase, you can migrate one application at a time. ## Frequently Asked Questions ### What does `ciTool: gitlab` do in KubeRocketCI? It switches a Codebase's CI engine from Tekton to GitLab CI. Instead of wiring the pipeline into Tekton, KubeRocketCI generates a `.gitlab-ci.yml` in the repository, and the pipeline runs as native GitLab CI jobs on a GitLab Runner. The only valid values for the field are `tekton` and `gitlab`. ### Does KubeRocketCI replace GitLab CI, or run on top of it? It runs on top of it. With GitLab CI integration, KubeRocketCI manages the Codebase lifecycle, owns the injected `.gitlab-ci.yml`, and provides the Portal and GitOps deployment - but the pipeline itself executes in GitLab CI, and the job logs live in GitLab. There is no Tekton in the pipeline path at all. ### Do I need a GitLab Runner for the GitLab CI path? Yes. KubeRocketCI bundles Tekton but not a GitLab Runner. With `ciTool: gitlab`, jobs run as GitLab CI and need a registered runner to execute. The testbed installs the official `gitlab/gitlab-runner` Helm chart with the Kubernetes executor, registered via a GitLab 17.x runner authentication token. Without a runner, pipelines stay `pending`. ### Why do I have to onboard the CI/CD components into my own GitLab instance? Because GitLab CI/CD components only resolve on the same instance (`$CI_SERVER_FQDN`). A reference like `$CI_SERVER_FQDN/kuberocketci/ci-java17-mvn/build@0.1.1` is looked up on *your* GitLab, not `gitlab.com`. So you must mirror (onboard) the KubeRocketCI components - or your own forks - into your instance and tag them at the version your template pins, before enabling GitLab CI on any Codebase. ### What is the ConfigMap for in GitLab CI integration? The ConfigMap holds the `.gitlab-ci.yml` template that KubeRocketCI injects. You create one per tech stack (for example `gitlab-ci-java-maven`), and the Codebase points at it with an annotation. KubeRocketCI substitutes your codebase name and commits the result. The template is thin orchestration - it `include`s the onboarded CI/CD components and gates them with `workflow` rules. ### How does the MR-to-build flow work without a webhook? GitLab CI integration relies on GitLab's own `workflow` rules instead of a webhook. The injected `.gitlab-ci.yml` includes the `review` component gated on `$CI_PIPELINE_SOURCE == "merge_request_event"` and the `build` component gated on the protected/default branch. Opening a merge request runs review; merging pushes to the protected branch and runs build. GitLab triggers both natively, so no project webhook is created. ### Can Tekton and GitLab CI codebases coexist on the same cluster? Yes. The choice is per-Codebase, and the same `gitlab` GitServer is reused for both engines. On the testbed, Tekton codebases (the Go app from parts one and two) run side by side with the GitLab-CI Java app on the same KubeRocketCI platform. You can adopt GitLab CI for one application without touching the rest. ### Where does the container image go in the GitLab CI build pipeline? The `buildkit-build` job builds and pushes the image to GitLab's bundled Container Registry using the CI job token - no external registry or Docker Hub credentials required. On the testbed it pushes to the in-cluster GitLab registry, and the `git-tag` job creates a matching release tag. From there the image feeds the same CodebaseImageStream and Argo CD deployment flow as the Tekton path. ## Summary GitLab CI integration in KubeRocketCI lets a Codebase run its pipeline in **GitLab CI instead of Tekton** by setting `spec.ciTool: gitlab`. KubeRocketCI then **generates a `.gitlab-ci.yml`** that pulls in reusable GitLab CI/CD components, gated so that a merge request runs a review pipeline and a merge runs a build pipeline. There is no webhook and no Tekton in the path - the proof is **zero `PipelineRun`s** for the Codebase. Before you enable it, set up three things: a **GitLab Runner** (KubeRocketCI does not bundle one), the **onboarded CI/CD components** in your own instance (they only resolve on the same GitLab), and the **template ConfigMap** KubeRocketCI injects. Complete these once, and subsequent GitLab-CI Codebases require no additional setup. I ran the full flow on the local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed: review pipeline green in 91s (10 jobs), build pipeline green in 100s (12 jobs), image pushed to GitLab's Container Registry, release tags created, and zero Tekton PipelineRuns - every command and result above is from that run. The most useful next steps from here: - Spin up the testbed with [Try KubeRocketCI Locally](/blog/try-kuberocketci-locally), then run `make gitlab-ci` and `make e2e-gitlabci` to reproduce this flow. - Read the [3.13 upgrade notes](/docs/operator-guide/upgrade/upgrade-krci-3.13) for the multi-CI `ciTool` enum and the [Codebase API](/docs/api/codebase) for the field reference. - Compare the Tekton path in [Kubernetes-native CI/CD with Tekton](/blog/kubernetes-native-cicd-tekton-kuberocketci) and the GitOps delivery in [Ephemeral Preview Environments](/blog/ephemeral-preview-environments-kubernetes-feature-branch). - Browse the [KubeRocketCI documentation](https://docs.kuberocketci.io) and the [basic concepts](/docs/basic-concepts) for Codebases, CodebaseImageStreams, and deployment flows. KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the testbed are all on [GitHub](https://github.com/KubeRocketCI/try-kuberocketci). {/* cspell:ignore gitlabci buildkit kaniko configmap mvn temurin xr9cm */} --- ## A Portable Kubernetes Audit Trail: Who Changed What # A Portable Kubernetes Audit Trail: Who Changed What | KubeRocketCI You trigger pipelines. You update a `GitServer`, promote a deployment, adjust a `Codebase`. Much of that flows through GitOps and lands in your Git history - but not all of it. Some changes you make directly against the cluster, from the portal or `kubectl`, because that's the quickest way to get something done. That's normal, everyday platform work. What's harder is answering a fair question that comes up afterward: *what changed, when, and who made the change?* - especially for the changes that never went through a pull request. On managed Kubernetes the raw answer exists - the API server writes an audit record for every request - but getting to it is another story. On Amazon EKS those records stream to CloudWatch. Answering a simple operational question then means writing a Logs Insights query, knowing the retention window, and having the right IAM. Great for a SIEM pipeline; heavy for a Tuesday-afternoon "what just changed?". KubeRocketCI ships a lighter, portable alternative called **krci-audit** - a self-contained, queryable audit trail that works the same on EKS, on-prem, or a local kind cluster. The catch, which most of this post is about: an audit trail is not free, so you should be deliberate about what you capture. Here's the reasoning - why native audit logs leave a gap for day-to-day questions, how krci-audit fills it without a cloud dependency, the performance trap of auditing everything (with real numbers from a running cluster), and where this is going next. ## Why Aren't Native Kubernetes Audit Logs Enough for "Who Changed What?" **Short answer: they're built for security forensics and SIEM, not for a quick operational lookup.** The data exists - every managed Kubernetes distribution can emit [API server audit logs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/), a structured record of who called the API, what they changed, and when. On EKS you enable the `audit` log type and the control plane ships it to CloudWatch Logs. As a foundation that's exactly right: immutable, comprehensive, centrally retained, and easy to alert on. But for *operational* questions it has friction: - **Quick access is a query, not a lookup.** "Who created this PipelineRun?" becomes a CloudWatch Logs Insights query against a large, verbose stream - filtered by `objectRef`, `verb`, and time - not a one-line answer. - **It's cloud-locked.** The retrieval path on EKS (CloudWatch), on AKS (Azure Monitor), and on-prem (a file sink you wire up yourself) are all different. Your runbook doesn't port. - **It's undifferentiated.** The native stream captures *everything* the audit policy allows - most of which is control-plane noise you'll never ask about (more on that below). - **It lives outside your platform.** Developers using the KubeRocketCI portal can't see "who triggered this run" without leaving the tool and querying a cloud console they may not have access to. None of this makes native audit logging wrong - it makes it the *wrong tool for the quick, in-platform question*. That's the gap krci-audit targets. ## How Does krci-audit Give You a Portable Audit Trail? **It captures a scoped set of admission changes inside your cluster and makes them instantly queryable** - no dependency on any cloud provider's logging service. krci-audit is a small, self-contained component that does exactly that. ```mermaid flowchart LR A[Resource change] --> B[Admission webhook] B --> C[Vector] C --> D[(PostgreSQL)] D --> E[Read API] E --> F[Portal / consumers] ``` A resource change hits a validating admission webhook, which ships the record through Vector into an append-only PostgreSQL table; a read-only API then serves it to the portal and other consumers. The pieces: | Component | Role | | --- | --- | | **Capture** | A `ValidatingWebhookConfiguration` (backed by [kube-audit-rest](https://github.com/RichardoC/kube-audit-rest)) receives the `AdmissionReview` for each matching change. It runs with `failurePolicy: Ignore` and a 1-second timeout, so it **never blocks a platform operation**. | | **Ship** | A [Vector](https://vector.dev/) sidecar tails the capture log, keeps only the events you asked for, and writes them to PostgreSQL. | | **Store** | A dedicated PostgreSQL database holds events in a monthly-partitioned, **append-only** table. Writes use a least-privilege role that cannot update or delete existing rows. | | **Read API** | A separate, read-only service exposes initiator lookup and a filterable events query. It connects as a SELECT-only role, so the read path can never alter the trail. | Because the whole thing runs *inside* your cluster, the experience is identical on EKS, AKS, GKE, OpenShift, or a laptop kind cluster - the same install, the same API, the same portal feature. It's an [add-on](/docs/next/operator-guide/monitoring-and-observability/audit-trails-setup), off by default, that you turn on when you have a reason to. ## Why Shouldn't You Audit Everything? **Because an admission webhook sits in the write path of the API server** - so a catch-all policy taxes every write *and* buries the trail in noise. Every request matching the webhook's rules is paused, sent to the webhook, and awaited (within the timeout) before the write completes. Point it at `*` - all resources - and you pay that cost on everything. How much noise? To put real numbers on it, I measured the [local **try-kuberocketci** testbed](/blog/try-kuberocketci-locally) - a single-node [kind](https://kind.sigs.k8s.io/) cluster (Kubernetes v1.36.1) running the full platform and otherwise idle. Comparing each `Lease` object's `resourceVersion` over a 30-second window shows the churn a catch-all audit policy would capture (reproduce it with `kubectl get leases -A`, twice, 30 seconds apart): - **31 `Lease` objects, and all 31 changed within those 30 seconds** - pure leader-election heartbeat traffic. Extrapolated: **~3,720 `Lease` UPDATE admissions per hour**, none of which anyone will ever audit. - And that's before `Events` (13), `EndpointSlices` (50), `Pods` (54), and `ConfigMaps` (122) - all high-churn, all captured by a `*` policy. Now compare that to the **scoped** trail the same testbed actually recorded over ~36 hours, using krci-audit's default configuration (shown below): | Resource | Events | | --- | --- | | PipelineRuns | 97 (71 UPDATE, 18 DELETE, 8 CREATE) | | CodebaseBranches | 12 | | GitServers | 4 | | CodebaseImageStreams | 2 | | CDStageDeployments / Stages | 3 | That's **118 meaningful events over a day and a half** - roughly **80 a day** - versus **~3,720 an hour** of lease noise from a single un-scoped resource type. A catch-all policy would be on the order of a thousand times larger, almost entirely with records no one wants. **The design that avoids this:** scope at the webhook *rules*, not downstream. When a resource isn't in the rules, the API server **never calls the webhook at all** - zero added latency, zero stored noise. krci-audit's default rules capture only what the platform cares about: ```yaml capture: failurePolicy: Ignore # never block a platform operation timeoutSeconds: 1 rules: - apiGroups: ["v2.edp.epam.com"] # KubeRocketCI CRDs resources: ["*"] - apiGroups: ["tekton.dev"] resources: ["pipelineruns"] level: metadata # capture object metadata, not full bodies (bounds size and PII) ``` The guidance is simple: **audit with a goal, not by default.** Decide what question you need the trail to answer - "who triggered pipelines", "who touched Git server config", "who deleted deployments" - and capture *those* resources. You can widen the filter later with a `helm upgrade`; you can't un-spend the latency budget you burned auditing lease renewals. ## How Do You See Who Triggered a Pipeline? **The KubeRocketCI portal shows a "Triggered By" field on the PipelineRun details page** - the first thing krci-audit powers, answering a question every platform user eventually asks: *who started this run?* ![KubeRocketCI PipelineRun details page showing the Triggered By field resolved to a service account](../assets/krci-audit-triggered-by-field.png) It resolves the run's creator from the audit trail and classifies them: - A **human** user (for runs started from the portal), or - An **automation** identity - the service account behind a Git webhook or scheduled build (as shown above), or - **N/A**, when the initiator can't be resolved (for example, historical runs created before auditing was enabled). It's deliberately distinct from the existing **Author** column, which shows the Git commit author. The person who wrote the code is not always the identity that launched the run. Under the hood the portal calls the krci-audit read API with the run's `kind`, `namespace`, and `name`; no extra annotations, no changes to how pipelines are created. ![KubeRocketCI PipelineRuns list in the CI/CD Pipelines section](../assets/krci-audit-pipelineruns-list.png) ## How Do You Query the Audit Trail Directly? **Through a plain HTTP API** - the same data the portal uses, handy for incident triage, scripts, and compliance exports. It's a `ClusterIP` service (in-cluster only), so port-forward it to try from your workstation: ```bash kubectl -n krci-audit port-forward svc/krci-audit-api 18080:8080 ``` **"Who created this resource?"** - an initiator lookup by `kind`, `namespace`, and `name`: ```bash curl "http://localhost:18080/api/v1/audit/initiator?kind=PipelineRun&namespace=krci&name=build-test-go-app-main-8a2c" ``` ```json { "actor": "system:serviceaccount:krci:krci-admin", "operation": "CREATE", "found": true, "timestamp": "2026-07-07T09:41:13Z" } ``` **"What happened to this kind of object?"** - a filterable, paginated events query (by `kind`, `operation`, `actor`, time): ```bash curl "http://localhost:18080/api/v1/audit/events?kind=PipelineRun&operation=DELETE&limit=20" ``` Real questions this answers without a cloud console: - *"Who deleted my Codebase last night?"* - filter by `operation=DELETE` and the object. - *"Show me everything this service account did."* - filter by `actor`; the same query backs a user's own activity view. - *"Produce the change history for this resource for an audit."* - export the filtered result. Because the store is **append-only** and time-stamped, it doubles as compliance evidence: an immutable record of who changed what. It maps cleanly to **SOC 2**, **ISO 27001**, **GDPR**, and **PCI DSS** change-tracking controls, and it's retrievable through the API without handing anyone direct database access. ## When Should You Still Use Native Audit and SIEM? **When you need cluster-wide forensics, tamper-proof off-cluster retention, or cross-signal correlation - keep shipping the API server audit log to your SIEM.** To be clear about scope: krci-audit **complements** native audit logging, it doesn't replace a security program. krci-audit is for the **operational, in-platform** question - fast, portable, developer-facing "who/what/when" - not the security org's system of record. ## Future Work This is the first slice of a broader capability. Next up: - **Authentication and authorization on the API.** Today the read API is unprotected and reachable only in-cluster (ClusterIP, no Ingress), with network scope as the interim boundary. Role-based access - so the trail respects who's allowed to see what - is the next milestone. - **"Who changed what" across every object, in the portal.** The initiator today is surfaced for PipelineRuns; the same audit data can enrich *every* KubeRocketCI resource - a change history and last-modified-by on Codebases, environments, Git servers, and more - turning the portal into a single place to answer accountability questions for the whole platform. ## Next Steps - **Operators:** follow the [Audit Trails Setup guide](/docs/next/operator-guide/monitoring-and-observability/audit-trails-setup) to enable the add-on, choose what to capture, and set retention. - **Users:** see the [Pipelines Overview](/docs/next/user-guide/pipelines) for the Triggered By field in context. - **New to the platform?** Start with [What is KubeRocketCI](/docs/about-platform) or [try it locally](/docs/quick-start/quick-start-overview). Audit trails are one of those features that's invisible until the moment you need it - and then it's the only thing that matters. Turn it on with a goal, keep it scoped, and it'll answer "who changed what?" in one call instead of one afternoon. --- ## Claude Code Telemetry: Token and Cost Metrics In the last 24 hours, one engineer's Claude Code sessions burned **$166.97**, **833K input/output tokens**, and **959 API requests** - and I can tell you exactly which Jira story got that money, which model spent it, which subagents did the heavy lifting, and that the prompt-cache hit ratio stayed at **97.2%** the whole time. Not from a vendor console: from a self-hosted Grafana stack fed by telemetry Claude Code already emits. That last part is the point. Claude Code has native [OpenTelemetry support](https://code.claude.com/docs/en/monitoring-usage) - rich metrics and events for tokens, cost, sessions, tools, skills, and agents - but nothing to look at them with. As soon as AI-assisted development becomes a line item, every engineering leader asks the same questions: *what are we spending, on what, and is it worth it?* This post shows how we answer them at KubeRocketCI with a small OTel Collector → Prometheus + Loki → Grafana pipeline that runs identically on a laptop and on a team Kubernetes cluster - the **claude-code-telemetry** stack - without ever capturing a single prompt or line of your code. Here's the path we'll take: why an "AI Factory" needs production-grade metering, what Claude Code emits out of the box, how to stand the stack up locally in minutes, how to attribute cost to projects and Jira tickets, how to deploy it for a whole team on Kubernetes (including CI/CD pipelines), the cardinality trap to avoid - and what a real day of data actually says. ## Why Should You Measure Claude Code Usage? **Because AI agents are production workers now, and production workers get metered.** When developers and CI pipelines run Claude Code daily, token spend becomes real budget - and without attribution you can't answer which project, epic, or team consumed it, whether the spend produced code, or how adoption is trending. Think of it as the observability layer of an **AI Factory**: the same discipline you apply to build minutes, cloud bills, and test flakiness, applied to AI-assisted engineering. Concretely, the questions that come up in every rollout: - **FinOps:** cost by project, epic, story, user, and model - who spends, on what work. - **Efficiency:** cost per commit, cost per 1K lines changed, cache hit ratio - is the spend producing output. - **Adoption:** daily/weekly active users, sessions, active hours - is the investment being used. - **Governance:** which plugins, skills, and MCP servers run where, in which versions - what your fleet actually executes. And one hard constraint shapes everything: this must work **without capturing prompts, responses, or file contents**. Usage analytics is not surveillance - only aggregate counters and structured metadata should ever cross the wire. ## What Telemetry Does Claude Code Emit Out of the Box? **Claude Code natively exports OTLP metrics and events once you set a few environment variables** - no wrappers, no scraping, no plugins. The metric names are fixed by the CLI, but every one carries dimensions you can slice by, and you can add your own. The metrics that matter most for tokenomics: | Metric | What it answers | | --- | --- | | `claude_code.token.usage` | tokens by `type` (input/output/cacheRead/cacheCreation), `model`, `agent.name`, `skill.name` | | `claude_code.cost.usage` | USD by the same dimensions, plus plugin and marketplace | | `claude_code.session.count` / `active_time.total` | adoption and engagement | | `claude_code.lines_of_code.count`, `commit.count`, `pull_request.count` | output to divide cost by | Alongside metrics, Claude Code emits **events** - `api_request` (per-call cost and token detail), `skill_activated`, `tool_result`, `tool_decision` - which carry the high-cardinality forensics: per-session, per-prompt, per-tool detail. Two facts drive the whole design. First, **you cannot invent new metric names**, but you *can* inject new **dimensions** and derive anything downstream with PromQL or LogQL. Second, content capture is opt-in and off by default in our setup: prompts, assistant responses, file contents, and raw API bodies stay dark - we enable only `OTEL_LOG_TOOL_DETAILS` so skill, agent, and tool *names* are visible. ## Which Claude Products Support OpenTelemetry? **Only Claude Code emits OTel today - the CLI, its IDE extensions, and the Agent SDK all share the same environment-variable configuration.** Claude Desktop's chat has no telemetry export (only its Cowork feature offers an admin-configured OTLP endpoint on Team/Enterprise plans), Claude Code on the web currently emits nothing, and raw API usage is served by Anthropic's pull-based Usage and Cost APIs instead. | Product | OTel export | Mechanism | | --- | --- | --- | | Claude Code CLI | Yes | `CLAUDE_CODE_ENABLE_TELEMETRY` + `OTEL_*` env vars | | VS Code / JetBrains extensions | Yes | inherit the CLI's `settings.json` | | Agent SDK | Yes | same pipeline, runs as a child process | | Claude Desktop (chat) | No | Cowork feature only, via admin portal | | Claude Code on the web | Not yet | managed sandbox exposes no `OTEL_*` config | | Claude API | No | pull-based [Usage & Cost / Analytics APIs](https://code.claude.com/docs/en/monitoring-usage) | Two practical consequences for a rollout. First, the stack described here captures **CLI, IDE-extension, and Agent SDK sessions** - if part of your team works in Claude Code on the web, that spend is invisible to OTel until Anthropic closes the gap, so budget reconciliation still needs the Usage and Cost API as the financial source of truth. Second, if you later enable Cowork telemetry, note its content is **not redacted by default** - unlike Claude Code - so apply filtering in the collector before it shares a backend with this pipeline. ## How Do You Build a Self-Hosted Telemetry Stack? **With one OTel Collector fanning out to Prometheus for metrics and Loki for events, visualized in Grafana** - the standard trio, packaged twice from the same config so a laptop and a Kubernetes cluster behave identically. That's the entire [claude-code-telemetry](https://github.com/KubeRocketCI/claude-code-telemetry) architecture: ```mermaid flowchart LR A[Claude Code - laptops] --> C[OTel Collector] B[Claude Code - CI/CD pipelines] --> C C --> P[(Prometheus - metrics)] C --> L[(Loki - events)] P --> G[Grafana dashboards] L --> G ``` The Collector is the heart: it receives OTLP (gRPC/HTTP), stamps every datapoint with a `collector.env` attribute so laptop and cluster traffic never get confused, converts resource attributes into Prometheus labels (`resource_to_telemetry_conversion`), and ships full-fidelity events to Loki. On top sit six pre-built dashboards, organized into three Grafana folders by audience: ![Grafana dashboard folders for Claude Code telemetry: usage, governance, and operational scopes](../assets/claude-code-telemetry/grafana-dashboards-overview.png) - **Claude Code** - Usage Audit, Efficiency & ROI, Operations & Adoption, Session Explorer: the daily-driver views. - **Governance** - Governance & Security: plugins, skills, permission decisions across the fleet. - **Operational** - Collector Health: is the pipeline itself alive and ingesting. ## How Do You Enable It for Local Development? **Start the docker-compose testbed, merge one `env` block into your Claude Code settings, and data flows within a minute.** The local stack is the source of truth we iterate on - same collector pipeline, same dashboards as the cluster deployment: ```bash git clone https://github.com/KubeRocketCI/claude-code-telemetry.git cd claude-code-telemetry/local docker compose up -d # Grafana :3000 · Prometheus :9090 · Loki :3100 · OTLP :4317 ``` Then merge `local/claude-settings.snippet.json` into `~/.claude/settings.json` - the essentials: ```json { "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_METRICS_EXPORTER": "otlp", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4317", "OTEL_LOG_TOOL_DETAILS": "1", "OTEL_LOG_USER_PROMPTS": "0", "OTEL_LOG_ASSISTANT_RESPONSES": "0", "OTEL_LOG_TOOL_CONTENT": "0" } } ``` Restart Claude Code, open Grafana, and the Usage Audit dashboard starts filling with your own traffic. This loop - real usage in, dashboard iteration out - is how every panel in this post was built. ## How Do You Attribute Cost to Projects and Jira Tickets? **By injecting business dimensions through `OTEL_RESOURCE_ATTRIBUTES` before each session starts.** Claude Code reads the variable once at startup and stamps every metric and event with it - so the discipline is simple: *one Jira ticket ≈ one session*. ```bash export OTEL_RESOURCE_ATTRIBUTES="project=krci-portal,jira.epic=EPMDEDP-15000,jira.story=EPMDEDP-17147" claude ``` Three custom keys are enough to start: `project` (the git repository), `jira.epic`, and `jira.story`. User identity comes free - Claude Code natively attaches `user.email`, `user.id`, and `organization.id` to everything, so per-person breakdowns need no injection at all. The Session Explorer dashboard shows what this buys you. Every session becomes a row with its epic, story, project, user, cost, and event count - sortable, filterable, drillable: ![Claude Code Session Explorer dashboard showing per-session cost attributed to Jira epics and stories](../assets/claude-code-telemetry/session-explorer-dashboard.png) Reading my own last 24 hours: **$166.97 total**, and the table names the culprits - two long `krci-workspace` sessions on story EPMDEDP-17188 at **$67.68** and **$60.66**, a `edp-keycloak-operator` triage on EPMDEDP-16043 at **$24.93**, and the session that built the telemetry stack itself at **$1.99**. When a manager asks "what did that epic cost in AI assistance?", this table *is* the answer - down to the session. ## How Do You Deploy It for a Team on Kubernetes? **With the repo's self-contained Helm bundle - one `helm install`, no operators, no CRDs, no cluster-monitoring prerequisites.** Local PoC is for one laptop; team consumption means everyone's Claude Code - and your CI runners - pushing OTLP to a single in-cluster endpoint with shared dashboards: ```bash helm dependency update deploy-templates helm install claude-code-telemetry deploy-templates \ -n claude-code-telemetry --create-namespace ``` The bundle wraps the upstream OTel Collector, Prometheus, Loki, and Grafana charts, and it is **deliberately isolated**: its Prometheus scrapes exactly two static targets (the collector's data and self-telemetry ports), all cluster-wide discovery is disabled, no node-exporter or kube-state-metrics tag along, and RBAC is namespace-scoped. It won't fight the observability stack you already run - and if you have one, each component toggles off (`grafana.enabled: false`, `prometheus.enabled: false`, …) so you can wire just the collector into existing storage. Three things change at team scale compared to the laptop: 1. **Distribution** - ship the `env` block through a managed `settings.json` (MDM), so telemetry is on for everyone, pointed at the team endpoint, and can't be silently unset. 2. **Authentication** - expose the OTLP endpoint behind auth (`OTEL_EXPORTER_OTLP_HEADERS` with a bearer token, or Claude Code's `otelHeadersHelper` for short-lived tokens). 3. **Enforcement** - the Collector becomes the governance point users cannot bypass: inject defaults for missing attributes, validate `jira.*` values against `^[A-Z]+-[0-9]+$`, and drop or route telemetry that arrives without a `project`. Every datapoint's `collector.env` label (`local-poc` vs `k8s`) keeps sources distinguishable, so laptops and the cluster can even feed shared dashboards during a migration without mixing data. ## How Do You Collect Metrics from CI/CD Pipelines? **Exactly the same way - headless Claude Code in a pipeline step is still Claude Code, and the same OTLP env vars apply.** This is where the AI Factory framing stops being a metaphor: automated agents doing code review, test generation, or triage inside [Tekton pipelines](/blog/kubernetes-native-cicd-tekton-kuberocketci) spend tokens too, and they should land in the same ledger as human sessions. In a pipeline task, set the same telemetry variables plus attribution that identifies the machine work: ```yaml env: - name: CLAUDE_CODE_ENABLE_TELEMETRY value: "1" - name: OTEL_METRICS_EXPORTER value: "otlp" - name: OTEL_LOGS_EXPORTER value: "otlp" - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://otel-collector.claude-code-telemetry:4317" - name: OTEL_RESOURCE_ATTRIBUTES value: "project=$(params.codebase),jira.epic=none,jira.story=$(params.ticket)" ``` The pipeline populates `project` from the codebase it builds and the ticket from the change it processes - attribution comes from parameters you already have. One caveat worth knowing: Claude Code does **not** propagate `OTEL_*` variables to subprocesses, so only the CLI process itself is instrumented - which is exactly what you want in a build pod. The result: a single Grafana view where "cost of AI per merge request" and "cost of AI per developer day" sit side by side. ## What Is the Cardinality Trap? **Every unique label combination in Prometheus is a stored time series - so unbounded labels like Jira story IDs will eventually blow up your metrics store.** This is the telemetry twin of the lesson from our [Kubernetes audit trail post](/blog/kubernetes-audit-trail-who-changed-what): collect with a goal, not by default. The rule we apply, enforced in the Collector: - **Bounded dimensions** (`model`, `type`, `agent.name`, `skill.name`, `project`, `jira.epic`) → safe as **metric labels**, fast Grafana breakdowns. - **Unbounded dimensions** (`jira.story`, `session.id`, `prompt.id`) → **events in Loki only**; derive per-story cost with LogQL over `api_request` events. A single-user PoC can keep `jira.story` on metrics for convenience - ours does. At team scale, the Collector strips it from the metrics pipeline while Loki keeps the full detail. Same data, right store: Prometheus answers "how much, by which bounded dimension" in milliseconds; Loki answers "what exactly happened in that session" when you drill in. ## What Do the Numbers Actually Say? **That efficiency metrics are the fastest way to turn raw spend into decisions - including uncomfortable ones.** The Efficiency & ROI dashboard divides cost by output, and a real day of data is more instructive than any synthetic demo: ![Claude Code Efficiency and ROI Grafana dashboard with cache hit ratio, cost per commit, and cost per active hour](../assets/claude-code-telemetry/efficiency-roi-dashboard.png) - **Cache hit ratio: 97.2%.** Prompt caching is doing enormous work - at roughly a tenth of the input price, those cache reads are the difference between $167 and a number several times larger. If this ratio drops, something in your workflow broke; it's the first efficiency metric worth alerting on. - **Cost per active hour: $31.10.** A defensible, explainable unit price for AI pairing - the number to put next to an engineer's loaded hourly cost when someone asks about ROI. - **Cost per 1K lines changed: $79.28.** Noisy on any single day, meaningful as a trend across weeks and projects. - **Cost per commit: $∞.** Total cost divided by `claude_code.commit.count`, which counts only commits made by Claude Code itself. We review and commit by hand, so the counter stays at zero and the panel shows infinity. If you commit through the agent, this becomes a real spend-per-delivered-commit number. The Usage Audit dashboard tells you *where* the tokens went: ![Claude Code Usage Audit dashboard showing tokens by model, cost by project and epic, and subagent token usage](../assets/claude-code-telemetry/usage-audit-dashboard.png) Over the two-day range: cost concentrates in **krci-workspace / EPMDEDP-16757 ($97.6)** and **edp-keycloak-operator / EPMDEDP-16043 ($80.7)**, with the telemetry stack's own development a modest **$10.4**. Token type over time is dominated by `cacheRead` - visual confirmation of that 97.2%. And the subagent panel is the sleeper hit: **custom agents consumed 18.2M tokens** against 4.08M for built-in Explore and 3.13M for general-purpose - the multi-agent workflows you build are where the budget actually goes. Tool usage (2,059 Bash invocations vs 475 Reads and 288 Edits in the window) rounds out the picture of what the agent fleet physically does all day. ## How Do You Govern Plugins and Skills Across a Team? **The same event stream doubles as a software inventory - who runs which plugin, in which version, triggered how.** The Governance & Security dashboard breaks down skill activations by trigger, which quietly answers an adoption question every platform team has: ![Grafana panel showing Claude Code skill activations by source and plugin skill usage by trigger](../assets/claude-code-telemetry/skill-plugin-usage.png) In our data, most activations are `claude-proactive` or `nested-skill` rather than `user-slash` - the agent invokes the team's KubeRocketCI plugins on its own more often than humans type slash commands. That's exactly what you want from packaged expertise, and now it's measurable rather than anecdotal. The plugin inventory closes the loop for fleet management: ![Claude Code plugin inventory panel listing installed plugins, versions, and scope per user](../assets/claude-code-telemetry/plugin-inventory-governance.png) Every user session reports its loaded plugins with marketplace, version, and install scope. When you ship a fixed `krci-godev` and need to know who's still loading 0.6.0 - the dashboard already knows. ## Next Steps - **Try it on your laptop:** clone [claude-code-telemetry](https://github.com/KubeRocketCI/claude-code-telemetry), run `docker compose up -d` in `local/`, merge the settings snippet, and watch your own session appear. - **Deploy for your team:** install the Helm bundle from `deploy-templates/` and distribute the `env` block via managed settings. - **Go deeper on the design:** the repo's `docs/analytics.md` covers the attribution model, cardinality guidance, and rollout plan; `spec/` catalogs the full Claude Code OTEL vocabulary. - **See the platform context:** Claude Code is one worker in the factory - [KubeRocketCI](/docs/about-platform) provides the CI/CD floor it works on, from [Tekton pipelines](/blog/kubernetes-native-cicd-tekton-kuberocketci) to [audit trails](/blog/kubernetes-audit-trail-who-changed-what). New here? [Try it locally](/blog/try-kuberocketci-locally). Token spend is the new build minutes: invisible until the invoice arrives, obvious once it's on a dashboard. Claude Code already emits everything you need - stand up the backend, inject three attributes, and "what does AI development cost us?" becomes a Grafana query instead of a guess. --- ## Stale Branch Cleanup on Kubernetes: Keep Your Platform in Sync with Git Every feature branch you create in KubeRocketCI becomes a `CodebaseBranch` resource in Kubernetes - the object that drives its CI pipelines, records its built images, and feeds [ephemeral preview environments](/blog/ephemeral-preview-environments-kubernetes-feature-branch). But when that branch merges and someone deletes it in GitLab or GitHub, the Kubernetes side never hears about it. The `CodebaseBranch` lives on: it clutters the Portal, offers a Build button for a branch that no longer exists, and on a busy platform these orphans accumulate by the hundreds. KubeRocketCI 3.14 closes that gap. The codebase-operator now periodically verifies every branch against the real git repository, marks missing ones with a **Stale** condition (and a badge in the Portal), and - if you opt in - deletes them automatically, while refusing to touch any branch that is still wired into a deployment. This post shows the whole lifecycle live on the local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed: every screenshot and command output below is from a real run. ## Why Stale Branches Accumulate The branch-per-feature workflow that makes [preview environments](/blog/ephemeral-preview-environments-kubernetes-feature-branch) so effective is also what produces the litter. The lifecycle of a healthy feature branch ends with a merge and a `git branch -d` - usually a checkbox in the merge request. That deletion happens *in git*. The platform objects created for the branch - the `CodebaseBranch` resource, its `CodebaseImageStream` (the record of images built from it) - are Kubernetes resources, and nothing in the git provider tells Kubernetes to remove them. The result is drift between two sources of truth: - **The Portal shows branches that do not exist.** New team members trigger builds against them and get confusing clone failures. - **Resources pile up.** Platforms with dozens of codebases and a branch-per-ticket habit accumulate hundreds of dead `CodebaseBranch` and `CodebaseImageStream` objects. - **Nobody knows which ones are safe to delete.** A `CodebaseBranch` may look abandoned and still be the input of a live deployment - deleting the wrong one breaks an environment. Manual cleanup does not scale, and a naive cron that deletes anything old is dangerous. What you want is exactly what a Kubernetes operator is for: continuously observe external state (git), record it on the resource, and act on it with guardrails. ## How Stale Branch Detection Works The codebase-operator runs a **stale branch checker** - a leader-elected background loop, not a webhook - because branch deletion in git produces no Kubernetes event to react to. On every tick it makes one cheap `git ls-remote` call per repository (no cloning, no working tree) and compares the listed branches against the `CodebaseBranch` resources in the namespace: ```mermaid flowchart LR A[Tickerevery 24h] --> B[git ls-remoteone call per repo] B --> C{Branch existsin git?} C -- yes --> D[Clear Stale conditionif it was set] C -- no --> E[Mark: Stale condition +stale label + event] E --> F{Cleanup strategy?} F -- markdefault --> G[Keep resource,visible in Portal] F -- auto --> H{Used by adeployment?} H -- no --> I[Delete branch,image stream GC'd] H -- yes --> J[Retain + recordthe reason] ``` The verdict is recorded in three places, so both humans and tooling can consume it: - **A `Stale` status condition** on the `CodebaseBranch` - the source of truth, with a reason (`BranchNotFoundInGit`) and a human-readable message. - **The `app.edp.epam.com/stale` label**, mirroring the condition so you can filter with a plain label selector. - **Kubernetes events** (`BranchStale`, `StaleBranchDeleted`, `StaleBranchRetained`, `BranchStaleResolved`) for audit trails and alerting. Just as important is what the checker will *not* do: - **No marking on infrastructure errors.** If the git server is unreachable or credentials are rejected, the sweep skips that repository entirely - a GitLab outage will never mass-mark your branches. - **Default branches are never checked.** `main` cannot go stale. - **Branches not yet pushed or already terminating are skipped.** - **A branch that reappears in git is un-marked** on the next sweep (`BranchStaleResolved`). Detection is always on and needs no configuration - the interval defaults to 24 hours (`branchStaleCheckInterval` Helm value; `0` disables the checker). Cleanup is a separate, opt-in decision per codebase, which we will get to below. ## Walkthrough: Marking Stale Branches on a Live Cluster The setup is the same local testbed as the [preview environments post](/blog/ephemeral-preview-environments-kubernetes-feature-branch): a [kind](https://kind.sigs.k8s.io) cluster with KubeRocketCI, Tekton, Argo CD, and a self-hosted GitLab, with the sample Go application `test-go-app` already onboarded. The only deviation from defaults: I set the check interval to `2m` so the demo does not wait a day between sweeps. The operator confirms the checker is running on startup: ```text INFO stale-branch-checker Starting stale branch checker {"interval": "2m0s"} INFO stale-branch-checker Listing remote branches {"repository": "git@gitlab.127.0.0.1.nip.io:krci/test-go-app.git"} INFO stale-branch-checker Remote branches listed successfully {"count": 2} INFO stale-branch-checker Codebase branches staleness check finished {"codebases": 2} ``` ### Step 1: Create Two Feature Branches In the `test-go-app` component: **Branches → Create Branch**. I create `feature-tt-123` and `feature-tt-456` from `main`, keeping the default pipelines. ![KubeRocketCI Portal Create Branch dialog for test-go-app, creating feature-tt-123 from main](../assets/stale-branch-cleanup-kubernetes-kuberocketci/create-feature-branch.png "Creating a feature branch in the KubeRocketCI Portal") Each click creates a `CodebaseBranch` resource and pushes the branch to GitLab. Note the new **STALE** column on the resource - empty for now, because every branch exists in git: ```bash $ kubectl -n krci get codebasebranch NAME RESULT STATUS CODEBASE NAME BRANCH STALE krci-gitops-main success created krci-gitops main test-go-app-feature-tt-123-02e67 success created test-go-app feature-tt-123 test-go-app-feature-tt-456-045fe success created test-go-app feature-tt-456 test-go-app-main success created test-go-app main ``` ![KubeRocketCI Portal branch list for test-go-app showing main, feature-tt-456 and feature-tt-123, all healthy](../assets/stale-branch-cleanup-kubernetes-kuberocketci/branches-created.png "Both feature branches created and healthy") ### Step 2: Delete a Branch in Git - and Watch It Get Marked Now the everyday event this feature exists for: the branch is merged and removed *in the git provider* - the "delete source branch" checkbox on a merge request, or an explicit API call. I delete `feature-tt-123` straight in GitLab: ```bash $ curl -X DELETE -H "PRIVATE-TOKEN: $PAT" \ "https://gitlab.127.0.0.1.nip.io/api/v4/projects/krci%2Ftest-go-app/repository/branches/feature-tt-123" # HTTP 204 ``` Kubernetes knows nothing about this - until the next sweep. Within two minutes: ```bash $ kubectl -n krci get codebasebranch NAME RESULT STATUS CODEBASE NAME BRANCH STALE krci-gitops-main success created krci-gitops main test-go-app-feature-tt-123-02e67 success created test-go-app feature-tt-123 True test-go-app-feature-tt-456-045fe success created test-go-app feature-tt-456 False test-go-app-main success created test-go-app main ``` The condition carries the full story: ```json { "type": "Stale", "status": "True", "reason": "BranchNotFoundInGit", "message": "Branch was not found in the git repository", "lastTransitionTime": "2026-07-13T10:33:09Z" } ``` An event is emitted for the audit trail, and the mirrored label makes fleet-wide queries trivial: ```bash $ kubectl -n krci get events --field-selector involvedObject.name=test-go-app-feature-tt-123-02e67 LAST SEEN TYPE REASON MESSAGE 86s Warning BranchStale Branch feature-tt-123 was not found in the git repository and is marked as stale $ kubectl -n krci get codebasebranch -l app.edp.epam.com/stale=true NAME ... BRANCH STALE test-go-app-feature-tt-123-02e67 ... feature-tt-123 True ``` The Portal surfaces the same information as a warning badge in the branch list - hover it and the tooltip shows the condition message: ![KubeRocketCI Portal branch list showing an orange Stale badge next to branches deleted from git, with a tooltip explaining the branch was not found in the git repository](../assets/stale-branch-cleanup-kubernetes-kuberocketci/stale-branches-marked.png "Stale badges in the Portal branch list") At this point nothing has been deleted - the default cleanup strategy is **mark**: make staleness visible, let a human decide. For many teams that is already the win. But we can go further. ## Auto Cleanup - With a Deployment-Aware Safety Net Before turning on automatic deletion, let's build the exact scenario that makes naive cleanup dangerous: a stale branch that is *still deployed*. Following the [preview environment flow](/blog/ephemeral-preview-environments-kubernetes-feature-branch), I build `feature-tt-456` and create a Deployment named `preview` with an Auto-triggered `dev` environment consuming that branch's image stream: ![KubeRocketCI deployment wizard with the test-go-app application selected and branch feature-tt-456 chosen as the deployment input](../assets/stale-branch-cleanup-kubernetes-kuberocketci/deployment-select-app-branch.png "Selecting the feature branch as the deployment input") After the deploy pipeline finishes, the preview environment is live in its own namespace, running the image built from `feature-tt-456`: ![KubeRocketCI environment applications view showing test-go-app Healthy and Synced with deployed version feature-tt-456-20260713-103618](../assets/stale-branch-cleanup-kubernetes-kuberocketci/preview-environment-deployed.png "The preview environment running the feature branch image") Now I delete `feature-tt-456` in GitLab too. Next sweep: **both** branches are stale - but one of them is backing a running environment. ### Guardrail 1: The Admission Webhook Even a well-meaning human with `kubectl` cannot break the deployment by "cleaning up" the stale branch. The codebase-operator's admission webhook now rejects deleting any `CodebaseBranch` that participates in a deployment (a rule the Portal used to enforce only client-side): ```bash $ kubectl -n krci delete codebasebranch test-go-app-feature-tt-456-045fe Error from server (Forbidden): admission webhook "codebasebranch.epam.com" denied the request: CodebaseBranch test-go-app-feature-tt-456-045fe cannot be deleted because it is used by CDPipeline preview (inputDockerStreams); remove it from the deployment first ``` ### Guardrail 2: Retention in Auto Cleanup Automatic cleanup is opt-in **per codebase**, via a single annotation: ```bash $ kubectl -n krci annotate codebase test-go-app app.edp.epam.com/branch-cleanup-strategy=auto ``` With `auto`, the next sweep deletes stale branches that nothing references - and retains the ones that a `CDPipeline` (or a Stage autotest quality gate) still uses. One sweep later, the two stale branches got exactly opposite treatment: ```bash $ kubectl -n krci get events --sort-by=.lastTimestamp | grep Stale Warning BranchStale Branch feature-tt-456 was not found in the git repository and is marked as stale Normal StaleBranchDeleted Branch feature-tt-123 was not found in the git repository and is not used by any deployment, deleting $ kubectl -n krci get codebasebranch NAME RESULT STATUS CODEBASE NAME BRANCH STALE krci-gitops-main success created krci-gitops main test-go-app-feature-tt-456-045fe success created test-go-app feature-tt-456 True test-go-app-main success created test-go-app main ``` `feature-tt-123` is gone, and because the `CodebaseBranch` owns its `CodebaseImageStream`, the image stream was garbage-collected with it - no orphaned children. `feature-tt-456` stays, and its condition message names the exact resource that is keeping it alive: ```text Branch was not found in the git repository; retained because it is used by CDPipeline preview (inputDockerStreams) ``` The Portal tooltip shows the same reason, so the "why is this still here?" question answers itself: ![KubeRocketCI Portal stale badge tooltip stating the branch was not found in the git repository and is retained because it is used by CDPipeline preview](../assets/stale-branch-cleanup-kubernetes-kuberocketci/stale-branch-retained-tooltip.png "A retained stale branch names the deployment that references it") ### Releasing the Hold The retained branch is cleaned up the moment it stops being needed. I remove the preview Deployment in the Portal (**Actions → Delete**, type the name to confirm) - which, as covered in the [previous post](/blog/ephemeral-preview-environments-kubernetes-feature-branch), tears down the environment namespace and the Argo CD Application: ![KubeRocketCI confirm deletion dialog for the preview deployment, requiring the deployment name to be typed](../assets/stale-branch-cleanup-kubernetes-kuberocketci/delete-deployment-confirm.png "Deleting the preview deployment releases the stale branch") Next sweep, the last stale branch goes too: ```bash $ kubectl -n krci get events --sort-by=.lastTimestamp | grep StaleBranchDeleted Normal StaleBranchDeleted Branch feature-tt-123 ... not used by any deployment, deleting Normal StaleBranchDeleted Branch feature-tt-456 ... not used by any deployment, deleting $ kubectl -n krci get codebasebranch NAME RESULT STATUS CODEBASE NAME BRANCH STALE krci-gitops-main success created krci-gitops main test-go-app-main success created test-go-app main ``` The platform is back to exactly what git says it should be - no manual bookkeeping, and at no point could the cleanup have taken down a running environment. ## Configuration Reference | Setting | Where | Default | Effect | |---|---|---|---| | `branchStaleCheckInterval` | codebase-operator Helm values (env `BRANCH_STALE_CHECK_INTERVAL`) | `24h` | Sweep interval; `0` disables detection entirely | | `app.edp.epam.com/branch-cleanup-strategy` | Annotation on a `Codebase` | `mark` | `mark` = only flag stale branches; `auto` = delete unreferenced stale branches | | `app.edp.epam.com/stale` label | Set by the operator on `CodebaseBranch` | - | Mirrors the `Stale` condition for label-selector queries | Events emitted by the checker: | Event reason | Type | Meaning | |---|---|---| | `BranchStale` | Warning | Branch missing in git, marked as stale | | `BranchStaleResolved` | Normal | Branch reappeared in git, mark removed | | `StaleBranchDeleted` | Normal | Auto cleanup deleted an unreferenced stale branch | | `StaleBranchRetained` | Warning | Stale branch kept because a deployment references it | Handy one-liners: ```bash # All stale branches across the platform namespace kubectl -n krci get codebasebranch -l app.edp.epam.com/stale=true # Why is this branch still around? kubectl -n krci get codebasebranch -o jsonpath='{.status.conditions[?(@.type=="Stale")].message}' ``` ## Frequently Asked Questions ### What marks a CodebaseBranch as stale in KubeRocketCI? A periodic checker in the codebase-operator lists the repository's branches with `git ls-remote` and compares them against the `CodebaseBranch` resources. If a branch no longer exists in the git repository, the resource gets a `Stale` status condition (reason `BranchNotFoundInGit`), the `app.edp.epam.com/stale` label, a Kubernetes event, and a warning badge in the Portal. ### Can the default branch ever be marked stale or deleted? No. Default branches (for example `main`) are excluded from staleness checks entirely, as are branches that have not been pushed yet and branches already being deleted. ### What happens if GitLab or GitHub is down during a sweep? Nothing. The checker never marks branches when the repository listing fails - connectivity problems, DNS errors, or rejected credentials cause the repository to be skipped for that sweep. Only a successful listing that positively lacks the branch produces a stale mark. ### Does stale detection delete branches automatically? Not by default. Out of the box the strategy is `mark`: branches are flagged and left for a human. Automatic deletion is enabled per codebase with the `app.edp.epam.com/branch-cleanup-strategy: auto` annotation, and even then a stale branch referenced by a CDPipeline or a Stage quality gate is retained, with the retaining resource recorded in the condition message. ### Can a stale branch that is still deployed be deleted by mistake? No - two independent guardrails prevent it. Auto cleanup skips any branch referenced by a deployment, and the codebase-operator's admission webhook rejects manual deletion (`kubectl delete`, API calls) of a `CodebaseBranch` that participates in a deployment, naming the CDPipeline in the error message. ### What happens to the images built from a deleted branch? The `CodebaseBranch` owns its `CodebaseImageStream`, so when auto cleanup deletes the branch resource, Kubernetes garbage collection removes the image stream record with it. The container images themselves stay in your registry - registry retention is a separate concern. ### How do I find all stale branches across the platform? `kubectl get codebasebranch -l app.edp.epam.com/stale=true -A`. The label mirrors the status condition specifically so that plain label selectors - and any tooling built on them - can find stale branches without parsing conditions. ## Summary Feature branches are ephemeral; the Kubernetes resources behind them used to be forever. With KubeRocketCI 3.14, the codebase-operator closes the loop. One `git ls-remote` per repository per sweep detects branches deleted from git, and a `Stale` condition plus label plus Portal badge make them visible. An opt-in per-codebase `auto` strategy deletes the ones nothing depends on - while an admission webhook and a deployment-aware retention check make sure the cleanup can never take down a running environment. In this run we created two branches, deleted them in GitLab at different points of their lifecycle, and watched the operator mark one, auto-delete it, retain the other while its [preview environment](/blog/ephemeral-preview-environments-kubernetes-feature-branch) was alive, and finish the job the moment the environment was gone - ending with a platform that matches git exactly. Next steps from here: - Spin up the same environment with [Try KubeRocketCI Locally](/blog/try-kuberocketci-locally) and replay the flow - the whole demo fits in a lunch break. - Read [Manage Branches](/docs/user-guide/manage-branches) for the day-to-day branch workflow in the Portal. - See [Ephemeral Environments on Kubernetes](/blog/ephemeral-preview-environments-kubernetes-feature-branch) for the preview-environment lifecycle that produces these branches in the first place. - Explore the [codebase-operator on GitHub](https://github.com/epam/edp-codebase-operator) - the stale checker lives in `controllers/codebasebranch/stalecheck/`. KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the local testbed are all on [GitHub](https://github.com/KubeRocketCI/try-kuberocketci). {/* cspell:ignore stalecheck 02e67 045fe */} --- ## Cancel In-Progress Tekton Pipelines: Stop Wasting CI on Outdated Commits Push a commit to a merge request and your review pipeline starts. Spot a typo, push again thirty seconds later - and now *two* pipelines are running, one of them validating a commit that no longer matters. GitHub Actions solved this years ago with `concurrency.cancel-in-progress`; Tekton has no built-in equivalent, so on a Tekton-based platform those superseded runs keep burning CPU until they finish, producing results nobody will read. KubeRocketCI 3.14 closes that gap with a single chart flag: `pipelines.cancelInProgress`. When enabled, the KRCI [Tekton interceptor](/blog/kubernetes-native-cicd-tekton-kuberocketci) cancels still-running review PipelineRuns for the same pull request before triggering the replacement. The cancellation is graceful (`CancelledRunFinally`): the superseded run still executes its `finally` tasks, so your git provider gets a status for the old commit instead of an eternally "pending" check. This post shows the whole flow live on the local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed - every screenshot and command output is from a real run. ## The Problem: Superseded Review Runs Pile Up A review (pull request) pipeline exists to answer one question: *is this commit mergeable?* The moment a newer commit lands on the same merge request, that question changes - and every pipeline still validating the old commit is wasted work. The pattern is worst exactly where CI matters most: active MRs with quick fix-up pushes, `/recheck` comments, rebases. Each event spawns a full pipeline - clone, build, lint, SonarQube scan, image verification - and on a shared cluster those zombie runs compete with useful work for CPU, memory, and runner capacity. GitHub Actions users express the fix in two lines of YAML: ```yaml title="GitHub Actions equivalent (for comparison)" concurrency: group: review-${{ github.ref }} cancel-in-progress: true ``` Tekton, by design, has no such primitive - a `PipelineRun` is a standalone Kubernetes resource with no concept of "group" or "supersedes". The [Tekton community has discussed pipeline concurrency for years](https://github.com/tektoncd/community/pull/716) without a shipped core feature. So the platform layer has to provide it - and in KubeRocketCI, the natural place is the **krci interceptor**, the component that already sits between every git webhook and every triggered pipeline. ## How Cancel-In-Progress Works The interceptor enriches webhook payloads for all four supported git providers (GitHub, GitLab, Bitbucket, Gerrit) before Tekton's EventListener instantiates a PipelineRun. That position makes it the perfect referee: it sees the *new* event before the new run exists, and it can find every *old* run for the same change because review PipelineRuns are now stamped with a `app.edp.epam.com/git-change-number` label at creation time. ```mermaid sequenceDiagram participant Dev as Developer participant Git as GitLab / GitHub participant EL as EventListener participant I as krci interceptor participant K8s as Kubernetes API Dev->>Git: push commit #2 to MR !2 Git->>EL: webhook (Merge Request Hook) EL->>I: intercept (cancelInProgress: true) I->>K8s: list PipelineRunscodebase + type=review + change=2 K8s-->>I: review-...-2p446 (Running) I->>K8s: patch spec.status =CancelledRunFinally I-->>EL: continue: true EL->>K8s: create replacement PipelineRun Note over K8s: old run executes finally tasks,reports status for commit #1 ``` The design choices matter as much as the mechanism: - **Graceful, not brutal.** The interceptor patches `spec.status: CancelledRunFinally` - Tekton stops scheduling new tasks and cancels running ones, but the pipeline's `finally` section still executes. In KubeRocketCI's pipelines-library, that is where commit status is reported back to the git provider - so the superseded commit ends up marked *failed*, never stuck *pending*. - **Review pipelines only.** The `cancelInProgress` parameter is injected into review Triggers exclusively. Build pipelines - the ones producing versioned artifacts after a merge - are never cancelled. - **Best-effort by contract.** If listing or patching fails, the error is logged and the new pipeline triggers anyway. Cancellation can never break your CI. - **Scoped to one change.** The label selector matches codebase + pipeline type + change number, so parallel MRs on the same repository never interfere with each other. - **RBAC follows the flag.** The interceptor's Role gains `patch` on `pipelineruns` only when the feature is enabled. Enabling it is one Helm value on the pipelines-library chart: ```yaml title="values.yaml (pipelines-library)" pipelines: # Cancel in-progress review PipelineRuns when a new commit is pushed to the same # Pull Request / Merge Request / Gerrit change. Cancellation is graceful # (spec.status: CancelledRunFinally), so finally tasks of the superseded run still execute. cancelInProgress: true ``` ## Walkthrough: Two Commits, One Surviving Pipeline The environment is the same local testbed as my previous posts: a [kind](https://kind.sigs.k8s.io) cluster with KubeRocketCI, Tekton, and self-hosted GitLab, with the Go sample application `test-go-app` onboarded and `pipelines.cancelInProgress: true` set. ### Step 1: Open a Merge Request I create a branch, add a commit, and open a merge request through the GitLab API - the same thing a developer does from the terminal: ```bash $ git checkout -b feature-cancel-demo && git commit -m "feat: add feature (attempt 1)" && git push # MR !2 "Add feature" opened against main ``` The webhook fires, and the review pipeline starts. Note the change-number label - this is the handle the interceptor will use later: ```bash $ kubectl -n krci get pipelinerun --show-labels | grep review review-test-go-app-main-2p446 Unknown Running 2s app.edp.epam.com/codebase=test-go-app, app.edp.epam.com/pipelinetype=review, app.edp.epam.com/git-change-number=2, ... ``` ### Step 2: Push a Fix While the Pipeline Is Still Running Ninety seconds in - the pipeline is mid-flight, fetch, build and sonar already done, image verification in progress - I notice the typo and push the fix to the same branch: ```bash $ git commit -m "feat: add feature (attempt 2, fixes review)" && git push ``` GitLab sends the `update` webhook. The interceptor finds the still-running pipeline for change 2, cancels it, and lets the new one trigger. Twenty seconds later the picture is exactly what you want: ```bash $ kubectl -n krci get pipelinerun NAME SUCCEEDED REASON STARTTIME COMPLETIONTIME review-test-go-app-main-2p446 False Cancelled 104s 15s review-test-go-app-main-8th5r Unknown Running 22s ``` One log line in the interceptor tells the story: ```text Canceled in-progress PipelineRun review-test-go-app-main-2p446 superseded by a new event for codebase test-go-app change 2 ``` The Portal shows both runs side by side - the superseded run **Cancelled** at 1m 29s instead of running to completion, the replacement already underway: ![KubeRocketCI Portal pipeline runs list for test-go-app showing the superseded review run Cancelled and the replacement review run Completed for the same pull request](../assets/cancel-in-progress-pipelines-tekton-kuberocketci/pipeline-runs-cancelled-and-completed.png "The superseded run is cancelled; only the run for the latest commit survives") ### Step 3: Verify the Cancellation Was Graceful Open the cancelled run in the Portal. The status is `Cancelled`, the spec says `CancelledRunFinally`, and the task list is the interesting part: ![KubeRocketCI Portal PipelineRun details of the cancelled review run, showing completed tasks, the cancelled dockerbuild-verify task and the executed finally tasks](../assets/cancel-in-progress-pipelines-tekton-kuberocketci/cancelled-run-details.png "Inside the cancelled run: finished tasks stay finished, the in-flight task is cancelled") ```bash $ kubectl -n krci get taskrun -l tekton.dev/pipelineRun=review-test-go-app-main-2p446 NAME OK REASON ...-fetch-repository True Succeeded ...-build True Succeeded ...-sonar True Succeeded ...-dockerbuild-verify False TaskRunCancelled <- was in flight ...-gitlab-set-failure-status True Succeeded <- finally task ran ``` `dockerbuild-verify` - the task that was running when the second commit arrived - was cancelled. But the `finally` task `gitlab-set-failure-status` executed *after* the cancellation and reported the outcome to GitLab: ![KubeRocketCI Portal showing the gitlab-set-failure-status finally task of the cancelled run succeeded, reporting the failed state for the superseded commit to GitLab](../assets/cancel-in-progress-pipelines-tekton-kuberocketci/cancelled-run-finally-task.png "The finally task still runs: GitLab gets a status for the superseded commit") The git provider's view confirms it - no dangling "pending" check anywhere: ```bash # commit #1 (superseded) $ glab api projects/krci%2Ftest-go-app/repository/commits/7fb8e40/statuses Review Pipeline -> failed # commit #2 (current) $ glab api projects/krci%2Ftest-go-app/repository/commits/5ffb585/statuses Review Pipeline -> success ``` The replacement run finished green, the MR shows a passing pipeline for the latest commit, and the cluster spent zero extra minutes on the outdated one. ## What This Saves Back-of-the-envelope for a real team: the review pipeline above takes about 4 minutes of pod time (clone, build, lint, SonarQube, image verification). A ten-developer team pushing an average of one fix-up commit per MR wastes one full pipeline per MR - dozens of pipeline-hours per month on a busy repository, all competing with useful runs for cluster capacity. Concurrency limits and cluster autoscaling treat the symptom; cancelling superseded runs removes the work itself. It is the same reasoning that makes [ephemeral environments](/blog/ephemeral-preview-environments-kubernetes-feature-branch) cheaper than permanent staging: the cheapest workload is the one that stops existing when it stops being useful. ## Configuration Reference | Setting | Where | Default | Effect | |---|---|---|---| | `pipelines.cancelInProgress` | pipelines-library Helm values | `false` | Cancel in-progress review PipelineRuns superseded by a new event for the same change | | `app.edp.epam.com/git-change-number` | Label stamped on review PipelineRuns | - | Selector the interceptor uses to find superseded runs | Behavior summary: - **Triggers cancellation:** a new commit pushed to the same PR/MR/Gerrit change, and re-trigger comments (`/recheck`, `/ok-to-test`). - **Cancels:** review PipelineRuns for the same codebase and change number that are not already done or cancelled. - **Never touches:** build pipelines, runs for other changes, runs for other codebases. - **On failure:** logs the error and proceeds - the new pipeline always triggers. Handy one-liner to see every run for a given pull request, newest first: ```bash kubectl -n krci get pipelinerun \ -l app.edp.epam.com/git-change-number=2,app.edp.epam.com/codebase=test-go-app \ --sort-by=.metadata.creationTimestamp ``` ## Frequently Asked Questions ### Does Tekton have a built-in equivalent of GitHub Actions cancel-in-progress? No. Tekton PipelineRuns are independent Kubernetes resources with no native concurrency groups; pipeline concurrency has been a long-running community discussion without a shipped core feature. KubeRocketCI implements the pattern at the platform layer, in the interceptor that already processes every git webhook. ### Why CancelledRunFinally instead of just deleting the PipelineRun? Deleting (or hard-cancelling) a run would leave the superseded commit with a forever-pending status in GitLab or GitHub, and would skip cleanup logic. `CancelledRunFinally` stops the useless work but still executes the pipeline's `finally` tasks - in KubeRocketCI's pipeline library that is where the commit status is reported, so the old commit is cleanly marked as failed. ### Are build pipelines ever cancelled? No. The `cancelInProgress` interceptor parameter is injected into review Triggers only. Build pipelines - triggered by merges and producing versioned artifacts - always run to completion. ### Which git providers are supported? All four KubeRocketCI providers: GitHub, GitLab, Bitbucket, and Gerrit. The change number is parsed from each provider's webhook payload (including Gerrit's, which sends it as either an integer or a string), and the same label-based cancellation logic applies everywhere. ### What happens if the cancellation itself fails? Nothing visible to the developer. Cancellation is best-effort: a failure to list or patch PipelineRuns is logged by the interceptor and the new pipeline is triggered regardless. The worst case is the old behavior - two runs in parallel - never a blocked pipeline. ### Can two different merge requests cancel each other's pipelines? No. The interceptor matches on codebase *and* pipeline type *and* change number. Parallel MRs on the same repository each have their own change number, so their pipelines coexist untouched. ### How do I enable this on an existing KubeRocketCI installation? Set `pipelines.cancelInProgress: true` in the pipelines-library chart values and upgrade the release. The chart adds the interceptor parameter to review Triggers, stamps the change-number label on review runs, and extends the interceptor's RBAC with `patch` on `pipelineruns` - all gated by the same flag, nothing else changes. ## Summary Every push to an active merge request used to leave a zombie review pipeline burning cluster resources on a commit nobody cares about anymore. With `pipelines.cancelInProgress: true`, the KRCI interceptor now cancels superseded review runs the instant the replacing event arrives - gracefully, so `finally` tasks still report a status for the old commit, scoped by a change-number label so parallel MRs never collide, and best-effort so CI can never be blocked by its own cleanup. In this run, the second commit landed 90 seconds into the first commit's pipeline: the in-flight task was cancelled on the spot, GitLab received a *failed* status for the superseded commit and a *success* for the new one, and only one pipeline ran to completion. The whole feature is one Helm value. Next steps from here: - Stand up the same environment with [Try KubeRocketCI Locally](/blog/try-kuberocketci-locally) and replay the flow. - Read [Kubernetes-Native CI/CD with Tekton](/blog/kubernetes-native-cicd-tekton-kuberocketci) for the architecture this builds on - the interceptor, triggers, and the pipelines library. - See [Manage Tekton Pipelines](/docs/user-guide/tekton-pipelines) for the day-to-day pipeline workflow in the Portal. - Explore the implementation in [edp-tekton on GitHub](https://github.com/epam/edp-tekton) - the cancellation logic lives in `pkg/interceptor/cancel_pipelineruns.go`. KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the local testbed are all on [GitHub](https://github.com/KubeRocketCI/try-kuberocketci). {/* cspell:ignore pipelineruns dockerbuild 2p446 8th5r 7fb8e40 5ffb585 COMPLETIONTIME glab Ftest */} --- ## Isolate Deployment Environments with vcluster: Multi-Cluster CD Without a Second Cluster Every platform team hits the same wall: the `qa` environment needs a different CRD version than `dev`, a teammate's experiment needs cluster-admin, or a preview environment needs to be disposable without a change-advisory meeting. Kubernetes namespaces don't isolate any of that - CRDs, admission webhooks, and cluster-scoped RBAC are shared by every namespace on the cluster. The textbook answer is a second cluster, but a second cluster means a second control plane to pay for, patch, and secure. [vcluster](https://www.vcluster.com/) sits exactly in that gap: a certified Kubernetes distribution that runs *inside* a namespace of your existing cluster, with its own API server, its own CRDs, and its own RBAC - while the pods it schedules land on the host nodes you already own. That makes it a perfect deployment target for a CI/CD platform: real cluster isolation, zero new hardware. This post walks through the complete flow on KubeRocketCI: create a vcluster, register it through the Portal, wire Argo CD, and promote an application into it. Everything below - every command, screenshot, and error message - is from a real run on the local [try-kuberocketci](/blog/try-kuberocketci-locally) testbed. ## Why a Virtual Cluster Instead of a Namespace or a Real Cluster? | | Namespace | vcluster | Separate cluster | |---|---|---|---| | Workload isolation | Yes | Yes | Yes | | Own CRDs & API versions | No - shared | Yes | Yes | | Own RBAC / cluster-admin | No - shared cluster scope | Yes | Yes | | Blast radius of a bad operator | Whole cluster | The vcluster | The other cluster | | Extra infrastructure cost | None | ~1 pod (control plane) | Full control plane + nodes | | Setup time | Seconds | Minutes | Hours to days | | Teardown | `kubectl delete ns` | `vcluster delete` | Ticket to infra team | If you need hard isolation of the *control plane* - separate cloud account, separate failure domain, compliance boundary - use a real cluster. For everything else (per-team sandboxes, qa environments with diverging CRDs, disposable preview targets like the ones in our [ephemeral environments post](/blog/ephemeral-preview-environments-kubernetes-feature-branch)), a vcluster gives you cluster-grade isolation at namespace-grade cost. ## Spin Up the vcluster and Mint a Deployer Token Two commands stand up the virtual cluster next to the platform (the testbed runs KubeRocketCI in a kind cluster, but the flow is identical on any Kubernetes): ```bash vcluster create dev2 --namespace vcluster-dev2 --connect=false ``` The vcluster's API server is now reachable from inside the host cluster at a stable Service URL: `https://dev2.vcluster-dev2.svc:443`. That in-cluster address is all the platform needs - no ingress, no load balancer. Next, create a ServiceAccount inside the vcluster for the platform to deploy with, following the [remote cluster via token](/docs/operator-guide/cd/deploy-application-in-remote-cluster-via-token) guide: ```bash vcluster connect dev2 --namespace vcluster-dev2 -- bash -c ' kubectl create sa krci-deployer -n kube-system kubectl create clusterrolebinding krci-deployer-admin \ --clusterrole=cluster-admin --serviceaccount=kube-system:krci-deployer kubectl apply -f - < **Deployment** -> **Clusters**, click **+ Add Cluster**, and fill in the Bearer form: name `dev2`, host `https://dev2.vcluster-dev2.svc:443`, the token from above, and **Skip TLS verification** for the first pass (you can wire the CA properly later). ![KubeRocketCI Portal Add Cluster form filled with the vcluster name dev2, in-cluster HTTPS endpoint and bearer token, Skip TLS verification enabled](../assets/vcluster-deployment-isolation-kuberocketci/add-cluster-form.png "The whole registration: name, endpoint, token") Here is what actually happens when you click Save - understanding it will save you a debugging session later: 1. The Portal creates **one Secret** named `dev2-cluster` (your name plus a `-cluster` suffix) in the platform namespace, containing a kubeconfig. 2. The cd-pipeline-operator picks it up, performs a **live connectivity probe** against the API endpoint, and stamps the result on the Secret as annotations (`app.edp.epam.com/cluster-connected`, `app.edp.epam.com/cluster-error`). 3. Only after the probe succeeds, it generates a ready-to-use **Argo CD cluster secret** next to it, named `dev2-cluster-argocd-cluster`. The "Secret has been created" toast therefore does *not* mean your credentials work. The truth is on the cluster card, which flips to **Connected** or **Disconnected** about a minute later: ![KubeRocketCI Portal cluster cards showing the vcluster Connected and a misconfigured cluster Disconnected with the connection error displayed](../assets/vcluster-deployment-isolation-kuberocketci/cluster-connected-disconnected.png "Trust the badge, not the toast: the operator probes the cluster asynchronously") That Disconnected card on the left is a deliberate demo - a cluster registered with an unreachable host. The Portal accepts it silently; the badge and error text are the only signals. Always wait for **Connected** before moving on. ## Wire Argo CD: Three Steps and One Golden Rule The Portal registration alone is not enough to deploy. Three pieces of glue connect the new cluster to the delivery machinery, and all three must agree on **one string: the cluster Secret name** (`dev2-cluster`). This is the part where most integrations go sideways, so here is the full naming chain from our live run: | Step | Object | Value | |---|---|---| | You type in the Portal | form field | `dev2` | | Portal creates | Secret (platform ns) | `dev2-cluster` | | Operator generates | Argo CD Secret (platform ns) | `dev2-cluster-argocd-cluster`, `data.name: dev2-cluster` | | You add | `krci-config` ConfigMap | `available_clusters: dev2-cluster` | | Environment wizard writes | `Stage.spec.clusterName` | `dev2-cluster` | | Operator renders | Argo CD Application destination | `name: dev2-cluster` | | Deploy pipeline receives | `KUBECONFIG_SECRET_NAME` param | `dev2-cluster` | The `clusterName` is resolved *as a Secret name* by the operator and *as a destination name* by Argo CD - one string, two lookups. With that in mind, the three steps: **1. Offer the cluster to the Environment wizard** - add the Secret name to the `krci-config` ConfigMap: ```bash kubectl -n krci patch configmap krci-config --type merge \ -p '{"data":{"available_clusters":"dev2-cluster"}}' ``` **2. Let Argo CD see the cluster.** Argo CD only discovers cluster secrets in its own control-plane namespace, so copy the generated secret there (keep the `argocd.argoproj.io/secret-type: cluster` label): ```bash kubectl -n krci get secret dev2-cluster-argocd-cluster -o json \ | jq '.metadata = {"name":"dev2-cluster","namespace":"argocd", "labels":{"argocd.argoproj.io/secret-type":"cluster"}}' \ | kubectl apply -f - ``` **3. Allow the destination in the Argo CD AppProject** - without this, the first deploy fails (we will meet the exact error below): ```bash kubectl -n argocd patch appproject krci --type json \ -p '[{"op":"add","path":"/spec/destinations/-", "value":{"name":"dev2-cluster","namespace":"krci-*"}}]' ``` Full reference for these steps lives in the [Add Cluster](/docs/user-guide/add-cluster) guide. ## Create the Environment on the Virtual Cluster Back in the Portal, open your Deployment Flow and create a new Environment. The **Cluster** dropdown now offers `dev2-cluster` next to `in-cluster` - pick it, and the wizard derives the deploy namespace as usual: ![KubeRocketCI Portal Create Environment wizard with the dev2-cluster virtual cluster selected and the deploy namespace prefilled](../assets/vcluster-deployment-isolation-kuberocketci/create-environment-wizard.png "Same wizard, different cluster: environment qa targets the vcluster") The moment the Environment is created, the operator reaches into the vcluster and prepares everything a tenant needs: the namespace (labelled with the tenant name), a `tenant-admin` RoleBinding, and a copy of the registry pull secret. You can verify from outside: ```bash vcluster connect dev2 --namespace vcluster-dev2 -- \ kubectl -n krci-demo-qa get rolebinding,secret ``` ```text NAME ROLE AGE rolebinding.rbac.authorization.k8s.io/tenant-admin ClusterRole/admin 24s NAME TYPE DATA AGE secret/regcred kubernetes.io/dockerconfigjson 1 24s ``` ## Deploy and Prove the Isolation Open the environment's **Applications** tab, click **Configure Deploy**, pick the image version, and hit **Start Deploy**: ![KubeRocketCI Portal Configure Deploy view with the latest built image version selected for deployment to the vcluster environment](../assets/vcluster-deployment-isolation-kuberocketci/configure-deploy-version.png "Promote the latest verified build to the vcluster environment") The regular deploy pipeline runs, Argo CD resolves the destination by name, and the application lands *inside* the virtual cluster: ```bash vcluster connect dev2 --namespace vcluster-dev2 -- \ kubectl -n krci-demo-qa get deploy,pods ``` ```text NAME READY UP-TO-DATE AVAILABLE deployment.apps/test-go-app 1/1 1 1 NAME READY STATUS RESTARTS pod/test-go-app-84cd86f69-5xj4c 1/1 Running 0 ``` From the host cluster's point of view, this workload lives in the vcluster's namespace under synthetic names; from the vcluster's point of view, it is a normal Deployment in a normal namespace with its own RBAC universe around it. Install a conflicting CRD version in there, grant someone cluster-admin, delete the whole thing - the host cluster and every other environment never notice. ![KubeRocketCI Portal environment Applications tab showing the test-go-app application deployed and healthy on the vcluster](../assets/vcluster-deployment-isolation-kuberocketci/environment-applications-tab.png "Deployed, synced, healthy - and fully isolated") ## The Three Errors You Will Actually Hit Every one of these appeared during our validation run - deliberately or not. Search engines love exact error strings, and so do humans at 2 a.m. ### `failed to get cluster secret: secrets "dev2" not found` The Environment's `clusterName` does not match the cluster Secret name. Almost always this means `available_clusters` contains the plain name you typed in the Portal (`dev2`) instead of the Secret name (`dev2-cluster`). Fix the ConfigMap value; the Environment reconciles on the next loop. ### `InvalidSpecError: application destination server 'dev2-cluster' ... do not match any of the allowed destinations in project 'krci'` The Argo CD AppProject does not allow the new cluster as a destination - step 3 above was skipped. Patch the AppProject, then re-run the deploy; the Application resolves immediately. ### The cluster card stays "Disconnected" (or "Unknown" forever) Read the error on the card - it is the operator's live probe result. Wrong token, unreachable host, and TLS failures all surface there within a minute. If the badge never changes at all, check that the Secret was created in the platform namespace: the Portal writes it to whatever namespace is currently selected, and the operator only watches its own. ## Tear It Down in the Right Order Order matters, because the operator needs the cluster Secret to clean up the remote side: 1. Delete the Environments that target the cluster. 2. Remove the name from `available_clusters`. 3. Delete the cluster in the Portal (the generated Argo CD secret is owned by it and goes too). 4. Delete your copy in the `argocd` namespace and the AppProject destination. 5. `vcluster delete dev2 --namespace vcluster-dev2` Delete the Secret before the Environments and the operator silently skips the remote cleanup, leaving orphaned namespaces inside the vcluster - harmless here, since step 5 vaporizes the whole thing, but a real remote cluster would keep the litter. ## Wrap-Up A vcluster turns "we need another environment" from an infrastructure request into a two-command operation - and to KubeRocketCI it is just another remote cluster: registered in the Portal, deployed to by the same pipelines, governed by the same [GitOps machinery](/blog/kubernetes-native-cicd-tekton-kuberocketci). The only real tax is the naming chain, and now you have the map. Try the whole flow yourself on the [local testbed](/blog/try-kuberocketci-locally), and check the [Add Cluster](/docs/user-guide/add-cluster) guide for the reference procedure, including IRSA-based clusters on EKS.