External Secrets Operator Integration

External Secrets Operator (ESO) can be integrated with KubeRocketCI.

There are multiple Secrets Providers that can be used within ESO. On this page, we detail the integration of KubeRocketCI with the two foremost providers:

• Kubernetes Secrets
• AWS Systems Manager Parameter Store

KubeRocketCI leverages a variety of secrets to facilitate the integration of different solutions. This document provides a comprehensive overview of the secrets utilized within the KubeRocketCI platform, along with their respective descriptions. Please note that all secrets are encoded in Base64 format.

Secret Name	Fields	Description	Used by
keycloak	username password	Username and password with specific permissions for the Platform tenant management in Keycloak	keycloak-operator
ci-defectdojo	token url	DefectDojo token DefectDojo URL	edp-tekton
kaniko-docker-config	.dockerconfigjson	Serialized JSON that follows docker config patterns	edp-tekton
regcred	.dockerconfigjson	Serialized JSON that follows docker config patterns	cd-pipeline-operator
ci-github	id_rsa token secretString	Private key from github repo API token Random string	edp-tekton
ci-gitlab	id_rsa token secretString	Private key from gitlab repo API token Random string	edp-tekton
ci-jira	username password	Jira username Jira password	edp-codebase-operator
ci-sonarqube	token url	SonarQube token SonarQube URL	edp-tekton
ci-nexus	username password url	Nexus username Nexus password Nexus URL	edp-tekton
ci-dependency-track	token url	Dependency-Track token Dependency-Track URL	edp-tekton
oauth2-proxy-cookie-secret	cookie-secret	Secret key for oauth2-proxy	edp-install
keycloak-client-headlamp-secret	clientSecret	Secret key for keycloak client	keycloak-operator
ci-argocd	token url	Argo CD token Argo CD URL	edp-tekton

Platform Core Secrets

The list below represents the baseline required for full operation within platform core components:

• kaniko-docker-config: Used for pushing container images to a specific registry.
• ci-sonarqube: Used in the CI process for SonarQube integration.
• ci-nexus: Used for pushing artifacts to the Nexus storage.

warning

These secrets are mandatory for Tekton pipelines to work properly.

Kubernetes Provider

All secrets are stored in Kubernetes in pre-defined namespaces. Platform suggests using the following approach for secrets management:

• EDP_NAMESPACE-vault, where EDP_NAMESPACE is a name of the namespace where KubeRocketCI is deployed, such as edp-vault. This namespace is used by the platform. Access to secrets in the edp-vault is granted only for Administrators.

• EDP_NAMESPACE-cicd-vault, where EDP_NAMESPACE is a name of the namespace where KubeRocketCI is deployed, such as edp-cicd-vault. Development team uses secrets in the edp-cicd-vault for microservices development.

See a diagram below for more details:

In order to Install KubeRocketCI, a list of passwords must be created. Secrets are provided automatically when using ESO.

1. Create a common namespace for secrets and platform:

   kubectl create namespace edp-vault
   kubectl create namespace edp

2. Create secrets in the edp-vault namespace:

   apiVersion: v1
   kind: Secret
   metadata:
     name: keycloak
     namespace: edp-vault
   data:
     password: cGFzcw==  # pass in base64
     username: dXNlcg==  # user in base64
   type: Opaque

3. In the edp-vault namespace, create a Role with a permission to read secrets:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     namespace: edp-vault
     name: external-secret-store
   rules:
   - apiGroups: [""]
     resources:
     - secrets
     verbs:
     - get
     - list
     - watch
   - apiGroups:
     - authorization.k8s.io
     resources:
     - selfsubjectrulesreviews
     verbs:
     - create

4. In the edp-vault namespace, create a ServiceAccount used by SecretStore:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: secret-manager
     namespace: edp

5. Connect the Role from the edp-vault namespace with the ServiceAccount in the edp namespace:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: eso-from-edp
     namespace: edp-vault
   subjects:
     - kind: ServiceAccount
       name: secret-manager
       namespace: edp
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: external-secret-store

6. Create a SecretStore in the edp namespace, and use ServiceAccount for authentication:

   apiVersion: external-secrets.io/v1beta1
   kind: SecretStore
   metadata:
     name: edp-vault
     namespace: edp
   spec:
     provider:
       kubernetes:
         remoteNamespace: edp-vault  # namespace with secrets
         auth:
           serviceAccount:
             name: secret-manager
         server:
           caProvider:
             type: ConfigMap
             name: kube-root-ca.crt
             key: ca.crt

7. Each secret must be defined by the ExternalSecret object. A code example below creates the keycloak secret in the edp namespace based on a secret with the same name in the edp-vault namespace:

   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   metadata:
     name: keycloak
     namespace: edp
   spec:
     refreshInterval: 1h
     secretStoreRef:
       kind: SecretStore
       name: edp-vault
     # target:
     #   name: secret-to-be-created  # name of the k8s Secret to be created. metadata.name used if not defined
     data:
     - secretKey: username       # key to be created
       remoteRef:
         key: keycloak           # remote secret name
         property: username      # value will be fetched from this field
     - secretKey: password       # key to be created
       remoteRef:
         key: keycloak           # remote secret name
         property: password      # value will be fetched from this field

Apply the same approach for enabling secrets management in the namespaces used for microservices development, such as sit and qa on the diagram above.

AWS Systems Manager Parameter Store

AWS SSM Parameter Store can be used as a Secret Provider for ESO. For the platform, it is recommended to use the IAM Roles For Service Accounts approach (see a diagram below).

AWS Parameter Store Scenario in KubeRocketCI

In order to Install KubeRocketCI, a list of passwords must be created. Follow the steps below, to get secrets from the SSM:

1. In the AWS, create an AWS IAM policy and an IAM role used by ServiceAccount in SecretStore. The IAM role must have permissions to get values from the SSM Parameter Store.

   a. Create an IAM policy that allows to get values from the Parameter Store with the edp/ path. Use your AWS Region and AWS Account Id:

   {
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": "ssm:GetParameter*",
           "Resource": "arn:aws:ssm:eu-central-1:012345678910:parameter/edp/*"
       }
   ]
   }

   b. Create an AWS IAM role with trust relationships (defined below) and attach the IAM policy. Put your string for Federated value (see more on IRSA enablement for EKS Cluster) and AWS region.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::012345678910:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXX"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringLike": {
                       "oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:edp:*"
                   }
               }
           }
       ]
   }

2. Create a secret in the AWS Parameter Store with the name /edp/my-json-secret. This secret is represented as a parameter of type string within the AWS Parameter Store:

   View: Parameter Store JSON

   {
     "keycloak":
     {
       "username": "keycloak-username",
       "password": "keycloak-password"
     },
     "ci-defectdojo":
     {
       "token": "XXXXXXXXXXXX",
       "url": "https://defectdojo.example.com"
     },
     "kaniko-docker-config":
     {
       "auths" :
       {
         "registry.com":
         {
           "username":"registry-username",
           "password":"registry-password",
           "auth": "<base64 encoded 'user:secret' string>"
         }
     }},
     "regcred":
     {
         "auths":
         {
           "registry.com":
           {
             "username":"registry-username",
             "password":"registry-password",
             "auth":"<base64 encoded 'user:secret' string>"
           }
     }},
     "ci-github":
     {
       "id_rsa": "id-rsa-key",
       "token": "github-token",
       "secretString": "XXXXXXXXXXXX"
     },
     "ci-gitlab":
     {
       "id_rsa": "id-rsa-key",
       "token": "gitlab-token",
       "secretString": "XXXXXXXXXXXX"
     },
     "ci-jira":
     {
       "username": "jira-username",
       "password": "jira-password"
     },
     "ci-sonarqube":
     {
       "username": "<ci-user>",
       "secret": "<secret>"
     },
     "ci-nexus":
     {
       "username": "<ci.user>",
       "password": "<secret>"
     },
     "oauth2-proxy-cookie-secret":
     {
       "cookie-secret": "XXXXXXXXXXXX"
     },
     "keycloak-client-headlamp-secret": "XXXXXXXXXXXX",
     "ci-argocd":
     {
       "token": "argocd-token",
       "url": "https://argocd.example.com"
     },
     "ci-dependency-track":
     {
       "token": "dependency-track-token",
       "url": "https://dependency-track.example.com"
     }
   }

3. Set External Secret operator enabled by updating the values.yaml file:

   externalSecrets:
     enabled: true

4. Install/upgrade edp-install:

   helm upgrade --install edp epamedp/edp-install --wait --timeout=900s \
   --version <edp_version> \
   --values values.yaml \
   --namespace edp \
   --atomic

Related Articles

• Install External Secrets Operator