Skip to main content
Version: 3.9

External Secrets Operator Integration

External Secrets Operator (ESO) can be integrated with KubeRocketCI.

There are multiple Secrets Providers that can be used within ESO. On this page, we detail the integration of KubeRocketCI with the two foremost providers:

KubeRocketCI leverages a variety of secrets to facilitate the integration of different solutions. This document provides a comprehensive overview of the secrets utilized within the KubeRocketCI platform, along with their respective descriptions. Please note that all secrets are encoded in Base64 format.

Secret NameFieldsDescriptionUsed by
keycloakusername
password
Username and password with specific permissions for the Platform tenant management in Keycloakkeycloak-operator
ci-defectdojotoken
url
DefectDojo token
DefectDojo URL
edp-tekton
kaniko-docker-config.dockerconfigjsonSerialized JSON that follows docker config patternsedp-tekton
regcred.dockerconfigjsonSerialized JSON that follows docker config patternscd-pipeline-operator
ci-githubid_rsa
token
secretString
Private key from github repo
API token
Random string
edp-tekton
ci-gitlabid_rsa
token
secretString
Private key from gitlab repo
API token
Random string
edp-tekton
ci-jirausername
password
Jira username
Jira password
edp-codebase-operator
ci-sonarqubetoken
url
SonarQube token
SonarQube URL
edp-tekton
ci-nexususername
password
url
Nexus username
Nexus password
Nexus URL
edp-tekton
ci-dependency-tracktoken
url
Dependency-Track token
Dependency-Track URL
edp-tekton
oauth2-proxy-cookie-secretcookie-secretSecret key for oauth2-proxyedp-install
keycloak-client-headlamp-secretclientSecretSecret key for keycloak clientkeycloak-operator
ci-argocdtoken
url
Argo CD token
Argo CD URL
edp-tekton

Platform Core Secrets​

The list below represents the baseline required for full operation within platform core components:

  • kaniko-docker-config: Used for pushing container images to a specific registry.
  • ci-sonarqube: Used in the CI process for SonarQube integration.
  • ci-nexus: Used for pushing artifacts to the Nexus storage.
warning

These secrets are mandatory for Tekton pipelines to work properly.

Kubernetes Provider​

All secrets are stored in Kubernetes in pre-defined namespaces. Platform suggests using the following approach for secrets management:

  • EDP_NAMESPACE-vault, where EDP_NAMESPACE is a name of the namespace where KubeRocketCI is deployed, such as edp-vault. This namespace is used by the platform. Access to secrets in the edp-vault is granted only for Administrators.

  • EDP_NAMESPACE-cicd-vault, where EDP_NAMESPACE is a name of the namespace where KubeRocketCI is deployed, such as edp-cicd-vault. Development team uses secrets in the edp-cicd-vault for microservices development.

See a diagram below for more details:

eso-with-kubernetes

In order to Install KubeRocketCI, a list of passwords must be created. Secrets are provided automatically when using ESO.

  1. Create a common namespace for secrets and platform:

    kubectl create namespace edp-vault
    kubectl create namespace edp
  2. Create secrets in the edp-vault namespace:

    apiVersion: v1
    kind: Secret
    metadata:
    name: keycloak
    namespace: edp-vault
    data:
    password: cGFzcw== # pass in base64
    username: dXNlcg== # user in base64
    type: Opaque
  3. In the edp-vault namespace, create a Role with a permission to read secrets:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
    namespace: edp-vault
    name: external-secret-store
    rules:
    - apiGroups: [""]
    resources:
    - secrets
    verbs:
    - get
    - list
    - watch
    - apiGroups:
    - authorization.k8s.io
    resources:
    - selfsubjectrulesreviews
    verbs:
    - create
  4. In the edp-vault namespace, create a ServiceAccount used by SecretStore:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: secret-manager
    namespace: edp
  5. Connect the Role from the edp-vault namespace with the ServiceAccount in the edp namespace:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
    name: eso-from-edp
    namespace: edp-vault
    subjects:
    - kind: ServiceAccount
    name: secret-manager
    namespace: edp
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: external-secret-store
  6. Create a SecretStore in the edp namespace, and use ServiceAccount for authentication:

    apiVersion: external-secrets.io/v1beta1
    kind: SecretStore
    metadata:
    name: edp-vault
    namespace: edp
    spec:
    provider:
    kubernetes:
    remoteNamespace: edp-vault # namespace with secrets
    auth:
    serviceAccount:
    name: secret-manager
    server:
    caProvider:
    type: ConfigMap
    name: kube-root-ca.crt
    key: ca.crt
  7. Each secret must be defined by the ExternalSecret object. A code example below creates the keycloak secret in the edp namespace based on a secret with the same name in the edp-vault namespace:

    apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    metadata:
    name: keycloak
    namespace: edp
    spec:
    refreshInterval: 1h
    secretStoreRef:
    kind: SecretStore
    name: edp-vault
    # target:
    # name: secret-to-be-created # name of the k8s Secret to be created. metadata.name used if not defined
    data:
    - secretKey: username # key to be created
    remoteRef:
    key: keycloak # remote secret name
    property: username # value will be fetched from this field
    - secretKey: password # key to be created
    remoteRef:
    key: keycloak # remote secret name
    property: password # value will be fetched from this field

Apply the same approach for enabling secrets management in the namespaces used for microservices development, such as sit and qa on the diagram above.

AWS Systems Manager Parameter Store​

AWS SSM Parameter Store can be used as a Secret Provider for ESO. For the platform, it is recommended to use the IAM Roles For Service Accounts approach (see a diagram below).

eso-with-ssm

AWS Parameter Store Scenario in KubeRocketCI​

In order to Install KubeRocketCI, a list of passwords must be created. Follow the steps below, to get secrets from the SSM:

  1. In the AWS, create an AWS IAM policy and an IAM role used by ServiceAccount in SecretStore. The IAM role must have permissions to get values from the SSM Parameter Store.

    a. Create an IAM policy that allows to get values from the Parameter Store with the edp/ path. Use your AWS Region and AWS Account Id:

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": "ssm:GetParameter*",
    "Resource": "arn:aws:ssm:eu-central-1:012345678910:parameter/edp/*"
    }
    ]
    }

    b. Create an AWS IAM role with trust relationships (defined below) and attach the IAM policy. Put your string for Federated value (see more on IRSA enablement for EKS Cluster) and AWS region.

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
    "Federated": "arn:aws:iam::012345678910:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXX"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
    "StringLike": {
    "oidc.eks.eu-central-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:edp:*"
    }
    }
    }
    ]
    }
  2. Create a secret in the AWS Parameter Store with the name /edp/my-json-secret. This secret is represented as a parameter of type string within the AWS Parameter Store:

    View: Parameter Store JSON
    {
    "keycloak":
    {
    "username": "keycloak-username",
    "password": "keycloak-password"
    },
    "ci-defectdojo":
    {
    "token": "XXXXXXXXXXXX",
    "url": "https://defectdojo.example.com"
    },
    "kaniko-docker-config":
    {
    "auths" :
    {
    "registry.com":
    {
    "username":"registry-username",
    "password":"registry-password",
    "auth": "<base64 encoded 'user:secret' string>"
    }
    }},
    "regcred":
    {
    "auths":
    {
    "registry.com":
    {
    "username":"registry-username",
    "password":"registry-password",
    "auth":"<base64 encoded 'user:secret' string>"
    }
    }},
    "ci-github":
    {
    "id_rsa": "id-rsa-key",
    "token": "github-token",
    "secretString": "XXXXXXXXXXXX"
    },
    "ci-gitlab":
    {
    "id_rsa": "id-rsa-key",
    "token": "gitlab-token",
    "secretString": "XXXXXXXXXXXX"
    },
    "ci-jira":
    {
    "username": "jira-username",
    "password": "jira-password"
    },
    "ci-sonarqube":
    {
    "username": "<ci-user>",
    "secret": "<secret>"
    },
    "ci-nexus":
    {
    "username": "<ci.user>",
    "password": "<secret>"
    },
    "oauth2-proxy-cookie-secret":
    {
    "cookie-secret": "XXXXXXXXXXXX"
    },
    "keycloak-client-headlamp-secret": "XXXXXXXXXXXX",
    "ci-argocd":
    {
    "token": "argocd-token",
    "url": "https://argocd.example.com"
    },
    "ci-dependency-track":
    {
    "token": "dependency-track-token",
    "url": "https://dependency-track.example.com"
    }
    }
  3. Set External Secret operator enabled by updating the values.yaml file:

    externalSecrets:
    enabled: true
  4. Install/upgrade edp-install:

    helm upgrade --install edp epamedp/edp-install --wait --timeout=900s \
    --version <edp_version> \
    --values values.yaml \
    --namespace edp \
    --atomic