KubeRocketCI 3.14 closes that gap with a single chart flag: pipelines.cancelInProgress. When enabled, the KRCI Tekton interceptor cancels still-running review PipelineRuns for the same pull request before triggering the replacement. The cancellation is graceful (CancelledRunFinally): the superseded run still executes its finally tasks, so your git provider gets a status for the old commit instead of an eternally "pending" check. This post shows the whole flow live on the local try-kuberocketci testbed - every screenshot and command output is from a real run.
A review (pull request) pipeline exists to answer one question: is this commit mergeable? The moment a newer commit lands on the same merge request, that question changes - and every pipeline still validating the old commit is wasted work. The pattern is worst exactly where CI matters most: active MRs with quick fix-up pushes, /recheck comments, rebases. Each event spawns a full pipeline - clone, build, lint, SonarQube scan, image verification - and on a shared cluster those zombie runs compete with useful work for CPU, memory, and runner capacity.
GitHub Actions users express the fix in two lines of YAML:
GitHub Actions equivalent (for comparison)
concurrency:
group: review-${{ github.ref }}
cancel-in-progress:true
Tekton, by design, has no such primitive - a PipelineRun is a standalone Kubernetes resource with no concept of "group" or "supersedes". The Tekton community has discussed pipeline concurrency for years without a shipped core feature. So the platform layer has to provide it - and in KubeRocketCI, the natural place is the krci interceptor, the component that already sits between every git webhook and every triggered pipeline.
The interceptor enriches webhook payloads for all four supported git providers (GitHub, GitLab, Bitbucket, Gerrit) before Tekton's EventListener instantiates a PipelineRun. That position makes it the perfect referee: it sees the new event before the new run exists, and it can find every old run for the same change because review PipelineRuns are now stamped with a app.edp.epam.com/git-change-number label at creation time.
The design choices matter as much as the mechanism:
Graceful, not brutal. The interceptor patches spec.status: CancelledRunFinally - Tekton stops scheduling new tasks and cancels running ones, but the pipeline's finally section still executes. In KubeRocketCI's pipelines-library, that is where commit status is reported back to the git provider - so the superseded commit ends up marked failed, never stuck pending.
Review pipelines only. The cancelInProgress parameter is injected into review Triggers exclusively. Build pipelines - the ones producing versioned artifacts after a merge - are never cancelled.
Best-effort by contract. If listing or patching fails, the error is logged and the new pipeline triggers anyway. Cancellation can never break your CI.
Scoped to one change. The label selector matches codebase + pipeline type + change number, so parallel MRs on the same repository never interfere with each other.
RBAC follows the flag. The interceptor's Role gains patch on pipelineruns only when the feature is enabled.
Enabling it is one Helm value on the pipelines-library chart:
values.yaml (pipelines-library)
pipelines:
# Cancel in-progress review PipelineRuns when a new commit is pushed to the same
The environment is the same local testbed as my previous posts: a kind cluster with KubeRocketCI, Tekton, and self-hosted GitLab, with the Go sample application test-go-app onboarded and pipelines.cancelInProgress: true set.
The webhook fires, and the review pipeline starts. Note the change-number label - this is the handle the interceptor will use later:
$ kubectl -n krci get pipelinerun --show-labels |grep review
review-test-go-app-main-2p446 Unknown Running 2s
app.edp.epam.com/codebase=test-go-app,
app.edp.epam.com/pipelinetype=review,
app.edp.epam.com/git-change-number=2, ...
Step 2: Push a Fix While the Pipeline Is Still Running
Ninety seconds in - the pipeline is mid-flight, fetch, build and sonar already done, image verification in progress - I notice the typo and push the fix to the same branch:
GitLab sends the update webhook. The interceptor finds the still-running pipeline for change 2, cancels it, and lets the new one trigger. Twenty seconds later the picture is exactly what you want:
Open the cancelled run in the Portal. The status is Cancelled, the spec says CancelledRunFinally, and the task list is the interesting part:
$ kubectl -n krci get taskrun -l tekton.dev/pipelineRun=review-test-go-app-main-2p446
NAME OK REASON
...-fetch-repository True Succeeded
...-build True Succeeded
...-sonar True Succeeded
...-dockerbuild-verify False TaskRunCancelled <- was in flight
...-gitlab-set-failure-status True Succeeded <- finally task ran
dockerbuild-verify - the task that was running when the second commit arrived - was cancelled. But the finally task gitlab-set-failure-status executed after the cancellation and reported the outcome to GitLab:
The git provider's view confirms it - no dangling "pending" check anywhere:
# commit #1 (superseded)
$ glab api projects/krci%2Ftest-go-app/repository/commits/7fb8e40/statuses
Review Pipeline -> failed
# commit #2 (current)
$ glab api projects/krci%2Ftest-go-app/repository/commits/5ffb585/statuses
Review Pipeline -> success
The replacement run finished green, the MR shows a passing pipeline for the latest commit, and the cluster spent zero extra minutes on the outdated one.
Back-of-the-envelope for a real team: the review pipeline above takes about 4 minutes of pod time (clone, build, lint, SonarQube, image verification). A ten-developer team pushing an average of one fix-up commit per MR wastes one full pipeline per MR - dozens of pipeline-hours per month on a busy repository, all competing with useful runs for cluster capacity. Concurrency limits and cluster autoscaling treat the symptom; cancelling superseded runs removes the work itself. It is the same reasoning that makes ephemeral environments cheaper than permanent staging: the cheapest workload is the one that stops existing when it stops being useful.
Does Tekton have a built-in equivalent of GitHub Actions cancel-in-progress?
No. Tekton PipelineRuns are independent Kubernetes resources with no native concurrency groups; pipeline concurrency has been a long-running community discussion without a shipped core feature. KubeRocketCI implements the pattern at the platform layer, in the interceptor that already processes every git webhook.
Why CancelledRunFinally instead of just deleting the PipelineRun?
Deleting (or hard-cancelling) a run would leave the superseded commit with a forever-pending status in GitLab or GitHub, and would skip cleanup logic. CancelledRunFinally stops the useless work but still executes the pipeline's finally tasks - in KubeRocketCI's pipeline library that is where the commit status is reported, so the old commit is cleanly marked as failed.
No. The cancelInProgress interceptor parameter is injected into review Triggers only. Build pipelines - triggered by merges and producing versioned artifacts - always run to completion.
All four KubeRocketCI providers: GitHub, GitLab, Bitbucket, and Gerrit. The change number is parsed from each provider's webhook payload (including Gerrit's, which sends it as either an integer or a string), and the same label-based cancellation logic applies everywhere.
Nothing visible to the developer. Cancellation is best-effort: a failure to list or patch PipelineRuns is logged by the interceptor and the new pipeline is triggered regardless. The worst case is the old behavior - two runs in parallel - never a blocked pipeline.
Can two different merge requests cancel each other's pipelines?
No. The interceptor matches on codebase and pipeline type and change number. Parallel MRs on the same repository each have their own change number, so their pipelines coexist untouched.
How do I enable this on an existing KubeRocketCI installation?
Set pipelines.cancelInProgress: true in the pipelines-library chart values and upgrade the release. The chart adds the interceptor parameter to review Triggers, stamps the change-number label on review runs, and extends the interceptor's RBAC with patch on pipelineruns - all gated by the same flag, nothing else changes.
Every push to an active merge request used to leave a zombie review pipeline burning cluster resources on a commit nobody cares about anymore. With pipelines.cancelInProgress: true, the KRCI interceptor now cancels superseded review runs the instant the replacing event arrives - gracefully, so finally tasks still report a status for the old commit, scoped by a change-number label so parallel MRs never collide, and best-effort so CI can never be blocked by its own cleanup.
In this run, the second commit landed 90 seconds into the first commit's pipeline: the in-flight task was cancelled on the spot, GitLab received a failed status for the superseded commit and a success for the new one, and only one pipeline ran to completion. The whole feature is one Helm value.
Explore the implementation in edp-tekton on GitHub - the cancellation logic lives in pkg/interceptor/cancel_pipelineruns.go.
KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the local testbed are all on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[Stale Branch Cleanup on Kubernetes: Keep Your Platform in Sync with Git]]>https://docs.kuberocketci.io/blog/stale-branch-cleanup-kubernetes-kuberocketci2026-07-13T00:00:00.000ZEvery feature branch you create in KubeRocketCI becomes a CodebaseBranch resource in Kubernetes - the object that drives its CI pipelines, records its built images, and feeds ephemeral preview environments. But when that branch merges and someone deletes it in GitLab or GitHub, the Kubernetes side never hears about it. The CodebaseBranch lives on: it clutters the Portal, offers a Build button for a branch that no longer exists, and on a busy platform these orphans accumulate by the hundreds.
KubeRocketCI 3.14 closes that gap. The codebase-operator now periodically verifies every branch against the real git repository, marks missing ones with a Stale condition (and a badge in the Portal), and - if you opt in - deletes them automatically, while refusing to touch any branch that is still wired into a deployment. This post shows the whole lifecycle live on the local try-kuberocketci testbed: every screenshot and command output below is from a real run.
The branch-per-feature workflow that makes preview environments so effective is also what produces the litter. The lifecycle of a healthy feature branch ends with a merge and a git branch -d - usually a checkbox in the merge request. That deletion happens in git. The platform objects created for the branch - the CodebaseBranch resource, its CodebaseImageStream (the record of images built from it) - are Kubernetes resources, and nothing in the git provider tells Kubernetes to remove them.
The result is drift between two sources of truth:
The Portal shows branches that do not exist. New team members trigger builds against them and get confusing clone failures.
Resources pile up. Platforms with dozens of codebases and a branch-per-ticket habit accumulate hundreds of dead CodebaseBranch and CodebaseImageStream objects.
Nobody knows which ones are safe to delete. A CodebaseBranch may look abandoned and still be the input of a live deployment - deleting the wrong one breaks an environment.
Manual cleanup does not scale, and a naive cron that deletes anything old is dangerous. What you want is exactly what a Kubernetes operator is for: continuously observe external state (git), record it on the resource, and act on it with guardrails.
The codebase-operator runs a stale branch checker - a leader-elected background loop, not a webhook - because branch deletion in git produces no Kubernetes event to react to. On every tick it makes one cheap git ls-remote call per repository (no cloning, no working tree) and compares the listed branches against the CodebaseBranch resources in the namespace:
The verdict is recorded in three places, so both humans and tooling can consume it:
A Stale status condition on the CodebaseBranch - the source of truth, with a reason (BranchNotFoundInGit) and a human-readable message.
The app.edp.epam.com/stale label, mirroring the condition so you can filter with a plain label selector.
Kubernetes events (BranchStale, StaleBranchDeleted, StaleBranchRetained, BranchStaleResolved) for audit trails and alerting.
Just as important is what the checker will not do:
No marking on infrastructure errors. If the git server is unreachable or credentials are rejected, the sweep skips that repository entirely - a GitLab outage will never mass-mark your branches.
Default branches are never checked.main cannot go stale.
Branches not yet pushed or already terminating are skipped.
A branch that reappears in git is un-marked on the next sweep (BranchStaleResolved).
Detection is always on and needs no configuration - the interval defaults to 24 hours (branchStaleCheckInterval Helm value; 0 disables the checker). Cleanup is a separate, opt-in decision per codebase, which we will get to below.
Walkthrough: Marking Stale Branches on a Live Cluster
The setup is the same local testbed as the preview environments post: a kind cluster with KubeRocketCI, Tekton, Argo CD, and a self-hosted GitLab, with the sample Go application test-go-app already onboarded. The only deviation from defaults: I set the check interval to 2m so the demo does not wait a day between sweeps.
The operator confirms the checker is running on startup:
INFO stale-branch-checker Starting stale branch checker {"interval": "2m0s"}
INFO stale-branch-checker Listing remote branches {"repository": "git@gitlab.127.0.0.1.nip.io:krci/test-go-app.git"}
INFO stale-branch-checker Remote branches listed successfully {"count": 2}
INFO stale-branch-checker Codebase branches staleness check finished {"codebases": 2}
In the test-go-app component: Branches → Create Branch. I create feature-tt-123 and feature-tt-456 from main, keeping the default pipelines.
Each click creates a CodebaseBranch resource and pushes the branch to GitLab. Note the new STALE column on the resource - empty for now, because every branch exists in git:
$ kubectl -n krci get codebasebranch
NAME RESULT STATUS CODEBASE NAME BRANCH STALE
krci-gitops-main success created krci-gitops main
test-go-app-feature-tt-123-02e67 success created test-go-app feature-tt-123
test-go-app-feature-tt-456-045fe success created test-go-app feature-tt-456
test-go-app-main success created test-go-app main
Step 2: Delete a Branch in Git - and Watch It Get Marked
Now the everyday event this feature exists for: the branch is merged and removed in the git provider - the "delete source branch" checkbox on a merge request, or an explicit API call. I delete feature-tt-123 straight in GitLab:
The Portal surfaces the same information as a warning badge in the branch list - hover it and the tooltip shows the condition message:
At this point nothing has been deleted - the default cleanup strategy is mark: make staleness visible, let a human decide. For many teams that is already the win. But we can go further.
Auto Cleanup - With a Deployment-Aware Safety Net
Before turning on automatic deletion, let's build the exact scenario that makes naive cleanup dangerous: a stale branch that is still deployed.
Following the preview environment flow, I build feature-tt-456 and create a Deployment named preview with an Auto-triggered dev environment consuming that branch's image stream:
After the deploy pipeline finishes, the preview environment is live in its own namespace, running the image built from feature-tt-456:
Now I delete feature-tt-456 in GitLab too. Next sweep: both branches are stale - but one of them is backing a running environment.
Even a well-meaning human with kubectl cannot break the deployment by "cleaning up" the stale branch. The codebase-operator's admission webhook now rejects deleting any CodebaseBranch that participates in a deployment (a rule the Portal used to enforce only client-side):
With auto, the next sweep deletes stale branches that nothing references - and retains the ones that a CDPipeline (or a Stage autotest quality gate) still uses. One sweep later, the two stale branches got exactly opposite treatment:
$ kubectl -n krci get events --sort-by=.lastTimestamp |grep Stale
Warning BranchStale Branch feature-tt-456 was not found in the git repository and is marked as stale
Normal StaleBranchDeleted Branch feature-tt-123 was not found in the git repository and is not used by any deployment, deleting
$ kubectl -n krci get codebasebranch
NAME RESULT STATUS CODEBASE NAME BRANCH STALE
krci-gitops-main success created krci-gitops main
test-go-app-feature-tt-456-045fe success created test-go-app feature-tt-456 True
test-go-app-main success created test-go-app main
feature-tt-123 is gone, and because the CodebaseBranch owns its CodebaseImageStream, the image stream was garbage-collected with it - no orphaned children. feature-tt-456 stays, and its condition message names the exact resource that is keeping it alive:
Branch was not found in the git repository; retained because it is used by CDPipeline preview (inputDockerStreams)
The Portal tooltip shows the same reason, so the "why is this still here?" question answers itself:
The retained branch is cleaned up the moment it stops being needed. I remove the preview Deployment in the Portal (Actions → Delete, type the name to confirm) - which, as covered in the previous post, tears down the environment namespace and the Argo CD Application:
Next sweep, the last stale branch goes too:
$ kubectl -n krci get events --sort-by=.lastTimestamp |grep StaleBranchDeleted
Normal StaleBranchDeleted Branch feature-tt-123 ... not used by any deployment, deleting
Normal StaleBranchDeleted Branch feature-tt-456 ... not used by any deployment, deleting
$ kubectl -n krci get codebasebranch
NAME RESULT STATUS CODEBASE NAME BRANCH STALE
krci-gitops-main success created krci-gitops main
test-go-app-main success created test-go-app main
The platform is back to exactly what git says it should be - no manual bookkeeping, and at no point could the cleanup have taken down a running environment.
What marks a CodebaseBranch as stale in KubeRocketCI?
A periodic checker in the codebase-operator lists the repository's branches with git ls-remote and compares them against the CodebaseBranch resources. If a branch no longer exists in the git repository, the resource gets a Stale status condition (reason BranchNotFoundInGit), the app.edp.epam.com/stale label, a Kubernetes event, and a warning badge in the Portal.
Can the default branch ever be marked stale or deleted?
No. Default branches (for example main) are excluded from staleness checks entirely, as are branches that have not been pushed yet and branches already being deleted.
What happens if GitLab or GitHub is down during a sweep?
Nothing. The checker never marks branches when the repository listing fails - connectivity problems, DNS errors, or rejected credentials cause the repository to be skipped for that sweep. Only a successful listing that positively lacks the branch produces a stale mark.
Does stale detection delete branches automatically?
Not by default. Out of the box the strategy is mark: branches are flagged and left for a human. Automatic deletion is enabled per codebase with the app.edp.epam.com/branch-cleanup-strategy: auto annotation, and even then a stale branch referenced by a CDPipeline or a Stage quality gate is retained, with the retaining resource recorded in the condition message.
Can a stale branch that is still deployed be deleted by mistake?
No - two independent guardrails prevent it. Auto cleanup skips any branch referenced by a deployment, and the codebase-operator's admission webhook rejects manual deletion (kubectl delete, API calls) of a CodebaseBranch that participates in a deployment, naming the CDPipeline in the error message.
What happens to the images built from a deleted branch?
The CodebaseBranch owns its CodebaseImageStream, so when auto cleanup deletes the branch resource, Kubernetes garbage collection removes the image stream record with it. The container images themselves stay in your registry - registry retention is a separate concern.
How do I find all stale branches across the platform?
kubectl get codebasebranch -l app.edp.epam.com/stale=true -A. The label mirrors the status condition specifically so that plain label selectors - and any tooling built on them - can find stale branches without parsing conditions.
Feature branches are ephemeral; the Kubernetes resources behind them used to be forever. With KubeRocketCI 3.14, the codebase-operator closes the loop. One git ls-remote per repository per sweep detects branches deleted from git, and a Stale condition plus label plus Portal badge make them visible. An opt-in per-codebase auto strategy deletes the ones nothing depends on - while an admission webhook and a deployment-aware retention check make sure the cleanup can never take down a running environment.
In this run we created two branches, deleted them in GitLab at different points of their lifecycle, and watched the operator mark one, auto-delete it, retain the other while its preview environment was alive, and finish the job the moment the environment was gone - ending with a platform that matches git exactly.
Next steps from here:
Spin up the same environment with Try KubeRocketCI Locally and replay the flow - the whole demo fits in a lunch break.
Read Manage Branches for the day-to-day branch workflow in the Portal.
Explore the codebase-operator on GitHub - the stale checker lives in controllers/codebasebranch/stalecheck/.
KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the local testbed are all on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[Claude Code Telemetry: Token and Cost Metrics]]>https://docs.kuberocketci.io/blog/claude-code-telemetry-opentelemetry-token-cost-tracking2026-07-12T00:00:00.000ZIn the last 24 hours, one engineer's Claude Code sessions burned $166.97, 833K input/output tokens, and 959 API requests - and I can tell you exactly which Jira story got that money, which model spent it, which subagents did the heavy lifting, and that the prompt-cache hit ratio stayed at 97.2% the whole time. Not from a vendor console: from a self-hosted Grafana stack fed by telemetry Claude Code already emits.
That last part is the point. Claude Code has native OpenTelemetry support - rich metrics and events for tokens, cost, sessions, tools, skills, and agents - but nothing to look at them with. As soon as AI-assisted development becomes a line item, every engineering leader asks the same questions: what are we spending, on what, and is it worth it? This post shows how we answer them at KubeRocketCI with a small OTel Collector → Prometheus + Loki → Grafana pipeline that runs identically on a laptop and on a team Kubernetes cluster - the claude-code-telemetry stack - without ever capturing a single prompt or line of your code.
Here's the path we'll take: why an "AI Factory" needs production-grade metering, what Claude Code emits out of the box, how to stand the stack up locally in minutes, how to attribute cost to projects and Jira tickets, how to deploy it for a whole team on Kubernetes (including CI/CD pipelines), the cardinality trap to avoid - and what a real day of data actually says.
Because AI agents are production workers now, and production workers get metered. When developers and CI pipelines run Claude Code daily, token spend becomes real budget - and without attribution you can't answer which project, epic, or team consumed it, whether the spend produced code, or how adoption is trending.
Think of it as the observability layer of an AI Factory: the same discipline you apply to build minutes, cloud bills, and test flakiness, applied to AI-assisted engineering. Concretely, the questions that come up in every rollout:
FinOps: cost by project, epic, story, user, and model - who spends, on what work.
Efficiency: cost per commit, cost per 1K lines changed, cache hit ratio - is the spend producing output.
Adoption: daily/weekly active users, sessions, active hours - is the investment being used.
Governance: which plugins, skills, and MCP servers run where, in which versions - what your fleet actually executes.
And one hard constraint shapes everything: this must work without capturing prompts, responses, or file contents. Usage analytics is not surveillance - only aggregate counters and structured metadata should ever cross the wire.
What Telemetry Does Claude Code Emit Out of the Box?
Claude Code natively exports OTLP metrics and events once you set a few environment variables - no wrappers, no scraping, no plugins. The metric names are fixed by the CLI, but every one carries dimensions you can slice by, and you can add your own.
The metrics that matter most for tokenomics:
Metric
What it answers
claude_code.token.usage
tokens by type (input/output/cacheRead/cacheCreation), model, agent.name, skill.name
claude_code.cost.usage
USD by the same dimensions, plus plugin and marketplace
Alongside metrics, Claude Code emits events - api_request (per-call cost and token detail), skill_activated, tool_result, tool_decision - which carry the high-cardinality forensics: per-session, per-prompt, per-tool detail.
Two facts drive the whole design. First, you cannot invent new metric names, but you can inject new dimensions and derive anything downstream with PromQL or LogQL. Second, content capture is opt-in and off by default in our setup: prompts, assistant responses, file contents, and raw API bodies stay dark - we enable only OTEL_LOG_TOOL_DETAILS so skill, agent, and tool names are visible.
Only Claude Code emits OTel today - the CLI, its IDE extensions, and the Agent SDK all share the same environment-variable configuration. Claude Desktop's chat has no telemetry export (only its Cowork feature offers an admin-configured OTLP endpoint on Team/Enterprise plans), Claude Code on the web currently emits nothing, and raw API usage is served by Anthropic's pull-based Usage and Cost APIs instead.
Two practical consequences for a rollout. First, the stack described here captures CLI, IDE-extension, and Agent SDK sessions - if part of your team works in Claude Code on the web, that spend is invisible to OTel until Anthropic closes the gap, so budget reconciliation still needs the Usage and Cost API as the financial source of truth. Second, if you later enable Cowork telemetry, note its content is not redacted by default - unlike Claude Code - so apply filtering in the collector before it shares a backend with this pipeline.
With one OTel Collector fanning out to Prometheus for metrics and Loki for events, visualized in Grafana - the standard trio, packaged twice from the same config so a laptop and a Kubernetes cluster behave identically. That's the entire claude-code-telemetry architecture:
The Collector is the heart: it receives OTLP (gRPC/HTTP), stamps every datapoint with a collector.env attribute so laptop and cluster traffic never get confused, converts resource attributes into Prometheus labels (resource_to_telemetry_conversion), and ships full-fidelity events to Loki. On top sit six pre-built dashboards, organized into three Grafana folders by audience:
Claude Code - Usage Audit, Efficiency & ROI, Operations & Adoption, Session Explorer: the daily-driver views.
Governance - Governance & Security: plugins, skills, permission decisions across the fleet.
Operational - Collector Health: is the pipeline itself alive and ingesting.
Start the docker-compose testbed, merge one env block into your Claude Code settings, and data flows within a minute. The local stack is the source of truth we iterate on - same collector pipeline, same dashboards as the cluster deployment:
Restart Claude Code, open Grafana, and the Usage Audit dashboard starts filling with your own traffic. This loop - real usage in, dashboard iteration out - is how every panel in this post was built.
How Do You Attribute Cost to Projects and Jira Tickets?
By injecting business dimensions through OTEL_RESOURCE_ATTRIBUTES before each session starts. Claude Code reads the variable once at startup and stamps every metric and event with it - so the discipline is simple: one Jira ticket ≈ one session.
Three custom keys are enough to start: project (the git repository), jira.epic, and jira.story. User identity comes free - Claude Code natively attaches user.email, user.id, and organization.id to everything, so per-person breakdowns need no injection at all.
The Session Explorer dashboard shows what this buys you. Every session becomes a row with its epic, story, project, user, cost, and event count - sortable, filterable, drillable:
Reading my own last 24 hours: $166.97 total, and the table names the culprits - two long krci-workspace sessions on story EPMDEDP-17188 at $67.68 and $60.66, a edp-keycloak-operator triage on EPMDEDP-16043 at $24.93, and the session that built the telemetry stack itself at $1.99. When a manager asks "what did that epic cost in AI assistance?", this table is the answer - down to the session.
With the repo's self-contained Helm bundle - one helm install, no operators, no CRDs, no cluster-monitoring prerequisites. Local PoC is for one laptop; team consumption means everyone's Claude Code - and your CI runners - pushing OTLP to a single in-cluster endpoint with shared dashboards:
The bundle wraps the upstream OTel Collector, Prometheus, Loki, and Grafana charts, and it is deliberately isolated: its Prometheus scrapes exactly two static targets (the collector's data and self-telemetry ports), all cluster-wide discovery is disabled, no node-exporter or kube-state-metrics tag along, and RBAC is namespace-scoped. It won't fight the observability stack you already run - and if you have one, each component toggles off (grafana.enabled: false, prometheus.enabled: false, …) so you can wire just the collector into existing storage.
Three things change at team scale compared to the laptop:
Distribution - ship the env block through a managed settings.json (MDM), so telemetry is on for everyone, pointed at the team endpoint, and can't be silently unset.
Authentication - expose the OTLP endpoint behind auth (OTEL_EXPORTER_OTLP_HEADERS with a bearer token, or Claude Code's otelHeadersHelper for short-lived tokens).
Enforcement - the Collector becomes the governance point users cannot bypass: inject defaults for missing attributes, validate jira.* values against ^[A-Z]+-[0-9]+$, and drop or route telemetry that arrives without a project.
Every datapoint's collector.env label (local-poc vs k8s) keeps sources distinguishable, so laptops and the cluster can even feed shared dashboards during a migration without mixing data.
Exactly the same way - headless Claude Code in a pipeline step is still Claude Code, and the same OTLP env vars apply. This is where the AI Factory framing stops being a metaphor: automated agents doing code review, test generation, or triage inside Tekton pipelines spend tokens too, and they should land in the same ledger as human sessions.
In a pipeline task, set the same telemetry variables plus attribution that identifies the machine work:
The pipeline populates project from the codebase it builds and the ticket from the change it processes - attribution comes from parameters you already have. One caveat worth knowing: Claude Code does not propagate OTEL_* variables to subprocesses, so only the CLI process itself is instrumented - which is exactly what you want in a build pod. The result: a single Grafana view where "cost of AI per merge request" and "cost of AI per developer day" sit side by side.
Every unique label combination in Prometheus is a stored time series - so unbounded labels like Jira story IDs will eventually blow up your metrics store. This is the telemetry twin of the lesson from our Kubernetes audit trail post: collect with a goal, not by default.
The rule we apply, enforced in the Collector:
Bounded dimensions (model, type, agent.name, skill.name, project, jira.epic) → safe as metric labels, fast Grafana breakdowns.
Unbounded dimensions (jira.story, session.id, prompt.id) → events in Loki only; derive per-story cost with LogQL over api_request events.
A single-user PoC can keep jira.story on metrics for convenience - ours does. At team scale, the Collector strips it from the metrics pipeline while Loki keeps the full detail. Same data, right store: Prometheus answers "how much, by which bounded dimension" in milliseconds; Loki answers "what exactly happened in that session" when you drill in.
That efficiency metrics are the fastest way to turn raw spend into decisions - including uncomfortable ones. The Efficiency & ROI dashboard divides cost by output, and a real day of data is more instructive than any synthetic demo:
Cache hit ratio: 97.2%. Prompt caching is doing enormous work - at roughly a tenth of the input price, those cache reads are the difference between $167 and a number several times larger. If this ratio drops, something in your workflow broke; it's the first efficiency metric worth alerting on.
Cost per active hour: $31.10. A defensible, explainable unit price for AI pairing - the number to put next to an engineer's loaded hourly cost when someone asks about ROI.
Cost per 1K lines changed: $79.28. Noisy on any single day, meaningful as a trend across weeks and projects.
Cost per commit: $∞. Total cost divided by claude_code.commit.count, which counts only commits made by Claude Code itself. We review and commit by hand, so the counter stays at zero and the panel shows infinity. If you commit through the agent, this becomes a real spend-per-delivered-commit number.
The Usage Audit dashboard tells you where the tokens went:
Over the two-day range: cost concentrates in krci-workspace / EPMDEDP-16757 ($97.6) and edp-keycloak-operator / EPMDEDP-16043 ($80.7), with the telemetry stack's own development a modest $10.4. Token type over time is dominated by cacheRead - visual confirmation of that 97.2%. And the subagent panel is the sleeper hit: custom agents consumed 18.2M tokens against 4.08M for built-in Explore and 3.13M for general-purpose - the multi-agent workflows you build are where the budget actually goes. Tool usage (2,059 Bash invocations vs 475 Reads and 288 Edits in the window) rounds out the picture of what the agent fleet physically does all day.
How Do You Govern Plugins and Skills Across a Team?
The same event stream doubles as a software inventory - who runs which plugin, in which version, triggered how. The Governance & Security dashboard breaks down skill activations by trigger, which quietly answers an adoption question every platform team has:
In our data, most activations are claude-proactive or nested-skill rather than user-slash - the agent invokes the team's KubeRocketCI plugins on its own more often than humans type slash commands. That's exactly what you want from packaged expertise, and now it's measurable rather than anecdotal.
The plugin inventory closes the loop for fleet management:
Every user session reports its loaded plugins with marketplace, version, and install scope. When you ship a fixed krci-godev and need to know who's still loading 0.6.0 - the dashboard already knows.
Try it on your laptop: clone claude-code-telemetry, run docker compose up -d in local/, merge the settings snippet, and watch your own session appear.
Deploy for your team: install the Helm bundle from deploy-templates/ and distribute the env block via managed settings.
Go deeper on the design: the repo's docs/analytics.md covers the attribution model, cardinality guidance, and rollout plan; spec/ catalogs the full Claude Code OTEL vocabulary.
Token spend is the new build minutes: invisible until the invoice arrives, obvious once it's on a dashboard. Claude Code already emits everything you need - stand up the backend, inject three attributes, and "what does AI development cost us?" becomes a Grafana query instead of a guess.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[A Portable Kubernetes Audit Trail: Who Changed What]]>https://docs.kuberocketci.io/blog/kubernetes-audit-trail-who-changed-what2026-07-07T00:00:00.000ZYou trigger pipelines. You update a GitServer, promote a deployment, adjust a Codebase. Much of that flows through GitOps and lands in your Git history - but not all of it. Some changes you make directly against the cluster, from the portal or kubectl, because that's the quickest way to get something done. That's normal, everyday platform work. What's harder is answering a fair question that comes up afterward: what changed, when, and who made the change? - especially for the changes that never went through a pull request.
On managed Kubernetes the raw answer exists - the API server writes an audit record for every request - but getting to it is another story. On Amazon EKS those records stream to CloudWatch. Answering a simple operational question then means writing a Logs Insights query, knowing the retention window, and having the right IAM. Great for a SIEM pipeline; heavy for a Tuesday-afternoon "what just changed?". KubeRocketCI ships a lighter, portable alternative called krci-audit - a self-contained, queryable audit trail that works the same on EKS, on-prem, or a local kind cluster. The catch, which most of this post is about: an audit trail is not free, so you should be deliberate about what you capture.
Here's the reasoning - why native audit logs leave a gap for day-to-day questions, how krci-audit fills it without a cloud dependency, the performance trap of auditing everything (with real numbers from a running cluster), and where this is going next.
Short answer: they're built for security forensics and SIEM, not for a quick operational lookup. The data exists - every managed Kubernetes distribution can emit API server audit logs, a structured record of who called the API, what they changed, and when. On EKS you enable the audit log type and the control plane ships it to CloudWatch Logs. As a foundation that's exactly right: immutable, comprehensive, centrally retained, and easy to alert on.
But for operational questions it has friction:
Quick access is a query, not a lookup. "Who created this PipelineRun?" becomes a CloudWatch Logs Insights query against a large, verbose stream - filtered by objectRef, verb, and time - not a one-line answer.
It's cloud-locked. The retrieval path on EKS (CloudWatch), on AKS (Azure Monitor), and on-prem (a file sink you wire up yourself) are all different. Your runbook doesn't port.
It's undifferentiated. The native stream captures everything the audit policy allows - most of which is control-plane noise you'll never ask about (more on that below).
It lives outside your platform. Developers using the KubeRocketCI portal can't see "who triggered this run" without leaving the tool and querying a cloud console they may not have access to.
None of this makes native audit logging wrong - it makes it the wrong tool for the quick, in-platform question. That's the gap krci-audit targets.
How Does krci-audit Give You a Portable Audit Trail?
It captures a scoped set of admission changes inside your cluster and makes them instantly queryable - no dependency on any cloud provider's logging service. krci-audit is a small, self-contained component that does exactly that.
A resource change hits a validating admission webhook, which ships the record through Vector into an append-only PostgreSQL table; a read-only API then serves it to the portal and other consumers. The pieces:
Component
Role
Capture
A ValidatingWebhookConfiguration (backed by kube-audit-rest) receives the AdmissionReview for each matching change. It runs with failurePolicy: Ignore and a 1-second timeout, so it never blocks a platform operation.
Ship
A Vector sidecar tails the capture log, keeps only the events you asked for, and writes them to PostgreSQL.
Store
A dedicated PostgreSQL database holds events in a monthly-partitioned, append-only table. Writes use a least-privilege role that cannot update or delete existing rows.
Read API
A separate, read-only service exposes initiator lookup and a filterable events query. It connects as a SELECT-only role, so the read path can never alter the trail.
Because the whole thing runs inside your cluster, the experience is identical on EKS, AKS, GKE, OpenShift, or a laptop kind cluster - the same install, the same API, the same portal feature. It's an add-on, off by default, that you turn on when you have a reason to.
Because an admission webhook sits in the write path of the API server - so a catch-all policy taxes every write and buries the trail in noise. Every request matching the webhook's rules is paused, sent to the webhook, and awaited (within the timeout) before the write completes. Point it at * - all resources - and you pay that cost on everything.
How much noise? To put real numbers on it, I measured the local try-kuberocketci testbed - a single-node kind cluster (Kubernetes v1.36.1) running the full platform and otherwise idle. Comparing each Lease object's resourceVersion over a 30-second window shows the churn a catch-all audit policy would capture (reproduce it with kubectl get leases -A, twice, 30 seconds apart):
31 Lease objects, and all 31 changed within those 30 seconds - pure leader-election heartbeat traffic. Extrapolated: ~3,720 Lease UPDATE admissions per hour, none of which anyone will ever audit.
And that's before Events (13), EndpointSlices (50), Pods (54), and ConfigMaps (122) - all high-churn, all captured by a * policy.
Now compare that to the scoped trail the same testbed actually recorded over ~36 hours, using krci-audit's default configuration (shown below):
Resource
Events
PipelineRuns
97 (71 UPDATE, 18 DELETE, 8 CREATE)
CodebaseBranches
12
GitServers
4
CodebaseImageStreams
2
CDStageDeployments / Stages
3
That's 118 meaningful events over a day and a half - roughly 80 a day - versus ~3,720 an hour of lease noise from a single un-scoped resource type. A catch-all policy would be on the order of a thousand times larger, almost entirely with records no one wants.
The design that avoids this: scope at the webhook rules, not downstream. When a resource isn't in the rules, the API server never calls the webhook at all - zero added latency, zero stored noise. krci-audit's default rules capture only what the platform cares about:
capture:
failurePolicy: Ignore # never block a platform operation
timeoutSeconds:1
rules:
-apiGroups:["v2.edp.epam.com"]# KubeRocketCI CRDs
resources:["*"]
-apiGroups:["tekton.dev"]
resources:["pipelineruns"]
level: metadata # capture object metadata, not full bodies (bounds size and PII)
The guidance is simple: audit with a goal, not by default. Decide what question you need the trail to answer - "who triggered pipelines", "who touched Git server config", "who deleted deployments" - and capture those resources. You can widen the filter later with a helm upgrade; you can't un-spend the latency budget you burned auditing lease renewals.
The KubeRocketCI portal shows a "Triggered By" field on the PipelineRun details page - the first thing krci-audit powers, answering a question every platform user eventually asks: who started this run?
It resolves the run's creator from the audit trail and classifies them:
A human user (for runs started from the portal), or
An automation identity - the service account behind a Git webhook or scheduled build (as shown above), or
N/A, when the initiator can't be resolved (for example, historical runs created before auditing was enabled).
It's deliberately distinct from the existing Author column, which shows the Git commit author. The person who wrote the code is not always the identity that launched the run. Under the hood the portal calls the krci-audit read API with the run's kind, namespace, and name; no extra annotations, no changes to how pipelines are created.
Through a plain HTTP API - the same data the portal uses, handy for incident triage, scripts, and compliance exports. It's a ClusterIP service (in-cluster only), so port-forward it to try from your workstation:
Real questions this answers without a cloud console:
"Who deleted my Codebase last night?" - filter by operation=DELETE and the object.
"Show me everything this service account did." - filter by actor; the same query backs a user's own activity view.
"Produce the change history for this resource for an audit." - export the filtered result.
Because the store is append-only and time-stamped, it doubles as compliance evidence: an immutable record of who changed what. It maps cleanly to SOC 2, ISO 27001, GDPR, and PCI DSS change-tracking controls, and it's retrievable through the API without handing anyone direct database access.
When you need cluster-wide forensics, tamper-proof off-cluster retention, or cross-signal correlation - keep shipping the API server audit log to your SIEM. To be clear about scope: krci-audit complements native audit logging, it doesn't replace a security program. krci-audit is for the operational, in-platform question - fast, portable, developer-facing "who/what/when" - not the security org's system of record.
This is the first slice of a broader capability. Next up:
Authentication and authorization on the API. Today the read API is unprotected and reachable only in-cluster (ClusterIP, no Ingress), with network scope as the interim boundary. Role-based access - so the trail respects who's allowed to see what - is the next milestone.
"Who changed what" across every object, in the portal. The initiator today is surfaced for PipelineRuns; the same audit data can enrich every KubeRocketCI resource - a change history and last-modified-by on Codebases, environments, Git servers, and more - turning the portal into a single place to answer accountability questions for the whole platform.
Audit trails are one of those features that's invisible until the moment you need it - and then it's the only thing that matters. Turn it on with a goal, keep it scoped, and it'll answer "who changed what?" in one call instead of one afternoon.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[GitLab CI Integration in KubeRocketCI]]>https://docs.kuberocketci.io/blog/gitlab-ci-integration-kuberocketci2026-06-08T00:00:00.000ZGitLab CI integration in KubeRocketCI lets a single application run its CI pipeline in GitLab CI - on a GitLab Runner - instead of Tekton, while still being managed as a first-class Codebase on the platform. You set one field on the Codebase (spec.ciTool: gitlab), and KubeRocketCI generates a .gitlab-ci.yml in the repository; GitLab then runs the pipeline, with no Tekton involved. From then on, every merge request runs a review pipeline and every merge runs a build pipeline - all native GitLab CI, all on your own cluster.
This is part three of my hands-on series on the local try-kuberocketci testbed. In part one I stood up the full platform in two commands; in part two I built ephemeral preview environments from a feature branch. Both ran their CI in Tekton. This post takes the same kind cluster running KubeRocketCI 3.13 and shows the multi-CI path: how GitLab CI integration works, the three things you must set up before you enable it - a Runner, the onboarded CI/CD components, and a ConfigMap - and a full review-to-build run with real output.
What "GitLab CI Integration" Means in KubeRocketCI
By default, KubeRocketCI runs your CI in Tekton. Version 3.13 added multi-CI: a per-Codebase choice of engine, set with one field - spec.ciTool: tekton (the default) or spec.ciTool: gitlab (see the 3.13 notes and Codebase API). Choosing gitlab is what this post is about.
When you set ciTool: gitlab, KubeRocketCI handles the wiring. Instead of the Tekton EventListener and webhook it would normally create, it generates a .gitlab-ci.yml and commits it to the repository; GitLab CI then runs the pipeline. The file is generated rather than written by hand, and no Tekton is involved - the build runs as native GitLab CI jobs on a GitLab Runner, and the logs reside in GitLab.
Other aspects are unchanged: the application remains a first-class Codebase, with the same Portal, branches, and GitOps deployment as a Tekton application. GitLab-CI and Tekton applications can run side by side on the same platform; only the CI execution moves to GitLab (see the comparison below).
On a real codebase the file lands in the repo, and there is no project webhook - GitLab CI does not need one:
$ kubectl -n krci get codebase java-gitlabci-app -ojsonpath='{.spec.ciTool}'
gitlab
# .gitlab-ci.yml is committed to the repo; project webhooks: 0
The generated .gitlab-ci.yml is intentionally minimal: it contains no build logic of its own and instead references reusable GitLab CI/CD components. Those components must be in place first.
Before You Enable It: Three Things to Set Up First
GitLab CI integration has three prerequisites that the Tekton path does not - a one-time platform setup, not something you repeat per app. Skip any one and the first pipeline fails to start or cannot resolve its components; complete them once, and subsequent GitLab-CI Codebases require no additional setup.
1. Install a GitLab Runner (KubeRocketCI Does Not Bundle One)
Tekton ships with KubeRocketCI; a GitLab Runner does not. GitLab CE already serves CI, but with no runner registered, jobs sit pending forever. On the testbed, make gitlab-ci installs the official gitlab/gitlab-runner Helm chart with the Kubernetes executor and registers it.
The Kubernetes executor runs each job as a pod in the cluster - the same model Tekton uses, so your CI capacity scales with the cluster. The testbed uses Helm for simplicity, but on a real cluster you install the runner the GitOps way, like every other platform component: KubeRocketCI ships a gitlab-runner add-on in the edp-cluster-add-ons repository that Argo CD reconciles, with the runner's registration token supplied through External Secrets. Either way you end up with one registered runner, sized to your job volume, ready to execute GitLab CI jobs.
2. Onboard the KubeRocketCI CI/CD Components Into Your Instance
This prerequisite is easy to overlook. The injected .gitlab-ci.yml does not contain the build steps directly; it references a GitLab CI/CD component - the kuberocketci/ci-java17-mvn library, which exposes review and build entry points over a 7-stage flow (prepare → test → build → verify → package → publish → release).
The catch: GitLab CI/CD components only resolve on the same GitLab instance ($CI_SERVER_FQDN). A component reference like $CI_SERVER_FQDN/kuberocketci/ci-java17-mvn/build@0.1.1 will not reach out to gitlab.com - it resolves against your GitLab. So you must onboard (mirror) the components into your instance before enabling GitLab CI, and tag them at the version your template pins.
On the testbed, make gitlab-ci mirrors the component into the local GitLab as kuberocketci/ci-java17-mvn at tag 0.1.1 (with two local-only patches: native arm64 builds and pushing to the in-cluster GitLab Container Registry instead of Docker Hub). In your own environment, this is where you publish the KubeRocketCI components - or your own forked components - to your GitLab CI/CD Catalog so every project can include them.
Onboard the components first, enable the Codebase second
If you set ciTool: gitlab before the component exists at the pinned tag on your instance, KubeRocketCI still injects .gitlab-ci.yml, but the very first pipeline fails at the include: stage with an unresolved-component error. Mirror and tag the components first.
KubeRocketCI does not invent the pipeline - it fills in a template stored in a ConfigMap, one per tech stack. The Codebase points at the template with an annotation (or KubeRocketCI falls back to a gitlab-ci-{lang}-{buildtool} naming convention), and your codebase name is substituted in before the file is committed.
The ConfigMap for Java/Maven contains only orchestration - a workflow rule set, variables, and two included component entry points (review and build) gated by rules:
These two rules are the entire control flow: the review component runs on merge_request_event, and the build component runs on the protected/default branch. That split produces the "MR → review, merge → build" behavior, with no webhook required.
Enable GitLab CI on a Codebase (spec.ciTool: gitlab)
With the Runner online, the components onboarded, and the ConfigMap applied, enabling GitLab CI is one Codebase manifest. The ciTool: gitlab field selects the engine; the annotation selects the template:
manifests/sample-gitlabci-codebase.yaml (excerpt)
apiVersion: v2.edp.epam.com/v1
kind: Codebase
metadata:
name: java-gitlabci-app
namespace: krci
annotations:
app.edp.epam.com/gitlab-ci-template: gitlab-ci-java-maven # pick the ConfigMap
spec:
type: application
lang: java
framework: java17
buildTool: maven
ciTool: gitlab # <-- run CI in GitLab CI, not Tekton
strategy: import
gitServer: gitlab
gitUrlPath: /krci/java-gitlabci-app
disablePutDeployTemplates:true
In the KubeRocketCI Portal, the component is a normal Codebase - same lifecycle, same branches, same deployment story as a Tekton app - but its CI Tool reads GitLab. That is the platform side of the integration: you manage the app on KubeRocketCI, even though its pipeline executes in GitLab.
Once KubeRocketCI finishes, the injected .gitlab-ci.yml is visible in the GitLab project - the {{.CodebaseName}} placeholder now reads java-gitlabci-app, and the two component includes are wired and ready:
The Pipeline in Action: Review on MR, Build on Merge
The flow mirrors the Tekton path from the previous posts - open a merge request to validate, merge to build and publish - but every job runs in GitLab CI on the Runner.
In the GitLab project, both pipelines show green - one triggered by the merge request, one by the merge to main:
Opening a merge request triggers a merge_request_event pipeline that runs the review component - validation only, no image push. On the live run it finished green in 91 seconds, 10 jobs across four stages:
review pipeline #4 (merge_request_event) - 91s - GREEN
prepare init-values
test test compile-check lint
test helm-docs helm-lint dockerfile-lint
build build sonar
verify dockerbuild-verify # buildkit builds the image but does NOT push
The sonar job is a real SonarQube quality gate - the same engine the Tekton pipelines use - and dockerbuild-verify proves the image builds without publishing it. GitLab renders the MR with its pipeline result attached:
Merging the MR pushes to the protected main branch, which trips the second workflow rule and runs the build component. It is the review jobs plus the publishing stages - buildkit-build (build and push), maven-deploy, and git-tag. On the live run it finished green in 100 seconds, 12 jobs:
build pipeline #5 (push, protected main) - 100s - GREEN
The buildkit-build job builds the container image and pushes it to GitLab's bundled Container Registry using the CI job token - no external registry, no Docker Hub credentials. After the build pipeline, the image is published with a content-addressed tag, and git-tag has created the matching release tag:
From here, the GitOps delivery story is identical to the Tekton path - the published image feeds the same CodebaseImageStream and Argo CD deployment flow covered in part two of this series.
Both engines are first-class. The choice is about where your team already lives and what you want to own.
Dimension
Tekton (default)
GitLab CI (ciTool: gitlab)
Pipeline definition
Tekton PipelineRun (KRCI-managed)
.gitlab-ci.yml (operator-injected, GitLab-native)
Trigger mechanism
Tekton webhook
GitLab workflow rules (no webhook)
Where job logs live
KubeRocketCI Portal / Tekton
GitLab CI UI
Reusable build logic
Tekton tasks / pipeline library
GitLab CI/CD components (must be on your instance)
Image build
kaniko
buildkit
Runner needed
No (Tekton bundled)
Yes - install a GitLab Runner
Best when
you want a unified, K8s-native CI surface in the Portal
your org standardizes on GitLab CI and its component catalog
Use Tekton when you want every Codebase to share one Kubernetes-native CI surface inside the Portal, with pipeline history in Tekton Results. Use GitLab CI when your organization has standardized on GitLab CI/CD components and wants pipeline authoring and logs to stay in GitLab - while still getting the KubeRocketCI Codebase lifecycle, Portal, and GitOps deployment around them. And because the choice is per-Codebase, you can migrate one application at a time.
It switches a Codebase's CI engine from Tekton to GitLab CI. Instead of wiring the pipeline into Tekton, KubeRocketCI generates a .gitlab-ci.yml in the repository, and the pipeline runs as native GitLab CI jobs on a GitLab Runner. The only valid values for the field are tekton and gitlab.
Does KubeRocketCI replace GitLab CI, or run on top of it?
It runs on top of it. With GitLab CI integration, KubeRocketCI manages the Codebase lifecycle, owns the injected .gitlab-ci.yml, and provides the Portal and GitOps deployment - but the pipeline itself executes in GitLab CI, and the job logs live in GitLab. There is no Tekton in the pipeline path at all.
Do I need a GitLab Runner for the GitLab CI path?
Yes. KubeRocketCI bundles Tekton but not a GitLab Runner. With ciTool: gitlab, jobs run as GitLab CI and need a registered runner to execute. The testbed installs the official gitlab/gitlab-runner Helm chart with the Kubernetes executor, registered via a GitLab 17.x runner authentication token. Without a runner, pipelines stay pending.
Why do I have to onboard the CI/CD components into my own GitLab instance?
Because GitLab CI/CD components only resolve on the same instance ($CI_SERVER_FQDN). A reference like $CI_SERVER_FQDN/kuberocketci/ci-java17-mvn/build@0.1.1 is looked up on your GitLab, not gitlab.com. So you must mirror (onboard) the KubeRocketCI components - or your own forks - into your instance and tag them at the version your template pins, before enabling GitLab CI on any Codebase.
What is the ConfigMap for in GitLab CI integration?
The ConfigMap holds the .gitlab-ci.yml template that KubeRocketCI injects. You create one per tech stack (for example gitlab-ci-java-maven), and the Codebase points at it with an annotation. KubeRocketCI substitutes your codebase name and commits the result. The template is thin orchestration - it includes the onboarded CI/CD components and gates them with workflow rules.
How does the MR-to-build flow work without a webhook?
GitLab CI integration relies on GitLab's own workflow rules instead of a webhook. The injected .gitlab-ci.yml includes the review component gated on $CI_PIPELINE_SOURCE == "merge_request_event" and the build component gated on the protected/default branch. Opening a merge request runs review; merging pushes to the protected branch and runs build. GitLab triggers both natively, so no project webhook is created.
Can Tekton and GitLab CI codebases coexist on the same cluster?
Yes. The choice is per-Codebase, and the same gitlab GitServer is reused for both engines. On the testbed, Tekton codebases (the Go app from parts one and two) run side by side with the GitLab-CI Java app on the same KubeRocketCI platform. You can adopt GitLab CI for one application without touching the rest.
Where does the container image go in the GitLab CI build pipeline?
The buildkit-build job builds and pushes the image to GitLab's bundled Container Registry using the CI job token - no external registry or Docker Hub credentials required. On the testbed it pushes to the in-cluster GitLab registry, and the git-tag job creates a matching release tag. From there the image feeds the same CodebaseImageStream and Argo CD deployment flow as the Tekton path.
GitLab CI integration in KubeRocketCI lets a Codebase run its pipeline in GitLab CI instead of Tekton by setting spec.ciTool: gitlab. KubeRocketCI then generates a .gitlab-ci.yml that pulls in reusable GitLab CI/CD components, gated so that a merge request runs a review pipeline and a merge runs a build pipeline. There is no webhook and no Tekton in the path - the proof is zero PipelineRuns for the Codebase.
Before you enable it, set up three things: a GitLab Runner (KubeRocketCI does not bundle one), the onboarded CI/CD components in your own instance (they only resolve on the same GitLab), and the template ConfigMap KubeRocketCI injects. Complete these once, and subsequent GitLab-CI Codebases require no additional setup.
I ran the full flow on the local try-kuberocketci testbed: review pipeline green in 91s (10 jobs), build pipeline green in 100s (12 jobs), image pushed to GitLab's Container Registry, release tags created, and zero Tekton PipelineRuns - every command and result above is from that run.
The most useful next steps from here:
Spin up the testbed with Try KubeRocketCI Locally, then run make gitlab-ci and make e2e-gitlabci to reproduce this flow.
KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the testbed are all on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[Ephemeral Preview Environments on Kubernetes]]>https://docs.kuberocketci.io/blog/ephemeral-preview-environments-kubernetes-feature-branch2026-06-07T00:00:00.000ZAn ephemeral preview environment is an isolated, temporary Kubernetes deployment created from a single feature branch and torn down when the work is done. Every branch gets its own namespace, its own image, its own URL - and zero of it lingers afterward. Ephemeral environments on Kubernetes make this pattern available on your own cluster - but a hands-on, open-source, portal-native version - feature branch to isolated namespace to one-click destroy, backed by real Tekton CI and Argo CD GitOps - is conspicuously missing from the public record.
So I built one, end to end, on the same local try-kuberocketci testbed from my last post: a kind cluster running KubeRocketCI 3.14.0 with Tekton, Argo CD, and self-hosted GitLab. This post is the full walkthrough - every screenshot, every line of terminal output, captured from a live run. We will take a stable main deployment, branch off it, ship a change that is visible only in the preview environment, inject per-environment config through GitOps, prove the two environments never touch each other, and then destroy the whole thing - leaving the baseline exactly as it was.
An ephemeral preview environment is a short-lived, namespace-isolated copy of your application, provisioned automatically from a feature branch or pull request and destroyed when that branch merges or is no longer needed. Unlike a shared staging environment - where one branch at a time monopolizes the slot - a preview environment is dedicated to a single branch, carries no long-term state, and releases its resources (namespace, pods, ingress) the moment you tear it down.
The terms preview environment and ephemeral environment are used interchangeably. "Preview" emphasizes the use case (review a branch before merge); "ephemeral" emphasizes the lifecycle (temporary by design). In KubeRocketCI they are the same object: a Deployment Stage with its own isolated Kubernetes namespace, created for a branch and removed on demand.
A preview environment lifecycle has five stages:
Trigger - create or push a branch in the Portal.
Build - Tekton pipeline builds and pushes a versioned image.
Provision - namespace created, Argo CD syncs the application.
Validate - test the live URL.
Destroy - namespace and Argo CD application deleted.
Ephemeral vs. Staging: Why Shared Environments Become Bottlenecks
A shared staging environment serializes your team: only one branch can occupy it at a time, so parallel feature work cannot be validated simultaneously, and "who broke staging?" becomes a recurring standup question. Ephemeral preview environments give every branch its own namespace, so several features can be tested concurrently without interfering - and when a branch merges, its namespace is deleted, so there is no idle staging box quietly accruing cost.
That last point matters more than it looks. Industry surveys consistently find a large share of provisioned Kubernetes capacity sits idle, and the trend is getting worse - CAST AI's 2026 State of Kubernetes Optimization Report measured average CPU utilization at just 8% across roughly 23,000 production clusters (down from 10% a year earlier), and Flexera's 2026 State of the Cloud report pegs self-estimated cloud waste at ~29%, its first rise in five years. Permanent staging environments are a textbook source of that waste. An environment that deletes itself cannot become idle.
Namespace-per-Branch, vCluster, or Dedicated Cluster?
There are three common isolation models. Namespace-per-branch reuses one shared control plane, provisions in seconds, and is sufficient for most teams - it is the model used here. vCluster adds a lightweight virtual API server per environment for stronger isolation at higher overhead. A dedicated cluster per branch is maximum isolation but impractical at scale. KubeRocketCI uses namespace-per-branch, with Argo CD managing each namespace's application manifests.
Namespace isolation is not network isolation
A separate namespace scopes names, RBAC, and quotas - but by default pods in different namespaces can still reach each other over the cluster network. If your preview environments need network isolation too, add NetworkPolicy rules; the namespace boundary alone does not provide them.
Before the walkthrough, a quick map of the moving parts. KubeRocketCI wraps a few core concepts that turn a Git branch into a running environment:
Codebase / CodebaseBranch - your application (test-go-app) and a branch within it (feature-tt-123). Creating a branch in the Portal creates the CodebaseBranch resource and mirrors the branch into Git.
CodebaseImageStream (CBIS) - the record of container image tags built for a branch. The build pipeline writes new tags here; the CD side reads from it. This handoff is the key to ordering (more on that below).
Deployment (CDPipeline) and Environment (Stage) - the deployment flow and the individual environments within it. A Stage owns a namespace and an Argo CD Application.
Trigger type - Manual, Auto, or Auto-stable. We use Auto: the environment redeploys whenever a new image tag lands in the CBIS, so the preview env always tracks the branch head with no manual step. (See Deployment Strategies.)
Here is the full path from a feature-branch build to a live, isolated workload:
The crucial detail is the CBIS handoff: Tekton writes the image tag after the image is actually pushed, and the Auto trigger only fires once that tag exists. The image is therefore guaranteed to be present before Argo CD tries to deploy it - which avoids the ImagePullBackOff race that bare deploy-only tooling is prone to.
The starting point is the end state of the previous post: the sample Go/Gin app test-go-app built from main and deployed by the demo Deployment into namespace krci-demo-dev.
$ kubectl get applications -n krci
NAMESPACE NAME SYNC STATUS HEALTH STATUS
krci demo-dev-test-go-app Synced Healthy
$ kubectl -n krci-demo-dev get deploy test-go-app -ojsonpath='{.spec.template.spec.containers[0].image}'
One property of main matters for the rest of this post: the Gin app only handles /hello, so the root path returns 404. That 404 is our control - the feature environment will return 200 on the same path, and main will keep returning 404 the entire time.
Step 2: Make a Change That Only the Preview Will Show
To prove isolation, the feature branch needs to behave differently from main. I add a root / handler that returns 200 and echoes a NAME environment variable (we will set that variable per-environment in Step 7), plus the Helm extraEnv plumbing the override needs. The whole change is 30 lines across four files:
The default extraEnv: {} goes into values.yaml, and a matching unit test goes into main_test.go. Crucially, this change lives only on feature-tt-123 - main is never touched.
Feature branches are built on demand: on the Branches tab, the feature-tt-123 row has a Build button. One click starts the gitlab-go-gin-app-build-default pipeline.
The build is the familiar Tekton DAG - fetch, version, SonarQube quality gate, kaniko image build, Git tag, and the all-important update-cbis step that records the new tag:
$ kubectl -n krci get pipelinerun |grep build-test-go-app-feature-tt-123-0afaf
Reading the New Image Tag from the CodebaseImageStream
With defaultversioning, the tag is BRANCH-DATETIME. The build writes it to the CBIS:
$ kubectl -n krci get codebaseimagestream test-go-app-feature-tt-123-0afaf \
-ojsonpath='{.spec.tags[*].name}'
feature-tt-123-20260607-060023
Step 4: Create the Deployment Flow and Auto-Trigger Environment
Now the environment. I follow the recommended convention of naming the deployment flow after my initials - sk - so personal preview flows are easy to find. In Deployments → Create Deployment, I select test-go-app and set its branch to feature-tt-123.
Then I add an environment named feat with trigger type Auto:
Naming Conventions: How KRCI Builds the Namespace
The namespace is derived as krci-<deployment>-<environment>, which the form pre-fills as krci-sk-feat. The environment overview confirms the wiring - Auto trigger, in-cluster, namespace krci-sk-feat, deploy and clean pipeline templates, quality gate Manual:
Creating the environment before the next build is deliberate: with Auto, the deploy fires on a CBIS update. Arm the environment first, and the build that follows will deploy itself.
Triggering one more build updates the CBIS, the Auto trigger fires, and the cd-pipeline-operator creates the deploy run - no clicks. The deploy pipeline runs its four tasks and goes green in 47 seconds:
The whole preview environment now shows in one card - the feat Stage, its namespace, the deploy/clean pipelines, and the application Healthy + Synced on the feature tag:
krci sk-feat-test-go-app Synced Healthy # the new preview env
$ kubectl -n krci-sk-feat get deploy,pods
NAME READY UP-TO-DATE AVAILABLE
deployment.apps/test-go-app 1/1 11
NAME READY STATUS
pod/test-go-app-666f6c6994-rltdl 1/1 Running
Step 6: Proof of Isolation - Feature Gets 200, Main Gets 404
This is the part no one publishes with real output. Two namespaces, two image tags, two versions of the code, running side by side on the same cluster:
$ kubectl -n krci-demo-dev get deploy test-go-app -ojsonpath='{..image}'
...test-go-app:main-20260606-132413@sha256:a39d120b... # main
$ kubectl -n krci-sk-feat get deploy test-go-app -ojsonpath='{..image}'
And the behavioral proof - same path, two answers:
# feature env (krci-sk-feat) -> GET /
HTTP/1.1 200 OK
{"name":"","status":"ok"}
# main env (krci-demo-dev) -> GET /
HTTP/1.1 404 Not Found
# sanity: both still serve /hello
feature /hello ->200{"message":"Hello, World!"}
main /hello ->200{"message":"Hello, World!"}
The feature branch's new code is live in krci-sk-feat and answers 200. The exact same request to main in krci-demo-dev still returns 404. Nothing on main changed. That is the entire promise of a preview environment, demonstrated rather than asserted.
Step 7: Per-Environment Config via GitOps Values Override
Preview environments usually need their own configuration - a different feature flag, API endpoint, or, here, a NAME variable. The GitOps way to do this is a values file in the GitOps repository at the path <deployment>/<environment>/<application>-values.yaml, with Values override enabled for the environment.
sk/feat/test-go-app-values.yaml (in the GitOps repo)
extraEnv:
-name: NAME
value:"Hello from the sk/feat preview environment"
On the environment's Applications tab, I open Configure Deploy, flip the per-app Values override toggle, and Start Deploy:
$ kubectl -n krci get pipelinerun |grep sk-feat
deploy-sk-feat-pjsxx True Succeeded
deploy-sk-feat-ec0e True Succeeded
KubeRocketCI pulls the values file from the GitOps repo and applies it to the Argo CD Application at sync time - no image rebuild required. The pod picks up the variable, and because our root handler echoes it, the override is now visible over HTTP too:
Step 8: Destroy the Environment - Namespace Gone, Zero Residual Cost
Teardown is a first-class action, not a side effect of closing a PR. On the deployment, Actions → Delete asks me to type the name to confirm:
The cd-pipeline-operator deletes the namespace and its Argo CD Application. In about 5 seconds the preview environment is gone - and the main baseline is exactly as it was:
$ kubectl get ns krci-sk-feat
Error from server (NotFound): namespaces "krci-sk-feat" not found
Cleanup Checklist: Stage, Branch, GitOps Entry Are Independent
One thing to know: destroying the Stage does not delete the branch.
$ kubectl -n krci get codebasebranch |grep feature
test-go-app-feature-tt-123-0afaf created feature-tt-123 # still here
The Stage, the CodebaseBranch, the Git branch, and the GitOps values file are independent lifecycle objects. To fully clean up after a feature, delete the branch on the Branches tab and remove the <deployment>/<environment>/ entry from the GitOps repo. That independence is a feature: you can spin a preview environment up and down repeatedly for the same branch without ever touching Git history.
KubeRocketCI vs. Other Preview Environment Approaches
Capability
KubeRocketCI
Argo CD ApplicationSet (PR generator)
Uffizzi
Okteto / Qovery / Vercel (SaaS)
License
Apache-2.0
Apache-2.0
Proprietary (SaaS only since 2024)
Proprietary / freemium
Self-hosted
Yes
Yes
No
Control plane is SaaS
Builds the image
Yes (Tekton)
No - deploy only
Yes
Yes
GitOps CD
Yes (Argo CD)
Yes (Argo CD)
No (Compose/Helm)
Varies
Developer Portal UI
Yes
No (YAML)
Limited
Yes
Per-branch namespace
Yes
Yes
Yes
Yes
Per-env values override
Yes (GitOps)
Manual
Limited
Yes
One-click destroy
Yes
Manual
Yes
Yes
Cost per environment
Compute only
Compute only
Compute only
Per-env / per-seat fees
The Argo CD ApplicationSet PR Generator: Powerful, but Incomplete
The Argo CD ApplicationSet Pull Request generator is the closest pure-OSS pattern, and it is genuinely useful - it watches open PRs and creates an Argo CD Application for each. But it is a deployment tool only: it does not build container images. The CI pipeline must build and push the tagged image first, and if the ApplicationSet reconciles before the image exists, the pods hit ImagePullBackOff. KubeRocketCI sidesteps that ordering problem with the CodebaseImageStream handoff described earlier - Tekton writes the tag, and the Auto trigger only fires once it is there. You also get a Portal, a Build button, and a Destroy button instead of hand-written ApplicationSet YAML.
Vercel and Netlify nail preview deployments for frontend/static workloads, and Qovery/Okteto/Bunnyshell offer polished managed experiences - but they put a SaaS control plane (and a per-environment or per-seat bill) between you and your cluster. Demand is real, but the tooling still frustrates teams: in Port's 2025 State of Internal Developer Portals survey, just 6% of developers were satisfied with the self-service tooling they use to spin up environments on demand. The open question that leaves is how to get that developer experience on your own cluster, with no SaaS bill - exactly the gap an Apache-2.0 IDP like KubeRocketCI fills.
What is the difference between a preview environment and an ephemeral environment?
They are used interchangeably. "Preview environment" emphasizes the use case - reviewing a feature branch before merge - while "ephemeral environment" emphasizes the lifecycle: it is temporary and torn down when done. In KubeRocketCI both refer to the same object: a Deployment Stage with its own isolated Kubernetes namespace, created for a branch and destroyed on demand.
Do I need a SaaS subscription to get preview environments on Kubernetes?
No. KubeRocketCI is Apache-2.0 open source and runs entirely on your own cluster - kind locally, or EKS/GKE/AKS in production. There is no per-environment fee and no SaaS control plane. You bring the cluster; KRCI provides the Portal, CI (Tekton), and GitOps CD (Argo CD) to manage the full preview-environment lifecycle.
Does setting up ephemeral environments require YAML expertise?
Not for the day-to-day flow. Creating a branch, building it, and deploying to an isolated namespace are all Portal form actions. The only YAML you write is the optional per-environment values file in the GitOps repo - and only when you actually need to override a value for a specific preview. The CDPipeline and Argo CD Application resources are managed by KRCI operators.
How does per-environment configuration override work in a GitOps workflow?
You add a file at <deployment>/<environment>/<application>-values.yaml in the GitOps repository - for example sk/feat/test-go-app-values.yaml - containing Helm values that override the application's defaults. Enable the Values override toggle on the environment, then deploy. KubeRocketCI applies those values to the Argo CD Application at sync time, so the pod reflects them without rebuilding the image.
What happens to the namespace when I delete the environment in KRCI?
The cd-pipeline-operator deletes the Kubernetes namespace and the associated Argo CD Application - all pods, services, and configmaps in that namespace are removed. The CodebaseBranch and the Git branch are separate objects, so destroying the Stage does not delete or merge the branch; clean those up separately if you want to.
Does the Argo CD ApplicationSet Pull Request generator build container images?
No. It is a deployment tool that watches open pull requests and creates Argo CD Applications for them. Your CI pipeline (Tekton, GitHub Actions, GitLab CI) must build and push the image first. If the ApplicationSet fires before the image is ready, pods hit ImagePullBackOff. KubeRocketCI avoids this with the CodebaseImageStream handoff: Tekton writes the tag, and the Auto trigger deploys only after it exists.
How much does it cost to run ephemeral environments on Kubernetes with KubeRocketCI?
The software cost is zero - KubeRocketCI is Apache-2.0 with no per-environment fee. The only cost is the compute the environment consumes while it runs, and because preview environments are destroyed when the branch work is done, they do not accumulate idle compute the way permanent staging environments do. On the local kind testbed in this post, the preview environment cost nothing beyond the Docker Desktop resources already allocated.
Starting from a stable main deployment, we created a feature branch in the Portal, shipped a change visible only on that branch, built it through Tekton, and let an Auto environment deploy it to its own isolated namespace through Argo CD GitOps. We proved the isolation with real output - feature returns 200, main keeps returning 404 - injected per-environment config through a GitOps values override, and then destroyed the environment in one action, leaving the baseline untouched and zero residual cost behind.
No SaaS bill, no hand-written ApplicationSet YAML, no kubectl required for the happy path - just a feature branch, a few portal clicks, and a real isolated Kubernetes environment that cleans up after itself. The pattern - feature branch deployment on Kubernetes with isolated namespaces - is entirely self-hosted and free. I ran the entire flow on June 7, 2026 on the local try-kuberocketci testbed; every screenshot and command output above is from that run.
The most useful next steps from here:
Spin up the testbed yourself with Try KubeRocketCI Locally, then follow this flow on your own machine.
KubeRocketCI is open source under Apache License 2.0. The platform, Helm charts, and the testbed are all on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[Try KubeRocketCI Locally in 2 Commands]]>https://docs.kuberocketci.io/blog/try-kuberocketci-locally2026-06-06T00:00:00.000ZEvaluating an internal developer platform without a working instance is like buying a car from a brochure. Every KubeRocketCI install path in the official docs assumes a cluster you already have - AWS EKS, GKE, an on-prem control plane. Today I ran the try-kuberocketci testbed end-to-end on my Apple Silicon Mac using Docker Desktop and two commands: make testbed (approximately 18–20 minutes) and make e2e (approximately 12 minutes). The result is a fully wired KubeRocketCI local install - Tekton, Argo CD, SonarQube, self-hosted GitLab CE, Prometheus, Grafana, Tekton Results, and the Portal - running in a disposable kind cluster. No cloud account. No /etc/hosts edits. No clicking through UIs to trigger pipelines. This post walks through exactly what happened, command by command, screenshot by screenshot.
KubeRocketCI (KRCI) is an open-source, Kubernetes-native internal developer platform for cloud-native CI/CD, developed and maintained under Apache 2.0. It assembles Tekton, Argo CD, SonarQube, and your Git provider into a cohesive, opinionated developer experience - managing the lifecycle of your Codebases (applications, libraries, autotests) from source through review, build, and GitOps-based deployment, exposed through a single Portal UI.
KubeRocketCI markets itself as cutting time-from-project-initiation-to-active-development from days to hours. The testbed lets you verify that claim in 30 minutes on a laptop. On our own dogfooding cluster over a recent 90-day window, the platform logged 18,397 pipeline runs with a 93% build success rate - and it runs the same Tekton pipeline stack you are about to stand up locally.
This repository is not KubeRocketCI itself - it is a local installer and test harness that brings the whole platform up on your machine, ready for evaluation, development, and demo preparation.
To run a KubeRocketCI Docker Desktop setup, you need only Docker Desktop - no cloud account, no pre-existing Kubernetes cluster, and no registry account are required.
Requirement
Minimum
Recommended
Notes
Docker Desktop
4.x+
Latest
Allocate RAM in Docker Settings → Resources
RAM allocation
8 GB (core only)
12 GB+
Full bed with GitLab + SonarQube needs 12 GB+
Disk space
~20 GB free
~30 GB
Container images and volumes
OS
macOS, Linux, WSL2
macOS/Linux
Windows via WSL2 supported
Apple Silicon
Supported
-
amd64 images run via Docker Desktop Rosetta 2
make
Any version
-
Pre-installed on macOS/Linux
kind, helm, kubectl
Latest stable
-
make tools installs kind via Homebrew
Internet access
Required
-
Images pulled from public registries during install
RAM is the most common failure mode
Docker Desktop with at least 12 GB of RAM allocated is required for the full bed. Below this threshold, GitLab CE or SonarQube will OOM-kill during startup. My live run used ~11.7 GB allocated to the Docker engine - right at the edge, and it completed successfully, but the README's 12 GB+ recommendation is well-founded. Check Docker Settings → Resources → Memory before running make testbed.
On Apple Silicon (M1/M2/M3), Docker Desktop's Rosetta 2 emulation runs the amd64-only images (Portal, GitLab, sonar-operator) transparently. No configuration changes are needed in the testbed - it just works.
How to Try KubeRocketCI Locally: The Full Stack You Get
The fastest way to try KubeRocketCI locally is two commands on Docker Desktop. The try-kuberocketci testbed spins up a KubeRocketCI kind cluster with the full CI/CD platform - including Tekton, Argo CD, Prometheus/Grafana, Tekton Results, SonarQube, and a self-hosted GitLab CE - in two commands: make testbed (approximately 18–20 minutes) and make e2e (approximately 12 minutes). All versions are pinned through the edp-cluster-add-ons GitOps repository - the same source of truth used in production KubeRocketCI deployments - making the local testbed a faithful reproduction of a real cluster rather than a stripped-down demo.
Why is KubeRocketCI installed last? The edp-install chart renders provider resources - the GitServer CRD, EventListener, Ingress rules - at install time. If GitLab or Argo CD is not already running when the chart applies, the operator's SSH connection check fails and the platform never reaches a healthy state. Installing KubeRocketCI last, after every dependency is ready, is what makes the chart wire itself correctly on first reconcile.
That is the entire KubeRocketCI local install. One clone, one command.
Check Docker Desktop RAM allocation first
Before running make testbed, open Docker Desktop → Settings → Resources and verify the memory slider is set to 12 GB or higher. This is the single most common reason the install fails, and it takes 30 seconds to check.
make testbed builds in strict dependency order, installing KubeRocketCI last. The sequence:
GitLab CE is the slowest component to initialize - plan for it to take several minutes of the total. Once make testbed completes, run make status to see the live cluster state and print all service URLs with credentials:
All six services are immediately reachable in a browser. No /etc/hosts edits - I will explain why in the next section.
nip.io Wildcard DNS: No /etc/hosts Editing Required
The try-kuberocketci testbed uses nip.io wildcard DNS so every platform service gets a stable, browser-accessible URL - no /etc/hosts edits, no port-forwards, no local DNS configuration required. Any subdomain of <IP>.nip.io resolves to <IP> via a public DNS server; the testbed generates all ingress hostnames in the *.127.0.0.1.nip.io pattern, so every service is immediately reachable from your browser with zero local DNS configuration.
Every competing local Kubernetes tutorial either requires manual /etc/hosts edits to map ingress hostnames to 127.0.0.1, or skips DNS entirely and uses port-forwards. For a KubeRocketCI Docker Desktop setup that serves six platform UIs simultaneously, manual host-file management would be a non-starter - nip.io eliminates the entire problem class.
Service
URL
KubeRocketCI Portal
https://portal.127.0.0.1.nip.io
Argo CD
http://argocd.127.0.0.1.nip.io
GitLab CE
https://gitlab.127.0.0.1.nip.io
SonarQube
http://sonar.127.0.0.1.nip.io
Grafana
http://grafana.127.0.0.1.nip.io
Tekton Results API
http://tekton-results.127.0.0.1.nip.io
The DNS trick works for your browser on the host. Inside the cluster, pods resolve gitlab.127.0.0.1.nip.io to the in-cluster GitLab Service via CoreDNS rewrites, and the kind node's containerd uses a registry mirror config to pull images from GitLab's bundled registry. This split-horizon DNS design is one of the more interesting pieces of the testbed architecture - the full walkthrough is in docs/architecture.md.
How Does make e2e Run the Full CI/CD Pipeline Automatically?
make e2e automates the complete CI/CD cycle - opening a GitLab Merge Request, triggering a Tekton review pipeline, merging the MR, building and pushing a container image with kaniko to GitLab's bundled registry, and asserting the Go/Gin sample application is live via Argo CD sync - in approximately 12 minutes with zero UI clicks.
Here is the verbatim output from my run today:
==> Applying the sample Codebase (test-go-app)
==> Waiting for the Codebase to be provisioned in GitLab (project + template push)
==> Waiting for the codebase-operator to create the project webhook
==> Waiting for the CodebaseImageStream test-go-app-main to exist
==> Creating the demo CDPipeline + dev Stage (triggerType: Auto)
==> Opening a merge request (branch ci-e2e-...) to trigger the review pipeline
merge request !1 opened
==> Waiting for the webhook to create the review PipelineRun(s)
A common blocker in local CI/CD setups is the container registry: DockerHub rate limits, Harbor setup complexity, credentials management. GitLab CE ships with a built-in Container Registry served on port 5050. The testbed wires kaniko to push images directly to it - eliminating every external registry dependency. Kaniko trusts the self-signed cert via a chart flag (edp-tekton.kaniko.customCert: true), and a group deploy token backs both push and pull. The registry URL is gitlab.127.0.0.1.nip.io:5050/krci/<codebase>, which maps cleanly to each codebase's GitLab project registry. No DockerHub account, no Harbor, no separate registry pod.
A note on the review/build trigger split: the EventListener has two separate Trigger CRs - gitlab-review fires on MR open/reopen/update; gitlab-build fires on MR action=merge. This means merging the MR - not a push to the branch - kicks the build pipeline. It is an intentional design choice that avoids spurious build runs on every force-push. Also worth noting: GitLab can occasionally deliver the MR webhook twice, creating a duplicate review run that fails fast with a harmless HTTP 400 when it tries to post the same commit status context. The e2e script asserts at least one run is fully green - the duplicate does not affect the result.
After make e2e passes, every component has real data in it. Here is what I found navigating each UI - a practical KubeRocketCI getting started tour of each surface.
The Portal signs in with a Kubernetes bearer token rather than a username/password - run make token to mint a 24-hour token, then paste it into the Portal's "More options → Use Service Account Token" field. (OIDC is intentionally not wired up in the testbed - there is no identity issuer inside a kind cluster - so the service-account token is the local sign-in path. make status flags the in-cluster Portal's token login as still being wired up; I captured these views through the Portal using that same make token service-account flow.)
The home dashboard shows the pipeline metrics from the e2e run: 3 total runs, 100% success rate, 0 failed, average duration 2m 18s. This is the same observability surface that surfaces 18,397 runs on the production cluster - just seeded with a single e2e cycle.
The Projects page shows both registered codebases: krci-gitops (the GitOps/Helm system codebase that KubeRocketCI requires) and test-go-app (the Go/Gin application created by the e2e script). Both show status Created.
The PipelineRuns list shows all three runs - deploy, build, review - all green, all for test-go-app on PR #1.
The build pipeline DAG is the most visually satisfying part. Every node in the React Flow graph is green: report-pipeline-start → fetch-repository → get-version → sonar → build → container-build → git-tag → update-cbis → save-cache. The sonar step includes a quality gate wait - the build would have failed here if SonarQube reported issues.
The build run details confirm the pipeline name (gitlab-go-gin-app-build-default) and duration (1m 34s). This is the Helm-templated pipeline library at work - the name encodes the Git provider, language, framework, and pipeline type, as described in the Kubernetes-native CI/CD with Tekton post.
The Deployments page shows the demo CDPipeline created by the e2e script, with test-go-app as its application.
The Observability → Pipeline Metrics view breaks down the three runs: Build 1/100%, Review 1/100%, Deploy 1/100%, average 2m 18s across all types.
Argo CD shows demo-dev-test-go-app as Synced + Healthy, target revision main-20260606-132413, path deploy-templates, namespace krci-demo-dev. This is the GitOps delivery leg: after kaniko pushed the image and the CodebaseImageStream updated, the cd-pipeline-operator created a CDStageDeploy resource, which triggered a deploy PipelineRun, which applied the Helm chart through Argo CD. The full GitOps chain, locally.
The self-hosted GitLab project shows both pipelines as Passed - Tekton's gitlab-set-status task wrote commit statuses back to GitLab after each pipeline run completed. Pipeline #1 is the review run (MR open), pipeline #2 is the build run (MR merge). The Go/Gin app is live at test-go-app in namespace krci-demo-dev - its / route returns 404 by design (no root handler in the Gin app), but the server is live and the deployment is 1/1 Available.
The deploy PipelineRun details confirm the full GitOps delivery chain completed in 57 seconds - all four tasks green (pre-deploy → deploy-app → post-deploy → promote-images), the Argo CD sync applied, and the workload live in krci-demo-dev.
To explore Tekton pipelines in KubeRocketCI further, or start customizing pipelines for your own stack, the krci CLI daily use post covers how to drive the platform from the terminal and hand off to AI agents. For the foundational basic concepts, the docs cover Codebases, CDPipelines, CodebaseImageStreams, and Stages in depth.
make down deletes the kind cluster and all associated Kubernetes resources, leaving Docker Desktop in a clean state. Nothing persists outside the kind cluster - no volumes, no registry state, no credentials. The testbed is fully disposable and can be re-run from scratch in under 20 minutes.
This disposability is a first-class design goal, not an afterthought. For full from-zero validation: make down && make testbed && make e2e. The try-kuberocketci repository runs this sequence as part of its own CI regression.
Safe on shared machines
The testbed uses fixed local-only credentials, self-signed TLS, and broad RBAC - all intentionally, for a throwaway kind cluster bound to localhost. It is explicitly not safe to expose or use on any shared, internet-reachable, or production cluster. See SECURITY.md for the details.
The fastest way to try KubeRocketCI locally is also the safest way to evaluate it - no cloud spend, no shared infrastructure, no consequences if something breaks. Here are the four use cases where the testbed pays off immediately.
Evaluating KubeRocketCI before adoption. Before committing months of engineering time to an IDP, run make testbed and spend 30 minutes with the real platform - not a demo video or a vendor-configured sandbox. Compare hands-on against Backstage, kubriX, or other IDPs with a working instance under your own fingers. The official platform installation guide covers the cloud path when you are ready to move from local to a real cluster.
Learning platform engineering. The testbed is a coherent, fully-wired example of Tekton, Argo CD, GitOps, SonarQube, and Tekton Results working together. Every piece is independently inspectable with kubectl, helm, and the respective UIs. Productive learning by reading a real system, not a toy demo.
Contributing to KubeRocketCI. The testbed is designed as a contributor sandbox. Run make down && make testbed to reproduce an issue from scratch without a cloud account. Each component can be rebuilt independently (make sonar, make argocd, etc.), so debugging one piece does not require tearing down the rest.
Demo preparation. Spin up a fresh instance in under 20 minutes before a presentation; tear down afterward. The make e2e output gives you a verified, live end-state to present - not a screen recording.
The table below compares try-kuberocketci against the three closest local IDP alternatives across the features that matter most for local evaluation - install time, automated testing, component depth, and Apple Silicon support.
Feature
try-kuberocketci
CNOE idpBuilder
kubriX kind guide
Manual Tekton+ArgoCD
Commands to full stack
2 (make testbed, make e2e)
1 (idpbuilder create)
6 manual steps
15+ manual steps
Install time
~18–20 min
~5–10 min
~30 min
60+ min
Automated E2E test
Yes (zero UI clicks)
No
No
No
Tekton CI pipelines
Yes
No
No
Manual
Self-hosted GitLab CE
Yes
No (Gitea)
No
No
SonarQube
Yes
No
No
Optional/manual
Tekton Results
Yes
No
No
No
Prometheus + Grafana
Yes
No
No
No
Apple Silicon support
Explicit (Rosetta)
Yes
Not documented
Not documented
nip.io DNS (no /etc/hosts)
Yes
Partial
No
No
GitOps-pinned versions
Yes (edp-cluster-add-ons)
Partial
No
No
Teardown command
make down
Manual
Manual
Manual
CNOE idpBuilder is the closest open-source alternative - it installs Argo CD, Gitea, and ingress-nginx in approximately 5–10 minutes with a single command. It is a solid choice for a minimal GitOps sandbox. What it does not have: Tekton CI pipelines, SonarQube quality gates, Tekton Results for pipeline history, Prometheus/Grafana observability, or an automated end-to-end pipeline proof. The try-kuberocketci testbed installs a significantly larger stack and uniquely validates the full MR-to-deployed-workload cycle without any UI interaction. Also note: GitHub integration is fully supported in production KubeRocketCI (integrate GitHub with KubeRocketCI) - the testbed uses GitLab specifically because it can be self-hosted in the kind cluster without a cloud account.
KubeRocketCI (KRCI) is an open-source internal developer platform for cloud-native CI/CD on Kubernetes, developed by EPAM and released under Apache 2.0, that integrates Tekton, Argo CD, and GitLab (or GitHub, Bitbucket) into a unified developer workflow platform. It manages Codebases from source through review, build, and GitOps-based deployment, surfacing everything through a single Portal UI and CLI.
How do I try KubeRocketCI without a cloud account?
Clone the try-kuberocketci repository and try KubeRocketCI locally by running make testbed - it installs the full KubeRocketCI platform on a local kind cluster on Docker Desktop in approximately 18–20 minutes, with no cloud account or pre-existing Kubernetes cluster required. Run make e2e afterward to validate the full CI/CD pipeline automatically.
What are the hardware requirements to run KubeRocketCI locally?
Docker Desktop with at least 12 GB of RAM allocated to the Docker engine is required for the full bed (8 GB suffices if you skip GitLab and SonarQube). Approximately 20 GB of free disk space is also needed for container images and volumes. No other cloud or network infrastructure is required.
Does KubeRocketCI work on Apple Silicon (M1/M2/M3)?
Yes. Apple Silicon (M1/M2/M3) is fully supported: Docker Desktop's Rosetta 2 emulation runs the amd64 container images (Portal, GitLab CE, sonar-operator) transparently on ARM64 hardware, with no configuration changes required in the testbed. I ran the entire flow on an Apple Silicon Mac for this post - the testbed explicitly documents and tests ARM64 support.
How long does it take to install KubeRocketCI locally?
make testbed installs the full platform in approximately 18–20 minutes; make e2e runs a complete automated pipeline proof in approximately 12 minutes. Total time from zero to a verified live workload is under 35 minutes. GitLab CE is typically the slowest component to initialize - the rest of the stack comes up faster.
In dependency order: a single-node kind cluster, ingress-nginx, cert-manager, Tekton Pipelines/Triggers, Argo CD, Prometheus and Grafana, Tekton Results, SonarQube Community, self-hosted GitLab CE (with bundled container registry), and KubeRocketCI edp-install 3.14.0 - 10 components in a defined sequence, with KubeRocketCI installed last so the chart can wire itself to every running dependency on first reconcile.
How do I run an end-to-end CI/CD pipeline locally with KubeRocketCI?
Run make e2e after make testbed completes. It automates the full cycle: applies a sample Go/Gin Codebase, waits for the codebase-operator to provision the GitLab project and webhook, opens a Merge Request to trigger the Tekton review pipeline, merges the MR to trigger the build pipeline (kaniko pushes the image to GitLab's bundled registry), waits for Argo CD to sync the deployment, and asserts the workload is 1/1 Available - all in approximately 12 minutes with zero UI clicks.
How do I tear down the local KubeRocketCI environment?
Run make down to delete the kind cluster and all associated Kubernetes resources, leaving Docker Desktop in a clean state. Nothing persists outside the kind cluster. The testbed can be re-run from scratch in under 20 minutes and is designed to be fully disposable - suitable for evaluation, demo preparation, contributor testing, and automated platform CI regression.
The testbed uses nip.io wildcard DNS: any subdomain of <IP>.nip.io resolves to <IP> via public DNS, so all platform services get stable browser-accessible URLs (*.127.0.0.1.nip.io → 127.0.0.1) without any local DNS configuration. Inside the cluster, CoreDNS rewrites handle pod-to-service resolution, and the kind node's containerd uses a registry mirror config for image pulls - no host-level DNS changes at any point.
What is the difference between try-kuberocketci and CNOE idpBuilder?
CNOE idpBuilder spins up a minimal IDP stack (Argo CD + Gitea + ingress-nginx) in approximately 5–10 minutes but ships no Tekton CI pipelines, no SonarQube, no GitLab CE, no Tekton Results, and no automated e2e validation. The try-kuberocketci testbed installs a full production-parity platform with all of these in ~18–20 minutes and uniquely includes a fully automated end-to-end pipeline test that validates a real MR-to-deployed-workload cycle without any UI interaction.
The fastest way to try KubeRocketCI locally is two commands on Docker Desktop. make testbed brings up a complete KubeRocketCI platform - 10 components including Tekton, Argo CD, SonarQube, GitLab CE, Prometheus/Grafana, and Tekton Results - in approximately 18–20 minutes. make e2e validates the full MR-to-deployed-workload pipeline automatically in approximately 12 minutes. Total: under 35 minutes from zero to a verified live workload, on Docker Desktop, with no cloud account. make down leaves nothing behind.
I ran this end-to-end on June 6, 2026 on an Apple Silicon Mac with Docker Desktop. The e2e test passed with 3/3 green runs (review + build + deploy), 100% success rate, average duration 2m 18s, and the Go/Gin sample app deployed at tag main-20260606-132413.
The most useful next steps from here:
Explore the KubeRocketCI quick start overview to continue KubeRocketCI getting started and move from a local kind cluster to a real cloud deployment.
KubeRocketCI is open-source under Apache License 2.0. The testbed, platform source, and Helm charts are all on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[krci CLI: From Terminal to AI Agents]]>https://docs.kuberocketci.io/blog/krci-cli-daily-use2026-05-05T00:00:00.000ZMost of my day-to-day platform work happens in a conversation. I sit in a Claude Code session - or any AI assistant with shell access - and ask plain-language questions about the state of our delivery cluster: what's failing, what's drifting, what's vulnerable, what's stale. The agent answers by calling the krci CLI, the predictable, JSON-emitting client over the KubeRocketCI Portal's tRPC API. I read the answer, decide what to do, and when a question turns into a routine I drop the underlying invocation into a script and let it run on cron or /loop. This post is a snapshot of that workflow with one running example - operator vulnerability status - and a tour of the other questions the same pattern answers.
KubeRocketCI is built so the platform's primitives are reachable from every runtime that matters in modern engineering: the portal, the CLI, Tekton pipelines, GitLab CI, schedulers, and AI agents. A workflow I write once runs everywhere it needs to:
Pipelines. A krci invocation that answers a question in my terminal runs unchanged inside a Tekton task or a GitLab CI job. Quality gates and release governance read live platform state without bespoke API clients.
Schedulers. A short bash wrapper in cron, or a Kubernetes CronJob produces a daily report on a hands-off cadence - same command, same output, same trust boundary.
Chat and webhooks. JSON output drops straight into Slack, Teams, or PagerDuty without a translation layer.
AI agents in the SDLC. AI coding agents - Claude Code, Cursor, the Anthropic and OpenAI SDKs - read terminal output natively. Giving an agent shell access to krci turns the platform into a tool surface it can call: list codebases, inspect SBOMs, compare branches, summarize changes since the previous run. No MCP server to write, no proprietary protocol to learn, no stale API client to maintain.
The portal and the CLI are peers: the portal is the right interface for browsing, comparing, and approving; the CLI is the right interface for composing, scheduling, and handing off to agents. Both are first-class products and ship in lockstep.
A short tour of the top-level surface, because this is what the agent has access to:
$ krci --help
Available Commands:
auth Authentication commands
deployment Manage deployments (CDPipelines)
env Inspect KRCI environments (Stages)
pipelinerun Manage pipeline runs
project Manage projects (Codebases)
sca Inspect Dependency-Track projects, components, and vulnerability findings
sonar Inspect SonarQube projects, quality gates, and issues
version Print krci CLI version
When you run any of these commands - say, krci deployment list to inspect CD pipelines across your environments-the output follows the same disciplined structure. Columns are consistent, status values are predictable, and the table is human-readable and machine-parseable without any translation. Here's what that looks like in practice:
This table shows exactly what's deployed where. The VERSION column tracks the build artifact (0.5.0-SNAPSHOT.4), the ENV column shows which environment it's in, and the STATUS column tells you health at a glance -healthy or missing. The SYNC column flags when an environment has drifted (OutOfSync) from what GitOps expects. In seconds, you can answer: "What's actually running in qa? Is it in sync? What version?" The agent can answer it too, and answer the next question - "Show me all deployments older than 30 days" - without any special prompting.
Every group follows the same shape - list, get, sometimes a third verb - and the flags repeat (--branch, --severity, -o json|table). Predictability is the point: an agent can reason about the next command from the previous one, and so can I. Authentication is OIDC against the same Keycloak the portal uses; the agent inherits the existing session, so there is no long-lived API key embedded in any prompt.
Asking the Agent: "What's the Vulnerability Status of Our Operators?"
Here's a recent question, verbatim, from a Claude Code session:
"Using krci, give me the current vulnerability status of every operator registered on the platform. Main branch only, critical and high counts, top affected components per operator."
One row is doing all the work: codebase-operator master carries two high-severity findings, and both trace back to a single github.com/tektoncd/pipeline cluster. Everything else is clean. That's the kind of answer I can act on without scrolling - one operator, one component, one upgrade to plan.
The command I run by hand to verify the answer is one line:
That's the whole loop. I described what I wanted in the language of the work, the agent shaped a krci invocation, and I can re-run any step independently to validate the result. No prior knowledge of jq flags, subcommand layout, or the platform's internal data model was required on my side.
One component, both highs, two natural-language turns, two commands. I never had to remember the flag shape; the agent bridged intent to invocation and the CLI's predictable output bridged invocation to answer.
Vulnerability status is one of many. The agent + CLI handle the same shape of question across the platform - I describe it in plain language, the agent runs krci, I verify the output against the platform:
"Which pipeline runs failed in the last 24 hours, grouped by codebase?" → krci pipelinerun list -o json filtered on .status.conditions.
"Which environments are running an out-of-date image of code-assistant?" → krci env list -o json joined against the latest tag.
"What's the SonarQube quality gate state of every Java codebase?" → krci sonar list -o json filtered by language.
"Which projects on the platform haven't produced an SBOM yet?" → diff krci project list against krci sca list.
"Show me every CD pipeline whose latest deploy is older than 30 days." → krci deployment list -o json filtered on .status.lastDeploy.
Same pattern every time: ask in language, get an answer the CLI can verify, move on. The agent does the orchestration; the CLI does the work.
The moment a question becomes recurring, it stops belonging in chat. The same krci invocation the agent assembled in the conversation becomes the body of a script:
#!/usr/bin/env bash
# Daily operator vulnerability digest - main branches only.
Cron or Kubernetes CronJob for a daily markdown digest in ~/reports/operators-$(date +%F).md.
Slack/Teams webhook for "first new critical on a release branch" alerts. Pipe the JSON through jq, post on threshold.
Claude Code /loop for a six-hour heartbeat that reports only what changed:
/loop 6h Run cli/scripts/operators-digest.sh and tell me ONLY what
changed since the previous run. Reply in under 60 words.
Same script, same JSON, three runtimes. The CLI is unchanged; only the wrapper changes.
Why the krci CLI Works Well as an Agent Tool Surface
Three properties matter, and each one comes from deliberate design rather than accident:
Stable, structured output.krci ... -o json is shaped consistently across releases. The agent doesn't need a parser; it navigates JSON.
Identity is solved upstream. OIDC sessions are inherited from the developer's existing login. No API keys leak into prompts, transcripts, or scripts.
Predictable verb shape.list, get, sometimes a third verb, with the same flags everywhere. The agent generalizes from one command to the next, and so do humans.
The result is that the conversation with the agent stays in the language of the work - "vulnerability status of operators", "failed runs in the last 24 hours", "environments running an out-of-date image" - and the CLI handles the translation. Both pieces are independently inspectable: I can replay any command myself, the agent can replay it tomorrow, and the script that captures a routine is short enough to review in one sitting.
Not every codebase produces an SBOM yet. Some appear in project list but are absent from sca list because no CycloneDX upload has happened on their build pipelines. The CLI surfaces the gap rather than hiding it, which is itself useful, but means agent answers should always cross-check both lists.
Cross-project aggregation sometimes still uses a shell loop. There's no krci sca list --filter='critical>0' flag yet - native flags are on the roadmap, the agent compensates in the meantime.
BOM age (Last BOM) shows up in krci sca get but not on sca list. If staleness matters to your routine, pin it explicitly in your script.
I include this list because tools earn trust by being honest about what they don't do yet.
Open your AI assistant of choice, give it shell access, and ask your own version of the question above. Anything along the lines of "using krci, show me the vulnerability status of every operator on the platform, main branch only" will work on day one.
When the answer becomes a routine, drop the underlying command into a script and run it on cron, a CronJob, or /loop.
That's the entire loop: ask in language, get an answer the CLI can verify, schedule it when it matters.
The krci CLI is open-source under Apache License 2.0. Source, issues, and release binaries live on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[Kubernetes-Native CI/CD with Tekton]]>https://docs.kuberocketci.io/blog/kubernetes-native-cicd-tekton-kuberocketci2026-05-04T00:00:00.000ZBuilding CI/CD on Kubernetes used to mean running Jenkins or GitLab CI in a pod and calling it done. Tekton changed that by making pipelines first-class Kubernetes objects - Tasks and Pipelines are CRDs, PipelineRuns are namespaced resources, and every step log is a container log. KubeRocketCI goes a step further: it ships a complete, production-grade CI/CD platform on top of Tekton so your team gets sensible defaults, a portal UI, GitOps-managed pipeline definitions, and opinionated quality gates - without the months of plumbing work that comes with assembling those pieces from scratch. I've seen teams go from a bare cluster to a working build-deploy loop in under a day using this stack.
Here's how the layering works in practice - from Tekton's native primitives to the Helm-templated pipeline library, the webhook-to-PipelineRun chain, each pipeline type, and the portal UI that surfaces it all.
Traditional CI servers are external systems that talk to Kubernetes. They authenticate, schedule jobs, wait for responses, and manage their own scaling independently of the cluster. Tekton turns that model inside out - the cluster is the CI engine. Every pipeline run is a pod, every step is a container, and the Kubernetes control plane handles scheduling, scaling, and secrets injection natively.
The practical consequences:
No CI server to maintain. No Jenkins controller HA setup, no separate runner fleet. Tekton scales horizontally with your cluster.
Uniform RBAC. Pipeline permissions use Kubernetes ServiceAccounts - no separate credential management UI.
Pipeline as code. Pipeline definitions are YAML manifests, version-controlled alongside application code, and applied to any conformant Kubernetes cluster identically.
Native secret injection. Tekton Tasks consume Kubernetes Secrets and ConfigMaps directly - no plugin abstraction required.
Shift-left by default. Code quality, security scanning, and test gates run inside the cluster as ordinary containers, co-located with the workloads they protect.
KubeRocketCI is an open-source, cloud-agnostic SaaS/PaaS platform built on Tekton - a CNCF Graduated project - developed and maintained by EPAM Systems. Where Tekton gives you primitives, KubeRocketCI gives you a working system.
The platform deploys and manages the full Tekton component set:
Component
Role
Tekton Pipelines
Executes all CI and CD pipeline runs
Tekton Triggers
Listens for Git webhook events and fires the matching pipeline
Tekton Interceptors
Enriches and filters GitHub, GitLab, and Bitbucket payloads before routing
Tekton Chains
Signs pipeline artifacts for supply-chain provenance
Tekton Results
Stores run history for audit trails and portal display
On top of Tekton, KubeRocketCI adds:
Pipeline library - Helm-packaged Tasks and Pipelines for Java, Go, Node.js, Python, .NET, Helm, Terraform, and more.
Portal UI - full pipeline management with DAG visualization, live log streaming, and inline manual approval gates.
Argo CD integration - pipeline definitions are stored in Git and synchronized by Argo CD. Every change to a pipeline is a reviewed, audited commit.
Pipelines in KubeRocketCI are not hand-written one-offs - they are generated by Helm from a structured library in edp-tekton. The naming convention follows:
For example, a Java 17 Maven build pipeline triggered from GitHub with default versioning renders as:
github-maven-java17-app-build-default
The same pipeline for GitLab becomes gitlab-maven-java17-app-build-default. This pattern means every combination of git provider, language, framework, and versioning strategy gets a consistently named, independently configurable pipeline - without duplicating task logic. Shared task sequences live in common Helm templates (e.g., _common_java_maven.yaml) and are included by reference.
The sonar step blocks with sonar.qualitygate.wait=true - the build fails fast at the quality gate rather than discovering issues downstream. In my experience, this single flag is the difference between catching technical debt in minutes versus after a release candidate is already tagged. Code that doesn't pass SonarQube never produces an artifact.
Each task is a reusable Tekton Task CRD (maven, sonarqube-maven) - the pipeline only sets parameters and runAfter ordering. Swap the maven task image to change the JDK version; the pipeline logic stays the same.
When a pull request is merged on GitHub, the event travels through four Tekton resources before a PipelineRun starts:
The Custom Interceptor is the key piece. It enriches the raw GitHub payload with platform-specific extensions that the webhook payload alone doesn't contain:
extensions.codebase - the KubeRocketCI codebase name for this repository
extensions.codebasebranch - the branch object with its pipeline configuration
extensions.pipelines.build - the exact pipeline name to run (e.g., github-maven-java17-app-build-default)
The TriggerBinding then extracts these into named parameters: gitrevision, targetBranch, gitsha, codebase, codebasebranch, pipelineName, commitMessagePattern, and Jira integration fields. The TriggerTemplate stamps these into a PipelineRun spec and creates it in the cluster.
This architecture means the decision of which pipeline runs for which codebase is stored in the platform's Kubernetes objects - not hardcoded in webhook configuration.
Triggered by a pull request. Runs static analysis, linting, unit tests, and code-quality gates before a merge is allowed. Typically completes in under five minutes.
Trigger methods: open a PR against a configured branch, comment /recheck or /ok-to-test, or use the Run Again button in the portal.
Triggered on merge to the target branch. Executes the full task sequence (see above), builds and pushes a container image, and produces a versioned artifact. Versioning follows either BRANCH-[DATETIME] (default) or SemVer (MAJOR.MINOR.PATCH-BUILD_ID) depending on project configuration.
Takes a versioned artifact and applies it to a target environment. Trigger modes: manual (via portal), Auto (deploy automatically on new artifact), or Auto-stable (promote only the rebuilt component, keep others at stable versions).
Tears down an environment on demand. Paired with Auto deploys and dynamic environments, this enables full ephemeral workflows - spin up for a PR, validate, clean up on merge.
A standalone pipeline decoupled from build, purpose-built for vulnerability scanning. Scan results surface in the portal's Results tab and link through to DefectDojo. Decoupling security means scans can run at any cadence without affecting build cycle time.
Executes automated test suites against already-deployed environments, independent of the deploy cycle. Teams validate application behavior on demand without triggering a full redeploy.
Orchestrates approval and publishing for new releases. KubeRocketCI does not ship a pre-built release pipeline - release governance is organization-specific. The full Tekton API surface is available to build custom approval steps, external system integrations, and auditable release flows.
The pre-built library covers most tech stacks, but production workloads always have exceptions. KubeRocketCI supports two customization patterns without forking the platform.
Define a custom pipeline once with a naming pattern, and every project matching that pattern picks it up automatically. The Tekton custom pipelines use case walks through the full authoring flow.
When different branches need different logic - main with full integration tests, release with image signing - pipelines are defined explicitly per branch. See the custom pipelines flow guide.
Both approaches follow the same Git-first workflow: write Tekton YAML, apply to the cluster to verify, commit, and let Argo CD synchronize the authoritative state.
The KubeRocketCI portal ships a dedicated tekton module with full pipeline management across four page areas: pipeline list, pipeline details, PipelineRun list, and PipelineRun details.
Pipeline list with type filter. The pipeline list filters by app.edp.epam.com/pipelinetype label - select build, review, deploy, or any of the seven types to narrow the view. Each pipeline shows its type, last run status, and a Run with params button.
Run with params. Clicking Run with params fetches the TriggerTemplate linked to the pipeline (via the app.edp.epam.com/triggertemplate label), generates a PipelineRun draft pre-filled with correct parameters, opens a YAML editor for review, and creates the PipelineRun resource via the Kubernetes API on save. No kubectl required.
DAG visualization. The pipeline details page renders the task graph using React Flow - nodes for each task, directed edges from runAfter dependencies, isolated tasks shown separately. Colors map to live status: blue (running), green (succeeded), red (failed), amber (pending). Finally-tasks are distinguished visually from the main execution chain.
Live logs and history. The PipelineRun details page streams live container logs while a run is active. After completion, logs are served from Tekton Results - the same view, no separate log aggregation setup needed.
Inline manual approval. When a pipeline step is an ApprovalTask (a platform-native CRD), the portal renders Approve / Reject buttons directly in the task view - with an optional comment field. The approval decision (spec.action, spec.approve.approvedBy, spec.approve.comment) is written back to the ApprovalTask resource. This is how release gates and environment promotion approvals are implemented as Kubernetes-native objects.
Create Application registers your codebase. The platform provisions a review and build pipeline for the default branch automatically - the Helm-templated library selects the correct pipeline name based on your tech stack.
Once the platform is running, every onboarded application has:
A review pipeline on every pull request - compile, test, SonarQube quality gate.
A build pipeline on every merge - versioned artifact, container image pushed, Tekton Chains provenance attestation.
A CD pipeline connected to Argo CD - promoting artifacts through configured stages.
Pipeline definitions in Git - every change to a pipeline goes through code review, synchronized by Argo CD.
Live + archived logs - streaming during a run, served from Tekton Results after completion.
Manual approval gates - inline approve/reject in the portal for any pipeline step that requires human sign-off.
This is what our own dogfooding instance looks like - we run KubeRocketCI on KubeRocketCI itself. Over 90 days, the krci namespace logged 18,397 pipeline runs (~200/day across build, review, and deploy types), with build pipelines hitting a 93% success rate at an average duration of 14 minutes 38 seconds. No external CI server involved.
The entire path - review pipeline on PR open, build pipeline on merge, artifact versioning, image push, and Argo CD sync - runs without the developer writing a single line of Jenkins Groovy or pipeline YAML. The only required input is declaring the tech stack at onboarding time.
No manual wiring required. The opinionated defaults deliver a production-grade setup from day one, and the full Tekton API surface is available when you need to diverge.
KubeRocketCI is open-source under Apache License 2.0. Platform source, Helm charts, and the full Tekton pipeline library are on GitHub.
]]>Sergiy Kulanovhttps://github.com/sergk<![CDATA[Integrating OIDC Authentication with Microsoft Entra in AWS EKS]]>https://docs.kuberocketci.io/blog/integrating-oidc-authentication-microsoft-entra-aws-eks2024-12-03T00:00:00.000ZIn modern cloud environments, secure and efficient access management is essential, especially for platforms like Amazon EKS. This blog will guide you through integrating OpenID Connect (OIDC) authentication using Microsoft Entra, making it easier to manage access to your EKS clusters and KubeRocketCI Portal. By implementing this approach, you can simplify user authentication while ensuring strong security controls. Whether you're improving compliance or streamlining access for your team, this integration is a practical solution to enhance your cloud-native workflows.
info
Updated: July 2026 — Links verified and integration steps re-checked against the current Microsoft Entra admin center.
In the context of enhancing digital security and user experience, we prioritize the integration of three key elements: Single Sign-On (SSO), OpenID Connect (OIDC), and Microsoft Entra. Here’s how they connect:
Single Sign-On (SSO) serves as the foundation, enabling users to access multiple applications with one set of login credentials, significantly simplifying the authentication process.
OpenID Connect (OIDC) builds on the SSO framework by providing an authentication layer, which uses straightforward identity verification to ensure secure and seamless access across services.
Microsoft Entra (formerly known as Azure Active Directory) is Microsoft's comprehensive identity and access management solution. It supports the implementation of both Single Sign-On (SSO) and OpenID Connect (OIDC), enabling organizations to securely manage user identities and enforce access controls. With its reliable set of tools, Microsoft Entra simplifies authentication, enhances security, and ensures seamless access to applications and services, making it an essential platform for modern identity management.
Together, these technologies streamline the login process, reinforce security, and enhance the user experience by allowing secure, seamless navigation across our digital ecosystem.
Microsoft Entra, formerly known as Azure Active Directory (Azure AD), is a modern identity and access management solution designed for secure access to applications and services. It provides features like Single Sign-On (SSO), identity federation, and seamless integration with on-premises directories such as LDAP and Active Directory. Microsoft Entra supports industry-standard protocols, including OpenID Connect (OIDC), OAuth 2.0, and Security Assertion Markup Language (SAML) 2.0, making it a versatile solution for managing user identities. By leveraging Microsoft Entra, organizations can enhance security, simplify user access, and avoid the complexities of building identity management features from scratch. For more details, see the Microsoft Entra official documentation.
In the left sidebar menu, select Overview section and then navigate to Manage tenants tab.
Click on Create button to create a new tenant.
Select the configuration type Workforce.
Fill in the fields for Tenant Name, Domain Name, and Location.
note
The Tenant Name and Domain Namekuberocketci are used as a demonstration examples. In your case, it is recommended to choose names that align with your organization's specific needs and naming conventions.
The new tenant will be created, and you can start configuring it for OIDC integration. Ensure you have switched to the new tenant.
Creating and Configuring OIDC Application in Microsoft Entra
After creating the tenant, you need to set up an OIDC Application in Microsoft Entra. Here's how you can do it:
In the Microsoft Entra Admin Center, in the left sidebar menu, select Applications and then click on App registrations.
Click on the New registration button to create a new application.
Fill in the details for the application, such as Name, Supported account types, and Redirect URI (http://localhost:8000/).
note
The Namekuberocketci is used as a demonstration example. In your case, it is recommended to choose a name that aligns with your application's specific needs and naming conventions (e.g. your AWS EKS cluster name).
In the created application, navigate to the Authentication section from the left sidebar menu. In the Implicit grant and hybrid flows section, select ID tokens for the token type. In the Allow public client flows section, set the value to No.
Navigate to the Certificates & secrets section from the left sidebar menu. In the Client secrets tab, click on the New client secret button to create a new secret.
Copy the generated client secret value and store it securely.
Navigate to the Token configuration section and click on Add group claim button. Choose the group type as Security Groups and for the ID token type, select Group ID.
(Optional) Additionally, add an optional upn claim in the Token configuration section.
note
This step is optional and should only be performed if external users (i.e., users with User type: Guest) will be added to the Microsoft Entra Tenant and require access to the Application.
(Optional) After adding the upn claim, click on the three dots next to it, select Edit, and turn on the Externally authenticated toggle to Yes value.
note
This step is optional and should only be performed if external users (i.e., users with User type: Guest) will be added to the Microsoft Entra Tenant and require access to the Application.
Navigate to the API permissions section. Ensure that the User.Read permission is added under the Microsoft Graph API. If not, click on the Add a permission button, select Microsoft Graph, and add the User.Read permission. After adding the permission, click on the Grant admin consent for 'Tenant name' button to grant the required permissions.
(Optional) Additionally, for Microsoft Graph API, add the OpenId permissions, such as openid, email, and profile.
note
This step is optional and should only be performed if external users (i.e., users with User type: Guest) will be added to the Microsoft Entra Tenant and require access to the Application.
The OIDC Application in Microsoft Entra is now configured and ready for integration with AWS EKS.
In the Microsoft Entra Admin Center, in the left sidebar menu, select Groups and then All groups. Click on New group button to create a new group(s) for users who will have access to the AWS EKS cluster.
Fill in the details for the group, such as Group type and Group name. Click on the Create button to create the group.
The group will be created, and you can start adding users to it.
In the Microsoft Entra Admin Center, in the left sidebar menu, select Users and then click on All users. In the New user tab, click on the Create new user button to create a new user.
Fill in the details for the user, such as User principal name, Mail nickname, Display name, and temporary password. In the Properties tab you can set the First name, Last name, and other details.
In the Assignment tab, click on the Add group button. In the Select Group window, choose the group(s) you created earlier (e.g., oidc-cluster-admins) and click on the Select button.
Click on the Review + create button to create the user. The user will be created and added to the group(s) you selected.
Configuring Microsoft Entra as an Identity Provider in AWS EKS
There are two methods to configure Microsoft Entra as an Identity Provider in AWS EKS: through the AWS Management Console and using Terraform.
note
The Application data, such as Directory (tenant) ID, Application (client) ID, and Issuer URL, can be found in the Overview section of the OIDC Application in the Microsoft Entra Admin Center.
In the OIDC identity providers section, click on the Associate identity provider button.
Fill in the following details:
Name: Entra
Issuer URL: https://login.microsoftonline.com/<Tenant-ID>/, where <Tenant-ID> is the Directory (tenant) ID. Ensure that the URL ends with /.
Client ID: <Application (client) ID>, which corresponds to the Application (client) ID of the OIDC Application.
Username Claim: upn.
Groups Claim: groups.
The process of applying the changes may take a few minutes. Once completed, the Microsoft Entra OIDC identity provider will be associated with the AWS EKS cluster.
To configure Microsoft Entra as an Identity Provider in AWS EKS using Terraform, you can use AWS EKS Terraform module. Here's an example of how you can do it:
variables.tf:
variable "cluster_identity_providers" {
description="Configuration for OIDC identity provider"
You may notice the issuer URL format differs between methods: the AWS Console example above uses https://login.microsoftonline.com/<Tenant-ID>/ (Entra v2), while this Terraform example uses https://sts.windows.net/<Tenant ID>/ (the legacy Entra v1 format). Both are valid, but the issuer you configure must match the iss claim of the tokens your app registration actually issues, so check the app's manifest accessTokenAcceptedVersion setting before choosing one.
After applying the Terraform configuration, the Microsoft Entra OIDC identity provider will be associated with the AWS EKS cluster.
Configuring RBAC Resources in AWS EKS cluster for Microsoft Entra User Groups
In this section, user authorization will be configured using Kubernetes Role-Based Access Control (RBAC). Microsoft Entra groups will be linked to Kubernetes ClusterRoles through ClusterRoleBinding resources, enabling precise control over resource access within the EKS cluster. Additionally, Roles and RoleBindings can be used for more granular access control within specific namespaces.
note
The Object ID of the Microsoft Entra group can be found in the Overview section of the group in the Microsoft Entra Admin Center.
Log in to the AWS EKS cluster and create the following ClusterRoleBinding resource, which associates the Microsoft Entra group oidc-cluster-admins with the cluster-admin Kubernetes Cluster Role. Replace <your-microsoft-entra-admin-group-object-id> with the Object ID of the oidc-cluster-admins group, which can be found on the Group overview page in the Microsoft Entra admin center.
Save the above YAML to a file, for example, clusterrolebinding.yaml, and apply it to the EKS cluster using the following command:
kubectl apply -f clusterrolebinding.yaml
The oidc-cluster-admins group will now have cluster-admin permissions within the EKS cluster.
Authenticating to AWS EKS using Microsoft Entra with kubectl
To authenticate to the AWS EKS cluster using Microsoft Entra, you can use the kubectl CLI tool with the kubelogin plugin. The kubelogin plugin simplifies the OIDC authentication process by handling the token exchange and session management. Here's how you can authenticate to the EKS cluster:
Create or update the kubeconfig file with the OIDC configuration. Replace <cluster-name> with the name of your EKS cluster and <region-code> with the AWS region code where the cluster is located.
Replace <Tenant ID>, <Application (client) ID>, and <Application (client) Secret> with the corresponding values from the OIDC Application in the Microsoft Entra Admin Center.
note
You can test the authentication to the EKS cluster immediately by running the following command:
kubectl --user=eks get nodes
Set the context for the kubeconfig file to use the eks user and the OIDC configuration. Replace <cluster-name> with the name of your EKS cluster.
To switch to the eks context, execute the following command:
kubectl config use-context eks
Test the authentication by running the following command:
kubectl get nodes
Configuring KubeRocketCI Portal with Microsoft Entra OIDC Authentication
Starting from version 3.10, KubeRocketCI platform supports Microsoft Entra as an Identity Provider for OIDC authentication in the Portal UI. To configure Microsoft Entra OIDC authentication, navigate to the edp-install Helm chart repository and set the following values in the values.yaml file:
...
edp-headlamp:
enabled:true
config:
oidc:
enabled:true
issuerUrl:"https://sts.windows.net/<Tenant ID>/"
clientID:"<Application (client) ID>"
clientSecretName:"<name of the secret containing the client-secret value>"
clientSecretKey:"clientSecret"
...
Replace <Tenant ID> and <Application (client) ID> with the corresponding values from the OIDC Application in the Microsoft Entra Admin Center. Also, specify the name of the Kubernetes Secret containing the Application Client secret value in the clientSecretName field.
In the Microsoft Entra Admin Center, navigate to the created OIDC Application and select the Authentication section. In the Redirect URIs field, add the URL of the KubeRocketCI Portal, for example, https://portal-<krci-namespace>.<dns-wildcard>/oidc-callback.
After applying the changes, the KubeRocketCI Portal will be configured to use Microsoft Entra OIDC authentication. Users will be able to log in to the Portal using Sign In option.
Integrating OpenID Connect (OIDC) authentication with Microsoft Entra in AWS EKS is a powerful way to enhance security and streamline user access management. By leveraging the capabilities of SSO, OIDC, and Microsoft Entra, organizations can simplify authentication processes, enforce access controls, and ensure secure navigation across cloud-native environments. Whether you're managing user identities, enhancing compliance, or improving user experience, this integration provides a robust solution to meet your identity and access management needs. By following the steps outlined in this guide, you can configure Microsoft Entra as an Identity Provider in AWS EKS, authenticate users using OIDC, and secure access to your EKS clusters and KubeRocketCI Portal effectively.
