Self-Hosted GitLab: CI/CD Pipelines Without Cloud Vendor Lock-in

In December 2025, GitHub announced a $0.002 per minute orchestration fee for self-hosted runners on private repos. You supply the hardware, pay for the electricity, maintain the nodes, and GitHub charges you for job scheduling. They postponed the change after backlash, but the signal was clear: platforms that control your CI/CD pipeline can monetize it whenever they want. GitLab.com's free tier, meanwhile, caps shared runners at 400 minutes per month with a 5-user limit on private namespaces.

This cluster took a different path. GitLab CE v18.8.2 runs on the same three nodes that host everything else, using the same Longhorn storage, the same Cilium network policies, and the same Gateway API routes documented throughout this series. The runner spawns ephemeral Kubernetes pods for every CI job and destroys them when they finish. The container registry stores images in Minio. Three projects have real pipelines pushing real deployments, with hundreds of successful runs between them. No cloud minutes, no vendor deciding what your CI/CD costs next quarter.

This is Part 8 of the homelab series. Part 7 set up alerting with Discord, email, and a dead man's switch. Now we're closing the loop from code to deployment without leaving the cluster.

What Runs Where

GitLab's Helm chart (v9.8.2) deploys 13 pods across two namespaces. The gitlab namespace runs the platform: webservice, sidekiq, gitaly, postgresql, redis, minio (S3-compatible object storage for registry blobs), two registry replicas, gitlab-shell for SSH, two KAS instances, a metrics exporter, and a toolbox for admin tasks. The gitlab-runner namespace holds the runner and its ephemeral CI job pods.

The chart bundles PostgreSQL, Redis, and Minio in-cluster by default, which is not supported for production. Gitaly in particular is unsupported on Kubernetes due to IOPS requirements. For a homelab, in-cluster works.

$ helm list -n gitlab && helm list -n gitlab-runner
NAME    NAMESPACE      REVISION  STATUS    CHART           APP VERSION
gitlab  gitlab         4         deployed  gitlab-9.8.2    v18.8.2

NAME           NAMESPACE      REVISION  STATUS    CHART                  APP VERSION
gitlab-runner  gitlab-runner  3         deployed  gitlab-runner-0.85.0   18.8.0

Resource footprint

Each CI job pod adds up to 2 CPU and 4Gi memory on top of this baseline. On a 3-node cluster with 48GB RAM, GitLab claims roughly 8% of available resources at idle. During active CI runs, that spikes to 15-20%.

That Minio storage number was originally 10Gi. After a few weeks of CI/CD pushing container images, XMinioStorageFull blocked all registry pushes. Pipelines failed with cryptic errors that surfaced in the runner logs, not in the registry or Minio logs. Debugging took an hour; expanding the PVC to 20Gi took five minutes. The lesson: size your object storage for the images your pipelines will actually produce, not for the Helm chart's default.

Gateway API and SSH

Two Gateway API HTTPRoutes serve the web UI and container registry through Cilium:

# Web UI: gitlab.k8s.rommelporras.com → webservice:8181
# Registry: registry.k8s.rommelporras.com → registry:5000

SSH gets a dedicated LoadBalancer service with its own Cilium LB IPAM address (10.10.30.21), separate from the Gateway's VIP (10.10.30.20). Git operations over SSH hit port 22 externally, mapped to the shell container's port 2222.

No Cloudflare Tunnel route. GitLab is accessible only via the home network or Tailscale. Source code and CI/CD credentials stay on the local network.

The Runner

privileged = true is the cost of building container images. Docker-in-Docker needs access to the host kernel, so the Kubernetes executor spawns job pods in privileged mode. Rootless alternatives like Kaniko and BuildKit are the production-standard choice in 2026, but DinD was simpler to set up for a homelab. If you're running untrusted code, switch to Kaniko.

[[runners]]
  [runners.kubernetes]
    namespace = "gitlab-runner"
    image = "alpine:latest"
    privileged = true
    cpu_limit = "2"
    memory_limit = "4Gi"
    memory_request = "1Gi"
    pull_policy = "if-not-present"

That memory_limit = "4Gi" was originally 2Gi. TypeScript type-checking during bun run build peaks at ~3Gi for a full-stack Next.js app. The job pod hit its limit and the kubelet OOM-killed it, which briefly destabilized the node. The Kubernetes executor has no built-in OOM warning; it relies entirely on pod resource limits, and exceeding them affects the node, not just the job.

The initial deployment also set clusterWideAccess: true in the Runner Helm chart, which creates a ClusterRole granting the runner service account broad permissions across all namespaces, including access to secrets and pods. Phase 5.2 switched to clusterWideAccess: false, scoping the runner to its own namespace. Cross-namespace deployments use separate service accounts with namespace-scoped Roles and tokens stored as CI/CD variables.

Authentication tokens, not registration tokens

GitLab introduced the new runner token architecture in 15.10 and has recommended it as the default since 16.0. The old shared registration token workflow can be disabled starting in 17.0, and in practice, GitLab 18.x rejects registration attempts with the old tokens. The cluster got 410 Gone on the first attempt and the error message didn't explain what to do instead. You have to know to create the runner in the GitLab Admin UI first, grab the glrt- authentication token, and use the new flow.

One gotcha: the Helm chart's projected volume still expects both runner-token and runner-registration-token keys in the Kubernetes secret, even when using the new auth flow. The ExternalSecret templates an empty string for the legacy key:

# manifests/gitlab-runner/externalsecret.yaml
target:
  template:
    data:
      runner-token: "{{ .runnerToken }}"
      runner-registration-token: ""  # empty, but chart requires the key

Without that empty key, the chart fails to mount the secret volume.

Three Real Pipelines

This isn't a tutorial with echo "Hello from CI". Three projects run production pipelines through this GitLab instance.

Invoicetron (full-stack Next.js, pipeline #164)

A four-stage GitFlow pipeline: validate, build, deploy, verify.

The validate stage runs type-checking (Bun + Prisma), linting, security audit, and unit tests in parallel. Build creates per-environment Docker images using docker buildx with registry cache (mode=max), because NEXT_PUBLIC_APP_URL bakes the domain into the JavaScript bundle at build time. Deploy runs a Prisma migration as a Kubernetes Job before patching the Deployment image:

# Prisma migration runs as a K8s Job before the deployment update
- kubectl delete job invoicetron-migrate -n $DEV_NAMESPACE --ignore-not-found
- cat <<EOF | kubectl apply -f -
  apiVersion: batch/v1
  kind: Job
  metadata:
    name: invoicetron-migrate
    namespace: $DEV_NAMESPACE
  spec:
    template:
      spec:
        imagePullSecrets:
        - name: gitlab-registry
        containers:
        - name: migrate
          image: $IMAGE_BASE/dev:$CI_COMMIT_SHORT_SHA
          command: ["bunx", "prisma", "migrate", "deploy"]
          env:
          - name: DATABASE_URL
            valueFrom:
              secretKeyRef:
                name: invoicetron-app
                key: database-url
        restartPolicy: Never
    backoffLimit: 3
  EOF

Verify stage curls the health endpoint to confirm the deployment is alive.

Portfolio (Next.js, pipeline #128)

Four stages with a gated merge request workflow. Validate and test must pass before merging. The main branch skips those stages (already validated in the MR pipeline) and goes straight to build and deploy. Three environments: development (automatic on develop), staging (manual trigger), production (automatic on main).

The test stage generates JUnit reports and Cobertura coverage, which GitLab renders directly in merge request diffs. E2E runs Playwright against a built Next.js server inside the CI pod.

Eventually Consistent (this blog's Ghost theme, pipeline #83)

The simplest pipeline. Validate with gscan, build with gulp zip, then deploy by uploading the zip to Ghost's Admin API. No containers involved. The deploy script generates a JWT token from the Ghost Admin API key using openssl, then uploads via curl:

# JWT generation from Ghost Admin API key (id:secret format)
ID=$(echo "$GHOST_ADMIN_API_KEY" | cut -d':' -f1)
SECRET=$(echo "$GHOST_ADMIN_API_KEY" | cut -d':' -f2)
HEADER=$(printf '{"alg":"HS256","typ":"JWT","kid":"%s"}' "$ID" | base64url)
PAYLOAD=$(printf '{"iat":%d,"exp":%d,"aud":"/admin/"}' "$NOW" "$EXP" | base64url)

Each pipeline targets two Ghost environments (dev and prod) using environment-scoped CI/CD variables for GHOST_URL and GHOST_ADMIN_API_KEY.

Registry authentication

Pipelines authenticate to the GitLab Container Registry using predefined CI/CD variables ($CI_REGISTRY_USER, $CI_REGISTRY_PASSWORD). The password maps to $CI_JOB_TOKEN, a short-lived token scoped to the current job. Deploy tokens stored in Vault and synced via External Secrets Operator handle image pulls from application namespaces.

21 Network Policies

The gitlab namespace has 15 CiliumNetworkPolicies. The gitlab-runner namespace has 6. Together they enforce default-deny with explicit allow rules.

The baseline policies cover DNS egress, kubelet health probes, intra-namespace communication, and kube-apiserver access. On top of that, targeted policies control traffic between namespaces:

# Runner pods can reach GitLab API and registry, nothing else in the gitlab namespace
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-gitlab-egress
  namespace: gitlab-runner
spec:
  endpointSelector: {}
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: gitlab
      toPorts:
        - ports:
            - port: "8181"   # GitLab API (webservice)
            - port: "8080"   # Internal API
            - port: "5000"   # Container registry

The runner namespace allows internet egress for package downloads and image pulls. The gitlab namespace restricts internet egress to webservice (OAuth, webhooks) and sidekiq (SMTP on port 587 for email notifications). Everything else is internal-only.

SSH ingress to gitlab-shell allows LAN clients via the LoadBalancer. Monitoring ingress opens 7 specific ports for Prometheus scraping across different pod labels.

Pod Security Standards

The gitlab namespace runs at enforce=baseline with audit=restricted and warn=restricted. Baseline is sufficient because GitLab's pods don't need host-level access.

The gitlab-runner namespace runs at enforce=privileged because DinD build pods require elevated kernel capabilities. This is the deliberate trade-off: privileged pods for the ability to build container images. The audit=restricted and warn=restricted labels still flag violations in the audit log, so you can track exactly which pods escalate privileges and why.

Secrets

Every secret flows through External Secrets Operator and HashiCorp Vault. No kubectl create secret commands, no hardcoded values:

Deploy tokens are particularly useful for cross-namespace image pulls. The invoicetron namespaces need to pull images from the GitLab registry, so ESO syncs a deploy token from Vault into a kubernetes.io/dockerconfigjson secret that the Deployment references as imagePullSecrets.

The Bitnami Question

The bundled PostgreSQL and Redis subcharts use Bitnami container images. In September 2025, Broadcom moved the versioned Bitnami catalog behind a paid subscription. Free users get only :latest tags from a legacy repository with no security patches. GitLab has acknowledged this by deprecating the bundled subcharts starting with chart version 19.0, which will mandate external databases for all deployments.

For this homelab, the current chart (9.8.2) still pulls the legacy Bitnami images. They work, but they won't receive security updates. The eventual path is external PostgreSQL and Redis, whether that's a CloudNativePG operator on the cluster or a dedicated VM. When that migration happens, it'll be a post in this series.

What's Still Missing

Part 7 called out that GitLab has zero custom alerts despite being the most complex deployment in the cluster. That gap is still open. The CiliumNetworkPolicies already allow Prometheus to scrape 8 metrics endpoints across both namespaces (gitlab-exporter on 9168, PostgreSQL on 9187, Redis on 9121, runner on 9252, and more). The exporters are running and producing metrics. But no ServiceMonitors exist to tell Prometheus to discover them, and no PrometheusRules define GitLab-specific alert thresholds. The metrics pipeline is wired at the network level but not at the discovery level.

There are no automated backups either. The toolbox pod supports backup-utility --skip registry for manual backups, and Longhorn snapshots cover the PVCs. But there's no CronJob automating either path. For a learning environment this is acceptable. For anything resembling production, it isn't.

One component to watch: Sidekiq regularly hovers near 88% of its 1.5Gi memory limit. It hasn't OOM-killed yet, but it's the most constrained component in the stack and likely needs a bump to 2Gi.

This post is the eighth in the "Building a Production-Grade Homelab" series:

1. Why kubeadm Over k3s, RKE2, and Talos in 2026
2. HA Control Plane with kube-vip: No Load Balancer Needed
3. Cilium Deep Dive: What Replacing kube-proxy Actually Means
4. Gateway API vs Ingress: No Ingress Controller Needed
5. Distributed Storage with Longhorn: 2 Replicas Are Enough
6. The Modern Logging Stack: Loki + Alloy (Why Not Promtail)
7. Alerting That Actually Wakes You Up: Discord, Email, and Dead Man's Switches
8. Self-Hosted GitLab: CI/CD Pipelines Without Cloud Vendor Lock-in (you are here)