Deploy grafana cloud k8s monitoring with flux ============================================= Grafana Cloud ------------- When setting up k8s monitoring in Grafana Cloud, a (long) `helm` command is provided, that looks like the follow (sensitive information redacted): .. code-block:: bash helm repo add grafana https://grafana.github.io/helm-charts && helm repo update && helm upgrade --install --atomic --timeout 300s grafana-k8s-monitoring grafana/k8s-monitoring \ --namespace "grafanacloud-flux" --create-namespace --values - <<'EOF' cluster: name: do-tor1-c01 destinations: - name: grafana-cloud-metrics type: prometheus url: https://REDACTED/api/prom/push auth: type: basic username: REDACTED password: REDACTED - name: grafana-cloud-logs type: loki url: https://REDACTED/loki/api/v1/push auth: type: basic username: REDACTED password: REDACTED - name: grafana-cloud-otlp-endpoint type: otlp url: https://REDACTED/otlp protocol: http auth: type: basic username: REDACTED password: REDACTED metrics: enabled: true logs: enabled: true traces: enabled: true clusterMetrics: enabled: true opencost: enabled: true metricsSource: grafana-cloud-metrics opencost: exporter: defaultClusterId: REDACTED prometheus: existingSecretName: grafana-cloud-metrics-grafana-k8s-monitoring external: url: https://REDACTED/api/prom kepler: enabled: true clusterEvents: enabled: true podLogs: enabled: true applicationObservability: enabled: true receivers: otlp: grpc: enabled: true port: 4317 http: enabled: true port: 4318 zipkin: enabled: true port: 9411 integrations: alloy: instances: - name: alloy labelSelectors: app.kubernetes.io/name: - alloy-metrics - alloy-singleton - alloy-logs - alloy-receiver alloy-metrics: enabled: true alloy-singleton: enabled: true alloy-logs: enabled: true alloy-receiver: enabled: true alloy: extraPorts: - name: otlp-grpc port: 4317 targetPort: 4317 protocol: TCP - name: otlp-http port: 4318 targetPort: 4318 protocol: TCP - name: zipkin port: 9411 targetPort: 9411 protocol: TCP EOF Flux ---- In the repository files look like: .. code-block:: none :emphasize-lines: 8,11-14 . |-- clusters | `-- mycluster | |-- flux-system | | |-- gotk-components.yaml | | |-- gotk-sync.yaml | | `-- kustomization.yaml | `-- infra.yaml |-- infra | `-- monitoring | |-- grafanacloud.yaml | |-- kustomization.yaml | |-- kustomizeconfig.yaml | `-- my-values.enc.yaml - ``infra.yaml``: Flux Kustomization - ``kustomization.yaml``: (regular) Kustomization for the resources in ``grafanacloud.yaml`` - ``grafanacloud.yaml``: Namespace, HelmRepository, and HelmRelease objects - ``kustomizeconfig.yaml``: additional config for ``kustomization.yaml`` - ``my-values.enc.yaml``: encrypted values to be injected to the HelmRelease Note the Kustomizations in the first 2 files are of different type (ref. [flux-faq-1]_) .. code-block:: yaml :linenos: :caption: infra.yaml --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: infra-monitoring namespace: flux-system spec: interval: 1h retryInterval: 1m timeout: 5m sourceRef: kind: GitRepository name: flux-system path: ./infra/monitoring prune: true wait: true decryption: provider: sops secretRef: name: sops-age .. code-block:: yaml :linenos: :caption: kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: grafanacloud-flux resources: - grafanacloud.yaml secretGenerator: - name: grafanacloud-secrets files: - values.yaml=my-values.enc.yaml configurations: - kustomizeconfig.yaml .. code-block:: yaml :linenos: :caption: grafanacloud.yaml --- apiVersion: v1 kind: Namespace metadata: name: grafanacloud-flux --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: grafana namespace: grafanacloud-flux spec: interval: 24h url: https://grafana.github.io/helm-charts --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: grafana-k8s-monitoring namespace: grafanacloud-flux spec: interval: 30m chart: spec: chart: k8s-monitoring version: "3.0.2" sourceRef: kind: HelmRepository name: grafana namespace: grafanacloud-flux interval: 12h values: cluster: name: do-tor1-c01 # the whole destinations: block will be injected from Secret via valuesFrom below clusterMetrics: enabled: true opencost: enabled: true metricsSource: grafana-cloud-metrics opencost: exporter: defaultClusterId: do-tor1-c01 prometheus: existingSecretName: grafana-cloud-metrics-grafana-k8s-monitoring external: url: https://REDACTED/api/prom enabled: true clusterEvents: enabled: true podLogs: enabled: true applicationObservability: enabled: true receivers: otlp: grpc: enabled: true port: 4317 http: enabled: true port: 4318 zipkin: enabled: true port: 9411 integrations: alloy: instances: - name: alloy labelSelectors: app.kubernetes.io/name: - alloy-metrics - alloy-singleton - alloy-logs - alloy-receiver alloy-metrics: enabled: true alloy-singleton: enabled: true alloy-logs: enabled: true alloy-receiver: enabled: true alloy: extraPorts: - name: otlp-grpc port: 4317 targetPort: 4317 protocol: TCP - name: otlp-http port: 4318 targetPort: 4318 protocol: TCP - name: zipkin port: 9411 targetPort: 9411 protocol: TCP valuesFrom: - kind: Secret name: grafanacloud-secrets In the ``HelmRelease`` manifest the version for the chart was the latest at the time (not sure if there is a drawback of just using ``version: "*"`` instead): .. code-block:: bash → helm repo add grafana https://grafana.github.io/helm-charts → helm repo list NAME URL grafana https://grafana.github.io/helm-charts → helm search repo grafana NAME CHART VERSION APP VERSION DESCRIPTION grafana/grafana 9.2.7 12.0.2 The leading tool for querying and visualizing t... grafana/grafana-agent 0.42.0 v0.42.0 Grafana Agent grafana/grafana-agent-operator 0.5.1 0.44.2 A Helm chart for Grafana Agent Operator grafana/grafana-operator v5.18.0 v5.18.0 Helm chart for the Grafana Operator grafana/grafana-sampling 1.1.5 v1.7.5 A Helm chart for a layered OTLP tail sampling a... grafana/alloy 1.1.1 v1.9.1 Grafana Alloy grafana/alloy-crd 1.0.0 1.0.0 A Helm chart the Alloy CustomResourceDefinition... grafana/alloy-operator 0.3.2 1.1.1 A Helm chart the Alloy Operator, a project to i... grafana/beyla 1.8.0 2.2.3 eBPF-based autoinstrumentation HTTP, HTTP2 and ... grafana/cloudcost-exporter 1.0.4 0.8.1 Cloud Cost Exporter exports cloud provider agno... grafana/enterprise-logs 2.5.0 v1.5.2 Grafana Enterprise Logs grafana/enterprise-logs-simple 1.3.0 v1.4.0 DEPRECATED Grafana Enterprise Logs (Simple Scal... grafana/enterprise-metrics 1.10.0 v1.7.0 DEPRECATED Grafana Enterprise Metrics grafana/fluent-bit 2.6.0 v2.1.0 Uses fluent-bit Loki go plugin for gathering lo... grafana/k6-operator 3.13.1 0.0.21 A Helm chart to install the k6-operator grafana/k8s-monitoring 3.0.2 3.0.2 Capture all telemetry data from your Kubernetes... grafana/lgtm-distributed 2.1.0 ^7.3.9 Umbrella chart for a distributed Loki, Grafana,... .. code-block:: yaml :linenos: :caption: kustomizeconfig.yaml nameReference: - kind: Secret version: v1 fieldSpecs: - path: spec/valuesFrom/name kind: HelmRelease To create the ``my-values.enc.yaml``: - Set up sops/age as mentioned in :ref:`sops-age`, with the ``.sops.yaml`` configured to encrypt both standard secrets and special values files (:ref:`sops-age-flux-helm`) - Create a ``my-values.yaml`` file containing the values portion that need to be encrypted (for this example it contains all the values under ``destinations:`` which contains sensitive values) .. code-block:: yaml :caption: my-values.yaml (temporary file, not to be committed) destinations: - name: grafana-cloud-metrics type: prometheus url: https://REDACTED/api/prom/push auth: type: basic username: REDACTED password: REDACTED - name: grafana-cloud-logs - Run the sops command to create the encrypted file from ``my-values.yaml`` .. code-block:: bash sops -e --input-type=yaml --output-type=yaml my-values.yaml > my-values.enc.yaml .. code-block:: yaml :caption: my-values.enc.yaml destinations: - name: ENC[AES256_GCM,data:Jugvs... type: ENC[AES256_GCM,data:LXkJT... url: ENC[AES256_GCM,data:tAs8V... auth: type: ENC[AES256_GCM,data:fVqu6V... username: ENC[AES256_GCM,data:jxwlZ... password: ENC[AES256_GCM,data:Q9IRQ... ... - Delete the ``my-values.yaml`` as it should not be committed to the repository NOTE: Instead of encrypting the full ``destinations:`` block, the ideal approach would be to just create a secret that contains only the sensitive values, however I was not able to find a way to do it that worked, therefore had to end up encrypting the whole block. References ---------- .. [flux-faq-1] `Flux FAQ: Are there two Kustomization types? (fluxcd.io) `__