9. Deploy grafana cloud k8s monitoring with flux

9.1. Grafana Cloud

When setting up k8s monitoring in Grafana Cloud, a (long) helm command is provided, that looks like the follow (sensitive information redacted):

helm repo add grafana https://grafana.github.io/helm-charts &&
  helm repo update &&
  helm upgrade --install --atomic --timeout 300s grafana-k8s-monitoring grafana/k8s-monitoring \
    --namespace "grafanacloud-flux" --create-namespace --values - <<'EOF'
cluster:
  name: do-tor1-c01
destinations:
  - name: grafana-cloud-metrics
    type: prometheus
    url: https://REDACTED/api/prom/push
    auth:
      type: basic
      username: REDACTED
      password: REDACTED
  - name: grafana-cloud-logs
    type: loki
    url: https://REDACTED/loki/api/v1/push
    auth:
      type: basic
      username: REDACTED
      password: REDACTED
  - name: grafana-cloud-otlp-endpoint
    type: otlp
    url: https://REDACTED/otlp
    protocol: http
    auth:
      type: basic
      username: REDACTED
      password: REDACTED
    metrics:
      enabled: true
    logs:
      enabled: true
    traces:
      enabled: true
clusterMetrics:
  enabled: true
  opencost:
    enabled: true
    metricsSource: grafana-cloud-metrics
    opencost:
      exporter:
        defaultClusterId: REDACTED
      prometheus:
        existingSecretName: grafana-cloud-metrics-grafana-k8s-monitoring
        external:
          url: https://REDACTED/api/prom
  kepler:
    enabled: true
clusterEvents:
  enabled: true
podLogs:
  enabled: true
applicationObservability:
  enabled: true
  receivers:
    otlp:
      grpc:
        enabled: true
        port: 4317
      http:
        enabled: true
        port: 4318
    zipkin:
      enabled: true
      port: 9411
integrations:
  alloy:
    instances:
      - name: alloy
        labelSelectors:
          app.kubernetes.io/name:
            - alloy-metrics
            - alloy-singleton
            - alloy-logs
            - alloy-receiver
alloy-metrics:
  enabled: true
alloy-singleton:
  enabled: true
alloy-logs:
  enabled: true
alloy-receiver:
  enabled: true
  alloy:
    extraPorts:
      - name: otlp-grpc
        port: 4317
        targetPort: 4317
        protocol: TCP
      - name: otlp-http
        port: 4318
        targetPort: 4318
        protocol: TCP
      - name: zipkin
        port: 9411
        targetPort: 9411
        protocol: TCP
EOF

9.2. Flux

In the repository files look like:

.
|-- clusters
|   `-- mycluster
|       |-- flux-system
|       |   |-- gotk-components.yaml
|       |   |-- gotk-sync.yaml
|       |   `-- kustomization.yaml
|       `-- infra.yaml
|-- infra
|   `-- monitoring
|       |-- grafanacloud.yaml
|       |-- kustomization.yaml
|       |-- kustomizeconfig.yaml
|       `-- my-values.enc.yaml

  • infra.yaml: Flux Kustomization

  • kustomization.yaml: (regular) Kustomization for the resources in grafanacloud.yaml

  • grafanacloud.yaml: Namespace, HelmRepository, and HelmRelease objects

  • kustomizeconfig.yaml: additional config for kustomization.yaml

  • my-values.enc.yaml: encrypted values to be injected to the HelmRelease

Note the Kustomizations in the first 2 files are of different type (ref. [flux-faq-1])

infra.yaml
 1---
 2apiVersion: kustomize.toolkit.fluxcd.io/v1
 3kind: Kustomization
 4metadata:
 5  name: infra-monitoring
 6  namespace: flux-system
 7spec:
 8  interval: 1h
 9  retryInterval: 1m
10  timeout: 5m
11  sourceRef:
12    kind: GitRepository
13    name: flux-system
14  path: ./infra/monitoring
15  prune: true
16  wait: true
17  decryption:
18    provider: sops
19    secretRef:
20      name: sops-age
kustomization.yaml
 1apiVersion: kustomize.config.k8s.io/v1beta1
 2kind: Kustomization
 3namespace: grafanacloud-flux
 4resources:
 5  - grafanacloud.yaml
 6secretGenerator:
 7  - name: grafanacloud-secrets
 8    files:
 9      - values.yaml=my-values.enc.yaml
10configurations:
11  - kustomizeconfig.yaml
grafanacloud.yaml
  1---
  2apiVersion: v1
  3kind: Namespace
  4metadata:
  5  name: grafanacloud-flux
  6---
  7apiVersion: source.toolkit.fluxcd.io/v1
  8kind: HelmRepository
  9metadata:
 10  name: grafana
 11  namespace: grafanacloud-flux
 12spec:
 13  interval: 24h
 14  url: https://grafana.github.io/helm-charts
 15---
 16apiVersion: helm.toolkit.fluxcd.io/v2
 17kind: HelmRelease
 18metadata:
 19  name: grafana-k8s-monitoring
 20  namespace: grafanacloud-flux
 21spec:
 22  interval: 30m
 23  chart:
 24    spec:
 25      chart: k8s-monitoring
 26      version: "3.0.2"
 27      sourceRef:
 28        kind: HelmRepository
 29        name: grafana
 30        namespace: grafanacloud-flux
 31      interval: 12h
 32  values:
 33    cluster:
 34      name: do-tor1-c01
 35    # the whole destinations: block will be injected from Secret via valuesFrom below
 36    clusterMetrics:
 37      enabled: true
 38      opencost:
 39        enabled: true
 40        metricsSource: grafana-cloud-metrics
 41        opencost:
 42          exporter:
 43            defaultClusterId: do-tor1-c01
 44          prometheus:
 45            existingSecretName: grafana-cloud-metrics-grafana-k8s-monitoring
 46            external:
 47              url: https://REDACTED/api/prom
 48        enabled: true
 49    clusterEvents:
 50      enabled: true
 51    podLogs:
 52      enabled: true
 53    applicationObservability:
 54      enabled: true
 55      receivers:
 56        otlp:
 57          grpc:
 58            enabled: true
 59            port: 4317
 60          http:
 61            enabled: true
 62            port: 4318
 63        zipkin:
 64          enabled: true
 65          port: 9411
 66    integrations:
 67      alloy:
 68        instances:
 69          - name: alloy
 70            labelSelectors:
 71              app.kubernetes.io/name:
 72                - alloy-metrics
 73                - alloy-singleton
 74                - alloy-logs
 75                - alloy-receiver
 76    alloy-metrics:
 77      enabled: true
 78    alloy-singleton:
 79      enabled: true
 80    alloy-logs:
 81      enabled: true
 82    alloy-receiver:
 83      enabled: true
 84      alloy:
 85        extraPorts:
 86          - name: otlp-grpc
 87            port: 4317
 88            targetPort: 4317
 89            protocol: TCP
 90          - name: otlp-http
 91            port: 4318
 92            targetPort: 4318
 93            protocol: TCP
 94          - name: zipkin
 95            port: 9411
 96            targetPort: 9411
 97            protocol: TCP
 98  valuesFrom:
 99    - kind: Secret
100      name: grafanacloud-secrets

In the HelmRelease manifest the version for the chart was the latest at the time (not sure if there is a drawback of just using version: "*" instead):

→ helm repo add grafana https://grafana.github.io/helm-charts

→ helm repo list
NAME    URL
grafana https://grafana.github.io/helm-charts

→ helm search repo grafana
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
grafana/grafana                         9.2.7           12.0.2          The leading tool for querying and visualizing t...
grafana/grafana-agent                   0.42.0          v0.42.0         Grafana Agent
grafana/grafana-agent-operator          0.5.1           0.44.2          A Helm chart for Grafana Agent Operator
grafana/grafana-operator                v5.18.0         v5.18.0         Helm chart for the Grafana Operator
grafana/grafana-sampling                1.1.5           v1.7.5          A Helm chart for a layered OTLP tail sampling a...
grafana/alloy                           1.1.1           v1.9.1          Grafana Alloy
grafana/alloy-crd                       1.0.0           1.0.0           A Helm chart the Alloy CustomResourceDefinition...
grafana/alloy-operator                  0.3.2           1.1.1           A Helm chart the Alloy Operator, a project to i...
grafana/beyla                           1.8.0           2.2.3           eBPF-based autoinstrumentation HTTP, HTTP2 and ...
grafana/cloudcost-exporter              1.0.4           0.8.1           Cloud Cost Exporter exports cloud provider agno...
grafana/enterprise-logs                 2.5.0           v1.5.2          Grafana Enterprise Logs
grafana/enterprise-logs-simple          1.3.0           v1.4.0          DEPRECATED Grafana Enterprise Logs (Simple Scal...
grafana/enterprise-metrics              1.10.0          v1.7.0          DEPRECATED Grafana Enterprise Metrics
grafana/fluent-bit                      2.6.0           v2.1.0          Uses fluent-bit Loki go plugin for gathering lo...
grafana/k6-operator                     3.13.1          0.0.21          A Helm chart to install the k6-operator
grafana/k8s-monitoring                  3.0.2           3.0.2           Capture all telemetry data from your Kubernetes...
grafana/lgtm-distributed                2.1.0           ^7.3.9          Umbrella chart for a distributed Loki, Grafana,...
kustomizeconfig.yaml
1nameReference:
2- kind: Secret
3  version: v1
4  fieldSpecs:
5  - path: spec/valuesFrom/name
6    kind: HelmRelease

To create the my-values.enc.yaml:

  • Set up sops/age as mentioned in Configure SOPS and AGE for secret encryption, with the .sops.yaml configured to encrypt both standard secrets and special values files (Using SOPS/AGE with Flux Helm Releases)

  • Create a my-values.yaml file containing the values portion that need to be encrypted (for this example it contains all the values under destinations: which contains sensitive values)

    my-values.yaml (temporary file, not to be committed)
    destinations:
  - name: grafana-cloud-metrics
    type: prometheus
    url: https://REDACTED/api/prom/push
    auth:
      type: basic
      username: REDACTED
      password: REDACTED
  - name: grafana-cloud-logs

  • Run the sops command to create the encrypted file from my-values.yaml

    sops -e --input-type=yaml --output-type=yaml my-values.yaml > my-values.enc.yaml
    my-values.enc.yaml
    destinations:
    - name: ENC[AES256_GCM,data:Jugvs...
      type: ENC[AES256_GCM,data:LXkJT...
      url: ENC[AES256_GCM,data:tAs8V...
      auth:
        type: ENC[AES256_GCM,data:fVqu6V...
        username: ENC[AES256_GCM,data:jxwlZ...
        password: ENC[AES256_GCM,data:Q9IRQ...
...

  • Delete the my-values.yaml as it should not be committed to the repository

NOTE: Instead of encrypting the full destinations: block, the ideal approach would be to just create a secret that contains only the sensitive values, however I was not able to find a way to do it that worked, therefore had to end up encrypting the whole block.

9.3. References