Kubernetes

Registry

The registry has a Kubernetes Helm chart that provides a pre-configured Scraper and Topology with some common defaults

The kubernetes scraper collects all of the resources and events in a Kubernetes cluster, and then watches for changes.

kubernetes-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: kubernetes-scraper
spec:
  retention:
    changes:
      - name: PodCrashLooping
        count: 10
        age: 72h
  kubernetes:
    - clusterName: local-kind-cluster
      transform:
        relationship:
          # Link a service to a deployment (adjust the label selector accordingly)
          - filter: config_type == "Kubernetes::Service"
            type:
              value: 'Kubernetes::Deployment'
            name:
              expr: |
                has(config.spec.selector) && has(config.spec.selector.name) ? config.spec.selector.name : ''
          # Link Pods to PVCs
          - filter: config_type == 'Kubernetes::Pod'
            expr: |
              config.spec.volumes.
                filter(item, has(item.persistentVolumeClaim)).
                map(item, {"type": "Kubernetes::PersistentVolumeClaim", "name": item.persistentVolumeClaim.claimName}).
                toJSON()
          # Link Argo Application to the resources
          - filter: config_type == "Kubernetes::Application" && config.apiVersion == "argoproj.io/v1alpha1"
            expr: |
              config.status.resources.map(item, {
                "type": "Kubernetes::" + item.kind,
                "name": item.name,
                "labels": {
                  "namespace": item.namespace,
                },
              }).toJSON()
        mask:
          - selector: |
              has(config.kind) ? config.kind == 'Certificate' : false
            jsonpath: .spec.dnsNames
            value: md5sum
          - selector: 'config_type == "Kubernetes::Certificate"'
            jsonpath: .spec.commonName
            value: md5sum
        exclude:
          - types:
              - Kubernetes::*
            jsonpath: '.metadata.ownerReferences'
          - types:
              - Kubernetes::Pod
            jsonpath: '.metadata.generateName'
        changes:
          mapping:
            - filter: >
                change.change_type == 'diff' && change.summary == "status.containerStatuses" && 
                patch != null && has(patch.status) && has(patch.status.containerStatuses) && 
                patch.status.containerStatuses.size() > 0 &&
                has(patch.status.containerStatuses[0].restartCount)
              type: PodCrashLooping
          exclude:
            - 'config_type == "Kubernetes::Endpoints" && details.message == "metadata.annotations.endpoints.kubernetes.io/last-change-trigger-time"'
            - 'config_type == "Kubernetes::Node" && has(details.message) && details.message == "status.images"'
            - 'details.source.component == "canary-checker" && (change_type == "Failed" || change_type == "Pass")'
            - >
              change_type == "diff" && summary == "status.reconciledAt" && 
              config != null && 
              has(config.apiVersion) && config.apiVersion == "argoproj.io/v1alpha1" && 
              has(config.kind) && config.kind == "Application"
      properties:
        - filter: 'config_type == "Kubernetes::Pod"'
          name: Logs
          icon: opensearch
          links:
            - text: opensearch
              url: https://opensearch.svc/_dashboards/app/discover#/?_a=(query:(language:kuery,query:'kubernetes_pod_id:{{.id}}'))
        - filter: 'config_type == "Kubernetes::Node"'
          name: Grafana
          icon: grafana
          links:
            - text: grafana
              url: https://grafana.svc/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?var-namespace={{.name}}
      exclusions:
        name:
          - junit*
          - k6-junit*
          - newman-junit*
          - playwright-junit-*
          - hello-world*
        namespace:
          - canaries
          - monitoring
        kind:
          - Secret
          - ReplicaSet
          - APIService
          - PodMetrics
          - NodeMetrics
          - endpoints.discovery.k8s.io
          - endpointslices.discovery.k8s.io
          - leases.coordination.k8s.io
          - podmetrics.metrics.k8s.io
          - nodemetrics.metrics.k8s.io
          - customresourcedefinition
          - controllerrevision
          - certificaterequest
          - orders.acme.cert-manager.io
        labels:
          canary-checker.flanksource.com/generated: 'true'
      relationships:
        - kind:
            expr: "has(spec.claimRef) ? spec.claimRef.kind : ''"
          name:
            expr: "has(spec.claimRef) ? spec.claimRef.name : ''"
          namespace:
            expr: "has(spec.claimRef) ? spec.claimRef.namespace : ''"

        - kind:
            value: Kustomization
          name:
            label: kustomize.toolkit.fluxcd.io/name
          namespace:
            label: kustomize.toolkit.fluxcd.io/namespace

        - kind:
            value: HelmRelease
          name:
            label: helm.toolkit.fluxcd.io/name
          namespace:
            label: helm.toolkit.fluxcd.io/namespace

        # FluxCD Git relationships
        - name:
            expr: "has(spec.sourceRef) ? spec.sourceRef.name : '' "
          namespace:
            expr: "has(spec.sourceRef) && has(spec.sourceRef.namespace)  ? spec.sourceRef.namespace : metadata.namespace "
          kind:
            value: "GitRepository"
      event:
        exclusions:
          reason:
            - SuccessfulCreate
            - Created
            - DNSConfigForming
        severityKeywords:
          error:
            - failed
            - error
          warn:
            - backoff
            - nodeoutofmemory

Field	Description	Scheme
`logLevel`	Specify the level of logging.	`string`
`schedule`	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	`string`
`retention`	Settings for retaining changes, analysis and scraped items	`Retention`
`kubernetes`	Specifies the list of Kubernetes configurations to scrape.	`[]Kubernetes`

Kubernetes

Field	Description	Scheme
`clusterName`	Specify cluster name	`string`
`event`	Specify configuration to handle Kubernetes events.	`Event`
`exclusions`	Resources to be excluded from scraping	`[]string`
`fieldSelector`	Resources to be included e.g `status.Phase=Running`	`string`
`kubeconfig`	Kubeconfig to connect to the cluster	[]EnvVar
`namespace`	Include resources only from this namespace	`string`
`relationships`	Create relationships between kubernetes objects.	`[]Relationship`
`scope`	Specify scope for scrape. e.g `cluster` for scraping at Cluster level	`string`
`selector`	Include resources matching this selector only e.g `matchLabels`	`string`
`since`	Set time constraint for scraping resources within the set period	`string`
`properties`	Custom properties to be added for each item	`[]ConfigProperty`
`transform`	Custom transformations to apply	`Transform`
`tags`	Tags to set on each config item. `cluster` and `namespace` are set by default	`map[string]string`

Events

Kubernetes::Event resources are mapped to config changes. Events can be very verbose so they can be excluded or their severity level changed:

Field	Description	Scheme	Required
`exclusions`	A list of keywords used to exclude event objects based on the reason	`[]string`
`severityKeywords`	Specify keywords used to identify the severity of the Kubernetes Event based on the reason	`SeverityKeywords`

SeverityKeywords

Field	Description	Scheme	Required
`warn`	A list of keywords used to identify a warning severity from the reason. It could also be a match pattern: e.g. `*` to match all or `!badword` to exclude `badword`	`[]string`
`error`	Same as `warn` but used to map to error severity.	`[]string`

Relationships

You can create relationships between kubernetes objects on the basis of

info

Relationships can also be defined under transform.relationships, however by defining them under kubernetes.relationships is simpler with specific support for kind, name and namespace fields.

kubernetes-relationship.yaml
kubernetes:
  - clusterName: 'eks'
    relationships:
      # If object has spec.claimRef field, use its kind, name and namespace
      - kind:
          expr: "has(spec.claimRef) ? spec.claimRef.kind : ''"
        name:
          expr: "has(spec.claimRef) ? spec.claimRef.name : ''"
        namespace:
          expr: "has(spec.claimRef) ? spec.claimRef.namespace : ''"

      # If object flux kustomize labels, link it to the parent Kustomization object
      - kind:
          value: Kustomization
        name:
          label: kustomize.toolkit.fluxcd.io/name
        namespace:
          label: kustomize.toolkit.fluxcd.io/namespace

      # If object helm kustomize labels, link it to the parent HelmRelease object
      - kind:
          value: HelmRelease
        name:
          label: helm.toolkit.fluxcd.io/name
        namespace:
          label: helm.toolkit.fluxcd.io/namespace

Field	Description	Scheme	Required
`kind`	`kind` of Kubernetes Object	`Lookup`	`true`
`name`	`name` of Kubernetes Object	`Lookup`	`true`
`namespace`	`namespace` of Kubernetes Object	`Lookup`	`true`

Lookup

There are 3 different ways to specify which value to use when finding related configs:

Field	Description	Scheme
`expr`	Use an expression to get the value	`string`
`value`	Specify a static value	`string`
`label`	Get the value from a label	`string`

Kubernetes

Kubernetes​

Events​

SeverityKeywords​

Relationships​

Lookup​

Kubernetes

Events

SeverityKeywords

Relationships

Lookup