Operator SDK Part 1

Feature image source

The Operator-SDK is a command-line tool used by operator developers to build, test and package their operators. The SDK assists developers transform application business logic into a package which extends the Kubernetes API.

Initializing a new project

You need to provide the SDK with some necessary information to initialize the scaffolding for a new project. The initialization command is; operator-SDK init --domain $DOMAIN --plugin $PLUGIN.

Parameters

--domain sets the domain for API resources that this operator will create on the cluster.

--plugin specifies the scaffolding structure and type of operator being build. If a plugin is not specified, the value defaults to Go. You can see the different scaffolding structures here.

Plugins

The SDK can build operators based on one of three technologies; Ansible, Helm, and Go. The choice of plugin affects how to write the business logic and your operators capabilities and limitations.

At the time of writing Helm can only achieve level 2 maturity level where Ansible and Go can achieve a maturity level of 5. The below image is accurate at the time of writing.

Plugin Maturity

Image source: https://operatorframework.io/operator-capabilities/

Ansible

Business logic is written operator with Ansible playbooks; these playbooks are mapped to Kubernetes resources using the watches.yaml file. The mapping designates what playbooks to execute by resource type and other configured conditions.

There are two primary packages inside an Ansible operators container, the Ansible Operator Binary and the Ansible Runner.

The Ansible Operator Binary gets resources from the Kubernetes API service, manages the resource status and triggers the Ansible Runner.

The Ansible Runner executes the Ansible playbook and sends events to the controller about the running process.

Helm
A Helm based operator consumes a one or more helm charts to manage an application. You can create charts specifically for the operator or import existing charts. The operator obtains values for charts from the .spec field of custom resources.

The watches.yaml file creates mappings between Kubernetes resources and charts. Override values are also configured using watches.yaml; these allow you to override chart default values and supersede custom resource .spec values.

Override values can be configured as a static value or reference an environment variable from the operator pod. Environmental variables are configured on the manager pod by editing config/manager/manager.yaml and add flexibility when deploying the operator in different environments.

A Go-based operator provides the highest level of flexibility but is the most complex to learn. The Go-based operator does not place an additional level of abstraction on top of the SDK as the Ansible and Helm operators do.

To write a custom resource, you first define the structure in a _types.go located in api/$API_VERSION/ then run make generate to generate a deep copy.go file for each CRD. The deep file ensures that the new APIs (CRDs) implement the runtime.Object interface.

The next step is to run make manifests to generate manifests from code.

The controller and reconciliation loops still need to have application logic added before the operator is functional.

Creating APIs

Operators extend the Kubernetes API by adding customer resources to the cluster. The command operator-SDK create API adds a new CRD and updates metadata within the operator project.

Creating a CRD for either an Ansible or Helm based operator is done by writing the schema to the CRD manifest. For a Go-based operator, you defined the CRDs schema using Go code and generate the manifest.

The schema is defined in the manifest using the openAPIV3Schema specification. Specifying every data field and validation criteria in your CRD improves stability and security, but it can take a long time. For example, working on a Helm chart, you would need to transpose all the chart inputs into your CRD and then keep it aligned with future chart releases.

The field x-Kubernetes-preserve-unknown-fields: true can be added to a CRD to preserve fields which not in the schema. If this field is omitted or set to false, fields not in the schema are removed. You add this field directly to the manifest for Ansible and Helm based operators, but for a based operator, you add // +kubebuilder:validation:XPreserveUnknownFields to the field.

It is important to know the minimum Kubernetes version that you intend for your operator to be used on. There is a slightly different CRD structure between apiextensions.k8s.io/v1beta1 and apiextensions.k8s.io/v1. apiextensions.k8s.io/v1beta1 is deprecated and will be removed from Kubernetes in version 1.19; apiextensions.k8s.io/v1 requires Kubernetes 1.16 or later.

Below is an example of a custom resource written for an Ansible based operator.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: vaults.vault.sdbrett.com
spec:
  group: vault.sdbrett.com
  names:
    kind: Vault
    listKind: VaultList
    plural: vaults
    singular: vault
  scope: Namespaced
  version: v1alpha1
  validation: 
    openAPIV3Schema:
      description: Vault is the Schema for the vaults API
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          description: Spec defines the desired state of Vault
          type: object
          x-kubernetes-preserve-unknown-fields: true
          properties:
            targetNamespace:
              type: string
        status:
          description: Status defines the observed state of Vault
          type: object
          x-kubernetes-preserve-unknown-fields: true
      type: object
  served: true
  storage: true
  subresources:
    status: {}

The above produces the follow CRD manifest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: vaults.vault.sdbrett.com
spec:
  group: vault.sdbrett.com
  names:
    kind: Vault
    listKind: VaultList
    plural: vaults
    singular: vault
  scope: Namespaced
  served: true
  storage: true
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      description: Vault is the Schema for the vaults API
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          description: Spec defines the desired state of Vault
          properties:
            targetNamespace:
              type: string
          type: object
          x-kubernetes-preserve-unknown-fields: true
        status:
          description: Status defines the observed state of Vault
          type: object
          x-kubernetes-preserve-unknown-fields: true
      type: object
  version: v1alpha1

To write the same CRD for a Go-based operator, you would write the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
/*
Copyright 2021.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1alpha1

import (
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: JSON tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// VaultsSpec defines the desired state of Vaults
type VaultsSpec struct {
  // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
  // Important: Run "make" to regenerate code after modifying this file

  TargetNamespace string `json:"targetNamespace,omitempty"`
}

// VaultsStatus defines the observed state of Vaults
type VaultsStatus struct {
  // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
  // Important: Run "make" to regenerate code after modifying this file
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// Vaults is the Schema for the vaults API
type Vaults struct {
  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`

  Spec   VaultsSpec   `json:"spec,omitempty"`
  Status VaultsStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// VaultsList contains a list of Vaults
type VaultsList struct {
  metav1.TypeMeta `json:",inline"`
  metav1.ListMeta `json:"metadata,omitempty"`
  Items           []Vaults `json:"items"`
}

func init() {
  SchemeBuilder.Register(&Vaults{}, &VaultsList{})
}

The above code generates the following CRD manifest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.4.1
  creationTimestamp: null
  name: vaults.vault.sdbrett.com
spec:
  group: vault.sdbrett.com
  names:
    kind: Vaults
    listKind: VaultsList
    plural: vaults
    singular: vaults
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Vaults is the Schema for the vaults API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: VaultsSpec defines the desired state of Vaults
            properties:
              targetNamespace:
                type: string
            type: object
          status:
            description: VaultsStatus defines the observed state of Vaults
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

Scaffolding

Scaffolding is a projects directory and file structure, the structure different based on the plugin being used to write the operator.

The tables below describe the basic scaffolding for each operator plugin.

Ansible

File/Folders	Purpose
Dockerfile	The Dockerfile for building the container image for the operator.
Makefile	Contains make targets for building, publishing, deploying the container image that wraps the operator binary, and make targets for installing and uninstalling the CRD.
PROJECT	A YAML file containing meta information for the operator.
config/crd	The base CRD files and the customization settings.
config/default	Collects all operator manifests for deployment, used by `make deploy`.
config/manager	The controller manager deployment.
config/Prometheus	The ServiceMonitor resource for monitoring the operator.
config/RBAC	The role, role binding for leader election and authentication proxy.
config/samples	The sample resources created for the CRDs.
config/testing	Some sample configurations for testing.
playbooks/	A subdirectory for the playbooks to run.
roles/	A subdirectory for the roles tree to run.
watches.yaml	The Group, Version, and Kind of the resources to watch, and the Ansible invocation method. New entries are added via the ‘create api’ command.
requirements.yml	A YAML file containing the Ansible collections and role dependencies to install during build.
molecule/	The Molecule scenarios for end-to-end testing of your role and operator

Source

Helm

File/Folders	Purpose
config	Contains kustomize manifests for deploying this operator on a Kubernetes cluster.
helm-charts/	Contains a Helm chart initialized with `operator-SDK create api`.
Dockerfile	Used to build the operator image with `make docker-build`.
watches.yaml	Contains Group, Version, Kind, and Helm chart location.
Makefile	Contains the targets used to manage the project.
PROJECT	Contains meta-information about the project.

Source

File/Folders	Purpose
Dockerfile	The Dockerfile for building the container image for the operator.
Makefile	Contains make targets for building, publishing, deploying the container image that wraps the operator binary, and make targets for installing and uninstalling the CRD.
PROJECT	A YAML file containing meta information for the operator.
apis/$GROUP/$VERSION	APIs for multi API projects
config/crd	The base CRD files and the customization settings.
config/default	Collects all operator manifests for deployment, used by `make deploy`.
config/manager	The controller manager deployment.
config/Prometheus	The ServiceMonitor resource for monitoring the operator.
config/RBAC	The role, role binding for leader election and authentication proxy.
config/samples	The sample resources created for the CRDs.
config/testing	Some sample configurations for testing.
controllers/$GROUP	controller for multi API projects
hack/	Script directory
main.go	Operator entry point

RBAC

You need to configure RBAC so you operator can watch and manage resources on the cluster. For Ansible and Helm based operators, RBAC management is done by managing Role and RoleBinding or ClusterRole and ClusterRoleBinding manifests. For a Go base operator, RBAC markers are added to the controller.go file.

Permissions are automatically added for CRDs created using the operator-SDK create api command, but additional permissions need to be set manually.

This is the base config/rbac/role.yaml for an Ansible / Helm project

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: manager-role
rules:
  ##
  ## Base operator rules
  ##
  - apiGroups:
      - ""
    resources:
      - secrets
      - pods
      - pods/exec
      - pods/log
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - apps
    resources:
      - deployments
      - daemonsets
      - replicasets
      - statefulsets
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  ##
  ## Rules for vault.sdbrett.com/v1alpha1, Kind: Vaults
  ##
  - apiGroups:
      - vault.sdbrett.com
    resources:
      - vaults
      - vaults/status
      - vaults/finalizers
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
# +kubebuilder:scaffold:rules

This generates the following ClusterRole

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ans-op-manager-role
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  - pods
  - pods/exec
  - pods/log
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - vault.sdbrett.com
  resources:
  - vaults
  - vaults/status
  - vaults/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

In Go, the ClusterRole is written to the controllers/*_controller.go file for each controller.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
/*
Copyright 2021.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
  "context"

  "github.com/go-logr/logr"
  "k8s.io/apimachinery/pkg/runtime"
  ctrl "sigs.k8s.io/controller-runtime"
  "sigs.k8s.io/controller-runtime/pkg/client"

  vaultv1alpha1 "github.com/sdbrett/go-op/api/v1alpha1"
)

// VaultsReconciler reconciles a Vaults object
type VaultsReconciler struct {
  client.Client
  Log    logr.Logger
  Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=vault.sdbrett.com,resources=vaults,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=vault.sdbrett.com,resources=vaults/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=vault.sdbrett.com,resources=vaults/finalizers,verbs=update

// Reconcile is part of the main Kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Vaults object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.7.0/pkg/reconcile
func (r *VaultsReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  _ = r.Log.WithValues("vaults", req.NamespacedName)

  // your logic here

  return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *VaultsReconciler) SetupWithManager(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
    For(&vaultv1alpha1.Vaults{}).
    Complete(r)
}

This code generates the following ClusterRole manifest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - vault.sdbrett.com
  resources:
  - vaults
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - vault.sdbrett.com
  resources:
  - vaults/finalizers
  verbs:
  - update
- apiGroups:
  - vault.sdbrett.com
  resources:
  - vaults/status
  verbs:
  - get
  - patch
  - update
  

Summary

So far, we have looked at how the operator-SDK is to build operators based on Ansible, Helm and Go. We’ve also seen some of the difference between building operators based on different technologies.

The next post looks at using the SDK to build, push and run your operator.