This guide is for platform teams that run Kubernetes Operators. It covers software operations, not gambling-licence compliance or player protection. A gaming company should use these checks only for services it actually runs on Kubernetes.

For New Zealand platform engineering, DevOps and SRE teams, the same questions apply to managed Kubernetes, on-premises clusters and hybrid deployments. Data residency needs a separate check: backups, telemetry and support access may leave the region even when the workloads stay local.

When is an Operator the right choice?

The Kubernetes Operator Pattern extends the Kubernetes API with application-specific operational knowledge. A Custom Resource Definition (CRD) registers a resource type, a Custom Resource (CR) expresses the desired state in its spec , and a controller's reconciliation loop compares that intent with the actual state. The managed software is often called the operand .

Operators earn their place when someone otherwise has to perform recurring application-specific work: database failover, backup restoration, certificate rotation or coordinated stateful upgrades. For a fixed collection of manifests, a Deployment, StatefulSet, Helm chart or GitOps workflow is usually easier to operate. Helm manages releases; an Operator keeps checking application state. They can work together.

Operating requirement Prefer Reason to reconsider
Install static services with occasional planned changes Helm or GitOps-managed manifests No application-specific control loop is needed
Continuously restore application-specific desired state Operator Requires controller development and long-term maintenance
Manage a standard stateless service Deployment and native Kubernetes controllers Custom automation may duplicate built-in behaviour
Manage stateful recovery and coordinated upgrades Operator, if the operational knowledge is explicit and testable Recovery logic can cause irreversible damage if poorly designed

Decision test: write down the interventions your team makes after installation. Can each one be represented as a safe, repeatable state transition and verified? If not, an Operator may simply turn a manual problem into a harder-to-debug controller.

Network equipment and cabling inside a server cabinet.
Infrastructure ownership and repeatable operations are central to a useful Operator.

1. Keep ownership and the API boundary clear

Keep ownership narrow. Operator SDK guidance favours an Operator for each managed application type, with distinct controllers for separate CRDs. Redis and MySQL have different failure and upgrade paths; combining them behind a single controller makes both harder to reason about. Use an orchestration layer only for coordination the applications genuinely share.

Give each CRD one authoritative reconciler. When two controllers compete to write the same fields, they may overwrite each other and repeatedly trigger reconciliation. Sharing an API requires explicit delegation, not competing writers.

Choose the API group, version, kind and scope before implementing the controller. An OpenAPI structural schema should reject invalid fields; an admission webhook can handle cross-field rules when schema validation is insufficient. Put the requested configuration in spec , not the observed result. Write the latter to status . Avoid defaults that disguise material behaviour.

For a database Custom Resource, useful fields might include a supported version, storage class, capacity, backup schedule and recovery policy. A field such as ready: true belongs in status, not in the requested configuration. Use status conditions with meaningful reasons and observedGeneration to indicate whether the reported result reflects the latest spec.

Useful APIs expose clear intent in spec , reliable feedback in status and observable reconciliation outcomes.

Check: submit an invalid CR and inspect the API error. Then change a valid field and confirm the corresponding status transition. Repeat with every older resource version you still support.

2. Make reconciliation safe to repeat

A Reconciler must tolerate repeated work. Events can be duplicated or combined, the cache can lag, and an external API can time out after completing the requested operation. On each pass, read current state, calculate the difference and make only the changes needed. Do not assume an event represents a one-time command.

Prefer level-based logic over event-specific assumptions. For example, a controller managing PostgreSQL should check the StatefulSet, PersistentVolumeClaims, Services and application health before advancing an upgrade. A successful Kubernetes API write does not establish that the database is healthy. Separate infrastructure readiness from application-level acceptance checks. 

Reconcile request
  → read Custom Resource and dependencies
  → validate prerequisites and ownership
  → compare desired state with observed state
  → make one safe, bounded change if needed
  → update status condition and observedGeneration
  → return; retry transient failures with backoff

Retry transient errors with bounded backoff. A tight requeue loop can overload the API server; an invalid configuration should instead produce a useful status condition and an error that points to the failing field.

Failure drill: stop the controller after an upgrade Job is created but before its result reaches status. On restart, it should find that Job rather than create another migration.

3. Manage ownership, backups and deletion together

Operators often create Pods, Jobs, Deployments, Services, ConfigMaps, Secrets and StatefulSets. Where appropriate, attach OwnerReferences so Kubernetes garbage collection can remove dependants. Use a finalizer only when deletion requires a controlled external action, such as releasing cloud resources or completing a defined backup procedure.

A finalizer can also block deletion indefinitely when an external service does not respond. Decide who can override it, when manual intervention is justified, and which data must survive. An automatic timeout that silently discards unfinished cleanup is not a safe substitute.

Prune completed Jobs and temporary Pods, but keep what incident review or audit policy requires. Confirm the retention setting rather than relying on indefinite histories or aggressive deletion.

A successful backup Job is not evidence of recovery. Set a recovery point objective (RPO) and recovery time objective (RTO); restore the data in a separate environment and measure both. Check how the storage driver, credentials and application version affect the result.

4. Minimise privilege and respect multi-tenancy

Run the Operator's containers as non-root wherever possible, use a dedicated ServiceAccount and grant role-based access control (RBAC) permissions under the principle of least privilege. Prefer namespaced Roles to ClusterRoles where the design allows. Separate the rights needed by the controller from the permissions needed by an operand's Pods. Avoid giving the controller broad rights merely to simplify initial development.

Apply Pod Security Standards and check whether the target namespaces enforce Baseline or Restricted policies through Pod Security Admission. Configure security contexts, privilege escalation, capabilities and volume mounts deliberately. Handle Secrets through Kubernetes-supported secret mechanisms; do not place credentials in custom resource status, logs or metrics labels.

For multi-tenant environments, make the watched namespace configurable and do not assume fixed installation namespaces or resource names. Constrain informer watches, validate cross-namespace references and prevent one tenant's CR from controlling another tenant's resources. A NetworkPolicy should grant only required traffic; an Operator must not create an unrestricted allow-all policy that bypasses existing segmentation. Where teams own their own network controls, offer a documented option to leave NetworkPolicy management to them.

Security drill: run under production RBAC and admission settings. Attempt a disallowed cross-namespace action and revoke a needed permission; verify that the controller reports the denial instead of retrying silently or requesting broader privileges. Review the Kubernetes Security Checklist for cluster-level threats.

Operator design decisions become easier to maintain when CRD ownership, controller boundaries, observability and access control are agreed early.

5. Build observability around user impact

A running controller Pod can coexist with a failed database. Expose Prometheus-compatible metrics for reconciliation duration, error counts, workqueue depth and operand health. Monitor application outcomes rather than treating the controller Deployment as the only health signal.

The Operator SDK observability recommendations cover metrics, alerts, Kubernetes Events and runbooks. Alert on actionable failures: a resource stuck Degraded, an upgrade stalled past its expected window, or reconciliation errors sustained long enough to require intervention.

Combine metrics with structured logging, contextual Kubernetes Events, status conditions and, where useful, distributed tracing. Typical service-level indicators (SLIs) include availability, request latency, error rate and throughput, but the most useful Operator-specific signals often concern the delay between a spec change and successful convergence. Monitor CPU and memory consumption too, without treating infrastructure usage as the sole measure of success.

Check: break a dependency on purpose. The CR should report a meaningful failure, the alert should fire within the agreed window, and the runbook should identify a useful first action.

6. Make controllers lean without losing correctness

An informer cache reduces repetitive API server reads, but an unrestricted cache can consume substantial memory in large clusters. Limit watched namespaces where suitable, use label selectors or field selectors when the controller's design permits, and avoid unnecessary watches for unrelated resource types. Cache filtering must not make required dependencies invisible.

Set resource requests and limits for the controller's CPU and memory based on measured behaviour. Understand the effect of ResourceQuota and LimitRange policies on both the Operator and its operands. Measure memory growth against the number of managed CRs, reconcile duration against event rate and API requests against cluster scale. Consider leader election when using multiple controller replicas for high availability; additional replicas do not necessarily increase reconcile throughput if only one is active leader.

More concurrency will not fix slow convergence caused by repeated no-op updates, a large cache or external API latency. Measure where time is spent before changing worker counts; compare CPU, memory, API traffic and reconciliation latency under a similar load.

Engineer monitoring equipment in a server room.
Metrics, alerts and status conditions are essential for proving that an Operator is converging safely under real production load.

7. Plan upgrades as compatibility transitions

An Operator release and an operand release are different changes. Use Semantic Versioning (SemVer) for Operator releases, and follow Kubernetes API versioning conventions for CRDs. For API evolution, consider served and storage versions, conversion webhooks, migration of stored objects, schema compatibility and clients still using older resource versions. A CRD update should not silently invalidate existing resources.

Where the deployment model uses Operator Lifecycle Manager (OLM) , understand how catalogues, bundles, ClusterServiceVersion (CSV) metadata, Subscriptions, InstallPlans, dependency resolution and update channels interact. Channels such as candidate and stable express release intent; a channel label is not evidence that a migration has been tested. Follow the OLM channel guidance for upgrade-path semantics.

Specify which operand versions each Operator release supports. Database schema migrations may be irreversible even if the controller Deployment can be rolled back. A rollback plan must therefore say what happens to application data.

Upgrade drill: start with the previous supported Operator and realistic CRs, upgrade through the advertised path, interrupt reconciliation and verify API compatibility and operand health. Test a data restore separately.

8. Test behaviour, not only controller code

Use unit tests for pure reconciliation decisions and schema validation. Use integration tests with a Kubernetes API environment to check CRD registration, admission behaviour and status updates. End-to-end (E2E) tests should exercise real dependencies, storage and operational faults. Tools in the Operator SDK , Kubebuilder and controller-runtime ecosystem support development and test workflows; Go is common, but Java Operator SDK, Helm? and Ansible-based approaches are alternatives when they fit the operational model.

The test matrix should cover creation, modification, deletion, stale cache reads, duplicate reconciliation, missing permissions, dependency loss, node disruption and version upgrades. For StatefulSets and PersistentVolumes, test retention and any storage expansion supported by the driver. With Apache Airflow, check workers and scheduled workloads as well as the control plane.

Scenario Expected evidence Unacceptable result
Controller restarted mid-operation Existing work adopted; convergence resumes Duplicate migrations or resources
Invalid Custom Resource Clear validation failure Silent partial deployment
RBAC permission removed Visible error and bounded retry Privilege escalation or noisy loop
Dependency unavailable Accurate status, useful alert False Ready condition
Operator upgraded Existing operands and API versions remain supported Unplanned destructive migration
CR deleted Documented retention and cleanup policy enforced Unexpected data loss or stuck finalizer

9. Common anti-patterns and how to recognise them

  • One super-Operator for everything: tightly couples unrelated Redis, MySQL and PostgreSQL lifecycles. Split responsibility unless cross-service coordination is a real requirement.
  • Two controllers owning one CRD: creates competing writes and unclear incident ownership. Establish a single authority and explicit interfaces.
  • Event-driven scripts disguised as reconciliation: assumes every event arrives exactly once. Reconstruct state from the API and make operations idempotent.
  • Self-installing CRDs or child Operators: expands privileges and confuses lifecycle ownership. Use the installation system, such as OLM where applicable.
  • Fixed namespaces and permissive networking: breaks shared clusters. Parameterise scope and honour tenant boundaries.
  • Success defined as a running controller Pod: hides broken operands. Report workload conditions and application outcomes.
  • Unbounded retries and old Jobs: increases API load and obscures incidents. Apply backoff, garbage collection and explicit retention.
  • Rollback promised without data recovery: ignores irreversible migrations. Test restoration and state clearly what rollback covers.

Upgrade planning is a compatibility exercise that should be tested alongside backup, restore and failure recovery procedures.

Production readiness checklist

  1. Confirm that a continuous application-specific control loop is necessary; reject unnecessary custom automation.
  2. Assign one owner to each CRD and document controller and operand boundaries.
  3. Validate CR schemas, defaults, API versions and status conditions.
  4. Prove idempotent reconciliation through restarts, retries and partial failures.
  5. Define finalizers, OwnerReferences, retention and manual cleanup procedures.
  6. Restrict RBAC, ServiceAccounts, namespace watches, NetworkPolicies and security contexts.
  7. Test under the actual Pod Security Admission and quota policies.
  8. Publish meaningful metrics, alerts, Events and runbooks.
  9. Measure cache size, CPU, memory, API traffic and convergence latency at realistic scale.
  10. Test upgrades of both Operator and operand, including older CR versions and data restoration.
  11. Demonstrate RPO and RTO against a recovery drill rather than relying on backup-job success.
  12. Record supported Kubernetes and application versions, escalation ownership and release procedures.

Questions platform teams should settle before adoption

Should we build an Operator or adopt an existing one?

Before building, inspect existing implementations on OperatorHub, Artifact Hub and upstream repositories. Check maintenance, supported Kubernetes versions, CRD evolution, security fixes, release history and failure recovery. An active project may still be unsuitable for your data and support requirements.

Does every CRD require a separate Operator?

No. An Operator binary may contain multiple controllers, each responsible for a distinct custom resource or concern. The critical boundary is unambiguous reconciliation ownership, not one executable for every CRD.

Can an Operator guarantee high availability?

No. Leader election can protect controller ownership and additional replicas can improve controller availability, but application high availability also depends on topology, storage, network, quorum rules and recovery logic. A healthy controller cannot compensate for an unavailable storage backend.

How should NZ teams handle cloud regions and privacy?

Map data classes, backups, telemetry, support access and disaster-recovery replicas to the organisation's legal and contractual obligations. Verify actual cloud-region services and cross-border transfers with providers and advisers where needed. Running Kubernetes in a New Zealand region does not, by itself, establish where every log, backup or support artefact is stored.

What is the simplest proof that an Operator works?

Change a CR and watch the operand converge. Then interrupt the controller and a dependency. If it recovers without duplicate operations or manual repair-and status and alerts correctly explain the interruption-you have meaningful evidence of operational readiness.