Shutdown Hours Lab

Determine if an Ocean-GKE Shielded Node workaround is required

To implement shutdown hours in Ocean, we must first consider if the GKE cluster has Shielded Node enabled. To check this, refer to the GKE cluster settings to determine if Shielded Nodes are enabled.

To check Shielded Node status, refer to the the GKE cluster configuration page, look under the Security group and find the key “Shielded GKE nodes”.

If the Shielded Node configuration is enabled, a workaround must be applied to ensure the Ocean controller pod can continue to run during off hours.

gke-shielded-node-config

Implementing the Shielded Node shutdown hours workaround

To implement the workaround, consider the following:

Workaround –> Terraform a small, single zone GKE managed node pool that the ocean controller pod can run on when the Ocean managed nodes are shut down.

The following terraform code will:

  • Create a low cost, single node pool of VM type e2-small
  • Deploy the node in a single node pool in a (zonal) availability zone
  • This node pool will be managed exclusively by GKE and will remain running during shutdown hours
resource "google_container_node_pool" "shutdown_node" {
  name       = "ocean-node-for-shutdown-hours"
  location   = var.region
  node_locations = [var.zone]
  cluster    = google_container_cluster.primary.name
  node_count = 1
  project    = var.project_id

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]

    labels = {
      env = var.project_id
    }

    preemptible  = false
    machine_type = "e2-small"
    tags         = ["ocean--shutdown-hours-node", "${var.project_id}-gke"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

Implementing shutdown hours in terraform

Now its time to implement shutdown hours for you GKE cluster. Lucky for you, this step is nothing more than adding a single config element to your existing Ocean-GKE terraform code. In the example below you will see a new config parameter “shutdown_hours” that has been added to the configuration. In this example, we have instructed Ocean to shutdown during non-business hours. The times are defined in GMT format.

module "ocean-gcp-k8s" {
  source     = "spotinst/ocean-gcp-k8s/spotinst"

  cluster_name                      = var.cluster_name
  location                          = var.region
  shutdown_hours = {
    is_enabled = true
    time_windows = [ 
      "Fri:23:30-Mon:13:30", 
      "Mon:23:30-Tue:13:30", 
      "Tue:23:30-Wed:13:30",
      "Wed:23:30-Thu:13:30",
      "Thu:23:30-Fri:13:30",
    ]
  }
}

Implementing shutdown hours in the console

From within the spot console; open the Ocean cluster “Customize Scaling” settings from the “Actions” menu.

new_vng

From within the scaling setting panel, ensure shut downs hours is enabled.

new_vng

Now you can can click on each hour of the day in the schedule below to shutdown the instance for the given hour. The schedule below demonstrates what a cluster shutdown schedule may look like for a typical work week where applications only need to run during the business day.

new_vng