Stackbooster Node Templates

Node Templates Overview #

In the Stackbooster platform, node templates dictate the launch parameters for Kubernetes nodes within your clusters, facilitating optimal resource allocation and cost-efficiency. They allow users to define constraints, properties and launch specifications for nodes added during cluster scaling. Key functionalities include:

  • Specifying node launch configuration params, such as subnets, disk size, computer architecture, etc.
  • Define the lifecycle of the nodes (spot, on-demand)
  • Configuration of the instance types and families that are eligible in the node template.
  • Define taints and labels for limim the pods that can run on the node template

Node Template Variants #

  1. Universal Node Template

    • Purpose: The Universal Node Template is automatically applied when a user creates a Kubernetes cluster from the Stackbooster console. This template serves as a one-size-fits-all starting point for node configuration, streamlining initial setup.
    • Features:
      • Availability Zones: Includes all zones of the EKS cluster.
      • Security Groups: Utilizes the security group specifically created for the worker nodes of the newly created EKS cluster.
      • Lifecycle: Configured primarily for spot instances, with an automatic fallback to on-demand instances if necessary.
      • Instance Families: Excludes specific AWS instance families (e.g., dl, f, g, vt, hpc, inf, p, trn, x, u) to optimize cost and performance.
      • Architecture: Supports both amd64 and arm64 architectures, enhancing flexibility for different application needs.
      • Root Device: Utilizes the root device of the gp3 type and 32 GiB size

  2. Personalized Node Template

    • Purpose: The Customized Node Template allows users to customize node parameters via a CustomResource YAML configuration. This template is for users who need specific configurations that the Universal Node Template does not support.
    • Configuration: Customizable through the Stackbooster console and YAML definitions, providing flexibility for advanced users to define exact specifications and preferences.
    • Configuration Application: To apply a personalized node template, use the following kubectl command with your NodeTemplate YAML:
      kubectl apply -f your-custom-node-template.yaml

Managing Node Configurations #

Important Note on Node Configurations: Existing nodes cannot be directly upgraded or modified according the new node template applied. To apply a new configuration to a node or group of nodes, you must first delete the existing node. Stackbooster will then replace it with a node based on the new configuration. This ensures that configurations are consistently applied and managed.

Multiple Node Template Options #

It is possible to configure multiple node templates within your environment. When creating node templates, aim to design them so they are mutually exclusive. This ensures that no pod is eligible for more than one template. If a pod matches multiple templates, the system will select the template that offers the cheapest instance capable of scheduling the pod.
Each node template is uniquely identified by its name. Modifications to a node template’s configuration are detected automatically, so there is no need to change its name when making updates. This simplifies management and ensures consistent tracking of changes.

Attributes of Node Templates #

IAM Instance Profile #

Required. Specifies the Amazon Resource Name (ARN) of the IAM instance profile to associate with the nodes.

instanceProfileArn: arn:aws:iam::000000000000:instance-profile/private-NodeInstanceProfile-1.

SSH Key #

Optional.. Specifies the SSH key that can be used to access nodes.

keyName: "sandbox"

Subnet Selector #

Required. Selects the subnets for node placement using subnet IDs or tag key-value pairs.

  1. By Subnet IDs. Selects subnets directly by specifying their IDs. This method uses a comma-separated list of subnet IDs.
subnetSelector:
  ids: "subnet-xxxxxxxxxxxxxxx1,subnet-xxxxxxxxxxxxxxx2,subnet-xxxxxxxxxxxxxxx3"
  1. By Tag Key-Value Pair. Selects subnets based on a specific tag key-value pair. This method allows selection of subnets that have tags matching exactly the specified key and value.
subnetSelector:
  Environment: "production"
  Project: "Stackbooster"

Security Group Selector #

Required. Chooses the security groups to be applied to the nodes, selectable by IDs or tag key-value pairs.

  1. By Security Group IDs. Selects security groups directly by specifying their IDs. This method uses a comma-separated list of security groups IDs.
securityGroupSelector:
  ids: "sg-xxxxxxxxxxxxxxx1,sg-xxxxxxxxxxxxxxx2"
  1. By Tag Key-Value Pair. Selects security groups based on a specific tag key-value pair. This method allows selection of security groups that have tags matching exactly the specified key and value.
securityGroupSelector:
  Environment: "production"

Block Device Mappings #

Optional. Configures the block devices attached to nodes, including settings for volume size and type. Default is 32 GiB and gp3 type (AWS)

blockDeviceMappings:
  - deviceName: "/dev/xvda"
    ebs:
      volumeSize: 32
      volumeType: "gp3"

Lifecycle #

Optional. Defines the lifecycle of the instance, with options like spot, od (on demand), or spot-od (spot with on-demand fallback). Default is spot-od.

lifecycle: "spot-od"

Labels #

Optional. Assigns labels to the nodes for identification and management within the cluster.

labels:
  template: "dev"
  Project: "Stackbooster"

Annotations #

Optional. Assigns annotations to the nodes.

annotations:
  function.kubernetes.io/service-type: "database"

Machine Types #

Optional. Specifies a list of machine types (instance types) that are allowed for node creation. If this list is provided, only the specified machine types will be used for nodes.

machineTypes:
   - "m5.large"
   - "t3.medium"

Blacklist Instance Families #

Optional. Defines a list of instance family types that are explicitly prohibited for use with nodes. This blacklist ensures that the specified machine families are not selected for node allocation under any circumstances, overriding any families listed in the whitelist.

blackListInstanceFamilies:
   - "t2"
   - "m1"

Whitelist Instance Families #

Optional. Contains a list of preferred instance family types that are allowed for node allocation. This whitelist specifies which machine types can be considered for use with nodes, except those explicitly prohibited by the blacklist. If a machine type is on both the whitelist and the blacklist, the blacklist takes precedence, and that machine type will not be used.

whiteListInstanceFamilies:
  - "m5"
  - "r4"

Node Restrictions #

Optional. Node restrictions allow you to apply constraints on where nodes can be placed based on specified labels, which helps in enforcing policies for workload segregation and resource utilization. StackBooster integrates well-known Kubernetes labels and supports cloud-specific labels. You can view a comprehensive list of supported labels below. Also, user

Labels can be incorporated into either the Node Template or the workload configuration (such as a nodeSelector in a pod’s specification). This includes well-known cloud labels, which are detailed below. The process of node selection relies on matching the criteria set forth in both the Node Template and the pod’s specifications. If there’s no overlap in requirements, StackBooster will refrain from provisioning nodes, thus ensuring that pods are placed only on appropriate nodes. In scenarios where a specific label, important for pod placement and included among the well-known cloud labels, is not set in the Node Template but is included in the pod’s specification, the system might select any available value that fits the cluster’s placement options.

Users have the option to create custom labels. Should a nodeSelector or a necessary nodeAffinity include a label that isn’t specified in the list below, Stackbooster will not deploy nodes with these labels, resulting in pods that are not scheduled. To enable Stackbooster to recognize and schedule nodes based on these custom labels, they must be included in the NodeTemplate requirements.

Properties #

  • Key: Label key used for matching the restriction against node labels.
  • Operator: Defines the relationship between the node label and the specified values. Possible Operators: In, NotIn, Exists, DoesNotExist, Gt, Lt.
  • Values: An array of values relevant to the restriction applied based on the operator.
nodeRestrictions:
  - key: "topology.kubernetes.io/zone"
    operator: "In"
    values: ["us-east-1a", "us-east-1b"]

Supported labels #

Instance Type
node.kubernetes.io/instance-type The intersection of specified instance types results from the configurations in machineTypes, along with the exclusions and inclusions set in blackListInstanceFamilies and whiteListInstanceFamilies.
NOTE, it’s important to note that not all instance types are available across all Availability Zones and regions.

Availability Zones
topology.kubernetes.io/zone OR failure-domain.beta.kubernetes.io/zone

CPU Architecture
kubernetes.io/arch We support only amd64 and arm64 architecture for now.

Operating System
kubernetes.io/os We support only linux for now.

Taints #

Optional. Defines a list of taints to be applied to each node, influencing scheduling decisions.
Array of objects, each representing a single taint.

  • Properties:
    • Effect: Specifies the effect of the taint, which determines how the taint influences pod scheduling on the node.
      • Possible Values: NoSchedule, NoExecute.
    • Key: A unique identifier for the taint.
    • Value: Provides additional information about the taint, used to match the taint against pod tolerations.
  • Sample Configuration:
  taints:
    - key: "example-key"
      value: "example-value"
      effect: "NoSchedule"

StackBooster Node Template Example #

Here’s a sample configuration for a StackBooster Node Template:

apiVersion: stackbooster.io/v1alpha1
kind: StackBoosterNodeTemplate
metadata:
  name: qa
  namespace: stackbooster
spec:
  # SSH Key (Optional)
  # An SSH key to be added to new nodes.
  keyName: "sandbox"

  # IAM Instance Profile (Required or Optional)
  # The ARN of the IAM Instance Profile to assign as the node's identity.
  instanceProfileArn: arn:aws:iam::852008484434:instance-profile/private-NodeInstanceProfile-aZlgQ4Zce4ug

  # Subnet Selector (Required)
  subnetSelector:
    ids: subnet-xxxxxxxxxxxxxxx1,subnet-xxxxxxxxxxxxxxx2,subnet-xxxxxxxxxxxxxxx3,subnet-xxxxxxxxxxxxxxx4

  # Security Group Selector (Required)
  securityGroupSelector:
    ids: sg-0dc8f3cd4db83eb2b

  # Block Device Mappings (Optional)
  # Configuration for block devices attached to nodes.
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 20
        volumeType: gp2

  # Node Restrictions (Optional)
  # Restricts nodes based on labels such as instance-type, architecture, OS, zone.
  nodeRestrictions: 
   - key: "topology.kubernetes.io/zone"
     operator: "In"
     values: ["us-east-1a", "us-east-1b"]
  
  taints:
    - key: "example-key"
      value: "example-value"
      effect: "NoSchedule"

  # Lifecycle (Optional)
  # Defines the lifecycle of the instance. Default is 'spot-od'.
  lifecycle: "spot-od"

  # Labels (Optional)
  # Labels to be added to the nodes.
  labels:
    team: qa

  # Machine Types (Optional)
  # Specifies machine types. If not provided, all possible offerings are used.
  machineTypes: []
  blackListInstanceFamilies:
    - dl
    - f
    - g
    - vt
    - hpc
    - inf
    - p
    - trn
    - x
    - u
  whiteListInstanceFamilies: []